U.S. Department of Transportation
Federal Highway Administration
1200 New Jersey Avenue, SE
Washington, DC 20590
202-366-4000


Skip to content
Facebook iconYouTube iconTwitter iconFlickr iconLinkedInInstagram

California Division

Home / About / Field Offices / California Division / Systems Engineering Guidebook for ITS

Home What's New Systems Engineering Guidebook Views Search Glossary Resources Feedback Site Map

3.6.3 Verification [Sub-system and system level verification]

 

OBJECTIVE:

The verification process is used by the system's owner and by other stakeholders to show that the as-built system, sub-system, and components meet all of their requirements and design. This process is used by the system's owner and other stakeholders to accept the system products from the development team.

DESCRIPTION:

Verification is the process that proves the system [or sub-system or component] meets its requirements and matches the design. Since verification is based on requirements and design, one of the keys to successful and effective verification is well-written and complete requirements and design documents. These requirements and design elements are developed, reviewed, and approved earlier in the project timeline before the system is developed or procured. Planning for the verification activities starts with the System

Engineering Management Plan [or with the Project Plan if a SEMP is not needed]. At this level, the general structure of the verification tasks is identified and shown to be compatible with the desired deployment plan and with the system concept. The Verification Plans are best written at the same time the requirements of the system, sub-system, or component are developed. This is done to show that the requirements, as written, can be verified. At the end of the detailed design effort, verification procedures can be written. These procedures are the detailed steps to be taken to verify each requirement and design element. There must be a clear trace from each requirement, through the Verification Plan, down to a detailed step in the verification procedure. Verification is performed iteratively. It starts with the integration activities at the component level. It progresses through the sub-system development to the verification of the entire system. Final verification for system acceptance is done with the installed system. At this point, system development is complete and the deployed system is ready for operations. The system's owner and stakeholder involvement is essential for verification.

Verification answers the question "Was the system built ‘right'"

CONTEXT OF PROCESS:

Shows the flow for Phase [3] Task 3, Verification Process. Summaries are described for inputs, constraints, and enablers into the task; activities of the task; and outputs from the task. The flow is described in detail in the accompanying text.

VERIFICATION PROCESS

Inputs:

Concept of Operations describes the way the system is to operate and will assist in the verification and integration effort.

System and Sub-system Requirementscontain the functional and performance requirements to be verified.

Design Specificationscontain the design elements to be verified.

Integration Plan [optional] shows how the integration steps are to be done iteratively with verification.

Deployment Strategy [optional] defines when and where verification takes place.

Integrated sub-systems/system is ready for verification.

Control:

Project Plan/Systems Engineering Management Plan [SEMP] establishes a high level description of the project's plan for verification.

Configuration Management Plan sets the configuration controls needed during verification.

Enablers:

Stakeholder involvement is needed for verification conduct and to show critical stakeholders that the system meets its requirements.

Technical Reviews include a test readiness review to determine all resources needed for a verification step are available.

Traceability to the verification plan & procedures ensures that all requirement are being verified.

Outputs:

Verification Master Plan is included in the Project Plan/SEMP to establish general guidelines for this important part of the systems engineering process.

Verification Plan documents the plan for verifying system and sub-system requirements.

Verification Procedures document the details of each verification step.

Verification Reports document results of each verification step.

Verified sub-system/system ready for further integration, deployment, or operational use.

Process Activities:

Plan verification activities in SEMP / Project Plan

During the project planning stage, a strategy for verification is developed which is compatible with the system concept and the deployment objectives.

Develop Verification Plan

Verification Plan is written for each level [component, sub-system or system]. The plan will develop a verification case and method for each requirement and for each design element contained in the applicable Specification.

In addition to the verification cases, the Verification Plan will give general guidance for all of the verification activities. These include: the identification of all verification participants, descriptions of their roles and responsibilities, and a schedule for verification activities. Finally, it includes: the identification of test equipment needed and of software drivers or simulators needed to model the interfaces to the system under test.

Trace between specifications and test cases

Each test case is traced to a specific requirement to ensure all requirements are verified.

Develop Verification Procedures

These procedures are the detailed step-by-step actions and the expected outcome for each verification case.

Perform verification

When all needed resources are ready, verification is performed according to the approved procedures.

Document verification results

Prepare a Verification Report for each verification step.

Illustrates where Verification occurs in the Vee Development Model. The verification plans are defined and developed in the System Requirements and High-Level Design (Project Architecture) Subsystem Requirements sections. Verification procedures are developed and performed in the Subsystem Verification and System Verification and Initial Deployment sections.

Where does Verification take place in the project timeline?

Is there a policy or standard that talks about verification?

FHWA Final Rule does not specifically mention general verification of requirements. It does require inter-operability tests relating to use of ITS standards. IEEE std. 1012 talks about independent verification and validation. CMMI identifies best practices.

Which activities are critical for the system's owner to do?

  • Identify and recruit stakeholders who are needed to participate in verification
  • Review and approve all documents
  • Witness critical verification steps

How do I fit this step to my project? [Tailoring]

Some level of verification is needed to accept the system. The formality with which verification is performed can be tailored to the budget, size, and complexity of the project. For a small simple project with few stakeholders, it only may be necessary to use the requirement document itself as a checklist and extemporize the procedures on the fly. Thus, no verification documents are needed. The system's owner determines what level for verification formality and documentation is needed to satisfy the complexity of the project.

What should I track to reduce project risk and to get what is expected? [Metrics]

Number of verification failures and their cause [poor requirements, design errors, inadequate integration], is an indication of the quality of products from the development team.

Checklist: Are all the bases covered? [

check Was a Verification Plan developed and approved?
check Were all requirements traced to a Verification Plan test case?
check Were Verification Procedures developed and approved?
check Were the key participants identified and trained?
check Were all resources needed for testing in-place?
check Were all participants notified of the testing schedule?
check Was a Verification Report prepared?

Are there any other recommendations which can help?

A closer look at the stages of verification, verification techniques, and the rules for performing verification

Key stages of verification

A project may require three or more different stages of verification: sub-system, system, and commissioning. The first is iterative with integration. The last is iterative with deployment. System verification and acceptance falls between the two. Of course, special project situations may require some tailoring, and perhaps additional stages, for complete verification.

  • Sub-system Verification – As discussed in 3.4.2, High Level Design, a system is often divided into two or more sub-systems for ease of development. Once the integration process has produced one of these sub-systems, it is verified against its requirements. Once verified, the sub-system can be integrated with other sub-systems.
  • System Verification step covers all integrated sub-systems and is usually used to accept the entire system. For many requirements, this is the last time they are verified. As such, this step is the most formal, reviewed, witnessed, and where there are failures, receives the most attention. It may not be as exhaustive as sub-system verification. Yet, it still must be extensive enough to produce a solid feeling among the stakeholders that the system does what it is supposed to do.
  • Sub-system and system verification is best done in a highly controlled environment, especially with respect to external inputs to the system under test. This usually requires software to simulate or model the external world. For instance, a traffic signal simulator or roadway sensor simulator may be needed to test a new central control system.
  • Commissioning is accomplished after the system is deployed to verify that the system works when installed. Commissioning is generally more cursory than system verification. It is just enough to verify that everything is still working. However, in some circumstances, a part of system verification must be deferred to the time of commissioning; again using simulated inputs as needed to complete the needed verification prior to commissioning.
  • Tip. Verification of the system's ability to work with the complete set of real sensors must wait until after deployment.

Caution.It may be necessary to overlap the last two stages of verification. System verification can be started in a development environment using simulated inputs from sensors and external system then completed after deployment and commissioning using real sensors and real external systems. While verification with simulated inputs may be necessary, final verification with real inputs is almost always mandatory.

Verification techniques

Four techniques are used to verify requirements: inspection, analysis, test, and demonstration.

  • Inspection: the visual verification of a requirement, such as a color, a size, and model number.
  • Analysis: the mathematical analysis of collected data to verify a requirement
  • Demonstration: the use of the system itself to verify the expected output, such as a response to an operator input. This is the most commonly used verification technique.
  • Test: similar to a demonstration except external test equipment is used.

A special type of demonstration is called a burn-in is used to identify and resolve random or latent defects [thermal, memory leaks, and race conditions].

General rules for performing verification

  • notify all stakeholders of the schedule for verification and clarification of their roles and responsibilities
  • identify and document the configuration of the system under test
  • define the process for recording all test actions and the system's response
  • define the process for dealing with all unexpected responses
  • define the process to manage anomalies
  • define a plan of action based on this analysis. [e.g. repeat the test, revise procedure, change the requirement, suspend the test, fix the system and retest]

Be careful of requirements creep. During verification some stakeholders, especially if they have not been involved in the design activities, will want to rewrite or add to the system requirements. A typical example is a desire to change the operator interface. There is a cost and schedule risk of doing this and the best way to avoid these occurrences is to ensure that the correct stakeholders are involved in establishing the requirements and designing the system from the start.

Related Template Checklist  

 

Page last modified on September 20, 2016
Federal Highway Administration | 1200 New Jersey Avenue, SE | Washington, DC 20590 | 202-366-4000