U.S. Department of Transportation
Federal Highway Administration
1200 New Jersey Avenue, SE
Washington, DC 20590
The verification process is used by the system's owner and by other stakeholders to show that the as-built system, sub-system, and components meet all of their requirements and design. This process is used by the system's owner and other stakeholders to accept the system products from the development team.
Verification is the process that proves the system [or sub-system or component] meets its requirements and matches the design. Since verification is based on requirements and design, one of the keys to successful and effective verification is well-written and complete requirements and design documents. These requirements and design elements are developed, reviewed, and approved earlier in the project timeline before the system is developed or procured. Planning for the verification activities starts with the System
Engineering Management Plan [or with the Project Plan if a SEMP is not needed]. At this level, the general structure of the verification tasks is identified and shown to be compatible with the desired deployment plan and with the system concept. The Verification Plans are best written at the same time the requirements of the system, sub-system, or component are developed. This is done to show that the requirements, as written, can be verified. At the end of the detailed design effort, verification procedures can be written. These procedures are the detailed steps to be taken to verify each requirement and design element. There must be a clear trace from each requirement, through the Verification Plan, down to a detailed step in the verification procedure. Verification is performed iteratively. It starts with the integration activities at the component level. It progresses through the sub-system development to the verification of the entire system. Final verification for system acceptance is done with the installed system. At this point, system development is complete and the deployed system is ready for operations. The system's owner and stakeholder involvement is essential for verification.
Verification answers the question "Was the system built ‘right'"
CONTEXT OF PROCESS:
Concept of Operations describes the way the system is to operate and will assist in the verification and integration effort.
System and Sub-system Requirementscontain the functional and performance requirements to be verified.
Design Specificationscontain the design elements to be verified.
Integration Plan [optional] shows how the integration steps are to be done iteratively with verification.
Deployment Strategy [optional] defines when and where verification takes place.
Integrated sub-systems/system is ready for verification.
Project Plan/Systems Engineering Management Plan [SEMP] establishes a high level description of the project's plan for verification.
Configuration Management Plan sets the configuration controls needed during verification.
Stakeholder involvement is needed for verification conduct and to show critical stakeholders that the system meets its requirements.
Technical Reviews include a test readiness review to determine all resources needed for a verification step are available.
Traceability to the verification plan & procedures ensures that all requirement are being verified.
Verification Master Plan is included in the Project Plan/SEMP to establish general guidelines for this important part of the systems engineering process.
Verification Plan documents the plan for verifying system and sub-system requirements.
Verification Procedures document the details of each verification step.
Verification Reports document results of each verification step.
Verified sub-system/system ready for further integration, deployment, or operational use.
Plan verification activities in SEMP / Project Plan
During the project planning stage, a strategy for verification is developed which is compatible with the system concept and the deployment objectives.
Develop Verification Plan
Verification Plan is written for each level [component, sub-system or system]. The plan will develop a verification case and method for each requirement and for each design element contained in the applicable Specification.
In addition to the verification cases, the Verification Plan will give general guidance for all of the verification activities. These include: the identification of all verification participants, descriptions of their roles and responsibilities, and a schedule for verification activities. Finally, it includes: the identification of test equipment needed and of software drivers or simulators needed to model the interfaces to the system under test.
Trace between specifications and test cases
Each test case is traced to a specific requirement to ensure all requirements are verified.
Develop Verification Procedures
These procedures are the detailed step-by-step actions and the expected outcome for each verification case.
When all needed resources are ready, verification is performed according to the approved procedures.
Document verification results
Prepare a Verification Report for each verification step.
Where does Verification take place in the project timeline?
Is there a policy or standard that talks about verification?
FHWA Final Rule does not specifically mention general verification of requirements. It does require inter-operability tests relating to use of ITS standards. IEEE std. 1012 talks about independent verification and validation. CMMI identifies best practices.
Which activities are critical for the system's owner to do?
How do I fit this step to my project? [Tailoring]
Some level of verification is needed to accept the system. The formality with which verification is performed can be tailored to the budget, size, and complexity of the project. For a small simple project with few stakeholders, it only may be necessary to use the requirement document itself as a checklist and extemporize the procedures on the fly. Thus, no verification documents are needed. The system's owner determines what level for verification formality and documentation is needed to satisfy the complexity of the project.
What should I track to reduce project risk and to get what is expected? [Metrics]
Number of verification failures and their cause [poor requirements, design errors, inadequate integration], is an indication of the quality of products from the development team.
Checklist: Are all the bases covered? [
|Was a Verification Plan developed and approved?|
|Were all requirements traced to a Verification Plan test case?|
|Were Verification Procedures developed and approved?|
|Were the key participants identified and trained?|
|Were all resources needed for testing in-place?|
|Were all participants notified of the testing schedule?|
|Was a Verification Report prepared?|
Are there any other recommendations which can help?
A closer look at the stages of verification, verification techniques, and the rules for performing verification
Key stages of verification
A project may require three or more different stages of verification: sub-system, system, and commissioning. The first is iterative with integration. The last is iterative with deployment. System verification and acceptance falls between the two. Of course, special project situations may require some tailoring, and perhaps additional stages, for complete verification.
It may be necessary to overlap the last two stages of verification. System verification can be started in a development environment using simulated inputs from sensors and external system then completed after deployment and commissioning using real sensors and real external systems. While verification with simulated inputs may be necessary, final verification with real inputs is almost always mandatory.
Four techniques are used to verify requirements: inspection, analysis, test, and demonstration.
A special type of demonstration is called a burn-in is used to identify and resolve random or latent defects [thermal, memory leaks, and race conditions].
General rules for performing verification
Be careful of requirements creep. During verification some stakeholders, especially if they have not been involved in the design activities, will want to rewrite or add to the system requirements. A typical example is a desire to change the operator interface. There is a cost and schedule risk of doing this and the best way to avoid these occurrences is to ensure that the correct stakeholders are involved in establishing the requirements and designing the system from the start.