3.6.3 Verification [Sub-system and system level verification]

Is there a policy or standard that talks about verification?

FHWA Final Rule does not specifically mention general verification of requirements. It does require inter-operability tests relating to use of ITS standards. IEEE std. 1012 talks about independent verification and validation. CMMI identifies best practices.

Which activities are critical for the system's owner to do?

• Identify and recruit stakeholders who are needed to participate in verification
• Review and approve all documents
• Witness critical verification steps

How do I fit this step to my project? [Tailoring]

Some level of verification is needed to accept the system. The formality with which verification is performed can be tailored to the budget, size, and complexity of the project. For a small simple project with few stakeholders, it only may be necessary to use the requirement document itself as a checklist and extemporize the procedures on the fly. Thus, no verification documents are needed. The system's owner determines what level for verification formality and documentation is needed to satisfy the complexity of the project.

What should I track to reduce project risk and to get what is expected? [Metrics]

Number of verification failures and their cause [poor requirements, design errors, inadequate integration], is an indication of the quality of products from the development team.

Checklist: Are all the bases covered? [

	Was a Verification Plan developed and approved?
	Were all requirements traced to a Verification Plan test case?
	Were Verification Procedures developed and approved?
	Were the key participants identified and trained?
	Were all resources needed for testing in-place?
	Were all participants notified of the testing schedule?
	Was a Verification Report prepared?

Are there any other recommendations which can help?

A closer look at the stages of verification, verification techniques, and the rules for performing verification

Key stages of verification

A project may require three or more different stages of verification: sub-system, system, and commissioning. The first is iterative with integration. The last is iterative with deployment. System verification and acceptance falls between the two. Of course, special project situations may require some tailoring, and perhaps additional stages, for complete verification.

• Sub-system Verification – As discussed in 3.4.2, High Level Design, a system is often divided into two or more sub-systems for ease of development. Once the integration process has produced one of these sub-systems, it is verified against its requirements. Once verified, the sub-system can be integrated with other sub-systems.
• System Verification step covers all integrated sub-systems and is usually used to accept the entire system. For many requirements, this is the last time they are verified. As such, this step is the most formal, reviewed, witnessed, and where there are failures, receives the most attention. It may not be as exhaustive as sub-system verification. Yet, it still must be extensive enough to produce a solid feeling among the stakeholders that the system does what it is supposed to do.
• Sub-system and system verification is best done in a highly controlled environment, especially with respect to external inputs to the system under test. This usually requires software to simulate or model the external world. For instance, a traffic signal simulator or roadway sensor simulator may be needed to test a new central control system.
• Commissioning is accomplished after the system is deployed to verify that the system works when installed. Commissioning is generally more cursory than system verification. It is just enough to verify that everything is still working. However, in some circumstances, a part of system verification must be deferred to the time of commissioning; again using simulated inputs as needed to complete the needed verification prior to commissioning.
• Verification of the system's ability to work with the complete set of real sensors must wait until after deployment.

It may be necessary to overlap the last two stages of verification. System verification can be started in a development environment using simulated inputs from sensors and external system then completed after deployment and commissioning using real sensors and real external systems. While verification with simulated inputs may be necessary, final verification with real inputs is almost always mandatory.

Verification techniques

Four techniques are used to verify requirements: inspection, analysis, test, and demonstration.

• Inspection: the visual verification of a requirement, such as a color, a size, and model number.
• Analysis: the mathematical analysis of collected data to verify a requirement
• Demonstration: the use of the system itself to verify the expected output, such as a response to an operator input. This is the most commonly used verification technique.
• Test: similar to a demonstration except external test equipment is used.

A special type of demonstration is called a burn-in is used to identify and resolve random or latent defects [thermal, memory leaks, and race conditions].

General rules for performing verification

• notify all stakeholders of the schedule for verification and clarification of their roles and responsibilities
• identify and document the configuration of the system under test
• define the process for recording all test actions and the system's response
• define the process for dealing with all unexpected responses
• define the process to manage anomalies
• define a plan of action based on this analysis. [e.g. repeat the test, revise procedure, change the requirement, suspend the test, fix the system and retest]

Be careful of requirements creep. During verification some stakeholders, especially if they have not been involved in the design activities, will want to rewrite or add to the system requirements. A typical example is a desire to change the operator interface. There is a cost and schedule risk of doing this and the best way to avoid these occurrences is to ensure that the correct stakeholders are involved in establishing the requirements and designing the system from the start.