 |
|
OBJECTIVE: The verification process is used by the system’s owner and by other stakeholders to show that the as-built system, sub-system, and components meet all of their requirements and design. This process is used by the system’s owner and other stakeholders to accept the system products from the development team. |
|
DESCRIPTION: Verification is the process that proves the system [or sub-system or component] meets its requirements and matches the design. Since verification is based on requirements and design, one of the keys to successful and effective verification is well-written and complete requirements and design documents. These requirements and design elements are developed, reviewed, and approved earlier in the project timeline before the system is developed or procured. Planning for the verification activities starts with the System
Verification answers the question “Was the system built ‘right’” |
|
CONTEXT OF PROCESS:
VERIFICATION PROCESS |
|
Inputs: Concept of Operations describes the way the system is to operate and will assist in the verification and integration effort. System and Sub-system Requirements contain the functional and performance requirements to be verified. Design Specifications contain the design elements to be verified. Integration Plan [optional] shows how the integration steps are to be done iteratively with verification. Deployment Strategy [optional] defines when and where verification takes place. Integrated sub-systems/system is ready for verification. |
|
Control: Project Plan/Systems Engineering Management Plan [SEMP] establishes a high level description of the project’s plan for verification. Configuration Management Plan sets the configuration controls needed during verification. |
|
Enablers: Stakeholder involvement is needed for verification conduct and to show critical stakeholders that the system meets its requirements. Technical Reviews include a test readiness review to determine all resources needed for a verification step are available. Traceability to the verification plan & procedures ensures that all requirement are being verified. |
|
Outputs: Verification Master Plan is included in the Project Plan/SEMP to establish general guidelines for this important part of the systems engineering process. Verification Plan documents the plan for verifying system and sub-system requirements. Verification Procedures document the details of each verification step. Verification Reports document results of each verification step. Verified sub-system/system ready for further integration, deployment, or operational use. |
|
Process Activities: Plan verification activities in SEMP / Project Plan During the project planning stage, a strategy for verification is developed which is compatible with the system concept and the deployment objectives. Develop Verification Plan Verification Plan is written for each level [component, sub-system or system]. The plan will develop a verification case and method for each requirement and for each design element contained in the applicable Specification. In addition to the verification cases, the Verification Plan will give general guidance for all of the verification activities. These include: the identification of all verification participants, descriptions of their roles and responsibilities, and a schedule for verification activities. Finally, it includes: the identification of test equipment needed and of software drivers or simulators needed to model the interfaces to the system under test. Trace between specifications and test cases Each test case is traced to a specific requirement to ensure all requirements are verified. Develop Verification Procedures These procedures are the detailed step-by-step actions and the expected outcome for each verification case. Perform verification When all needed resources are ready, verification is performed according to the approved procedures. Document verification results Prepare a Verification Report for each verification step. |
|
Where does Verification take place in the project timeline? |
Is there a policy or standard that talks about verification?
FHWA Final Rule does not specifically mention general verification of requirements. It does require inter-operability tests relating to use of ITS standards. IEEE std. 1012 talks about independent verification and validation. CMMI identifies best practices.
Which activities are critical for the system’s owner to do?
§ Identify and recruit stakeholders who are needed to participate in verification
§ Review and approve all documents
§ Witness critical verification steps
How do I fit this step to my project? [Tailoring]
Some level of verification is needed to accept the system. The formality with which verification is performed can be tailored to the budget, size, and complexity of the project. For a small simple project with few stakeholders, it only may be necessary to use the requirement document itself as a checklist and extemporize the procedures on the fly. Thus, no verification documents are needed. The system’s owner determines what level for verification formality and documentation is needed to satisfy the complexity of the project.
What should I track to reduce project risk and to get what is expected? [Metrics]
Number of verification failures and their cause [poor requirements, design errors, inadequate integration], is an indication of the quality of products from the development team.
Checklist: Are all the bases covered? [
R Was a Verification Plan developed and approved?
R Were all requirements traced to a Verification Plan test case?
R Were Verification Procedures developed and approved?
R Were the key participants identified and trained?
R Were all resources needed for testing in-place?
R Were all participants notified of the testing schedule?
R Was a Verification Report prepared?
Are there any other recommendations which can help?
A closer look at the stages of verification, verification techniques, and the rules for performing verification
Key stages of verification
A project may require three or more different stages of verification: sub-system, system, and commissioning. The first is iterative with integration. The last is iterative with deployment. System verification and acceptance falls between the two. Of course, special project situations may require some tailoring, and perhaps additional stages, for complete verification.
§ Sub-system Verification – As discussed in 3.4.2, High Level Design, a system is often divided into two or more sub-systems for ease of development. Once the integration process has produced one of these sub-systems, it is verified against its requirements. Once verified, the sub-system can be integrated with other sub-systems.
§ System Verification step covers all integrated sub-systems and is usually used to accept the entire system. For many requirements, this is the last time they are verified. As such, this step is the most formal, reviewed, witnessed, and where there are failures, receives the most attention. It may not be as exhaustive as sub-system verification. Yet, it still must be extensive enough to produce a solid feeling among the stakeholders that the system does what it is supposed to do.
§ Sub-system and system verification is best done in a highly controlled environment, especially with respect to external inputs to the system under test. This usually requires software to simulate or model the external world. For instance, a traffic signal simulator or roadway sensor simulator may be needed to test a new central control system.
§ Commissioning is accomplished after the system is deployed to verify that the system works when installed. Commissioning is generally more cursory than system verification. It is just enough to verify that everything is still working. However, in some circumstances, a part of system verification must be deferred to the time of commissioning; again using simulated inputs as needed to complete the needed verification prior to commissioning.
§
Verification of the system’s ability to work with the
complete set of real sensors must wait until after deployment.
It may be necessary to overlap the last
two stages of verification. System verification can be started in a
development environment using simulated inputs from sensors and external system
then completed after deployment and commissioning using real sensors and real
external systems. While verification with simulated inputs may be necessary,
final verification with real inputs is almost always mandatory.
Verification techniques
Four techniques are used to verify requirements: inspection, analysis, test, and demonstration.
§ Inspection: the visual verification of a requirement, such as a color, a size, and model number.
§ Analysis: the mathematical analysis of collected data to verify a requirement
§ Demonstration: the use of the system itself to verify the expected output, such as a response to an operator input. This is the most commonly used verification technique.
§ Test: similar to a demonstration except external test equipment is used.
A special type of demonstration is called a burn-in is used to identify and resolve random or latent defects [thermal, memory leaks, and race conditions].
General rules for performing verification
§ notify all stakeholders of the schedule for verification and clarification of their roles and responsibilities
§ identify and document the configuration of the system under test
§ define the process for recording all test actions and the system’s response
§ define the process for dealing with all unexpected responses
§ define the process to manage anomalies
§ define a plan of action based on this analysis. [e.g. repeat the test, revise procedure, change the requirement, suspend the test, fix the system and retest]
Be careful of requirements creep. During verification some stakeholders, especially if they have not been involved in the design activities, will want to rewrite or add to the system requirements. A typical example is a desire to change the operator interface. There is a cost and schedule risk of doing this and the best way to avoid these occurrences is to ensure that the correct stakeholders are involved in establishing the requirements and designing the system from the start.
|
![Shows the flow for Phase [3] Task 3, Verification Process. Summaries are described for inputs, constraints, and enablers into the task; activities of the task; and outputs from the task. The flow is described in detail in the accompanying text.](31_files/image001.gif)
California Division