U.S. Department of Transportation
Federal Highway Administration
1200 New Jersey Avenue, SE
Washington, DC 20590
Validation starts with a clearly stated set of needs. These needs are the basis for the system requirements. When the system is developed, the system is assessed against these needs.
The validation process has three primary activities:
Planning: With stakeholder involvement planning starts at the beginning of the project timeline. The plan includes who will be involved, what will be validated, what is the schedule for validation, and where the validation will take place.
Validation strategy: This defines how the validation will take place and what resources will be needed. For example, whether a before and/or an after study will be needed. If so, the before study will need to be done prior to deployment of the system.
Perform validation: After the system has been accepted, the system should be assessed based on planning & strategy and the results documented.
The system’s owner and stakeholders are responsible for the validation of the system. The primary systems engineering activity is to assist in development and execution of all three activities.
Validation answers the question "Was the 'right' system built?"
CONTEXT OF PROCESS:
SYSTEM VALIDATION PROCESS
Verified systemafter the system has been verified [accepted by the system's owner]; it is ready for validation testing.
Concept of Operations provides the goals, objectives, and needs to be assessed.
Project Plan/Systems Engineering Management Plan [SEMP] includes the validation plan used to identify the strategy, schedule, and resources for validation.
Stakeholder involvement includes the system’s owner and its stakeholders. Each will have needs that the system is intended to address. When the assessment is performed, the stakeholders must be in agreement on the plan, strategy, and outcome of the assessment.
Traceability from the concept of operations to the validation plan & procedures ensures that the user needs are validated when the system is deployed.
Validation Master Plan specifies what needs to be validated, where, and when. This becomes part of the Systems Engineering Management Plan [SEMP].
Validation Plan defines how the validation will be performed. In particular, it specifies whether a before and after study is needed. If special environmental conditions or resources are needed to conduct the assessment.
Validated system [Assessment of the system] is one that has been assessed against the initially stated needs. It may have fallen short in some areas and exceeded in others. The short falls are used to identify new requirements for the evolution of the system.
Validation report documents the results of the validation process: the strengths and weaknesses of the system. It shows where improvement can be made.
Develop validation strategy
Validation planning occurs at the beginning of the project and is part of the Systems Engineering Management Plan. The plan includes the environment for validation resources. A validation plan is developed as part of the systems planning and concept of operations.
Strategies include alpha testing, beta testing, and an evaluation period for validation. If before and after studies are needed, it will be identified in the strategy.
Validate system [Assessment of the system capabilities in operations]
Once the system has been accepted and deployed, the functionality and performance of the system are validated [assessed] against the needs, goals, and objectives as stated in the concept of operations. Also, the system is assessed in the "real-world " operations to evaluate the system against expectations of the system’s owner and stakeholders. This evaluation can result in one of the following:
Case 1] System performs as expected.
Action: Expand the system to address additional needs and document the emergent qualities of the system as it is in operations. New requirements will be developed for the next evolution of the system.
Case 2] Needs were not clearly articulated and the system falls short of expectations.
Action: Improve the process used for the elicitation of needs and involvement of stakeholders and then correct the definition of needs. Develop the correct set of requirements for the next evolution of the system.
Case 3] The problem space was not understood and the needs were based on the ill-defined problem.
Actions: Improve the problem definition process and the elicitation processes. Re-evaluate the problem space and needs to ensure it is understood for the next evolution.
Where does the Validation take place in the project timeline?
Is there a policy or standard that talks about Validation?
FHWA Final Rule does not specifically mention general validation practices to be followed. IEEE-1012 Independent verification and validation and CMMI identify best practices.
Which activities are critical for the system’s owner to do?
How do I fit these activities to my project? [Tailoring]
There is great latitude in system validation. It is dependent on institutional agreements (State and FHWA requirements) on a per project basis. In signal upgrade systems a simple before and after study on selected intersections may be sufficient to validate. In a more complex system a number of evaluations may be needed. This validation may be needed for each stakeholder element, each sub-system [e.g., camera, CMS, and detection system]. It may be done on a sample area of the system or comprehensively. Getting this addressed with the stakeholders in the planning stage is very important.
What should I track in this process step to reduce project risks and get what is expected? [Metrics]
On the technical side:Each need, goal, and objective should have an element that can be measured and tracked throughout the development. For example, for an incident management system the goal of the planned system may be to reduce incident management time by 30%. The technical metric is "time ". This includes, for example, detect time, time to verify, response time, and time to clear. The time would be the metric to monitor throughout the development.
On the project management side:At this point the development is complete. As the project manager, it will be important to validate the systems as soon as possible and IAW the plan. If validation is delayed too long, the assessment may become more difficult to accomplish [lack of resources and interest] and [with the changing environment] the results of the assessment may become diluted. [E.g. change in traffic patterns, increase in congestion over time].
Checklist: Are all the bases covered?
|Were all the needs clearly documented?|
|With each need, goal, and objective is there an outcome that can be measured?|
|Are all the stakeholders involved in the validation planning and the definition of the validation strategy?|
|Are all the stakeholders involved in the performance of the validation and is there an agreement on the planned outcomes?|
|Are there adequate resources to complete the validation?|
|Are the system’s owner and stakeholders participating in the requirements walkthrough and approval process?|
|Is there adequate systems engineering support for the validation planning, strategy, and performing validation?|
Are there any other recommendations that can help?
This is an area of high stakeholder involvement. Ample time should be given to this activity. Clearly identifying measurable needs, goals, and objectives is critical for assessing the system as well as the development of a good set of requirements for the system.
The systems engineering mindset is to "start at the finish line " [what the system is to do and how well it is to do it]. This clear end point is essential for the successful completion of the system. The journey may encounter detours, road blocks, and it may be longer than expected. The validation process helps the system’s owner in making this "finish line " clear to the stakeholders and to the development team.
Validate the system as quickly as possible. There may be a tendency to lose interest once the system has been developed, accepted by the system’s owner, deployed, and commissioned into service, assuming that the system is doing the job it was intended to do. With Intelligent Transportation Systems [ITS], it is not only the delivery of the project [system] that is important, but that the project [system] delivered meets the users needs.[Was the right system delivered?] This can only be done through the validation process. The system’s owner and stakeholders should follow through as soon as possible with the assessment of the system.
What is the difference between Validation and Verification? First let us look at validation, then verification.
Validation determines if the system is being developed will meet the intended needs of the system’s owner and stakeholders when completed. Does the system solve the problem or issue that it was intended to solve? Does it solve it to the expected extent?
The needs, vision, goals, and objectives are the starting points for validating the system. It sets the "stake " in the ground and says this is what we want, what problem we intend to solve, and to what extent we want to address the issues. [Performance metrics]
The first part of validation is to make sure that the system development starts out on the right track. This is done by validating the requirements of the system this is done on the left side of the Vee during the requirements development phase. Are these the "right " requirements being implemented? This question needs to be addressed early in the project timeline. It requires high stakeholder involvement and an accurate translation of the needs, goals, and objectives into a set of system requirements that can be built. The system’s owner should take ample time to clarify the vision, goals, objectives, and needs. They need to be made measurable. The translation of the needs into system requirements is done using the elicitation process and other techniques. For example, similar systems, technology review, prototyping, and/or modeling. The second part of validation is at the end of development where the system has been accepted and is now put into operations. Does the system do what it was intended to do, and to what extent? Was the "right " system built?
Verification is the process which makes sure that what was built matches the requirements. Was the system built the way the requirements and design specified? Was the system built "right "? Both the verification and validation processes are important and necessary. However, it is the validation which views the system from the system’s owner and stakeholder perspective. The verification of the system is viewed from the development team’s perspective. Systems engineering’s goal is to unify these views.