U.S. Department of Transportation
Federal Highway Administration
1200 New Jersey Avenue, SE
Washington, DC 20590
202-366-4000
California Division
Purpose of these Documents
These documents plan, describe, and record the activity of validating that the system meets the intended purpose and needs of the systems’ owner and stakeholders. Since a complex system may involve a series of validation related activities, several sets of these documents may be needed. All of these validation documents follow the master plan for validation defined in the Systems Engineering Management Plan.
Usually, for even moderately complex systems, the following two levels of validation documents are prepared:
These documents are described in this section. Note that for this phase there is no set of detailed procedures. Validation as described in section 3.7.1 involves stakeholders and the actual users of the system. The Validation Plan will lay out the overall expectations for the assessment of the completed system but will let the users work the system as part of their every day job rather than imposing a strict step-by-step procedure. It is their system after all and they need to be comfortable with it and provide the final answer on how well it satisfies their needs. Reporting the results and any corrective actions needed will be part of this effort.
A critical issue is assuring that all user needs are included in the validation activity. This is best done by first tracing each of the needs documented in the Concept of Operations into a validation activity.
Additional Information is found in IEEE 1012-1998, Software Verification and Validation.
Tailoring these Documents to Your Project
A separate Validation Plan may not be required for the simplest projects, especially where the system is essentially COTS and does not involve any custom software development, and where the project office personnel have a very clear understanding of the purpose of the system. In some cases, it is possible to take a copy of the ConOps Document, improvise an outline or scenario to check-off the validated system, and annotate the ConOps with the results of the assessment. This can be a perfectly acceptable way to validate the performance of a simple system.
However, preparation of these validation documents is strongly advised if:
There is also the question of how comprehensive to make the validation effort. It is impossible to cover everything, that is, all possible combinations of actions under all possible operational situations. A good rule of thumb is: if it was important enough to write down as a need, then it should be validated, at least once, as part of using the system in a real-world environment. This may not, for example, test all possible failure mode conditions. If a good job was done in earlier phases of testing and verification then the most important and most likely scenarios are covered.
Checklist: Critical Information
Is there a documented Validation Plan for the system of interest? | |
Does the Validation Plan answer all the questions of who, what, where, and when concerning validation? | |
Does the Validation Plan make clear what needs to happen if a problem is encountered? | |
Does the Validation Plan define the environmental conditions and systems configuration needed for each scenario? | |
Are all applicable user needs traced to an event in the Validation Plan? Does each validation event define a realistic and doable scenario? | |
Was a Validation Report developed that documents the validation results? | |
Does the Validation Report describe, in detail, the resolution of every anomaly encountered during testing? | |
Does the Validation Report include recommendations from the users and stakeholders to address the situation in future system evolutions? |
VALIDATION PLAN TEMPLATE
IEEE 1012-1998 Independent Verification and Validation
section |
contents |
---|---|
Title Page |
The title page should follow the Transportation Agency procedures or style guide. At a minimum, it should contain the following information:
|
1.0 Purpose of Document |
This section identifies the type of validation activity to be performed within this Plan. For instance, this activity may validate the entire system, a sub-system, the deployment at a site, or any other validation activity called for in the Program Plan or in the SEMP. |
2.0 Scope of Project |
This section gives a brief description of the planned project and the purpose of the system to be built. Special emphasis is placed on the project’s complexities and challenges that must be addressed by the systems engineering efforts. This section also describes the environment in which the project operates. It identifies the organization structures that encompass all stakeholders. It also gives a brief description of the role to be played by each stakeholder. This includes ad hoc and existing management work groups and multi-disciplinary technical teams that should be formed for supporting the project. Such teams are critical to reaching successful system deployment. |
3.0 Referenced Documents |
This is a list of all documents used in the preparation of this Validation Plan. This almost always includes the Project Plan, the SEMP [if one was written], and the Concept of Operations. However, reference of other documents, such as descriptions of external systems, standards, and manuals may need to be included. |
4.0 Validation Conduct |
This section provides details on how the validation is accomplished. It defines: who does it; when and where it is to be done; the responsibilities of each participant before, during, and after each event/activity; the hardware and software to be used [and other systems as well]; and the documents to be prepared as a record of the activity. Another very important part of this section defines how anomalies are to be handled [that is, what to do when something fails or, in the case of Validation, does not match the documented needs or does not satisfactorily address the original problem]. In general, the following information should be included in this section:
- Notification of participants - Emphasis on the management role of the operators - Procedures for approving last minute changes to the scenarios
|
5.0 Validation Event Identification |
This section is where we identify the specific scenarios and other events to be performed. For Validation, scenarios can be clustered around a typical operator’s use of the system. It may also be structured around the operational needs defined in the baseline ConOps. There may also be events setup to exercise the final system during failure modes or even situations such as loss of power to the building or a flood near the field equipment. The actual grouping of Needs into a validation event is arbitrary. They should be related and easily combined into a reasonable set of repeatable actions. Each event should contain at least the following information:
|
section |
contents |
---|---|
Title Page |
The title page should follow the Transportation Agency procedures or style guide. At a minimum, it should contain the following information:
|
1.0 Purpose of Document |
This section identifies the type of validation performed. For instance, the activity may validate the entire system, a sub-system, the deployment at a site, or any other validation activity called for in the Program Plan or in the SEMP. This section can be taken from the applicable Validation Plan. |
2.0 Identification of the Configuration under test |
This section identifies the equipment and software validated. It also identifies all equipment and software necessary for this validation activity that is external to the system / sub-system configuration. This may include special test equipment and any external systems with an interface to the system. This section can be taken from the applicable Validation Plan and updated to reflect the actual system as delivered. |
3.0 Individual Validation Reports |
This section summarizes the purpose and results of each event performed in the applicable Validation Plan. Special attention is paid to any situation where a failure (or deviation from the expected System performance) occurred and how the failure was resolved. This section covers:
|