U.S. Department of Transportation
Federal Highway Administration
1200 New Jersey Avenue, SE
Washington, DC 20590
202-366-4000


Skip to content
Facebook iconYouTube iconTwitter iconFlickr iconLinkedInInstagram

California Division

Home / About / Field Offices / California Division / Systems Engineering Guidebook for ITS

Home What's New Systems Engineering Guidebook Views Search Glossary Resources Feedback Site Map

8.4.11 Validation Documents Template

Purpose of these Documents

These documents plan, describe, and record the activity of validating that the system meets the intended purpose and needs of the systems’ owner and stakeholders.  Since a complex system may involve a series of validation related activities, several sets of these documents may be needed. All of these validation documents follow the master plan for validation defined in the Systems Engineering Management Plan.

Usually, for even moderately complex systems, the following two levels of validation documents are prepared:

  • a plan to lay out the specific validation efforts
  • a report on the results of the validation activity

These documents are described in this section.  Note that for this phase there is no set of detailed procedures.  Validation as described in section 3.7.1 involves stakeholders and the actual users of the system.  The Validation Plan will lay out the overall expectations for the assessment of the completed system but will let the users work the system as part of their every day job rather than imposing a strict step-by-step procedure.  It is their system after all and they need to be comfortable with it and provide the final answer on how well it satisfies their needs.  Reporting the results and any corrective actions needed will be part of this effort.

A critical issue is assuring that all user needs are included in the validation activity. This is best done by first tracing each of the needs documented in the Concept of Operations into a validation activity.

Additional Information is found in IEEE 1012-1998, Software Verification and Validation.

Tailoring these Documents to Your Project

A separate Validation Plan may not be required for the simplest projects, especially where the system is essentially COTS and does not involve any custom software development, and where the project office personnel have a very clear understanding of the purpose of the system. In some cases, it is possible to take a copy of the ConOps Document, improvise an outline or scenario to check-off the validated system, and annotate the ConOps with the results of the assessment. This can be a perfectly acceptable way to validate the performance of a simple system.

However, preparation of these validation documents is strongly advised if:

  • the system is more complex
  • there are a number of separate validation activities
  • multiple deployment sites are involved
  • more than one or two stakeholders have to be satisfied

There is also the question of how comprehensive to make the validation effort. It is impossible to cover everything, that is, all possible combinations of actions under all possible operational situations. A good rule of thumb is: if it was important enough to write down as a need, then it should be validated, at least once, as part of using the system in a real-world environment. This may not, for example, test all possible failure mode conditions. If a good job was done in earlier phases of testing and verification then the most important and most likely scenarios are covered.

Checklist: Critical Information

check Is there a documented Validation Plan for the system of interest?
check Does the Validation Plan answer all the questions of who, what, where, and when concerning validation?
check Does the Validation Plan make clear what needs to happen if a problem is encountered?
check Does the Validation Plan define the environmental conditions and systems configuration needed for each scenario?
check Are all applicable user needs traced to an event in the Validation Plan?  Does each validation event define a realistic and doable scenario?
check Was a Validation Report developed that documents the validation results?
check Does the Validation Report describe, in detail, the resolution of every anomaly encountered during testing?
check Does the Validation Report include recommendations from the users and stakeholders to address the situation in future system evolutions?

 

VALIDATION PLAN TEMPLATE

IEEE 1012-1998 Independent Verification and Validation

section

contents

Title Page

The title page should follow the Transportation Agency procedures or style guide. At a minimum, it should contain the following information:

  • VALIDATION PLAN FOR THE [insert name of project] AND [insert name of transportation agency]
  • Contract number
  • Date that the document was formally approved
  • The organization responsible for preparing the document
  • Internal document control number, if available
  • Revision version and date issued

1.0 Purpose of Document

This section identifies the type of validation activity to be performed within this Plan. For instance, this activity may validate the entire system, a sub-system, the deployment at a site, or any other validation activity called for in the Program Plan or in the SEMP.

2.0 Scope of Project

This section gives a brief description of the planned project and the purpose of the system to be built. Special emphasis is placed on the project’s complexities and challenges that must be addressed by the systems engineering efforts.

This section also describes the environment in which the project operates. It identifies the organization structures that encompass all stakeholders. It also gives a brief description of the role to be played by each stakeholder. This includes ad hoc and existing management work groups and multi-disciplinary technical teams that should be formed for supporting the project. Such teams are critical to reaching successful system deployment.

3.0 Referenced Documents

This is a list of all documents used in the preparation of this Validation Plan. This almost always includes the Project Plan, the SEMP [if one was written], and the Concept of Operations. However, reference of other documents, such as descriptions of external systems, standards, and manuals may need to be included.

4.0 Validation Conduct

This section provides details on how the validation is accomplished. It defines: who does it; when and where it is to be done; the responsibilities of each participant before, during, and after each event/activity; the hardware and software to be used [and other systems as well]; and the documents to be prepared as a record of the activity. Another very important part of this section defines how anomalies are to be handled [that is, what to do when something fails or, in the case of Validation, does not match the documented needs or does not satisfactorily address the original problem].

In general, the following information should be included in this section:

  • A description of the participating organizations and personnel and identification of their roles and responsibilities. This may include for example, the operators, an event recorder, witnesses, and/or engineering support.  Some agencies prefer to have contractors not around during validation, others want access to them in case questions or problems arise.
  • Identification of the location of the activity, that is, the place, or places, where the progress must be observed. 
  • The schedule of when Validation will occur including a sequencing of the events that make up the Validation activity.
  • The system configuration for all of the activities, including the main system hardware and software and any supporting equipment, software, or external systems.  Several configurations may be used depending on the type of system and type of development that was just completed.  For instance, a signal upgrade may have a smaller configuration to validate than a new TMC.
  • Identification of the documents to be prepared to support the validation, including any special scenarios, a Validation Report and descriptions of special test equipment and software.
  • Details on the actual conduct of the activity, including:

-  Notification of participants

-  Emphasis on the management role of the operators

-  Procedures for approving last minute changes to the scenarios

  • The processes for handling anomalies, including recording of critical information, resolution of the cause of a failure [e.g. fix the software, reset the system, change the ConOps, record potential future changes], and determination of any retesting activities necessary.

 

5.0 Validation Event Identification

This section is where we identify the specific scenarios and other events to be performed.  For Validation, scenarios can be clustered around a typical operator’s use of the system.  It may also be structured around the operational needs defined in the baseline ConOps.  There may also be events setup to exercise the final system during failure modes or even situations such as loss of power to the building or a flood near the field equipment.  The actual grouping of Needs into a validation event is arbitrary. They should be related and easily combined into a reasonable set of repeatable actions.

Each event should contain at least the following information:

  • A description name and a reference number
  • A complete list of the needs to be validated. For ease of tracing into the Validation Plan and other documents, the Needs are given numbers. They can be accurately and conveniently referenced without repeating all the words from the ConOps.
  • A description of the objective of the event, usually taken from the wording of the Needs
  • Any data to be recorded or noted during the event.
  • A statement of the pass/fail criteria. Often, this is just a statement that the system satisfies the needs.
  • A description of the system configuration. That is a list of the hardware and software items needed and how they should be connected. Often, the same configuration is used for several events/scenarios
  • A list of any other important assumptions and constraints necessary for conduct of the event

 

section

contents

Title Page

The title page should follow the Transportation Agency procedures or style guide. At a minimum, it should contain the following information:

  • VALIDATION REPORT FOR THE [insert name of project] AND [insert name of transportation agency]
  • Contract number
  • Date that the document was formally approved
  • The organization responsible for preparing the document
  • Internal document control number, if available
  • Revision version and date issued

1.0 Purpose of Document

This section identifies the type of validation performed. For instance, the activity may validate the entire system, a sub-system, the deployment at a site, or any other validation activity called for in the Program Plan or in the SEMP. This section can be taken from the applicable Validation Plan.

2.0 Identification of the Configuration under test

This section identifies the equipment and software validated. It also identifies all equipment and software necessary for this validation activity that is external to the system / sub-system configuration. This may include special test equipment and any external systems with an interface to the system. This section can be taken from the applicable Validation Plan and updated to reflect the actual system as delivered.

3.0 Individual Validation Reports

This section summarizes the purpose and results of each event performed in the applicable Validation Plan. Special attention is paid to any situation where a failure (or deviation from the expected System performance) occurred and how the failure was resolved. This section covers:

  • Event overview and results
  • Completed Validation Plan pages annotated with results
  • Description of each anomaly, if any, from the expected result called for in the Validation Plan
  • Any back-up data or records related to the experience
  • Details of the resolution of each anomaly, including procedure modifications, software fix, re-testing and results, regression testing and results, and required document changes [including changes to the Conops, new requirements for next version].

 

Related Task Checklist  

 

Return to top
Page last modified on September 20, 2016
Federal Highway Administration | 1200 New Jersey Avenue, SE | Washington, DC 20590 | 202-366-4000