Systems Engineering Guidebook for ITS - Example Project 2 - Adding New Functionality to an Existing System

4.11.2 Example Project 2 - Adding New Functionality to an Existing System

[Please note that the solution given here is for this example only. Other viable solutions may be possible. Each must be evaluated for a given project.]

Building on Example 1, a new requirement was added. The changeable message signs were to have shared control with a partner agency [Agency B]. Primary agency A owns and operates the signs and the host system that controls them. This new requirement was driven by the development of a regional architecture. The existing CMS host system was deployed prior to the regional architecture. The requirement reads, “The changeable message sign system shall share control with agency B”. For this example, the smaller agency B manages events at two centers. As part of the installation, the primary agency will be installing six signs that would assist agency B for their event management. Agency B would use the CMS in coordination with their local control of traffic signals to divert traffic to appropriately get the attendees in and out of the event faster and more safely.

New software may need to be developed and integrated into the existing system. The project had an initial cost estimated at $10.5 million for the signs, new software, workstations, and communications for the participating agencies and systems engineering activities. With this new requirement, new risks and complexity are introduced relative to example 1. It is recommended that, for this example, the following systems engineering processes be used to clearly define and develop the shared control of the CMS. [In this example some of the steps needed for example project 1 [section 4.11.1] may be incorporated, e.g., Technical Reviews, others, e.g., Unit Verification by the vendor still needs to be performed ]

Process Step	Estimated Level of Effort	Check list of supporting activities	Check list of issues	Check list of risks	Examples
Feasibility	Medium 2-5 pages	R Procurement R Trade study R Stakeholder involvement R Elicitation R Risk mgt	R Procure the services of systems engineering services R Address all user needs R Definition of the problem R Scope of the problem R Possible solution concepts R Estimated cost and benefit R Identification of the portion of the regional architecture that this will fulfill R Institutional issues R Feasibility with existing system[s] R Feasibility with partner agencies R Identify project risks R Technical metrics and performance R Document the feasibility analysis	R Picking a point solution without considering the business case or cost benefit of alternatives R Selecting a solution that is not appropriate among participating agencies R Proposing a solution that is too costly R Incomplete solutions	Definition of the problem and need: “Sharing of CMS by Agency B for event management, and to provide alternate routing at the beginning and ending of the event”. Scope: Agency B needs shared control of 6 CMS that are in the event areas Feasibility: Can the existing software be modified to include this new requirement? How much reverse engineering is needed to integrate the new requirement into the existing system? Trade study and cost benefit: Evaluate stand-alone systems controlling the signs or integrate software functionality into legacy system at the primary agency. Institutional issues: Equipment standards different between the agencies. Limited support staff and maintenance at agency B. Cost Estimate: Reverse engineering effort increased the cost of the project to $10.7 million from the original $10.5 million Identified Risks: Interagency MOUs cannot be signed or delayed Reverse engineering will be more costly than expected Standards and license agreements Security Maintenance Limited solution [not general enough for region]
Planning	Low* to Medium SEMP framework developed 2-3 pages	R Stakeholder involvement R Risk mgt R Config. mgt R Project planning R Technical reviews	R Identification of expected plans for the project R Expected content and quality of the plans R Expectations on the effort needed for the development R Schedule & budget R Monitoring and controlling of effort	R Loss of control of the project deliverable, schedule & budget R Missing critical activities R Lack of long term maintenance & operations R Not meeting expectations	The identified technical plans include: Development Plan [Software, Hardware] Integration Plan Deployment Plan Verification and Validation Plans Development team CM Plan Project Plan Operations & Maintenance Plan Configuration Management Plan Risk Management Plan * Note: This effort will be low if plan frameworks have already been done, medium effort if they need to be developed.
Concept of Operations & Validation Plan	Medium 5-10 pages	R Elicitation R Stakeholder involvement R Risk mgmt R Trade studies R Technical reviews	R Definition of the way the system will operate and be maintained R Identification of the project level stakeholders e.g. R Maintenance R Supervisors R Operators R IT department R Agencies’ risk managers R Validation of the system R Limitations of shared control R Alternative operational concepts R Definition of user needs at the project level R Identification of risks R Target performance of the shared control R Revised cost estimate R Agency’s normal Operations & Maintenance Standards	R Lack of understanding on: R How shared control will operate with limitations R How the system will be maintained R Scope of the project R Who will be impacted by the control R How the system will be validated R What is needed for shared control R Project risks R Project needs R Operational & Maintenance Standards and agency limitations R City’s policies and risks regarding control of the CMS R Missing alternative operational strategies	How shared control will operate with limitations: Agency B staff needs to monitor the status of the 6 CMS and post messages on alternate routes for local events and emergency traffic conditions Be able to remotely control the signs from the supervisor’s home. Limited to the use of pre-developed canned messages by agency B How the system will be Maintained: Agency B maintenance is limited and lacks the skills to maintain the communications link The standard for Agency B is a Windows-based workstation or PC. Staff can install software if installation instructions are provided or there is a standard installation wizard. The primary agency will maintain the host and communications system and provide installation support to agency B. Operational Standards and Norms Agency B is a 5-day operation that is supported on weekends for events and emergencies from the supervisor’s home. The primary agency is a 7-day 24-hour operation. On weekends, if agency B cannot be reached, the primary agency has the authority to post messages on behalf of agency B [MOUs] as agreed. If any system fault occurs, the primary agency would need to identify and resolve the problem. Additional risks identified: Security of the remote link into the system; a security plan will be needed. [Update SEMP with a framework of a Security Plan.] Validation of the shared control The transfer of control between agencies will be in accordance with the scenario developed in the conops.
Development of System Level Requirements and Verification Plans	Medium 5-7 pages of Requirements and a 5-7 page Verification Plan	R Stakeholder involvement R Elicitation R Technical reviews R Trade studies R Risk mgmt R Config mgmt R Traceability	R Definition of what the system is to do to support the identified needs R What will be used as the basis for accepting the completed system? R New risks that may be uncovered R New requirements may be needed R Are all the needs addressed? R Is each need addressed completely? R What will be needed to support the development, operations & maintenance? R Project cost R The important things are implemented R Establish a baseline of Requirements that will be used to build the system R Validation of Requirements	R Not having a basis to accept the system when completed R Not completely defining what the system is to do R Scope creep R Expectations not met R Project cost R Losing control and visibility of the development R Requirements not validated by stakeholders	Definition of what the system is to do to support the identified needs “The changeable message sign system shall share control with agency B.” “Agency B shall have remote access to the CMS system.” “Remote access to the CMS host shall be secure.” “Remote access shall be limited to a pre-defined set of messages.” What will be used for the basis of verification and acceptance of the system? Verification Plan would contain: Demonstrate that only a pre-defined set of messages can be displayed. Analysis that the system is secure. What will be needed to support the development, operations & maintenance? Users and maintenance documentation shall be provided Installation documentation shall be developed for the host and remote users. Project Costs With the additional support documentation and security aspects the project costs have been revised to 10.8 million If only the high priority requirements are implemented, the estimated cost is 10.6 million. Establish a baseline of requirements that will be used to build the system Requirements walk-through and a review with the stakeholders is performed for acceptance of the requirements document to establish a system baseline
High Level Design Sub-system Requirements and Verification Plans	Medium 3-5 pages for each of the sub-systems	R Stakeholder involvement R Trade studies R Risk mgmt R Config mgmt R Technical reviews R Procurement	R What Project Architectures will be viable R Sub-system Requirements R Identification of candidate commercial off the shelf products R Establish a sub-system baseline for each sub-system - performance R Establish interface standards R Sub-system verification R Add content to the plans as appropriate R Selection of a systems integrator R Project costs and risks	R Not deployable R Not maintainable R Inflexible R Agency B staff cannot access internet from home or remotely R Lack of Standards R That the sub-systems are not verified R Project costs R Lack of facilities and services for new functionality R Decomposed incorrectly	What project architectures are viable: Centralized control with direct dial-in remote links Centralized control with access via internet Centralized control call in/email via operator [man in-the-loop] Distributed workstations direct to field controllers Recommended Architecture Centralized control with access via internet [Rational] Remote workstations are platform independent Flexible in a multi-agency environment Maintenance for remotes are minimized VPN technology offers fairly good security Remote software maintained at host [thin client] Project Costs Revised cost estimate based on responses from system integrator proposals
Development of Component Level Design	Defined by the SEMP – Development Plan	R Stakeholder involvement R Trade studies R Risk mgmt R Config mgmt R Technical reviews R Procurement	R Provide content to SEMP R Reverse engineer the legacy system software and restructure for remote application [As-builds] R Recommendations from the system integrator on alternatives R Definition on how to implement the recommended architecture R Develop the software architecture for the system R Develop build-to specifications R Establish a detailed design baseline R Verification of the units R Identification of configuration items R Critical design review prior to implementation R Define and document the development environment R Prototype user interface R Traceability between detailed design and requirements	R Lack of a specification to build the system R Are there alternatives that were missed? R Lack of a modular design R Lack of unit verification R Lack of documentation for legacy system to make the needed changes to the affected areas of the system R Not all requirements are addressed R Losing configuration control	Provide Content to SEMP Development Plan and schedules Configuration Management Plan Risk Plan Integration Plan Deployment Plan Security Plan Definition on how to implement the recommended architecture: Detailed design of software architecture Specify the internal interfaces between the central system software for new functionality Specify Java applets developed at the host for remote access Detailed design specifications [code-to] for the Java applets and user interface Specify VPN strategy Detailed design of Oracle application Specify an internet server using Apache technology and Oracle server Specify a T1 communications link with ISP Design data tables and schemas
Hardware and software development	Defined by the SEMP – Development Plan	R Technical reviews R Config mgmt R Risk mgmt	R Development of software R Purchase of commercial off-the-shelf products R Development of COTS application software R Configuration management of software at the developmental level R Code walk-through R Start development of user documentation R Develop Unit Verification Procedures R Developmental engineering reviews R Identify project risks	R Lack of traceability between the coding and the detailed design documentation R Not implementing all functionality requirements R Not meeting performance requirements R Losing configuration control	Development of Software Coding of individual units of software Coding libraries Checking in and checking out of software for CM Code data tables Purchase of COTS products Software license Maintenance contracts Communications links
Unit verification	Defined by the SEMP Development Plan	R Technical Reviews R Config mgmt R Risk mgmt	R Check out of the units of software and hardware R Communications R COTS products and applications R Traceability to detailed design R Complete unit functionality	R Propagating defects to a higher level R Inability to verify units and to complete development at the unit level	Check out the units of software and hardware Check out purchased servers Integrate basic COTS applications with server and verify operations Check units of software that it can perform as specified Installed communications check End-to-end test [Pinging messages] Evaluate data rates and delays
Unit integration	Defined by the SEMP Integration Plan	R Technical review – verification readiness review R Config mgmt R Risk mgmt	R Integrate units of software into sub-systems R Develop Sub-system Verification Procedures	R Interfaces are not compatible R Propagating defects to a higher level	Integrate units of software into sub-systems Application software for Oracle into the server Integrate Apache application with internet server
Sub-system verification	Defined by the SEMP Verification Master Plan and Verification Procedures	R Technical review – verification readiness review R Config mgmt R Risk mgmt	R Verification of sub-systems for functionality R Make ready for the next level of integration R Update user documentation R Complete sub-system functionality	R Interfaces are not compatible R Sub-system functions not complete R Not meeting performance R Propagating defects to a higher level	Verification of sub-systems for functionality Verify that the database management system is functional and that the data tables are populated and can be accessed within the performance requirements The Apache application is functional and accessibility of the server to the internet is functional
Sub-system integration	Defined by the SEMP Integration Plan	R Technical review R Verification readiness review R Config mgmt R Risk mgmt	R Integrate sub-systems together into the final system configuration R Update user documentation R Update Operations & Maintenance Plans R Initial deployment and transition into Operations Plan update R System Verification Procedures	R Operating agency staff not ready for operations & maintenance R Final configuration is not appropriate R Documentation for users is not ready R Propagating defects to a higher level	Integrate sub-systems into the final systems configuration Integrate the Apache server and internet communications with the Java applet exercise system, end-to-end check for memory leaks, fault conditions, browser compatibility, security, sign filtering [be able to access only the signs required for agency B]. Check Oracle database for agency profiles and login authority.
System verification	Defined by the SEMP Verification Plan	R Stakeholder involvement R Technical reviews R Verification readiness reviews R Config/ Risk mgmt	R Verify that the system meets all requirements R All documentation is updated and ready for users R Complete system functionality	R System not fully implemented R System does not meet requirements R System does not meet the needs of the user	All documentation is updated and ready for users All user training, maintenance, user manuals are completed
Deployment	Defined by the SEMP Deployment Plan	R Technical review R Deployment readiness review	R Determine if system is ready to be deployed R Evaluation period R Training updates	R System is not ready to be deployed R Latent defects that did not surface during verification	Determine if system is ready to be deployed Staff is trained, Internet access is available, VPN is configured, agency profiles are fully populated, access to the correct signs has been verified, remote users can read the 6 CMS status and post appropriate messages
Validation	Defined by the SEMP Validation	R Stakeholder involvement R Risk mgmt R Config mgmt	R Pre-system studies vs. post-system evaluation, effects on event management	R Not meeting the expectations of the stakeholders R Not meeting the needs as specified in Concept of Operations	Pre-system studies vs. post-system evaluation, effects on event management In the pre-system evaluation it took 7 staff members 2 hours to set up the event management process. The effects on the event - it took 30 minutes from the end of the event to move traffic out of the area. It took 45 minutes prior to the event to park the event attendees. In the post-system evaluation it took 1 staff member 10 minutes to set up the event management process. The effectiveness of dynamically changing the signs shows when it took only 15 minutes to clear the event and 30 minutes to park the event attendees.
Operations & maintenance	Defined by the Operations & Maintenance Plan	R Stakeholder involvement R Config mgmt

Process Step

Estimated
Level of Effort

Check list of supporting activities

Check list of issues

Check list of risks

Examples

Feasibility

Medium

2-5 pages

R Procurement

R Trade study

R Stakeholder involvement

R Elicitation

R Risk mgt

R Procure the services of systems engineering services

R Address all user needs

R Definition of the problem

R Scope of the problem

R Possible solution concepts

R Estimated cost and benefit

R Identification of the portion of the regional architecture that this will fulfill

R Institutional issues

R Feasibility with existing system[s]

R Feasibility with partner agencies

R Identify project risks

R Technical metrics and performance

R Document the feasibility analysis

R Picking a point solution without considering the business case or cost benefit of alternatives

R Selecting a solution that is not appropriate among participating agencies

R Proposing a solution that is too costly

R Incomplete solutions

Definition of the problem and need:

“Sharing of CMS by Agency B for event management, and to provide alternate routing at the beginning and ending of the event”.

Scope:

Agency B needs shared control of 6 CMS that are in the event areas

Feasibility:

Can the existing software be modified to include this new requirement?

How much reverse engineering is needed to integrate the new requirement into the existing system?

Trade study and cost benefit:

Evaluate stand-alone systems controlling the signs or integrate software functionality into legacy system at the primary agency.

Institutional issues:

Equipment standards different between the agencies. Limited support staff and maintenance at agency B.

Cost Estimate:

Reverse engineering effort increased the cost of the project to $10.7 million from the original $10.5 million

Identified Risks:

Interagency MOUs cannot be signed or delayed

Reverse engineering will be more costly than expected

Standards and license agreements

Security

Maintenance

Limited solution [not general enough for region]

Planning

Low* to Medium

SEMP framework developed

2-3 pages

R Stakeholder involvement

R Risk mgt

R Config. mgt

R Project planning

R Technical reviews

R Identification of expected plans for the project

R Expected content and quality of the plans

R Expectations on the effort needed for the development

R Schedule & budget

R Monitoring and controlling of effort

R Loss of control of the project deliverable, schedule & budget

R Missing critical activities

R Lack of long term maintenance & operations

R Not meeting expectations

The identified technical plans include:

Development Plan [Software, Hardware]

Integration Plan

Deployment Plan

Verification and Validation Plans

Development team

CM Plan

Project Plan

Operations & Maintenance Plan

Configuration Management Plan

Risk Management Plan

* Note:

This effort will be low if plan frameworks have already been done, medium effort if they need to be developed.

Concept of Operations & Validation Plan

Medium

5-10 pages

R Elicitation

R Stakeholder involvement

R Risk mgmt

R Trade studies

R Technical reviews

R Definition of the way the system will operate and be maintained

R Identification of the project level stakeholders e.g.

R Maintenance

R Supervisors

R Operators

R IT department

R Agencies’ risk managers

R Validation of the system

R Limitations of shared control

R Alternative operational concepts

R Definition of user needs at the project level

R Identification of risks

R Target performance of the shared control

R Revised cost estimate

R Agency’s normal Operations & Maintenance Standards

R Lack of understanding on:

R How shared control will operate with limitations

R How the system will be maintained

R Scope of the project

R Who will be impacted by the control

R How the system will be validated

R What is needed for shared control

R Project risks

R Project needs

R Operational & Maintenance Standards and agency limitations

R City’s policies and risks regarding control of the CMS

R Missing alternative operational strategies

How shared control will operate with limitations:

Agency B staff needs to monitor the status of the 6 CMS and post messages on alternate routes for local events and emergency traffic conditions

Be able to remotely control the signs from the supervisor’s home.

Limited to the use of pre-developed canned messages by agency B

How the system will be Maintained:

Agency B maintenance is limited and lacks the skills to maintain the communications link

The standard for Agency B is a Windows-based workstation or PC. Staff can install software if installation instructions are provided or there is a standard installation wizard.

The primary agency will maintain the host and communications system and provide installation support to agency B.

Operational Standards and Norms

Agency B is a 5-day operation that is supported on weekends for events and emergencies from the supervisor’s home.

The primary agency is a 7-day 24-hour operation. On weekends, if agency B cannot be reached, the primary agency has the authority to post messages on behalf of agency B [MOUs] as agreed. If any system fault occurs, the primary agency would need to identify and resolve the problem.

Additional risks identified:

Security of the remote link into the system; a security plan will be needed. [Update SEMP with a framework of a Security Plan.]

Validation of the shared control

The transfer of control between agencies will be in accordance with the scenario developed in the conops.

Development of System Level Requirements and Verification Plans

Medium 5-7 pages of Requirements and a 5-7 page Verification Plan

R Stakeholder involvement

R Elicitation

R Technical reviews

R Trade studies

R Risk mgmt

R Config mgmt

R Traceability

R Definition of what the system is to do to support the identified needs

R What will be used as the basis for accepting the completed system?

R New risks that may be uncovered

R New requirements may be needed

R Are all the needs addressed?

R Is each need addressed completely?

R What will be needed to support the development, operations & maintenance?

R Project cost

R The important things are implemented

R Establish a baseline of Requirements that will be used to build the system

R Validation of Requirements

R Not having a basis to accept the system when completed

R Not completely defining what the system is to do

R Scope creep

R Expectations not met

R Project cost

R Losing control and visibility of the development

R Requirements not validated by stakeholders

Definition of what the system is to do to support the identified needs

“The changeable message sign system shall share control with agency B.”

“Agency B shall have remote access to the CMS system.”

“Remote access to the CMS host shall be secure.”

“Remote access shall be limited to a pre-defined set of messages.”

What will be used for the basis of verification and acceptance of the system?

Verification Plan would contain:

Demonstrate that only a pre-defined set of messages can be displayed.

Analysis that the system is secure.

What will be needed to support the development, operations & maintenance?

Users and maintenance documentation shall be provided

Installation documentation shall be developed for the host and remote users.

Project Costs

With the additional support documentation and security aspects the project costs have been revised to 10.8 million

If only the high priority requirements are implemented, the estimated cost is 10.6 million.

Establish a baseline of requirements that will be used to build the system

Requirements walk-through and a review with the stakeholders is performed for acceptance of the requirements document to establish a system baseline

High Level Design
Sub-system Requirements and Verification Plans

Medium 3-5 pages for each of the sub-systems

R Stakeholder involvement

R Trade studies

R Risk mgmt

R Config mgmt

R Technical reviews

R Procurement

R What Project Architectures will be viable

R Sub-system Requirements

R Identification of candidate commercial off the shelf products

R Establish a sub-system baseline for each sub-system - performance

R Establish interface standards

R Sub-system verification

R Add content to the plans as appropriate

R Selection of a systems integrator

R Project costs and risks

R Not deployable

R Not maintainable

R Inflexible

R Agency B staff cannot access internet from home or remotely

R Lack of Standards

R That the sub-systems are not verified

R Project costs

R Lack of facilities and services for new functionality

R Decomposed incorrectly

What project architectures are viable:

Centralized control with direct dial-in remote links

Centralized control with access via internet

Centralized control call in/email via operator [man in-the-loop]

Distributed workstations direct to field controllers

Recommended Architecture

Centralized control with access via internet [Rational]

Remote workstations are platform independent

Flexible in a multi-agency environment

Maintenance for remotes are minimized

VPN technology offers fairly good security

Remote software maintained at host [thin client]

Project Costs

Revised cost estimate based on responses from system integrator proposals

Development of Component Level Design

Defined by the SEMP – Development Plan

R Stakeholder involvement

R Trade studies

R Risk mgmt

R Config mgmt

R Technical reviews

R Procurement

R Provide content to SEMP

R Reverse engineer the legacy system software and restructure for remote application [As-builds]

R Recommendations from the system integrator on alternatives

R Definition on how to implement the recommended architecture

R Develop the software architecture for the system

R Develop build-to specifications

R Establish a detailed design baseline

R Verification of the units

R Identification of configuration items

R Critical design review prior to implementation

R Define and document the development environment

R Prototype user interface

R Traceability between detailed design and requirements

R Lack of a specification to build the system

R Are there alternatives that were missed?

R Lack of a modular design

R Lack of unit verification

R Lack of documentation for legacy system to make the needed changes to the affected areas of the system

R Not all requirements are addressed

R Losing configuration control

Provide Content to SEMP

Development Plan and schedules

Configuration Management Plan

Risk Plan

Integration Plan

Deployment Plan

Security Plan

Definition on how to implement the recommended architecture:

Detailed design of software architecture

Specify the internal interfaces between the central system software for new functionality

Specify Java applets developed at the host for remote access

Detailed design specifications [code-to] for the Java applets and user interface

Specify VPN strategy

Detailed design of Oracle application

Specify an internet server using Apache technology and Oracle server

Specify a T1 communications link with ISP

Design data tables and schemas

Hardware and software development

Defined by the SEMP – Development Plan

R Technical reviews

R Config mgmt

R Risk mgmt

R Development of software

R Purchase of commercial off-the-shelf products

R Development of COTS application software

R Configuration management of software at the developmental level

R Code walk-through

R Start development of user documentation

R Develop Unit Verification Procedures

R Developmental engineering reviews

R Identify project risks

R Lack of traceability between the coding and the detailed design documentation

R Not implementing all functionality requirements

R Not meeting performance requirements

R Losing configuration control

Development of Software

Coding of individual units of software

Coding libraries

Checking in and checking out of software for CM

Code data tables

Purchase of COTS products

Software license

Maintenance contracts

Communications links

Unit verification

Defined by the SEMP Development Plan

R Technical Reviews

R Config mgmt

R Risk mgmt

R Check out of the units of software and hardware

R Communications

R COTS products and applications

R Traceability to detailed design

R Complete unit functionality

R Propagating defects to a higher level

R Inability to verify units and to complete development at the unit level

Check out the units of software and hardware

Check out purchased servers

Integrate basic COTS applications with server and verify operations

Check units of software that it can perform as specified

Installed communications check

End-to-end test [Pinging messages]

Evaluate data rates and delays

Unit integration

Defined by the SEMP Integration Plan

R Technical review – verification readiness review

R Config mgmt

R Risk mgmt

R Integrate units of software into sub-systems

R Develop Sub-system Verification Procedures

R Interfaces are not compatible

R Propagating defects to a higher level

Integrate units of software into sub-systems

Application software for Oracle into the server

Integrate Apache application with internet server

Sub-system verification

Defined by the SEMP Verification Master Plan and Verification Procedures

R Technical review – verification readiness review

R Config mgmt

R Risk mgmt

R Verification of sub-systems for functionality

R Make ready for the next level of integration

R Update user documentation

R Complete sub-system functionality

R Interfaces are not compatible

R Sub-system functions not complete

R Not meeting performance

R Propagating defects to a higher level

Verification of sub-systems for functionality

Verify that the database management system is functional and that the data tables are populated and can be accessed within the performance requirements

The Apache application is functional and accessibility of the server to the internet is functional

Sub-system integration

Defined by the SEMP Integration Plan

R Technical review

R Verification readiness review

R Config mgmt

R Risk mgmt

R Integrate sub-systems together into the final system configuration

R Update user documentation

R Update Operations & Maintenance Plans

R Initial deployment and transition into Operations Plan update

R System Verification Procedures

R Operating agency staff not ready for operations & maintenance

R Final configuration is not appropriate

R Documentation for users is not ready

R Propagating defects to a higher level

Integrate sub-systems into the final systems configuration

Integrate the Apache server and internet communications with the Java applet exercise system, end-to-end check for memory leaks, fault conditions, browser compatibility, security, sign filtering [be able to access only the signs required for agency B]. Check Oracle database for agency profiles and login authority.

System verification

Defined by the SEMP Verification Plan

R Stakeholder involvement

R Technical reviews

R Verification readiness reviews

R Config/ Risk mgmt

R Verify that the system meets all requirements

R All documentation is updated and ready for users

R Complete system functionality

R System not fully implemented

R System does not meet requirements

R System does not meet the needs of the user

All documentation is updated and ready for users

All user training, maintenance, user manuals are completed

Deployment

Defined by the SEMP Deployment Plan

R Technical review

R Deployment readiness review

R Determine if system is ready to be deployed

R Evaluation period

R Training updates

R System is not ready to be deployed

R Latent defects that did not surface during verification

Determine if system is ready to be deployed

Staff is trained, Internet access is available, VPN is configured, agency profiles are fully populated, access to the correct signs has been verified, remote users can read the 6 CMS status and post appropriate messages

Validation

Defined by the SEMP Validation

R Stakeholder involvement

R Risk mgmt

R Config mgmt

R Pre-system studies vs. post-system evaluation, effects on event management

R Not meeting the expectations of the stakeholders

R Not meeting the needs as specified in Concept of Operations

Pre-system studies vs. post-system evaluation, effects on event management

In the pre-system evaluation it took 7 staff members 2 hours to set up the event management process. The effects on the event - it took 30 minutes from the end of the event to move traffic out of the area. It took 45 minutes prior to the event to park the event attendees.

In the post-system evaluation it took 1 staff member 10 minutes to set up the event management process. The effectiveness of dynamically changing the signs shows when it took only 15 minutes to clear the event and 30 minutes to park the event attendees.

Operations & maintenance

Defined by the Operations & Maintenance Plan

R Stakeholder involvement

R Config mgmt