Document Description	Date	Version Number
First draft release of Configuration Management Plan for review	May 2 2006	0.1
Revised per comments from stakeholders	June 27, 2006	1.0

1 Introduction

The Sacramento Transportation Area Network, or STARNET, is an information exchange network that will be used by the operators of transportation facilities and emergency responders in the Sacramento region of California. STARNET will enable the real-time sharing of data and live video, and refinement of joint procedures pertaining to the operation of roadways and public transit, and public safety activities. It will thereby assist operations personnel in the coordination of their activities and in providing the public with comprehensive information about current travel conditions and options.

The configuration management plan provides guidelines for recording, and managing changes to, the configuration of hardware, software, and communications facilities used in STARNET. Among other things, it is intended to avoid a situation where one agency or person unilaterally makes a change that may have negative consequences for other users or for system maintenance, or where a repair or change is needed for the benefit of all, but is made very difficult by inadequate documentation of the current configuration.

2 Distributed Ownership

STARNET involves a data and video sharing network composed of field devices (e.g., transit vehicles, CCTV cameras, vehicle detectors, traffic signals, etc.), computers, software, and communications links, and procedures for its use and maintenance. Each component of STARNET is independently owned, operated, and maintained by one of the involved agencies. The agencies voluntarily cooperate in coordinating their STARNET-related activities for their mutual benefit.

One of the involved agencies, the Sacramento Area Council of Governments or SACOG, is a regional joint powers agency in which all six counties and 22 cities in the region have membership and voting rights via its Board of Directors. Caltrans District 3 is an ex officio member. As an associative agency, SACOG represents the shared interests of all member agencies, and is therefore suited to being the owner of the few STARNET components not already owned by one of the other STARNET agencies. For example, the Regional Transportation Management Display (including the incident tracker) software is assigned to SACOG.

3 Distributed Configuration Management

Just as ownership of STARNET components is distributed among the involved agencies, so is configuration management. For components it owns, each agency is responsible for the following key configuration management activities:

· Maintain comprehensive and accurate configuration records, including updating records after making any change.

· Consider potential impacts on other agencies, and consult with them when appropriate, before making any configuration change.

· Inform other agencies of any change.

Each agency has its own configuration management procedures, used in managing the configuration of the STARNET components that it owns or is responsible for. Generically, configuration management procedures include:

· Creation and maintenance of configuration records such as drawings or geographical information systems showing the location and connectivity of all components, related or separate tables and text describing each component and its history and current configuration, documentation of current operation and maintenance procedures, documentation of current inter-agency cooperation procedures, etc.

· Change control procedures such as categorization of components and procedures with regard to change control, level of authority needed for change approval for each category, a change control board for at least some categories, and change implementation and notification procedures.

Agencies provide configuration management for all components and procedures affecting STARNET, including field devices providing data or video, central software managing such devices or processing received data, interfaces to the STARNET network, communications equipment and links used in STARNET, and procedures to be followed by personnel that may affect the operation or maintenance of such components or interactions with other agencies.

4 Configuration Documentation

Each agency maintains accurate records of its resources and procedures that are used in regional coordination via STARNET. Such records are updated whenever any change is made. The following scenarios illustrate why it is important for the region that each agency maintains complete and accurate records of its facilities and procedures.

Scenario 1 – Some years after the initial implementation of STARNET, Agency A joins the STARNET network and sets about configuring its STARNET interface to obtain data from Agency B. Due to good record keeping, Agency B is able to supply Agency A with an up-to-date list of field devices and sufficient information about each for Agency A to decide which devices it needs data from.

Scenario 2 – Several of the STARNET agencies experience a reliability problem involving the STARNET interface to the local system at each agency. As part of troubleshooting procedures, each agency is asked to report the configuration of its interface including routers, firewalls, switches, and operating system network connections. Such information is readily available from all agencies and a comparison quickly indicates the differences and the cause of the problem.

Scenario 3 – As a result of the problem discussed in the previous scenario, the agencies jointly decide to change the interface configuration to avoid similar problems in the future. Good records enable each agency to easily determine exactly what changes it needs to make, to make the changes, and to record the changed condition.

Scenario 4 – During implementation of the interface configuration change described in the prior scenario, an unanticipated side effect emerges and the agencies agree to roll back to the previous configuration, at least until a better solution is determined. The good records kept by each agency, including storage of prior-state information, allow the former condition to be restored accurately and quickly.

SACOG is responsible for maintenance of the STARNET Interface Definition document. This document specifies the data elements, messages, protocol, and other aspects of the common (standard) interface between the STARNET network and each node system. This is the STARNET side of the interface to each node. The configuration of each node system, the specification of the node side of the STARNET interface, and the configuration of the STARNET interface software for a particular node, are specific to each node and therefore documented by the agency that owns the node system.

5 Change Control

In preparing procedures and training personnel, each agency considers the different degrees to which a local change can impact other STARNET agencies and therefore the amount of inter-agency coordination warranted. The following table describes the inter-agency coordination that is appropriate for local changes involving different degrees of regional impact.

Ideally, all operations, maintenance, design, and policy personnel that could affect any of the resources used in STARNET are provided with written guidelines and thorough training aimed at making them aware of the potential impacts of local changes on other agencies and of appropriate steps to take if an impact is anticipated. This need not involve significant additional effort or time on the part of agency personnel, just an awareness and consideration of potential impacts, and appropriate, common sense coordination with other agencies when needed. If an employee suspects a planned action may impact other STARNET users, they should at least raise the issue with their supervisor.

Regional Impact	Example Changes	Inter-Agency Coordination
None	1. Change the color of paint on a traffic signal cabinet.	None
Minor	1. Relocate a camera rarely used by other agencies. 2. Add more AVL-equipped buses. 3. Upgrade the operating system on a node system server computer. 4. Change presets on a camera.	Notify other agencies at the time of change if not before.
Potentially Significant	1. Decommission a fiber optic link used as a back-up path for STARNET communications. 2. Decommission a camera commonly used by other agencies. 3. Replace a broadband Internet link used by STARNET with a much slower link. 4. Change procedures involving interaction with other agencies during incidents.	Discuss with the most affected agencies prior to making the change. Notify all agencies well in advance of making the change. Notify other agencies of the actual changes made after successfully completed.
Potentially Major	1. Add a new node system to STARNET. 2. Remove a node system from STARNET. 3. No longer allow incident data to be exported to the regional display. 4. Change the look and feel, or functionality, of the Regional Transportation Management Display.	Formally notify the Region ITS Partnership of the proposed change and have it placed on the agenda for at least one meeting of the STARNET Technical Advisory Committee. Make a change only after thorough discussion, and attempts at consensus if controversial. Coordinate the changes with other agencies. Notify other agencies of the actual changes made after successfully completed.

Formal configuration management often includes a change control board that reviews and approves proposed changes. The STARNET Technical Advisory Committee performs that role for STARNET, but since each STARNET component is owned by one agency, only that agency can ultimately decide on changes to that component. Hence the STARNET Technical Advisory Committee serves as a change control forum, but does not formally approve changes. As with all aspects of STARNET, the mutual benefits that participating agencies derive from inter-agency cooperation is the motivation for adhering to the majority, if not consensus, opinion of the Technical Advisory Committee.

When a planned change warrants inter-agency discussion, the originating agency (or an appointed representative of multiple agencies planning a change) prepares a briefing document for the STARNET Technical Advisory Committee, records the major points of discussion and the conclusion, and archives such documentation as a record that others can refer to later if needing to understand the origin or reason for a particular change or abandonment or alternation of a planned change. Other agencies are kept apprised of the final form of the change, the schedule for its implementation, and confirmation of its implementation once completed. The agency making a change coordinates with other agencies as needed to facilitate their taking any concurrent actions needed to adjust for the change. Configuration documentation is updated to reflect the change as soon as the change is proven to be successful.

Good configuration management also allows for rolling back to the prior proven (baseline) state in the event that an implemented change causes unforeseen problems. The STARNET agencies therefore maintain records of the prior state and ensure that implementation procedures allow for roll back if needed.

Agencies perform periodic reviews of the effectiveness of configuration management documents, procedures, and change experiences. These reviews are an opportunity to address any shortcomings and enhance the effectiveness of configuration management.

6 Software Management

Deployment of STARNET involves implementation of new software to provide the interface to each agency’s node systems and to provide the Regional Transportation Management Display, including the shared incident tracker. The system integration contract will require thorough documentation of this software, its initial configuration, source component libraries, version control, and procedures for re-building the executable software from source components. It will also require provision of tools needed for building and maintaining the software, requirements documentation, bug tracking, source version management, and other software configuration management functions. SACOG will become the owner of this software and the supporting documentation and tools.

Each agency is similarly responsible for documentation and tools related to its node system software.

7 Guidelines for Agency Procedures

Each STARNET agency is responsible for configuration management for hardware and software that are part of the overall STARNET system. Each agency needs to establish procedures for configuration management relative to the components it owns. These procedures need to be tailored to the circumstances of that agency and the types of components it owns. To the extent that an agency does not already have sound procedures for such configuration management, the following guidelines are intended to assist in developing them.

7.1 Overview of Procedures

Configuration management procedures usually need to include at least the following:

· Configuration management scope

· Configuration documentation

· Configuration change request

· Configuration change review and approval

· Configuration change implementation

· Configuration change notification

· Configuration change documentation

Procedures need to be implemented and used. To this end they need to be part of routine employee training, and perhaps the subject of special training sessions. It is also good practice to regularly conduct a review of configuration management procedures and refine them based on feedback as their effectiveness.

7.2 Configuration Management Scope

Configuration management procedures should first identify what facilities are subject to configuration management. For STARNET, this includes all software, hardware, and communications facilities involved in the generation, receipt, transmission, or processing of data exchanged or displayed via STARNET. It includes field devices for which data are exchanged on STARNET.

The same configuration management procedures will likely be applied to other facilities that are not involved in STARNET. It is important that facilities involved in STARNET be so designated, as this will trigger additional considerations and steps for these components, involving review by other STARNET agencies. A good practice is to place a “STARNET component” or similar sticker on each piece of equipment that is important for STARNET. Splash screens and comment fields in software user interfaces can also include such a reminder.

Some types of changes to a component may need a high level of change control while other types of changes may only need a lower level or no control. Hence the inventory documentation should indicate which types of change are subject to which levels of change control. Making changes to such recordings of the change control required for each type of change should itself be subject to change control.

It is not always practical to predetermine the appropriate level of change control for every conceivable type of change that may need to be made. Procedures should state that unless the planned change is clearly in a category that does not require change control or is covered by a prior blanket change approval, then it should be assumed that change approval is needed and start by following the procedures for the lowest level of change control. The person to whom the request is referred should then decide if this type of change needs to be elevated to the next level, and so on.

7.3 Configuration Documentation

Procedures should specify the type and format of documentation to be used for facilities subject to configuration management. Configuration documentation normally includes some form of inventory database with fields such as; component description (including make/model), serial number or other unique identifier, date purchased, current location, current use or purpose served, pointer to another component of which this is a subcomponent, current status (e.g., awaiting installation, installed but not operational, operational, temporarily failed, stored as spare, etc.), date of acquisition, date first put into service, procurement source, procurement method, procurement cost, nature and expiration date of any warranty or maintenance agreement, identification of agency personnel with knowledge of this component, identification of any third party maintenance service used, description of current configuration (or pointer to other documentation containing same), description of last configuration change (or pointer to other documentation containing same), the types of changes subject to each level of change control (e.g., high, medium, low, none) or pointer to other documentation of same, and pointers to further documentation if any (e.g., drawings showing the spatial or logical integration of this component with others, drawings showing details of the component design or configuration, text documents describing the current configuration, users manual, maintenance manual, etc.).

Such information may be stored in a geographical information system, relational database, spreadsheet or any other suitable form. It is good practice to avoid replication of the same information, such as information common to every instance of a particular type of device or software package.

Most of this information is applicable to both hardware and software. There are two aspects of software configuration that need appropriate documentation and management. One is the source code and tools used by software developers to generate the software, and the other is the parameters and scripts entered or created by users to affect the operation of the software.

Procedures need to include measures for ensuring that the documentation is kept up to date, a new version identifier is assigned each time a document changes, the nature and date of changes is recorded, and that conflicting duplicate versions are avoided.

7.4 Configuration Change Request

Procedures should require that no change be made to a component until the proposed change is approved using the procedures applicable to the change control category (e.g., high, medium, low, none) for that type of change. The first step in gaining change approval is making a request for approval. For items in the low level control category, procedures may require only a verbal request to, and approval from, a supervisor or designated person. Medium and high level control categories should require a written request for approval, where that request includes information such as change description, change purpose, change priority (how urgent is it?), procedures to be used to make the change, other changes or actions on which this change is dependent or which depend on this change, proposed time schedule for the change, cost, risks and concerns, risk mitigation measures planned, planned testing prior to making the change, planned testing during and after the change, roll back plan, and any other relevant information.

Urgency should not be an excuse to circumvent the change approval process. Instead, it should motivate those involved in the approval process to expedite the review and approval. However, procedures may allow for urgent temporary changes to be made with a lower level of change control if the temporary change is needed to correct a serious problem and there is insufficient time to complete a higher level of change control even when done as quickly as possible. Such emergency procedures should require either restoration of the former configuration or formal approval of the change, as soon as practical. It is also important that users and maintenance personnel be trained to consider the repercussions of an emergency change on other STARNET users, and not only their own agency. In addition to seeking approval after the fact, it is important to inform all potentially affected parties of the change as soon as practical.

If a certain type of change needs to be made often, such as during routine maintenance, a one-time approval may be sought for all such occurrences rather than repeating the approval process each time. Such a change request might also request including this type of change a lower level change control category in the inventory.

Some changes may require simultaneous changes to components owned by other agencies. Such agencies should each follow its change approval processes, but attempt to coordinate them and ensure they are jointly reviewed and approved.

7.5 Configuration Change Review and Approval

Procedures need to identify the person to whom a change request should be referred at each level of change control. For example, at the low level, change requests may go to a particular person or position, at the medium level they may need to go to a higher level person within the organization and also be referred to other agencies potentially affected by the change, and at the high level may also be referred to the chair of the STARNET Technical Advisory committee. If a particular person is designated to receive change requests, alternates need to be named in case that person is unavailable.

A person receiving a change request should review the request and determine if it needs to be elevated to a higher level of approval and what other people need to review the request prior to making an approval decision. If it is appropriate to have the same group of people review all requests, those people can be referred to as a change control board.

Most agencies will want to limit the formal act of approving or rejecting a request to someone within their agency, no matter what other agencies are involved in the review and decision making process. The STARNET Technical Advisory committee or other agencies acting separately cannot override a decision by the agency owning the component, but the owning agency should attempt to adhere to the consensus view of all affected agencies.

The change review and approval process may result in changes to the proposed work plan (e.g., when the work is to be done, what testing is to be performed, the nature of the change itself, etc.). Those reviewing a proposed change may also request that certain people be notified when the change it about to be made so that they can monitor the system for related problems or inform their operators of the changed condition. The approval notification must clearly indicate what is being approved and any such additional instructions. A simple “approved” or “not approved” response is often not sufficient. Similarly, if a request is for multiple changes, some of those changes may be approved but not others, and the approval notification needs to be very explicit in this regard. Requests containing multiple unrelated changes should be discouraged.

Approval to a written request should always be in writing, in addition to any verbal communication. If a request is rejected, an explanation should be given.

7.6 Configuration Change Implementation

Procedures should require approved changes to be made according to the approved work plan (e.g., nature of the change, time schedule for making the change, testing to be performed, coordination with other agencies, roll back plan, etc.). Even if other agencies have not formally requested notification, it is good practice to require those making a change that has been reviewed by other agencies to notify them in advance of making the change so they can monitor the system as the change is made or notify their operators of the change and how it may affect them.

Procedures should stress the need for thorough testing after a change is made to ensure it has not caused any unforeseen problems. Such testing may involve interaction with operators or maintenance personnel in other agencies.

7.7 Configuration Change Notification

Procedures should require those making an approved change to notify others within their agency and in other agencies as appropriate, once the change is completed. If a change results in unexpected problems or cannot be completed as planned, all parties should be fully briefed on the nature of the problems. Those notified may request a further change or roll back to the former state if they consider the problems so warrant. Unless such further change is in accordance with an already approved roll back plan, it will need to follow the change approval process again. Problems encountered during a change should be treated as a learning opportunity for all parties.

7.8 Configuration Change Documentation

Procedures should require those making a change to immediately update the facilities inventory and other relevant documentation. It is good practice to require periodic reviews or audits of documentation to ensure it is being kept up to date. Such reviews may involve spot checks or review of the documentation of components known to have recently approved changes.

-oOo-