Table of Contents
Introduction pg. 3
Management Decision Points pg. 4
Summary of Findings pg. 5
Finding Number 1– Existing Process can accommodate SE pg. 6
Finding Number 2 – Implementing SE will affect multiple Department levels pg. 7
Finding Number 3 – OCIO and FHWA requirements can be met with SE pg. 8
Finding Number 4 – Institutionalizing SE requires improved capabilities pg. 15
Finding Number 5 – SE process must be tailored for type of ITS project pg. 17
Finding Number 6 – High performing relationships are essential pg. 21
Action Plan Roadmap pg. 23
Action Plan Phase 1 – Start‐up and concurrent activities pg. 24
Action Plan Phase 2 – Basic Systems Engineering pg. 26
Action Plan Phase 3 – Advanced Systems Engineering pg. 27
Action Plan Phase 4 – Institutionalizing SE at the Department level pg. 28
Action Plan Phase 5 –Specific ITS challenges pg. 29
Sample Action Plan Schedule pg. 30
Acknowledgements
This Executive Summary was the collaborative effort of the following individuals:
Frank Taylor – (Project Manager) ‐ Caltrans, Division of Transportation Planning
Frank Cechini – (FHWA) – ITS Manager, Federal Highway Administration, California Division
Michael E. Krueger – (ASE Consulting LLC) – Consultant Project Manager
Appreciation to the following for their review and comments:
Osama Elhamshary – CCIT Project Manager
Bill Tournay – Caltrans, Division of Transportation Planning
Reza Navai – Caltrans, Division of Transportation Planning
Contact Information:
Frank Taylor – Caltrans Project Manager (916) 651‐6006
Osama Elhamshary – CCIT Project Manager (510) 642‐3150
Michael E. Krueger – ASE Consulting LLC – Project Manager (714) 708‐2993
SYSTEMS ENGINEERING EVALUATION
This executive summary is based on the “Final Report: Systems Engineering Evaluation for ITS,” 2006 (Final Report). The strategic goals and objectives that guided the study were as follows:
Long Term Goal: Implement systems engineering (SE) “best practices” and capabilities for Intelligent Transportation Systems (ITS) projects within California Department of Transportation (Department) and integrate these best practices and capabilities into the existing project development process.
Primary Objective: Document and implement best practices in systems engineering for ITS projects and at a minimum meet the 23 CFR 940 part 11 of the FHWA Final Rule on the application of the Systems Engineering Analysis (SEA) for ITS projects.
Secondary Objective: Evaluate the benefit of the recommended systems engineering process for ITS to other processes within the Department.
The purpose of this systems engineering evaluation was to:
Capture best practices
Evaluate the existing practices
Develop a tailored project development model
Develop a Roadmap and Action Plan to successful implementation of SE best practices
The goal in building the Department’s SE capabilities is to improve the cost, schedule, and quality of ITS projects. The following illustration shows the dependencies among people, processes, and technology when working together to achieve this goal.
Developing expertise among Department staff is critical for building SE capabilities. By providing specialized SE training, documenting SE processes, and applying new technologies, the Department can build a workforce with the knowledge and tools to carry out SE integration.
“Everyone realizes the importance of having a motivated, quality work force but...even our finest people can’t perform at their best when the process is not understood or operating at its best.”*
*CMMI V1.1 and Appraisal Tutorial ‐ Mike Phillips, CMMI Program Manager Feb 2004 Slide
The above recommended management decisions are necessary to successfully accomplish the following Department goals:
Implement systems engineering best practices and build capabilities
Integrate best practices into the existing project development process
Comply with 23 CFR 940 Part 11
Comply with the Stewardship Agreement on ITS projects
The following pages will build the case for the changes necessary and layout an action plan to accomplish these goals. These management decisions will be revisited at the end of the report with further analysis of organizational responsibilities and recommendations for leadership of the proposed action plan.
The six “Findings” from the SE Evaluation Final Report are described here.
Each Finding is addressed in more detail in the following pages.
Existing project management and organizational processes are closely aligned with industry standards and can accommodate the systems engineering process for ITS projects.
Implementation of the SE process will have an impact on Caltrans Headquarters and District Divisions. The SE process will affect all ITS projects carried out by the Department.
A tailored SE process will address the Office of the Chief Information Officer (OCIO), State Administrative Manual (SAM), and 23 CFR 940 part 11 (FHWA Final Rule) requirements.
Integrating the SE process into capital project development is the means to institutionalize the SE process within Caltrans. This will address the “Caltrans/FHWA Joint Stewardship and Oversight Agreement.” It also ensures that the SE process flows to Division and District ITS projects.
The SE process must be tailored at the project level based on the risks and complexity of the project.
Management support is a critical element in successful project development. High performing relationships (with regard to the development of ITS projects) must exist within Caltrans Headquarters, Divisions, Districts, as well as OCIO and FHWA.
Caltrans has historically used systems engineering in establishing and documenting their Standard Specifications, Special Provisions, and Standard Plans for traditional roadway design and construction projects. These “standards” consist of a series of documented functional, performance, and testing requirements. ITS projects can be addressed more inclusively within the existing capital project development process by incorporating industry standards for systems engineering activities.
Currently, requirements from other state agencies and federal partners are handled as separate review and approval actions. OCIO has a set of requirements for developing Information Technology (IT) projects (State Administrative Manual). ITS projects need to adhere to these requirements, as well as requirements of the FHWA Final Rule 23 CFR 940 part 11. By integrating SE process activities into the capital development process tailored for ITS projects, Caltrans will address OCIO requirements and the Final Rule in a unified effort.
To effectively manage SE process activities Caltrans must foster an SE culture by providing access to SE resources at all levels of the Department.
For example:
Caltrans and FHWA website resources (e.g., System Engineering Guidebook for ITS, templates, documented SE activities, lessons learned)
Tools (e.g., development, requirements management, simulation)
Training (e.g., NHI, CITE, UC Berkeley Institute for Transportation Studies, private industry, proposed ITS Academy)
Specific Divisions within the Department will be impacted more than others.
The following are examples of Division level involvement:
Divisions (HQ and Districts) that implement ITS projects are directly impacted since they apply the SE process activities. Examples – Division of Traffic Ops, Division of Research and Innovation.
HQ Divisions that manage the capital development process (e.g., Design, Project Management) will be involved in managing the SE process.
DOTP will lead statewide ITS architecture activities, support the lead Division and/or regional partners in concept exploration, and sustain the SE environment (training, tools, and resources).
Construction and Maintenance Divisions will be more inclusive stakeholders in early project development discussions on concept and design activities.
OCIO requirements in this table are addressed by SE best practices.
The SE process can be tailored to include additional OCIO information (e.g. Data Center consolidation, procurement) required within the Feasibility Study Report (FSR).
OCIO requirements in this table and Project Management Oversight are elements of Systems Engineering best practices.
This table describes the tailoring that the OCIO recommends based on the criticality of a project. The table includes some, but not all, elements of standard SE best practices. For example, Concept of Operations, SE Planning, and Decision Gates are not mentioned. Incorporating SE best practices into a life cycle model will assure that all elements are considered.
The requirements needed by FHWA are addressed by SE best practices.
The SE process can be tailored to include additional FHWA information (e.g. Regional Architecture, procurement) required within the Final Rule.
Caltrans adoption of institutional‐level SE process best practices and capabilities addresses FHWA recommendations.
In its August, 2006 letter to Caltrans, FHWA highlighted specific recommendations with regard to the management of SE process activities. Each of the recommendations reflects SE best practices at the institutional, i.e. Department level. Management of the SE process at the Department level, and application across Divisions and Districts, will address responsibilities within the Caltrans/FHWA Joint Stewardship & Oversight Agreement. Integration of SE best practices into the Caltrans Project Development Procedures Manual (PDPM) will facilitate this institutional objective. Currently, FHWA decides which SE tasks and deliverables to review and/or approve, and specifically approves all systems engineering management plans and related technical plans. Improved Institutional level capability within Caltrans will potentially minimize FHWA project review/approval actions.
Use of the industry standard systems engineering life cycle puts the feasibility study, management oversight, and FHWA Rule requirements in a systems development framework.
The “Vee Development Model” is the recommended development model for ITS projects. This model correlates some key systems principles about the relationship of the early phases of project development to the end results. Each task should be performed sequentially. While the practice of overlapping tasks may appear to shorten the schedule, it can introduce unintentional risks to the project. When working with ITS projects, the decision to move forward prior to completing a previous phase must be done with great care, if at all.
It is important to integrate ITS with the capital project development process during the planning phase.
The systems engineering process has a number of phases throughout its life cycle. Two of these phases are important to integrate the Capital Development planning activities with FSR requirements. They are:
Study/FSR/Concept Exploration Phase ‐ This phase addresses traditional project planning activities as well as the FSR requirements for establishing the business case.
Preliminary Engineering Phase – This phase addresses traditional preliminary design as well as the activities on the left side of Vee. This phase, though, is currently overlooked by the feasibility study activities; instead these activities are expected to be addressed during the previous concept exploration phase.
This issue will be further explained in the next two pages.
Current FSR requirements specify some existing SE activities, but by‐passes others:
The feasibility study for an IT project is performed during the concept exploration phase as shown on the Vee life cycle model. After the FSR is approved by the OCIO it is included in the next state budget cycle for the design and implementation phases. OCIO FSR requirements include many existing SE activities but by‐pass several critical ones e.g., Concept of Operations.
Framing OCIO requirements within an SE process life cycle will result in successful project development
Systems engineering elaborates on the FSR requirements needed for ITS projects. Those FSR requirements, and the associated management oversight, can be better performed when placed as part of a total systems development framework. For ITS projects, preliminary engineering that includes the by‐passed activities should be performed before FSR approval and submittal into the state budget. (These phases are shown in green). Since each ITS project will have different needs and risks, executing the entire SE process will ensure project success. For example, stakeholder involvement in the development of functional requirements, as an early activity of the SE process is critical to obtaining system requirements that meet OCIO, as well as Caltrans, expectations.
The project management oversight and Post Implementation Evaluation Report (PIER) is performed as prescribed by OCIO.
The SE process addresses FHWA Final Rule requirements before Component Level Design. These Final Rule elements represent industry best practice and are recognized as standard SE activities.
The Joint Stewardship and Oversight Agreement can be addressed by institutionalizing the SE process at the Department level. This will maximize oversight responsibility within Caltrans for federally funded projects, minimizing FHWA involvement in approval actions.
Industry standard “levels of capability” were used in the SE Evaluation. The “levels” indicated an organization’s SE capabilities. The following are brief descriptions of each of the levels:
Level 0 SE process activities are not performed, are incomplete, or are performed in an “AD‐HOC” manner.
Level 1 Basic SE process activities are routinely performed on projects and meet the goals of the process activities.
Level 2 Advanced SE process activities are planned, tailored, performed, and managed at the project level.
Level 3 SE process activities are institutionalized at the organizational level. Process activities are clearly defined, documented, and applied across Caltrans ITS projects.
Level 4 SE process activities are quantitatively managed. Process metrics are monitored, measured, and managed through statistical analysis.
Level 5 SE process activities are optimized and continuously improved for ITS projects.
The OCIO and FHWA Final Rule requirements are addressed at the project level (Levels 0‐2). There is no requirement to institutionalize these processes as long as they are done for each ITS project.
The program management recommendations are at the Department level (Levels 3‐5). This addresses the goal of integrating the SE into the capital development process; and addresses the Joint Stewardship & Oversight Agreement.
The recommended project and Department level capabilities will improve system performance and management.
The transition of emphasis from project performance to Department level capabilities requires SE process management. The term “Managed Process” in Level 2 at the project level means that the SE process is managed by local Divisions and Districts. The term “Process Management” in Level 3 means that there is a defined process, across all ITS projects, which needs to be managed at the Department level. Process management evolves as SE capabilities mature ‐ over time‐while applying the process to ITS projects. This is analogous to the historical development of roadway design standards. In the early days, roadway designs were created in an ad‐hoc manner for individual projects based on local experiences. Designs that got locally documented were later viewed from an organizational perspective. This led to state‐wide guidelines for consistent design application. In California, this became what Caltrans now refers to as the “Gold Book”. Eventually a national perspective recognized the need for design standards, which led FHWA to adopt AASHTO standards for interstate application.
The traditional capital project development life‐cycle process is used for the design and installation of ITS field elements (e.g., Signals, CMS, CCTV, RWIS), expanding existing systems while adding no new capabilities or interfaces. These are defined as medium risk projects by FHWA. As with roadway elements (e.g., pavement, drainage), ITS field elements are designed and constructed with documented Standard Plans, Specifications, and Special Provisions.
The systems engineering life cycle is used for the design and implementation of ITS projects. These projects involve software & hardware development and integration. They include multi‐jurisdictional and/or multi‐modal system implementations. In summary, new functionality is being established. These are defined as high risk projects by FHWA. FHWA’s Systems Engineering Analysis (SEA) addresses all ITS projects – field equipment and system functionality. FHWA has categorized the development of field elements as medium risk projects and development of systems as high risk projects.
OCIO requirements apply only to the IT system elements. In order to secure State funding, regardless of the funding source, ITS projects must be included in the State budget.
ITS projects that are high risk, or have a high degree of complexity, require additional SE activities.
This diagram illustrates how increasing project risks are mitigated by additional system engineering activities. In this example, the project is adding field cameras to an existing closed circuit television system. The engineering for the communications and software functionality was done in an early stage of the project. The risk being incurred is in the placement of additional field CCTV cameras sites. In this case, the project would be considered medium risk (formerly called Minor ITS project) even though the dollar value of the project may be high. The traditional (roadway) project development process will be used for development because documented standard plans, specifications, and special provisions are available.
The project incurs a higher level of risk when camera control is shared with partner agencies. In this case, the value may be less than that of the expansion project in the example above. Software for sharing control will be needed. In addition, interagency agreements on how the partner agencies plan to control the cameras. The risks in this project include the software implementation, agreement on the conditions of camera control among the partner agencies, and a higher level of stakeholder involvement. This project would be considered high risk (formerly called a Major ITS project).
Ultimately, the development of a regional transportation management system that includes the camera functionality has a higher risk of failure because of its large number of stakeholders, increased functionality, and complexity. As a result, highly formalized SE process activities are needed.
Working through the SE life cycle phases improves implementation cost estimates*
This graph shows how working through the SE life cycle phases (SE process) improves cost estimates for development phases of software projects. In other words, a more defined project will provide better project estimates. For example, it has been shown that software cost estimates at the concept exploration phase have a greater margin of error than estimates done at the detailed design phase.
* Barry Boehm, Software Engineering Economics, ed. Raymond T. Yeh, Advances in Computing Science and Technology Series (New Jersey: Prentice Hall, 1989), 311.
Tailored SE process is designed to mitigate project risks.
There is a cost associated with performing systems engineering process activities. As the curve indicates, more formal SE activities result in higher costs. Concurrently, project risks are reduced as more formal SE activities are performed. There is an optimized point where the degree of formal SE performed crosses the risk curve. System designers strive for this threshold on each ITS project.
Clear understanding of approved and documented SE process activities between internal and external stakeholders will increase confidence in meeting the deployment goals of ITS projects.
On‐going, active relationships are essential among all parties for the successful deployment of ITS projects. These relationships indicate that a sound SE process is in place, which keeps the ITS project on schedule while meeting stakeholder requirements. Section 2.4 of the Final Report documents a range of relationship stages that exist among the internal and external stakeholders for ITS project developments. Some Districts tend to be independent, while others team up with Headquarters. The current ATMS Change Control Board is an example of a “high‐performing” relationship. The Board, made up of several Districts and Headquarters representatives, has established an SE process which manages ATMS software and hardware in a collaborative manner.
Engage key stakeholders in SE process development ‐ “Get Stakeholder buy‐in”
The Final Report recommends engaging OCIO, IT, FHWA, as well as Headquarters Divisions and Districts, in the development of the SE process activities. This entails setting up a separate management structure within the Department to support process improvement activities and provide needed resources. The Action Plan follows and provides details of this recommended management structure.
This roadmap will guide the Department in implementing an Action Plan for “process” and “capabilities” improvement.
The roadmap shows five (5) phases and a set of parallel activities. The first 3 phases address the SE process and activities at the project level. When using one or more pilot projects, it is necessary to document SE process activities, acquired or needed training and SE process activities.
Phase 4 integrates these SE activities into the capital development process.
Phase 5 calls for a review and update of the process by targeting specific project challenges ‐ over a period of time ‐ on a wide range of projects.
The following actions must be in‐place from the onset of Phases 2‐5.
Establish the Process Focused Organization to include:
A Department Manager for Systems Engineering
A Process Review Board – Representatives from Headquarters & District Divisions, and external stakeholders (including the Office of the Chief Information Officer, FHWA, and BT&H) who would perform periodic assessments, and recommend improvements to the SE process and corresponding capabilities within the Department.
Identify one (1) or more candidate pilot projects on which to perform the SE process activities.
A Process Action Team to identify pilot projects, perform and document the process activities, train stakeholders, and gather feedback. The Team may periodically be reactivated for specific SE undertakings.
These supporting activities occur throughout the Action Plan phases.
Enhance the on‐line interactive “Systems Engineering Guidebook for ITS”
(www.fhwa.dot.gov/cadiv/segb). Add examples, templates, and policies that are developed as a result of the implementation of this Action Plan.
Develop materials for SE training – (Materials that already exist from UC Berkeley Institute of Transportation Studies could be used as a starting point for internal Caltrans training).
Establish an SE training program as part of an “Intelligent Transportation Systems Academy” for Caltrans.
Establish a technology center for ITS, accessible by Caltrans’ Headquarters and District Division level staff. (Could be an addition to Caltrans Library)
Phase 2 documents and implements the basic SE process on pilot projects
Phase 2 identifies the following six (6) tasks to be performed by the Process Action Team:
Draft descriptions of the basic SE activities as illustrated on the “Vee” diagram. Leverage from the high level descriptions developed within the SE Guidebook for ITS projects, and add detail for pilot projects.
Train project stakeholders in Basic SE activities.
Perform the basic process activities on pilot projects. Start from the Concept Exploration and progress through the life cycle Phase.
At the end of each action phase collect user feedback on the process.
Identify the risks incurred, benefits gained, strengths, and weaknesses of each process activity of the phase.
Identify a set of tailoring options that can be used for future projects.
* The BOLDED text represents new activities.
Phase 3 documents and implements the advanced SE process on pilot projects
Phase 3 continues with performing the same of Phase 2 activities in addition to the following action items:
The Process Review Board (PRB) will review the output of Phase 2, assess the activities performed, and make any course corrections for future project phases.
The Process Action Team will write advanced SE process activity descriptions and apply them to pilot projects.
Phase 4 documents and implements the Department’s SE process at the Headquarters level and integrates it into the existing capital development process
Phase 4 actions are performed at the Headquarters’ level. This builds on Phases 2 and 3. Additional action items include:
Write descriptions for the management process activities.
Integrate the SE process activity descriptions into the capital development process. Format and included in the existing capital development and management documentation.
Provide additional training for SE process management activities.
Determine whether the Department needs higher level capability. These additional levels of capability exceed the requirements of the OCIO and FHWA/Caltrans Joint Stewardship and Oversight Agreement. They are:
Level 4 – Quantitative Management of the SE process
Level 5 – Continuously optimizing the SE process
Phase 5 addresses specific ITS project challenges for on‐going projects
Phase 5 describes on‐going activities to improve the capabilities for specific project challenges:
Monitor the process performance on all ITS projects over time, and gather feedback to continuously tailor the process activities over a broad range of ITS project types.
Improve the SE process activities by addressing specific project challenges.
This schedule is based on process improvement experiences in other industries
Phase 1 ‐ Start‐up activities less than 1 month.
Phase 2 ‐ Will take approximately 9‐12 months to achieve the performance of the basic process.
Phase 3 ‐ Will add a number of advanced SE process activities. This will take on the order of 15 ‐ 18 months to achieve.
Phase 4 ‐ Will institutionalize the SE process activities. This will take approximately 9‐12 months.
Phase 5 ‐ On‐going activities would continue until enough experience has been gained.
Options for accelerating the schedule:
Eliminate pilot projects from Phase 2 – Document the basic process and proceed to Phase 3. This should reduce the Phase 1 schedule by approximately 3‐4 months. [32 months total]
Perform option 1 and then divide and perform advanced process across 3‐4 pilot projects concurrently. This should reduce Phase 3 schedule (assuming projects are available) approximately 6 months and total of 9‐10 months from Phases 3 & 4. [26 months total]
No pilot project in Phases 2 and 3. Document the basic and advanced processes and proceed to Phase 4. This is the most aggressive of the options, saving 12‐16 months from Phase 2 and 3. [20 months total]