U.S. Department of Transportation
Federal Highway Administration
1200 New Jersey Avenue, SE
Washington, DC 20590
202-366-4000


Skip to content U.S. Department of Transportation/Federal Highway AdministrationU.S. Department of Transportation/Federal Highway Administration

California Division

Home / About / Field Offices / California Division / Systems Engineering Guidebook for ITS

Home What's New Systems Engineering Guidebook Views Search Glossary Resources Feedback Site Map

7.3 How Do You Measure Process Improvement?

CMMI provides a basis for measuring the overall “maturity” of an organization with respect to process and the “capability” of individual processes.  CMMI uses a series of well-defined levels to measure both of these aspects of process improvement.  Maturity levels between 1 and 5 are used to rate the overall process maturity of an organization and capability levels between 0 and 5 are used to rate the capability of specific process areas. 

CMMI Representations

These two perspectives – organizational maturity and process capability - are supported by two different “representations” in CMMI:

1.       The Staged Representationprovides a predefined road map for organizational process improvement that moves through a sequence of five maturity levels.  Each maturity level has a set of process areas associated with it.  An organization reaches a maturity level by satisfying all of the goals associated with the process areas that are in that maturity level.  In general, the predefined approach in the staged progression begins with improved project management, progresses to organization-wide process definition, and finally implements standardized process definitions that support quantitative analysis.   The staged representation provides a couple of key benefits: 1) it provides a means of comparing the process maturity of multiple organizations since each maturity level is specifically defined, and 2) it provides a proven path for process improvement that has been used successfully by many organizations.

 

Maturity levels are Initial, Managed, Defined, Quantitatively Managed, and Optimized.

Figure 7‑7  Maturity Levels in the Staged Representation

2.       The Continuous Representation lets you decide the order in which process improvement should be accomplished.  Using the continuous representation, you decide the process areas that should be improved and the capability levels that should be attained in each process area based on your organization's specific needs.  Within each process area, goals are associated with different capability levels.  You attain a specific capability level for a process area by meeting all of the goals at that capability level.  Since each capability level builds on previous capability levels, you also have to satisfy all goals at the lower capability levels.  Obviously, the benefit of the continuous representation is its flexibility.  The drawback is that there are no specific stages and no concept of overall organizational maturity, so it is difficult to compare two organizations based on CMMI if they are using the continuous representation.  Another drawback is the flip side of flexibility – too many choices can mire a process improvement effort in more debate and decision-making delay than if the predefined staged representation was used.

Capability levels are Incomplete, Performed, Managed, Defined, Quantitatively Managed, and Optimized.

Figure 7‑8 Capability Levels in the Continuous Representation

A quick comparison of Figure 7‑7 and Figure 7‑8 hints at the close parallels between the two representations, particularly at the higher levels.  In both representations, the focus through level 2 is on process performance at a project level.  At levels 3 and above, the focus shifts to the organization with institutionalization and management of the process at the organizational level. 

In ITS, the flexibility of the continuous representation makes it an attractive representation for many organizations considering a process improvement initiative.  As you take advantage of the flexibility of the continuous representation, it is also important to understand the “standard” path to process improvement that is embedded in the staged representation.  This path begins with a focus on project management and applying processes on selected projects, adding organizational policies, training and resources to support the processes, and then documenting and managing proven processes at the organizational level.  This proven path provides a good reference and a starting point for your process improvement plan.

Appraisals

There are two key reasons to perform an appraisal: 1) to accurately diagnose internal process problems and areas for process improvement, and 2) to create a benchmark for process capability/maturity that can be used to rate the organization.  In order to measure your organization's process capability, you compare your organization's processes to CMMI or another baseline process model.  Such an examination by a trained team of professionals using CMMI is called an appraisal.  While the terms appraisal and assessment are used interchangeably in everyday conversation, there is an important distinction within CMMI.  An assessment  is an appraisal that an organization does internally to support process improvement. 

Three types of appraisals are defined in the SEI document “Appraisal Requirements for CMMI” [ARC]:

  • Class A – Full comprehensive method, used for benchmarking
  • Class B – Less comprehensive, partial self- assessment
  • Class C – Quick look [check specific areas]

There are 40 requirements and constraints for process appraisals included in ARC.  Only the more comprehensive (expensive) Class A appraisal that meets all 40 ARC requirements can be used to generate benchmark capability/maturity levels that are published by SEI.  The Class B and Class C appraisals require fewer resources and still support process improvement objectives.

More information on the CMMI appraisal program can be found on the SEI website at http://www.sei.cmu.edu/cmmi/tools/appraisals/. 

While an experienced consultant can be hired to perform an appraisal, informal assessments may also be performed in-house by another department or quality team with assessment capabilities.   An internal assessment is carried out in much the same way as a formal CMMI appraisal, but the results are not officially reported to SEI as they are in a Class A appraisal. An internal assessment is less political and likely to be more realistic than a formal appraisal that gets advertised outside the organization.   Most transportation agencies, being new to CMMI, should keep it simple initially and start with an assessment geared towards process improvement.

Any appraisal should begin with an understanding of the sponsor's goals and objectives for the appraisal and a clear definition of the portion of the organization to be appraised.   Specific projects are identified and a combination of document reviews and interviews are used to understand the processes that are currently defined and how well each appraised project conforms to the defined processes.  Once the results are validated, a report is generated that identifies the level of maturity and/or areas for process improvement.

The cost of CMMI appraisals varies greatly and depends upon the type of appraisal, tailoring requested, and size of the team.   A minimum cost would be approximately $30K for a Class C appraisal to several hundred thousands of dollars for a full Class A appraisal.  

Formal appraisals are common for Defense, Federal Aviation Administration, and Department of Energy firms.  These appraisals are also being performed for banking and some information technology firms.  To date, few ITS firms have subjected their processes to CMMI appraisals that could lead to organizational comparisons, although a few ITS firms have undertaken their own assessments.  Internal assessments have also been performed by a few state DOTs.  FHWA has developed a Systems Engineering for ITS Workshop that includes a process improvement review  that focuses on systems engineering processes related to 23 CFR 940 requirements.  This assessment is a good way for transportation agencies to start a process improvement initiative since it is provided by FHWA at no cost to the agency and is focused on those process areas most germane to ITS and the underlying 23 CFR 940 requirements.

Return to top
Federal Highway Administration | 1200 New Jersey Avenue, SE | Washington, DC 20590 | 202-366-4000