7       Capabilities and Best Practices in System Development

Often, ITS development requires the project owner to build or select a team to define, design, develop, and deploy an ITS project. Candidate teams are evaluated against a set of criteria such as: past performance, dedication of key staff to the project, approach to the project, knowledge of the project, and schedule for completion. Usually, the proposed costs are not considered until an evaluation of capabilities has been made and a “short list” of teams has been determined.

This has been the traditional approach. In recent years, new criteria in the area of capabilities in software and systems development have emerged. Now, there is the ability to evaluate candidate development teams or internal agency organizations based on their capability to perform software and systems development. These criteria can be found in the tool called Capability Maturity Model Integration [CMMI].

Background:

The following is an excerpt from CMMI Distilled: a Practical Introduction to Integrated Process Improvement , a SEI Series textbook in Software Engineering, by Ahern, Clouse, and Turner which was published by Addison-Wesley in 2001.

“Model-based process improvement involves the use of a model to guide the improvement of an organization’s processes. Process improvement grew out of the quality management work of Deming [1], Crosby[2], and Juran[3] and this work was aimed at increasing the capability of work processes. Essentially, process capability is the inherent ability of a process to produce planned results. As the capability of the process increases, it becomes predictable and measurable, and the most significant causes of poor quality and productivity are controlled or eliminated. By steadily improving its process capability, the organization matures”.

The early 1990’s saw a proliferation of models for process assessment that included: acquisition, people, security, integrated product development, software, systems development, and project framework, in addition to the ISO 9000 series. This created a quagmire of process standards and quality models. To eliminate inconsistencies, duplications, and provide a common framework, terminology, and focus, Capability Maturity Model Integration [CMMI] was initiated by the U.S. Department of Defense and the National Defense Industrial Association [NDIA] in 1997. They teamed with the Software Engineering Institute at Carnegie Mellon to integrate the pertinent models for systems development together into a single model. It is called Capability Maturity Model Integration [CMMI]. The CMMI model uses source material from Software [SW-CMM, draft version 2c], Systems Engineering [EIA/IS 731], and integrated product and process development [IPD-CMM, version 0.98]. The team that put CMMI together included authors from the source models and other key industry experts. The final version was completed in 2000. To download the latest version of CMMI for free, go to http://www.sei.cmu.edu/cmmi/ CMMI supersedes previous capabilities standards such as SW-CMM and systems engineering EIA 731.

Best Practice Areas:

Figure 7‑1 illustrates how CMMI has integrated the best practices from source material into 24 Process Areas [EIA 731 has 19 of these and calls them Focus Areas] or best practices. In CMMI these process areas are divided into four categories as illustrated. These process areas cover the “waterfront” of best practices needed for systems development. The graphic illustrates how processes support the next level up. The Process management at the enterprise level supports the management processes for the program and project level. The management processes, in turn, support the engineering processes at the project level. Cross-cutting processes that support all levels are on the left side.

Figure 7‑1 CMMI Process Areas and Categories

As illustrated, a set of best practices is associated with the following categories: engineering, management, support, and process management. For each of the 24 process areas illustrated, there is a set of practices which is required to be performed by the organization to demonstrate a capability and achieve a level of maturity.

Rating Systems:

How is CMMI used? It is a rating system [developed by the Software Engineering Institute] used by software and systems development firms to rate how well their organization performs software and systems development. It is used by system’s owners, as an evaluation tool, for the selection of a candidate systems development organization. This rating is accomplished in two ways: 1] as a staged representation illustrated in Figure 7‑2 and 2] continuous representation illustrated in Figure 7‑3.

1] Staged representation provides a single number [0-5] for an organization that is an indicator of how well they perform software or systems development.

§         Level 0 [not on scale] means no processes are documented or followed.

§         Level 1 [Initial] Competent people, heroics of the individuals characterize the completion of projects. Processes are known and understood but performance is sometimes unpredictable, poorly controlled, and reactive in execution.

§         Level 2 [Repeatable] Basic project management is performed, some configuration, requirements, planning, and control is performed. Practices exist at the project level only and are reactive. Characterized as a good project team, working together. And producing repeatable results from project to project.

§         Level 3 [Defined] indicates that the organization has a standardized set of documented processes and proactively performs these processes.

§         Level 4 [Managed] indicates that the organization has statistical methods for analyzing the processes performed. The processes are measured and controlled.

§         Level 5 [Optimizing] Organizations have continual process improvement. The organization has a focus on process.

Figure 7‑2 Staged View or Representation of CMMI

2] Continuous representation is used to focus on specific best practices and is not concerned about an overall rating. In this case, an organization may focus on 20 or 24 best practices which are critical for the organization. The focus for this example is on performing the 20 best practice areas at a level 3 or higher. The others are not relevant to the organization. Figure 7‑3 illustrates a fractional score for configuration management. This is done in the continuous representation because an assessment credits an organization for individual sub-processes. For example, if 8 sub-processes make-up configuration management and 4 of them are performed at a level 3 while the other 4 are performed a level 2, then the organization will receive a 2.5 rating for the configuration management process. This results in fractional scores for individual processes in the continuous representation.

Levels of Maturity:

CMMI is a single model with two representations or views: staged and continuous, as discussed above. Organizations choose their representation for process improvement and thereby achieve a level of maturity for their organization.

Organizations may choose their representation depending upon their goals and objectives. For example, a company that provides systems development services may elect to use the staged view; since the results would be a simple number that identifies the organizations maturity level [1-5] as illustrated in Figure 7‑2. Other organizations may elect to use continuous representation to illustrate a “profile” of maturity across the process areas as illustrated in Figure 7‑3. These organizations may be more interested in achieving a profile which addresses specific needs. For example, it may be appropriate for a large agency that develops their own systems to use the continuous view in order to achieve maturity in specific areas. Other areas may not be applicable to them.

It should be noted that, in some cases, the higher levels of maturity are not needed or warranted. So, an organization may elect to stay at a level 2 or 3. The processes involved to achieve the higher levels of maturity [3, 4, and 5] may be too expensive for the return, or the domain of practice does not require it.

The following is an example of how the stages of maturity build on each other. A development company at a level 2 CMMI [staged representation] means they have a set of repeatable processes [e.g., estimating the cost for developing software]. If a company advertises that they are a level 3 [staged representation], this implies that they not only have repeatable processes required for level 2, but also have defined and documented processes required for level 3. In the staged representation, each level of competency builds on the previous level. CMMI provides a way to map the continuous representation into a staged representation.

Figure 7‑3 Continuous View or Representation of CMMI with Nine Example Process Areas

How would the ITS project owner use CMMI for contracting purposes?

Armed with this knowledge, a project sponsor can develop and publish a Request for Qualifications [RFQ] using a level of maturity for a systems development team as one of the evaluation criterion. When verified, this would demonstrate the development team’s capabilities. The caution here is for large companies where the maturity level may refer to a specific team within a company. It does not apply to the entire company unless the whole company performed on the assessed project at the desired maturity level. When reviewing the qualifications, make sure the team proposed at a maturity level is the assessed team .

How would an organization use CMMI to improve internal processes?

Within CMMI there is a process set-up for doing CMMI assessments. It is called Assessment Requirements for CMMI [ARC]. There are trained assessors who evaluate organizations and provide a report. The following are the major requirements being assessed:

§         Responsibilities – lists responsibilities of the assessment sponsor and assessment team

§         Assessment method defined

§         Planning and preparing for the assessment

§         Assessment data collection

§         Assessment data consolidation and validation

§         Rating

§         Reporting results

The team usually does an interview of the assessment sponsors to determine the goals and objectives of the assessment; and will tailor the assessment for that purpose.

Specific projects are identified and those project teams are interviewed using a set of assessment tools such as questionnaires and interviews. Once the results are validated, a report is generated that identifies the level of maturity and/or areas of needed improvement.

Types of assessments:

The following are the types of assessments that can be done:

Formal SEI -ARC allows the following:

§         Class A – Full comprehensive method

§         Class B – Less comprehensive, partial self- assessment

§         Class C – Quick look [check specific areas]

Internal Assessments are usually done in-house by another department, or quality team, with assessment capabilities. This assessment is carried out the same way as SEI assessors would do. However, it is not officially reported to SEI when completed. This type of assessment is less political and likely to be more realistic than a formal assessment that gets advertised outside the organization.

Internal assessment can be carried out in accordance with the formal SEI-ARC

Mini assessment is usually an internal assessment that performs a “quick evaluation” in key process areas. These popular assessments are inexpensive to conduct; using internal resources.

The cost of these assessments varies greatly and depends upon the type of assessment, tailoring requested, and size of the team. A minimum cost would be approximately $30K-60K [Quick look and mini-assessment] expanding to several hundred thousands of dollars. [Class A Formal full comprehensive method].

Formal and internal assessments are common for Defense, Federal Aviation Administration, and Department of Energy firms.  These assessment are also being performed for banking and some information technology firms.

For transportation agencies, these types of assessments are not well known nor widely-used. Current few, if any, ITS development firms been appraised against CMMI for ITS projects, or one of the earlier models for systems engineering [SE-CMM] or software [SW-CMM]?” The reason is that agencies have been, for the most part, unaware of these tools. They have not made it one of their evaluation criteria in RFP’s or RFQ’s. Since there has not been a significant push from the agencies to make this happen, development firms have not offered this in their proposals. Hopefully, as a result of this Guidebook, criteria for CMMI levels of maturity will start showing up in RFQ’s, RFP’s, and RFI’s. Over time, it should be common for ITS development teams to perform, at a minimum, level 2 or, preferred, level 3 on the CMMI staged representation. This will provide the project sponsor the confidence that the selected development team has the capability for performing well on an ITS project thereby reducing project risks.

When a system integrator makes the decision to achieve a CMMI maturity level 2, the average time is from 1-2 years to document, train for and implement the needed processes. Then, the assessment takes place after the level 2 practices have been applied to a real project. This may take another 2 years [assuming an 18-24 month project]. It may take from 3-5 years for a system integrator starting at level 0 to be assessed and recognized as being at CMMI level 2.

In the short term, it is recommended that evidence of progress work towards a CMMI level of maturity be shown as a demonstration of a best effort over the next 3 to 5 years. For example, this could be done by providing documented processes and procedures [drafts or final] showing staff with appropriate certifications or the systems integrators’ process improvement. After this period, System’s owners would give this criterion more weight in the qualification evaluation process.

Systems Engineering Technical Assistance [SETA] consultants or systems managers may not be currently considering any formal assessment to be performed. An internal assessment using continuous representation would be recommended. As an alternative, staff can demonstrate their expertise through professional certification programs like the INCOSE Certified Systems Engineering Professional [CSEP] and Project Management Institute [PMI] certification.

The following table identifies the suggested capabilities profile and best practices that each would consider having, at a minimum, for ITS development.

Table 7‑1 Suggested Minimum Capabilities Table for ITS

Legend for Table