Process Review Quality Improvement Team
Final Report
Word Version (153 kb) | PDF Version (87 kb)
December 15, 1998
Executive Summary
Purpose of Review:
The purpose of this review was to evaluate the quality of the FHWA/STA Joint Process Review program and to offer recommendations for improvements.
Background:
This review primarily consisted of interviews with individuals from the STA Central Office, STA District Offices, the .......... Division of FHWA, and other FHWA Division Offices. Discussions were held with management as well as with personnel who have had significant experience with the process review program. The following is a summary of the significant observations of the review.
Observation #1: Support of Process Review Program
The current process review program in .......... is well received and well supported by management at both STA and FHWA.
Recommendation:
The process review program should continue in .......... in its current basic format. It should be emphasized that these reviews are joint STA/FHWA reviews.
Observation #2: Process Review Program Coordinators
The lack of STA and FHWA process review program coordinators reduces the continuity, efficiency, and effectiveness of the joint process review program.
Recommendation:
FHWA and STA should each formally identify a process review program coordinator.
Observation #3: Overall Process Review Procedures
Process review teams are often uncertain as to the degree of formality and the proper procedures to be used in conducting process reviews.
Recommendation:
New guidance has been developed on the procedures for conducting process reviews. This guidance should be adopted by STA and FHWA, and a training seminar should be conducted to showcase the new guidance.
Observation #4: Compliance vs. Partnering
Joint process reviews are still often perceived to be FHWA compliance reviews.
Recommendation:
All process review correspondence should be co-signed by STA and FHWA review team leaders. Correspondence should be sent to both STA and FHWA and should use the newly developed process review letterhead.
Observation #5: Follow-up to Process Review Recommendations
Follow-up to process review recommendations is currently insufficient.
Recommendation:
A formal process review tracking system should be created and maintained in the FHWA .......... Division office, and the review teams and/or delegated specialists should improve follow-up of process review recommendations.
Observation #6: Selection of Process Review Topics
The annual process review topic selection meeting is the preferred method for developing the annual process review program. However, submitted topics are often too broad or generic to allow the review teams to properly focus their reviews.
Recommendation:
The annual meeting should continue and written briefings should be submitted when topics are suggested for review.
Observation #7: Team Member Selection
Identification of team members has traditionally been done several months after the selection of the process review topics, resulting in the loss of a significant portion of the review year.
Recommendation:
Review teams should be selected within 30 days after the annual topic selection meeting.
Observation #8: District Participation on Review Teams
STA Districts are not regularly being solicited to participate on process review teams.
Recommendation:
Standard practice should include solicitation of the Districts to participate on process review teams.
Observation #9: Need for Increased STA Participation
In the recent past, STA team members' participation on many District reviews has been inadequate.
Recommendation:
STA staff should only be assigned to one process review per year to allow for full, active participation in all the District reviews. STA should consider incorporating process review activities into their annual performance rating criteria.
Process Review Quality Improvement Team
Final Report
Purpose:
The purpose of this review was to evaluate the quality of the FHWA/STA Joint Process Review program and to offer recommendations for improvements.
Background:
The process review program in .......... has been in existence for over 20 years and has continually evolved during this time. Emphasis of reviews has moved from Federal compliance and cyclical reviews to joint agency quality improvement reviews based on selection of review topics through a partnering process. This change in emphasis, along with changing personnel, stewardship roles, and priorities, have all affected the quality of our process review program. Thus, there was a need for an examination of current process review procedures in ..........
Scope of Review:
This review explored the current procedures used in the process review program, including engineering, administration (financial), right-of-way, and planning. Items that were addressed included selection of review topics, team composition, development of purpose and scope, District review procedures, statewide close-out report and meeting procedures, evaluation of process review quality, and review follow-up procedures.
Review team:
............................................................
This review examined past, current, and desirable practices and procedures for the process review program in .......... Recommendations were made based on interviews with various FHWA Division Office personnel as well as STA personnel from the Central Office and selected Districts who have had significant involvement with previous process reviews. STA Bureaus interviewed (in separate group settings) were Bridges and Structures, Materials and Physical Research, Design and Environment, and Construction. All Deputy Directors of Highways were interviewed as well as staff from STA Districts 1, 3, 4, 5, 7, 8, and 9. The interview process also included discussions with numerous individuals from other FHWA Division Offices. This review focused on identifying best practices from successful quality improvement/process reviews and other review programs. This report offers recommendations for improvement based on a comprehensive assembly of "best practices."
Process Review Program in ..........:
Traditionally, process reviews in .......... have been conducted by FHWA personnel with limited team involvement of STA personnel. These reviews were required by FHWA stewardship policy in States with Certification Acceptance programs. Many of the reviews were perceived to be compliance oriented with the focus of a "gotcha" mentality. Even with this perception, many reviews were very successful and resulted in significant improvements in many areas of the highway program in .......... The process review program in .......... has always received favorable support from upper management at both STA and FHWA.
Several documents have been developed over the past years to serve as training tools and reference for properly conducting a process review. These documents have proven to be very useful to those involved in the program. As STA and FHWA have progressed and changed their stewardship roles, focus has shifted to joint process reviews with STA and FHWA sharing in the responsibilities of the reviews. With this change in philosophy, it has become necessary to revise and update the current procedures for process reviews. Hence, the need for this review. The following are the observations and recommendations made by this process review quality improvement team and are offered for consideration of adoption by both STA and FHWA management.
Observation #1: Support Of Process Review Program
The current process review program in .......... is well received and well supported by management at both STA and FHWA.
Discussion:
Based on feedback from numerous managers at STA, as well as FHWA, the existing process review program should continue. The program is beneficial for improving the quality of STA's highway program and fulfilling FHWA's stewardship responsibilities. Process reviews conducted over the past years have directly and/or indirectly resulted in major improvements in many areas and disciplines in the .......... highway program.
Recommendation:
The process review program should continue in .......... in its current basic format. However, an emphasis should be placed on the new philosophy that these are joint reviews rather than FHWA compliance reviews.
Observation #2: Process Review Program Coordinators
The lack of STA and FHWA process review program coordinators reduces the continuity, efficiency, and effectiveness of the joint process review program.
Discussion:
One of the primary concerns that has been identified through this quality improvement review was the lack of an individual in charge of the overall process review program. Continuity, quality, and uniformity have been suffering because of the lack of process review program coordinators. The team highly recommends the establishment of a process review program coordinator within the .......... Division of FHWA and within STA. These two individuals should be in charge of organizing and conducting the annual process review meeting, identifying individual review team leaders, recommending team members, tracking review progress, commenting on process review reports, commenting on the overall quality of process reviews, developing and maintaining a process review tracking system, ensuring timely follow-up to process review recommendations, disseminating process review findings as appropriate, developing an annual summary report of process review activities, and any other activities related to the process review program. These specific activities are discussed further in the attached Process Review Guidelines.
Recommendation:
The team strongly recommends that FHWA formally identify a process review program coordinator in the .......... Division Office. This person would be responsible for coordinating the entire process review program in .......... The team also recommended that STA identify an overall process review program coordinator. This individual should not require full-time dedication to the process review program, but it should be a significant collateral duty. Duties of the STA process review program coordinator would be similar to that of the FHWA process review program coordinator, but to a lesser degree.
Observation #3: Overall Process Review Procedures
Process review teams are often uncertain as to the degree of formality and the proper procedures to be used in conducting process reviews.
Discussion:
Based on discussions with past process review team members and investigation of process review files, the review team determined that no standard procedures are being used for conducting process reviews. Some teams have been very formal while other teams have taken a more informal approach. Issues such as initial distribution of guidelines, distribution of District and statewide reports, content of process review reports, presentation techniques at statewide close-out meetings, and many other minor procedural issues were found to vary significantly. In addition, with new FHWA staff and the expectation that STA will select some team members new to the process review program, the need for guidance on process review procedural matters has been identified. The last guidance on process reviews in .......... was issued in 1987. However, some of that information is now considered outdated and an updated version of the process review guidelines would therefore be beneficial.
Recommendation:
The Process Review Quality Improvement Team has developed guidance on the overall process review program. The review team recommends this guidance be adopted as the official procedures for process reviews in .......... and also recommends presenting a half-day training seminar on the adopted process review procedures. The guidance is included as an attachment to this report.
Observation #4: Compliance Versus Partnering
Joint process reviews are still often perceived to be FHWA compliance reviews.
Discussion:
Based on discussions with STA staff at both the Central Office and the District level, the team believes that many individuals still perceive these reviews to be a duty of FHWA focusing on compliance issues. Although this may have been true in the past, management has shifted emphasis from compliance issues to a joint effort looking for best practices and quality improvement potential. Traditionally, FHWA has led the reviews, drafted the reports, conducted the statewide close-out meetings, and signed all official process review correspondence to STA. The team believes that a shift in the process review responsibilities as indicated in the attached guidelines will help demonstrate that process reviews are truly joint reviews and STA has equal responsibility in all stages of the review.
Recommendation:
The team recommends that all process review correspondence be co-signed by STA and FHWA review team leaders and the correspondence be addressed to both STA and the .......... Division of FHWA. Standard process review letterhead with both agencies' logos (see example attached to the guidelines) has also been developed and should be used for all process review correspondence. The team also recommends that assignments should be divided among the FHWA and STA process review team members throughout the review as much as possible.
Observation #5: Follow-Up To Process Review Recommendations
Follow-up to process review recommendations is currently insufficient.
Discussion:
A high-quality process review requires follow-up on significant review observations to ensure that all resolutions to recommendations are implemented. Many times in the past, agreements were reached at the statewide close-out meetings but were not aggressively pursued. Ultimately, many actions were not implemented because of the lack of follow-up. In the past, the .......... Division of FHWA utilized the Management Information Control System (MICS) to track the status of recommendations, but because of limited capability, restricted access, and not being user friendly, the use of the system has been essentially discontinued. It was agreed that a process review tracking system is necessary to help ensure more timely implementation of resolutions from the statewide close-out meetings.
Recommendation:
A formal process review tracking system should be created and maintained in the .......... Division Office, and review teams and/or delegated specialists should improve follow-up of process review recommendations.
Observation #6: Selection Of Process Review Topics
The annual process review topic selection meeting is the preferred method for developing the annual process reviews. However, topics are occasionally selected with minimal evaluation and the most appropriate reviews may not always be selected. In addition, the review topics are often too broad and general for the review teams to properly establish a well defined purpose and scope.
Discussion:
Currently, numerous topics are suggested for possible review by both STA and FHWA management at the annual process review topic selection meeting. A consensus is reached on the most appropriate topics which usually results in five or six process reviews per year. There is seldom any communication between STA and FHWA prior to this meeting regarding which topics will be discussed. This does not allow for much evaluation of the proposed topics and, as a result, the meetings have been conducted essentially as brainstorming sessions.
The topics chosen for review are sometimes very broad issues. When review teams prepare purpose and scope statements, it is often difficult to identify the original intent of the review.
Recommendation:
A written briefing should be submitted whenever a topic is suggested for review. The briefing would explain why the topic should be selected and mention the desired outcome from the review. Requiring a one- or two-paragraph briefing with the topic would force more thought and preparation of a review topic and would ultimately result in more focused, quality reviews.
Observation #7: Team Member Selection
Identification of team members has traditionally been done several months after the selection of the process review topics, resulting in the loss of a significant portion of the year.
Discussion:
The annual process review topic selection meeting is usually held in September and the team members have not typically been selected until December. This leads to a loss of several months during the winter when STA and FHWA team members could devote substantial time to a process review. A typical process review is expected to last one year and, with this delay in team member identification, up to 25% of the year is lost.
Many of the Districts have indicated that this time between September and December is a good time for them to participate in a process review due to the lighter workload after the construction season. Many of the District reviews could be held during this time if teams were selected and finalized soon after the annual topic selection meeting.
Recommendation:
The review teams should be selected within 30 days after the annual topic selection meeting. This would allow review teams to begin the process review earlier in the year and possibly conduct some District reviews during the winter.
Observation #8: District Participation On Review Teams
The STA Districts are not regularly being solicited to participate on the process review teams.
Discussion:
The review teams have traditionally consisted of FHWA and STA Central Office staff. A list of the process review topics is sent to the Districts, but they are not regularly asked to participate in the reviews. When District staff have been on review teams in the past, they have provided valuable experience and offered a different and beneficial perspective on particular processes. The Districts have indicated they would like to have the opportunity to participate on a review team. However, unless the District has a strong interest in the topic, they may not want to commit their resources due to recent staffing reductions.
Recommendation:
The review team recommends developing a standard practice to solicit the Districts to participate on process review teams. If District personnel cannot fully participate on a review, they could be asked for limited participation by taking part in developing the guidelines, purpose and scope or by commenting on the observations of the review.
Observation #9: Need For Increased STA Participation
In the recent past, STA team members' participation on many District reviews has been inadequate to conduct a thorough, successful process review.
Discussion:
STA team members are chosen to participate in process reviews by STA upper management and are often selected to participate on more than one process review per year. These individuals have many other responsibilities which makes it difficult to fully participate on more than one process review at a time. STA staff have indicated that process reviews often are low priority and their immediate supervisors tend to support their assessments. In order to ensure the success of the joint process review program, it is essential that STA team members attend District reviews and participate as much as possible. The STA upper management support and direction in this area are critical in encouraging more participation by the STA team members.
In order to properly emphasize the importance of devoting efforts to the process review program, FHWA has traditionally included the accomplishments of the process reviews as a major portion of individual annual performance appraisals. Similar emphasis could be given from STA management to ensure full participation from STA team members. Careful selection of STA team members is important so that existing work loads can be taken into account to ensure there is time available and motivation for the team members to actively participate in the process reviews.
Recommendation:
STA staff should only be assigned to one process review per year to allow for full, active participation in all the District reviews. STA should consider incorporating process review activities into their annual performance rating criteria to help improve STA participation on process reviews and help demonstrate support of the process review program by STA upper management.
Guidelines for Conducting Process Reviews
The following is general guidance on how a typical process review should be conducted. This guidance is based on a compilation of "best practices" and recommendations from management at both FHWA and STA. Flexibility of these guidelines is necessary because many action items vary depending on the topic of the process review.
The guidance has been organized into the following categories:
- Selection of Process Review Topics
- Selection of Team Members/Identification of Review Team Leaders
- Development of Purpose and Scope
- Guideline Development
- District Reviews
- Statewide Close-out Report
- Conducting the Statewide Close-out Meeting
- Evaluation of Process Review Quality
- Follow-up to Process Review Observations
Selection of Process Review Topics
The current format of inviting STA and FHWA management together for an annual joint meeting in early September is the preferred method of selecting topics for process reviews. This September meeting is desirable in order for the review topics to be finalized by October 1 and announced and advertised at the annual STA Program Development Engineer's Meeting (September/October) and the Project Implementation Engineer's Meeting (January/February). The format of the annual process review topic selection meeting should remain the same as in the past. All STA Deputy Directors should attend as well as a representative from each STA Central Office Bureau. FHWA should be represented by the Division Administrator, Assistant Division Administrator, Team Leaders, and program specialists (i.e., bridge engineer, environmental specialist, etc.). Others interested in attending should be invited to listen to the discussion in a gallery format.
The FHWA and STA process review program coordinators will be responsible for collecting, organizing, and distributing a proposed list of topics. STA and/or FHWA personnel should submit topics to either of the process review program coordinators. These submittals should consist of a briefing (one or two paragraphs) explaining why the topic should be chosen, an example of any specific incidences for correction, and what outputs are anticipated from the process review (specification change, new policy, new product, etc.). The target deadline for submission of topics should be August 15 of each year. The process review program coordinators should distribute the list of potential topics (with briefings) to all STA Bureau Chiefs, Deputy Directors, and FHWA management at least two weeks prior to the annual topic selection meeting. This will allow for better preparation and evaluation of topics and will eliminate some of the "on-the-spot" decision making.
Before the annual topic selection meeting, all FHWA engineers/specialists will meet to discuss the proposed topics and provide feedback to management on recommended review activities. It is recommended that appropriate STA staff meet prior to the annual meeting for the same purpose.
FHWA transportation engineers and specialists should be gathering information for process review topics throughout the year from the Districts. The Districts should be solicited for topics on an informal basis, when the transportation engineers are in the Districts. At every FHWA Engineers/Specialists meeting (generally held every two months), there should be time set aside for discussion of future process review topics. A current list of possible topics (with briefings) should be kept on the common directory of FHWA's LAN system for review and comment.
STA should perform similar activities for soliciting topics from the Districts. In the past, memos were sent to all District Bureaus of Project Implementation asking for process review proposals. Other District Bureaus have indicated that they would also like to have an opportunity for input. It is recommended, therefore, that the Central Office continue to ask for topics from the Districts, but the invitation should be extended to all District Bureaus, including Program Development and Operations.
Selection of topics should be based on real or perceived opportunities for improvement, and consideration should be given to a fairly uniform distribution of review topics among the various STA Bureaus or program areas. Final topic selection should be based on consensus of both STA and FHWA management. The number of reviews chosen should take into consideration the workload on STA Districts, Central Office, and FHWA staff. The Districts have indicated they can handle one or two reviews per Bureau per year. There was general agreement that the number of engineering process reviews conducted per year should be five or six.
Selection of Team Members/Identification of Review Team Leaders
Teams should be selected shortly after the annual process review topic selection meeting in order to get the reviews organized and guidelines developed in a timely manner. Traditionally, teams have not been selected until December with the first team meeting not held until the first part of the following year. This leads to a loss of several months during the winter when most of the Districts have indicated they would have more time to devote to a process review.
Process reviews have been conducted in the past with District personnel on the review teams. Generally, this has been the exception rather than the rule. District representation has proven to be very beneficial in past reviews. Not only does it benefit the statewide review team, but it benefits the District through observation of other Districts' practices. All Districts should be given the opportunity to participate on the review teams. Solicitation of District team members should be done by STA Central Office management with input and recommendations from the individual STA and FHWA team leaders. Although the Districts would like the opportunity to participate, several Districts have indicated that unless there is a strong interest in the topic, they may not want to commit their staff to be on the reviews due to limited staff resources. This should be a consideration when discussing team composition.
Generally, two or three FHWA staff with two or three STA Central Office staff should make up a core process review team. District personnel should be considered for inclusion as discussed above. Efforts should be made to ensure that all affected STA Bureaus are represented on the team, and additional team members could be included for specific tasks throughout the review if necessary. Each team should have an assigned FHWA and STA team leader to lead the review. FHWA management should be responsible for assigning the FHWA process review team leaders and team members, with individual interest taken into account. The STA Deputy Directors should be responsible for selecting the STA team leaders and team members in consultation with the appropriate Bureau Chiefs.
The FHWA and STA process review team leaders should have substantial knowledge in the review area. No individual should be a team leader on more than one review per year. It is the responsibility of the review team leaders to organize the review and keep the review on track and within scope. FHWA management should choose an FHWA advisor if the assigned FHWA review team leader needs additional guidance and expertise in coordinating a particular process review.
The STA team leader should be selected as a central point of contact for STA and should provide guidance on the direction that the Central Office wants to take with the review. Communication between the STA team leader and STA management is essential to ensure that all affected Bureaus are informed of the progress and preliminary observations of the review.
A District contact person should be selected by the District Engineer for each of the Districts to be reviewed. This individual will serve as the focal point for all communication concerning the process review at the District level and be responsible for arranging a meeting time and place for the District review and coordinating with all District personnel involved in the review. The District contact person would also be responsible for gathering information on projects to be reviewed.
Purpose and Scope
The development of a purpose and scope is very important in defining a process review. As discussed above (in "Selection of Process Review Topics") a briefing should accompany a topic suggested for review which provides a description of the objective, an example of any specific incidences for correction, and the intention of the suggested review. This briefing will serve as the framework for the development of the purpose and scope.
In addition to the topic briefing, additional discussions held at the annual process review topic selection meeting should provide a substantial basis for the process review team to establish a purpose and scope. Based on the observations of the initial District review (pilot review), the team should reassess the purpose and scope to determine if they are on the right track to accomplish the goals of the review. If the team believes it is necessary to deviate from the intended purpose and scope identified through the topic briefing and the annual topic selection meeting, the team should meet with management before proceeding.
A review team should develop a finalized "Purpose" which clearly states the objective of the review. If the review includes a specific issue for improvement, the purpose should include background information and desired changes. As a note, it was often repeated through the interview process that the review teams should stay better focused on the review's original purpose throughout the entire review.
The Scope of Review should generally include the number of Districts involved, the number of projects to be reviewed, the specific action items (key steps in the process) to be reviewed in each District, and the review schedule, all as applicable. The "Scope" should also identify which District and/or Central Office Bureaus will be directly involved in the review. The need to re-analyze the scope after the pilot review can be determined by the review team on a case-by-case basis.
The number of Districts reviewed should be three or four; however, this also depends on the topic of the review. The process review teams should consider informally collecting key information from as many other Districts as possible. This can be done through telephone conversations, brief office visits, or by requesting that the Districts complete certain parts of the review guidelines. Districts should be chosen based on the availability of projects or action items related to the review topic. Try not to select Districts that have limited or no projects available for review or Districts that have only limited involvement in the particular review topic. The STA and FHWA process review program coordinators should monitor District selections of the various process review teams. Efforts should be made to ensure that no particular District is overburdened with too many process reviews while other Districts do not get reviewed at all. When scoping a review, consideration should also be given to selecting the Chicago Department of Transportation as a potential review candidate.
The number of projects selected for review in a particular District certainly depends on the topic. However, as a rule of thumb, a minimum of three projects should be selected for review. It is important to note that the more projects reviewed in a District, the more statistically valid the observations and recommendations will be.
Many process reviews will also require an STA Central Office review in order to assess the procedures and responsibilities of the Central Office staff. Since STA Central Office has the primary responsibility of policy development and project oversight, it is very important that the appropriate Bureaus be reviewed as a part of the overall process. It is suggested, but not necessary, that the Central Office review be conducted prior to any of the District reviews. Generally, an individual Central Office report would not need to be developed; however, the significant observations and recommendations should be included in the statewide close-out report for discussion and resolution.
Guideline Development
The first step in developing guidelines is to perform a search of any existing process review files that may be used or modified. There is no sense in "re-inventing the wheel" if it is not needed. The second step is to check with other State DOTs and FHWA Divisions to see if they have done similar reviews for which guidelines may be available for reference.
When preparing guidelines, every attempt should be made to avoid simple yes/no questions. Questions should be phrased to produce comprehensive feedback and be clear enough for everyone involved in the review to easily understand. The Districts often research answers ahead of time so it is important to be very clear as to what is really wanted from them. Guidelines should generally include the following: Purpose of Review, Scope of Review, Identification of the Review Team, and finally the Review Questions. Upon completion of the draft review guidelines, they should be placed in the common directory of the FHWA LAN system. The FHWA process review team leader should e-mail all appropriate personnel in the Division Office to alert them that the guidelines are available for review and comment. A two-week time frame for comments is adequate. The STA process review team leader should be responsible for ensuring that the guidelines are reviewed by proper STA personnel.
Upon finalizing the guidelines, the STA and FHWA team leaders will co-sign a cover letter to the FHWA Division Administrator and the STA Director of Highways with attention to the appropriate Deputy Director(s). The newly developed standard process review letterhead will be used for this transmittal as well as all other process review correspondence (see attachment for sample). A copy of this letter should be sent to all affected Bureau Chiefs in the Central Office. The time frame for formal distribution of guidelines should be at least 30 days prior to the first planned District review. The review guidelines should be transmitted to the Districts selected for review as an attachment to a formal cover letter to the District Engineer co-signed by the two team leaders.
The guidelines should also be sent to all of the other Districts not selected for review. This should be done through a formal cover letter to the District Engineer with instruction that it is for informational purposes only. If the review team determines that it is desirable to obtain certain information from Districts not selected for specific review, the cover letter should include a request that the appropriate sections of the guidelines be completed and returned to the team.
After sending guidelines to the Districts selected for review, the process review team should follow up approximately two weeks prior to the District review to remind the District of the upcoming review and to ensure there are no questions concerning the guidelines.
A copy of the final version of the guidelines should be placed in FHWA's electronic process review file. This file should be maintained by the FHWA process review program coordinator.
District Reviews
District reviews are intended to establish a statewide trend for the particular topic under review. A District review should consist of: Entrance Meeting, Interviews and Project Reviews (either ongoing projects or archived project files), Close-out Meeting and District Report.
From past experience, too many people conducting a District review can cause a loss in efficiency; therefore, the number of team members participating in a District review should be kept to three or four. It is important that there is an appropriate mix of both STA and FHWA team members on the reviews. All process review team members should attend the first (pilot) District review in order for the entire team to be familiar with what is expected in conducting subsequent reviews. The FHWA and STA team leaders should attend all the District reviews for continuity purposes.
Entrance Meeting
Entrance meetings are highly recommended in order to alert District management of the team's presence in the District and to explain the purpose and scope of the review. During this entrance meeting, a tentative time should be established for conducting the close-out meeting. Most Districts have indicated they want an entrance meeting with the District Engineer present along with all the affected Bureau Chiefs (or delegated substitutes). If the District Engineer is not available for an entrance meeting, it is important to at least inform him/her of the review team's presence in the District.
District Interviews and Project Reviews
The District interviews should be consistent throughout the course of the review to make sure the same information is obtained from each District. The process review guidelines should provide a good format for the interviews, and the Districts should review the guidelines in advance to be prepared to provide the desired information.
The purpose and scope of the review will define the number of projects to be reviewed in a District. Enough projects should be reviewed in order to get a good sample of the total projects fitting the topic of review. Usually three or more projects will establish a good trend of how the process under investigation is handled in the District. Because Districts spend a lot of time gathering project files for the review, careful project selection is important so as to not cause unnecessary project file search for the Districts.
Upon completion of interviews and project reviews, a preliminary report of observations should be drafted for use at the District close-out meeting.
District Close-out Meeting
If at all possible, the District Engineer should be at the District close-out meeting. The close-out meeting should be held before the team leaves the District and should be at a time when the District Engineer and the affected Bureaus can be represented. This meeting should cover all the observations the team made during the review. However, minor observations can be discussed and resolved informally and do not necessarily need to be included in the District report. During the close-out meeting, the team should discuss which observations are isolated incidences (i.e., occurred on only one project), which observations were found to occur only in that District, and which observations are of statewide concern and will be included in the statewide report. Generally, the observations of statewide concern are issues found in several Districts or issues that involve a Central Office function.
District Report
The purpose of the District report is to summarize the results of the District review and to document the team's observations and recommendations and the comments from the District. All Districts have indicated they want a written report in order to document the District review.
The review team should develop a draft District report within one week of the District review while the observations and discussions of the review are fresh in the minds of the review team as well as the District. The final District report should be completed and sent to the District as soon as possible but within 30 days of the review.
The format of the District report should be standardized as much as possible, similar to the statewide close-out report. Using a newly developed standard reporting form (see attachment for sample), the report should include the following in this recommended order:
- Title of Review
- Purpose of Review
- Scope of Review (including projects reviewed)
- Team Members
- District Staff involved
- Executive Summary (optional but recommended)
- Observations, including "best practices" in order of significance - each observation should have a recommendation (The use of "Observation" is preferred over "Finding")
- Conclusion (optional)
- Attachments (supporting documents, project information, etc.)
The FHWA and/or STA review team leader should ensure that all team members and the District have a chance to review the draft report before the final copy is released. The final version of the District report should be sent by formal cover letter to the District Engineer with a copy sent to all team members and others (if applicable) who participated in the review. The letter should be co-signed by the FHWA and STA team leaders, using the newly developed process review letterhead.
Statewide Close-out Report
The purpose of a statewide close-out report is to summarize the results of the process review, document statewide observations found in the Districts and to document the resolutions discussed at the statewide close-out meeting. Observations in the report can be either positive or negative. The review team should provide recommendations in the report that will resolve or improve the documented observations. The report also serves as an avenue to share "best practices" in .......... as well as from other states, if applicable.
The newly developed standard reporting form should be used for the statewide close-out report (similar to the District reports). The review team should complete an initial draft of the statewide close-out report within 30 days of the of the last District review. As a note, this will require finalization of the last District report concurrently with the preparation of the initial draft statewide report. This is necessary in order to complete the entire review by the November 30 target date.
Close-out reports can often get to be fairly lengthy documents. Therefore, there is a need and a clearly expressed desire to provide an executive summary of the process review. The executive summary should be concise and should provide information including purpose, scope, major observations, and recommendations.
Observations in a close-out report should be arranged in order of priority and significance. If the team considers an observation to be significant, it should be reported and addressed. There is no maximum number of observations in a report.
The format of the statewide close-out report should be standardized as much as possible. Using the new standard reporting form, the report should include the following in this recommended order:
- Title of Review
- Purpose of Review
- Scope of Review
- Identification of Team Members
- Executive Summary
- Observations, including "Best Practices," in order of significance - each observation should have a recommendation
- Conclusion (optional)
- Attachments (supporting documentation, example specifications, graphs, photos, etc.)
Upon completion of the initial draft statewide close-out report, the FHWA review team leader should ensure that the appropriate FHWA staff are provided an opportunity to review the report and provide comments. This should include all affected specialists, engineering team leaders, and the Assistant Division Administrator. The recommended procedure (internal to FHWA) is to place the initial draft report in the common directory of the FHWA LAN system and e-mail all engineers and affected specialists to inform them of the availability of the draft report for comment. A set time frame for review of the initial draft report of two weeks should be adequate. The STA team leader is expected to coordinate all necessary reviews at the STA Central Office.
The final draft version of the statewide close-out report should be sent by formal cover letter to the Director of Highways, with attention to the Deputy Director(s) and to the FHWA Division Administrator. The cover letter should be co-signed by the FHWA and STA review team leaders using the newly developed process review letterhead for the official transmittal. A copy of the cover letter and attached statewide close-out report should be sent to all affected Central Office Bureau Chiefs and to the process review team members.
Recommendations to improve the content of process review statewide close-out reports include the following:
- Use statistics whenever possible to justify observations (i.e., "In 7 out of 9 Districts, this procedure was followed... etc.")
- Recommendations should have substance and have specific actions or products ready for recommended implementation. Do not just hand over the issue to STA.
- Whenever possible, obtain information from other states on their processes, procedures, etc. These can be shared as "best practices" (sample specifications, procedures, forms, etc.).
- Use the term "Observation" rather than "Finding"
- Material to back up the observations should be included as attachments to the statewide report. Examples include charts, graphs, sample specifications, checklists, etc.
- The District reports do not need to be included as attachments to the statewide report.
Conducting the Statewide Close-out Meeting
Timely scheduling of statewide close-out meetings is critical for the proper dissemination of process review results. For example, it is important to close out all design-related process reviews by the end of August so that the results can be shared and discussed at the annual Program Development Engineers meeting in mid-September. Likewise, for construction-related process reviews, the close-out meeting should be held by the end of November. This will allow for the discussion of process review results at the annual Construction Engineer's meeting and the annual Materials Engineer's meetings scheduled in January and early February.
The STA review team leader should contact the appropriate Deputy Director(s) and the affected Bureau Chiefs to identify possible statewide close-out meeting dates. The FHWA team leader should likewise coordinate with the Division Administrator and the Assistant Division Administrator to determine possible meeting dates. One or two days before the statewide close-out meeting, the review team leaders should follow up with appropriate management as a reminder in order to assure attendance.
For those process review recommendations perceived to be highly controversial or considered a tough sell, it is recommended that a preliminary close-out meeting be held. This meeting should be held with STA Bureau Chiefs, Section Chiefs, and/or other appropriate STA staff, as well as FHWA Team Leaders/Specialists to agree on tentative resolutions. The need for a preliminary close-out meeting should be left to the discretion of the FHWA and STA review team leaders.
Close-out meetings should be as brief as possible and should not exceed two hours. Multimedia communication such as slides, photographs, videos, and computer generated presentations should be used whenever possible to increase the effectiveness of focusing attention on the observations and recommendations. Presentations which consist of a stand-alone reading of the observations and recommendations directly from the statewide close-out report are discouraged. In order to stress that these process reviews are joint efforts, presentation of observations should be done by the entire team, including both STA and FHWA team members.
At the end of the statewide close-out meeting, the FHWA and STA review team leaders should summarize the proposed resolutions made during the meeting. This is necessary to ensure that everyone in attendance has a complete understanding of what was agreed upon. It is also important that the review team leaders ensure that someone is assigned to take meeting minutes for the documentation of agreements. In the past, some close-out meetings were conducted and concluded without proper documentation, and later discussions resulted in confusion and uncertainty of the agreements reached at the meeting.
Within two weeks of the statewide close-out meeting, the STA review team leader should ensure that a formal written response is provided to the process review team who should document the resolutions to the process review recommendations. After this response is received, the review team should finalize the statewide report by inserting the resolutions after the appropriate recommendations. Resolutions may include immediate actions, action plans, proposed implementation dates, or other action items. The completed version of the final statewide report should be sent by formal cover letter (under dual signature) to the STA Director of Highways with attention to the Deputy Director(s). Copies should be sent to all affected Central Office Bureau Chiefs, all STA District Engineers, and to the FHWA Division Administrator. A copy of the final report should also be sent to the FHWA Resource Center for their information. An electronic copy of the final report should be placed in the FHWA LAN common directory with an e-mail notification of availability to all Division Office engineers and specialists.
Evaluation of Process Review Quality
The overall quality of the process review program is critical in order to gain wide acceptance, favorable recognition, and continued support from STA and FHWA upper management. Previous attempts have been made to quantify the "value added" from the process review program, but these efforts have had limited success. Currently, there is no standard format for assessing the quality of process reviews.
The quality assessment of the process review program should be enhanced with the establishment of FHWA and STA process review program coordinators. The program coordinators will facilitate in the continual attention on the progress of the process reviews and the follow-up activities from previous reviews. Through constant and direct involvement, the process review program coordinators should be able to easily identify the strengths and weaknesses of the entire program and also individual process reviews. The program coordinators should give guidance to the process review team leaders throughout the year. The program coordinators should conduct an annual program quality assessment as follows:
Approximately one month prior to the annual process review topic selection meeting, the program coordinators should meet with all the FHWA and STA team leaders of the ongoing process reviews. The discussions should focus on strengths, weaknesses, and successes of the ongoing process reviews. Many of the ongoing process reviews may not be entirely complete at this time, but the team leaders should have a good handle on the progress and direction of their review. Feedback and discussions between all process review team leaders will provide the benefit of learning from the successes and mistakes for future process reviews. The program coordinators should present to STA and FHWA management (at the annual process review topic selection meeting) any significant observations that would result in a major change to the current process review program.
This same meeting should be used to summarize the preliminary issues of the current reviews. The process review program coordinators should use the information gathered to determine if any of the ongoing reviews should be continued on or expanded into the next year's review program. If so, this need should be brought up at the annual topic selection meeting in September.
STA and FHWA management should also discuss process review quality at one or more of their regularly scheduled monthly "breakfast meetings." This item should at a minimum be discussed during the November "breakfast meeting" when purpose and scope statements have been developed for the following year's process reviews and when most of the current process reviews are nearing completion. The process review program coordinators should brief management on the status of current process reviews and provide purpose and scope statements for the following year's reviews prior to the meeting. It is essential for STA and FHWA management to discuss the purpose and scope of the new process reviews at this time to ensure that the process review teams are proceeding in the right direction.
Follow-up to Process Review Observations
A high-quality process review requires follow-up on observations to ensure that all resolutions to recommendations are implemented. Many times in the past, the resolutions reached at the statewide close-out meeting were not aggressively pursued and, ultimately, many actions were not implemented because of the lack of follow-up.
A key to ensuring adequate follow-up to review recommendations is the implementation of a comprehensive tracking system. Previously, the .......... Division used the Management Information Control System (MICS) which was useful to track the status of recommendations, but because of limited capability, restricted access, and not being user friendly, the use of the system has been essentially discontinued. A primary component of a new system should be the tracking of follow-up activities. If and when such a system is implemented, the process review program coordinator and the appropriate FHWA specialists and engineers should use this system to assist in following up on resolutions to ensure implementation.
In order to ensure adequate follow-up, it is necessary to establish clear action items with specified deadlines at the time of the statewide close-out meeting. Upon agreement of resolutions from the close-out meeting, the review team leaders should ensure that all follow-up items are documented and entered into the process review tracking system. The follow-up activities after this point should be assigned to the appropriate Division Office specialist. In cases where there is no specific specialist (i.e., plan quality, etc.) for the process review topic, the follow-up responsibility should be assigned by the FHWA process review program coordinator in consultation with FHWA management. The assigned individual should, at a minimum, perform follow-up activities at three months, six months, and one year after the statewide close-out meeting. Follow-up after one year should be left to the individual responsible for follow-up of that review. The tracking system should keep a process review status open until all the agreed activities have been addressed and/or accomplished.
Because of continual staff turnover, the FHWA process review program coordinator should monitor the follow-up/tracking system and establish contingency plans if a team leader or assigned specialist in charge of specific follow-up activities leaves the office or is reassigned to a different position. This contingency plan could require another team member from that particular review to be responsible for additional follow-up.
A common problem frequently identified was that many people are not informed of the proposed changes resulting from the process reviews. This is particularly true for District Office personnel. They are familiar with the review conducted in their District, but are often excluded in the distribution of all the final statewide reports. Communication of process review resolutions is extremely critical. The resolutions of process reviews should be discussed at appropriate STA annual meetings. This would include the annual Construction Engineer's Meeting, the Materials Engineer's Meeting, and the Program Development Engineer's Meeting. Process review resolutions should also be discussed at District Engineer/Bureau Chiefs Meetings, District Resident Engineers Meetings, .......... Road Builders Association Meetings, and other management meetings. For those reviews which have major issues and warrant special emphasis, members from the process review team should offer their services in visiting the Districts to discuss the review. For those observations which result in significant changes to specifications, policies, or procedures, the changes should be incorporated into STA's specific task training program, if applicable. Another good practice for ensuring that process review resolutions are disseminated is to provide copies of all construction-related process review reports to all Resident Engineers.
When changes are made at the Central Office level (i.e., specification changes, new policies, etc.) as a result of a process review, the FHWA Transportation Engineer should follow up at the District level to ensure they are incorporating the recommended changes. If the Districts are reluctant to make changes, reasons should be brought to the attention of the FHWA specialist and the appropriate Central Office staff for discussion.
It is also important to note that follow-up to positive observations is critical. This follow-up is necessary in order to ensure that "best practices" are shared and considered by others for adoption. Follow-up procedures for best practices would be similar to those provided in the discussion above.
A final step in the follow-up process is the development of an annual summary report of all the process reviews. This summary report should be produced by the FHWA and STA process review program coordinators and should be limited to two pages or less in length. It should concisely state the purpose and scope of the reviews along with the significant observations, recommendations, and the status of implementation of resolutions. Appropriate STA and FHWA staff should be involved in the review of this document. This annual report should be sent to the STA Director of Highways and to the FHWA Division Administrator. The report should be targeted for completion by January 15 of each year.
Attachments
1. Sample Statewide Close-out Report
(District Reports are Similar)
2. Sample Process Review Letterhead for Transmitting Process Review Correspondence
3. Process Review Flow Chart (not included)
STA/FHWA Joint Process Review Program
Statewide Summary Report
National Bridge Inspection Standards (NBIS) And Bridge Maintenance Review (1997)
Date Of Report: July 21, 1998
Purpose Of Review:
The purpose of this review was to assess the .......... Department of Transportation's (STA's) compliance with the requirements and procedures of 23 CFR 650, Subpart C - NBIS.
Scope Of Review:
This review focused on the NBIS programs in STA Districts 8 and 9. The District Bridge Maintenance Engineers and staff were interviewed to evaluate whether the District's procedures, policies, inspector's qualifications, and documentation were adequate to satisfy the NBIS requirements...
Review Personnel:
.................... FHWA Division Bridge Engineer
.................... STA Bureau of Bridges and Structures
.................... STA Bureau of Bridges and Structures
Executive Summary:
This review consisted of ..........
Observations And Recommendations:
Observation:
All of the Districts reviewed had ..........
Discussion:
The Code of Federal Regulations (CFR), 23 Highways - PART 650, Subpart C, contains language that ..........
Recommendation:
STA should revise ..........
Resolution:
Observation:
In two out of three Districts reviewed ..........
Discussion:
Recommendation:
Resolution:
Conclusion:
July 31, 1998
| M. .........., P.E. Director of Highways .......... Department of Transportation ........................................ ...................., .......... .......... |
M. .........., P.E. Division Administrator Federal Highway Administration .............................. .............................. |
| Attention: | M. .........., Deputy Director of Highways M. .........., Division Administrator |
Dear ....................:
Subject: Process Review QIT - Final Report
Enclosed for your information and action is the final report from the Process Review Quality Improvement Team. This report was written to establish clear and uniform guidelines for conducting process reviews in the State of .......... We would like to schedule a meeting with you and your staff to discuss this review. Meeting time and location can be established at a later date.
| Sincerely yours, ...................., FHWA Process Review Team Leader |
Sincerely yours, ...................., STA Process Review Team Leader |
Enclosure
cc: