United States Department of Transportation - Federal Highway Administration FHWA Home Feedback
   CALENDAR OF EVENTS divider TRAINING divider TECHNOLOGY DEPLOYMENT divider TECHNICAL ASSISTANCE
   Home   divider  end of menu

Photo: We the People

Civil Rights Home
space image
Civil Rights Programs - Good, Best, and Innovative Practices
space image
Civil Rights Reports and Other Information
space image
Civil Rights Training
space image
Civil Rights E-Learning and Technical Assistance
space image
Civil Rights Calendar of Events
space image
Civil Rights Links
space image
 

FHWA Resource Center

What's New in Civil Rights

Data Collection and Program Evaluation

Click here to view the PowerPoint Presentation

Slide - 1

…why should I care?

Slide - 2

Data Collection and Program Evaluation (Analysis)

Photograph of a construction work zone. A "T" intersection is visible. Several orange temporary traffic control barrels have been set up in a temporary traffic pattern with signage in the forground that includes: a stop sign, a no left turn, a no right turn and no thoroughfare, wrong way, do not enter and road closed signage.

Slide two has a photograph of a construction work zone. A "T" intersection is visible. Several orange temporary traffic control barrels have been set up in a temporary traffic pattern with signage in the forground that includes: a stop sign, a no left turn, a no right turn and no thoroughfare, wrong way, do not enter and road closed signage.

…can be confusing

Slide - 3

Objectives

• Identify the importance of data collection and program evaluation.
• Explain the basic structure of an evaluation plan or strategy.
• Design a specific program evaluation plan to
– Identify specific data that must be collected,
– Assess the implementation and effectiveness of various civil rights programs, and
– Ensure State DOT management is aware of issues, needs and potential problems associated with various civil rights programs.

Slide - 4

Why should I care?

• Reliable information about the impact of FHWA programs (program evaluation) is essential to ensuring non-discrimination.
• The relationship between FHWA programs and varying social factors have wide-ranging implications for any community, municipality,county, or state.

Slide - 5

Why is it important?

• Data collection and program evaluation (analysis) can:
– Help identify and prioritize problem areas,
– Initiate and evaluate the effectiveness of policies and programs to ensure non-discrimination,
– Assess the relationship between specific programs and beneficiaries to develop non-discrimination strategies.

Slide - 6

Importance (continued)

– Draw agency attention and resources to resolve issues and problems.
– Provide justification for certain programs or illustrate a need for new programs.
– Help justify continued or additional funding.
– Communicate the importance of the goals, solutions, strategies, or programs.

 

Slide - 7

“Program evaluation is like a bunch of feathers because it lifts you up to get a bird's eye view so you can see the whole picture” (unknown).

 

Slide - 8

Current FHWA Data Collection

• How do you collect statistical data?
• What do you do with it?
• Project-based or Program-based?
• Benefits vs. Burdens analysis?
• “Letter of the law” or “Spirit of the law”?

 

Each State DOT must “develop procedures for the collection of statistical data (race, color, religion, sex, and national origin) of participants in, and beneficiaries of State highway programs, i.e., relocatees, impacted citizens and affected communities.” (23 CFR 200.9(b)(4))

 

Slide - 9

Steps to look at the whole picture

• Step 1: Focus the Evaluation
• Step 2: Identify Barriers to Evaluation
• Step 3: Methods of Data Collection
• Step 4: Organizing, Analyzing, Interpreting, Summarizing and Reporting Results
• Step 5: Develop the Evaluation Plan

 

Slide - 10

Step 1: Focus the Evaluation

• What will be evaluated?
• What is the purpose for this evaluation?
• Who will be affected by or involved in the evaluation?
• What are the critical questions the evaluation must address?

Slide - 11

Step 1: Focus Example (Right-of-Way)

• What will be evaluated?
– Real estate appraisals (project-based or program-based)

 

• What is the purpose of the evaluation?
– To ensure real estate appraisals are equitable without regard to race, color, or national origin.

 

• Who will be affected by or involved in the evaluation?
– Certified Appraiser - Relocation Agents
– Review Appraisers - Property Management
– State DOT Appraisers - State DOT Leadership
– Acquisition Agents - FHWA Division/Headquarters

 

Slide - 12

Step 1: Focus Example (Right-of-Way)

• What are the critical questions the evaluation must address?
– Was the resulting appraisal amount fair and equitable?
– Are the land unit values consistent throughout the project area?
– Were relocatees required to move to a similar socio-demographic region?
– Were relocatees provided relocation advisory services to find replacement housing (i.e., transportation to/from available housing).

Slide - 13

Step 2: Identify Barriers

• Factors that may influence the evaluation
• Implications for the evaluation
• Strategies to overcome barriers

Evaluation is like an acorn because it must overcome adverse conditions to grow into a tree” (unknown).

Slide - 14

Step 2: Identify Barriers

• Organizational Politics:
– Is there support for the evaluation?
– Are there opponents to the evaluation?

 

• Project Leadership:
– Who has control over the program?
– How does the evaluation fit their goals?

 

• Professional Influences:
– Are professional groups interested?
– Are the supportive of an evaluation?

Slide - 15

Step 2: Identify Barriers

• History:
– What has been the tradition or history of self-evaluation of programs?
– Have evaluations been conducted before?

 

• Organizational Setting:
– Does the program fit into a larger organizational network? Where or how?
– What kind of information could jeopardize the program?

 

Slide - 16

Step 2: Identify Barriers

• Economics:
– Is fiscal support for the program and evaluation secure?

 

• Interpersonal Patterns:
– How much interagency or interpersonal conflict is likely to occur as a result?
– Could the evaluation be controversial to internal or external audiences?

 

Slide - 17

Step 2: Identify Barriers

• Legal Guidelines:
– Will legal restrictions limit collection of desired information/data?
– Will the project be affected by pending legislation?

 

• Resources:
– Are resources (human, financial, time, etc.) available to support the evaluation?

 

Slide - 18

Step 3: Data Collection Methods

• What type of data should be collected?
• What methods should be used to collect data?
• How much information should you collect?
• Questionnaires
• Validity and Reliability

 

Slide - 19

Step 3: Types of Data

• Descriptive information can include:
– Characteristics of the program.
– Demographic information on program participants, beneficiaries, etc.
– Actual benefits paid or realized by beneficiaries.
– Specific results of a program or service.

 

Slide - 20

Step 3: Types of Data

• Judgmental information can include:
– Opinions from consultants.
– Beneficiaries beliefs and values.
– Agency personnel's interpretation of laws.
– Stakeholders perceived priorities.
– Interpretations of policy, procedures and guidelines.

Slide - 21

Step 3: Data Collection Methods

• Opinion Surveys: as assessment of how a person or group feels about a particular issue.
• Questionnaire: a group of questions that people respond to verbally or in writing.
• Case Studies: experiences and characteristics of selected persons involved with a program.
• Individual interviews: individual's responses, opinions, or views.

Slide - 22

Step 3: Data Collection Methods

• Group Interviews: small groups' responses, opinions, and views.
• Records: information from records, files, or receipts.
• Advisory, Advocate Teams: ideas and viewpoints of selected persons.

 

Slide - 23

Step 3: Choosing a Method

• Availability: information may already be available to you that can help answer some questions. Review information in prior
records, reports and summaries.
• Interruption Potential: the more disruptive an evaluation is to the routine of the program, the more likely that it will be unreliable.
• Protocol Needs: you may need to obtain appropriate permission to collect information.

Slide - 24

Step 3: Choosing a Method

• Reactivity: you do not want “how” you ask something to alter the response you get.
• Bias: to be prejudiced in opinion or judgment.
• Reliability: will people interpret your questions the same way each time?
• Validity: Will the collection methods produce information that actually measures what you intend to be measured?

 

Slide - 25

Step 3: How Much Data?

• Sampling: a portion of subjects in order to learn something about the entire population without having to measure the whole group.
– Random: each individual in the group has an equal chance of being chosen for the sample.
– Purposive: a sample that will represent specific viewpoints.

Slide - 26

Step 3: When Choosing a Method, Consider…

• Should you use a sample of a population or a census (an entire population, such as all people living in a project area)?
• Should you use a random or purposive sample?
• How large a sample size do you need?
• Is your sample likely to be biased?

 

Slide - 27

Step 3: Questionnaires

• Questionnaires are a widely used method of collecting information. They can be a cost effective way to reach a large number of people or a
geographically diverse group.

 

Slide - 28

Step 3: Questionnaires

• Questionnaires should include the following key elements:
– Cover letter: include the purpose of the study, why and how participants were selected, etc.
– Introduction: a short recap of the information included in the cover letter.
– Instructions: clear and concise.

 

Slide - 29

Step 3: Questionnaires

• Questionnaires should include the following key elements:
– Grouping questions: group questions with similar topics together in a logical flow.
– Demographic questions: age, gender, ethnicity, etc.
– Thank you: thank the respondent for completing the questionnaire.
– Disposition: what to do with the questionnaire when complete.

 

Slide - 30

Step 3: Validity

• “Does the questionnaire measure what you want it to measure?” A questionnaire that is valid for a specific situation or audience may
not be valid in a different situation or for a different audience.
• Have a panel of experts (professionals familiar with the program) review the questionnaire and provide feedback.
• Test the questionnaire on a similar population.

 

Slide - 31

Step 3: Reliability

• Does the questionnaire consistently give the same results with the same group of people under the same condition?
• Can be shown when using a pilot sample with a questionnaire administered repeatedly over time.
• Are answers repeated over time?
• Revise as needed.

Slide - 32

Step 4: Organizing, Analyzing,
Interpreting, Summarizing and Reporting

• Organize Evaluation Data
• Analyze
• Quantitative Data Analysis
• Qualitative Data Analysis
• Interpret (summarize)
• Report Results

 

Slide - 33

Step 4: Organize Data

• Develop a system to organize your data (spreadsheet).
• As data are received, check to be sure it is complete.
• Set up protocol for who does/does not have access to the data.
• Track all data. Ensure data are not lost or overlooked as analysis and summarizing begin.

Slide - 34

Step 4: Quantitative Data

• Quantitative Data Analysis: information that contains numerical data.
– N: The total number of participants in the sample.
– Mean: The average score of the sample.
– Median: The score halfway between the high and low score.
– Mode: The response given the most.
– Standard Deviation: The range from the mean in which 68% of the responses are found.
– Frequency: How often a particular response was given.

Slide - 35

depicts a bell shaped curve with a normal distribution or Gaussian distribution

Slide number 35 depicts a bell shaped curve with a normal distribution or Gaussian distribution. This is a continuous probability distribution that describes data that cluster around a mean or average. The graph of the associated probability density function is bell-shaped, with a peak at the mean, and is known as the Gaussian function or bell curve. Step 4 is the Standard deviation. The mean score represents points of view that are the same as others. Each deviation to the left represents more. Each deviation to the right represents
less.

Step 4: Standard Deviation

Slide - 36

Step 4: Quantitative Analysis Example

• On a scale of 1 to 5, where 1=poor and
5=excellent, how would you rate the overall
quality of the workshop?
– Answers (10 people): 4,5,2,4,3,4,3,3,5,4
– Mean: 3.7 (the total divided by 10)
– Median: 3.5 (halfway between the lowest score of
2 and the highest score of 5)
– Mode: 4 (the score repeated the most)
– Standard Deviation: 0.95 (68% of the answers were between 2.75 and 4.65).
– Frequency: 1=0, 2=1, 3=3, 4=4, 5=2.

 

Slide - 37
Step 4: Qualitative Data

• Qualitative data analysis is the process of systematically searching and arranging the interview transcripts, field notes, and other materials to increase your own understanding and enable you to present what you have discovered.
(Bodgan and Biklen(1992)).

 

Slide - 38

Step 4: Qualitative Data

• Data resulting from interviews, focus groups, open-ended questions, or case studies.
• Develop common elements, themes, topics, sentiments, etc. from the qualitative data.

Slide - 39

Step 4: Interpret

• Identify trends, commonalities and testimony that will help answer the critical evaluation questions that were generated in Step 1.
• To be useful, the data must be interpreted so that the stakeholders will understand the results and know how to use them.

Slide - 40

Step 4: Interpret Example

• At the end of a public meeting to discuss a future construction project, persons were asked to rate their satisfaction with key components of the project on a scale of 1 to 5.
• 1=not satisfied and 5=very satisfied.

Slide - 41

Slide number 41 depicts a chart with survey responses, this is the (mean). This chart identifies: Project Location, Projected Cost,
Length Of Construction, Project Schedule & Detour Route.

 

Step 4: Interpret Example

• A total of 75 people responded.
– 40 Hispanic/Latino
– 34 White (non-Hispanic)
– 29 African-American
– 15 Disabled

Survey Responses (mean)

Project Location

3.15

Projected Cost

4.21

Length of
Construction

2.80

Project Schedule

2.55

Detour Route

3.43

Slide - 42

Slide number 42 depicts four charts: White Respondents, Hispanic Respondents, African-American Respondents and Disabled
Respondents.

Step 4: Interpret Example

White Respondents (mean)

Project Location

4.85

Projected Cost

4.10

Length of Construction

2.90

Project Schedule

1.85

Detour Route

1.44

 

Hispanic Respondents (mean)

Project Location

2.61

Projected Cost

4.48

Length of Construction

3.12

Project Schedule

2.98

Detour Route

3.85

 

African-American Respondents (mean)

Project Location

1.98

Projected Cost

3.99

Length of Construction

2.49

Project Schedule

1.97

Detour Route

4.10

 

Disabled Respondents (mean)

Project Location

3.16

Projected Cost

4.25

Length of Construction

2.68

Project Schedule

3.41

Detour Route

4.34

 

What does it all mean?

 

Slide - 43

Step 4: Reporting

• Serves as a basis for further program development and improvement.
• Provides support for continuing or expanding the program.
• Serves as a basis for public relations and promoting future programs.
• Identifies implications of the results, how they can be used, and recommendations for the overall program.

Slide - 44

Step 5: Evaluation Plan

• Who should lead the evaluation?
• How should tasks and responsibilities be formalized, organized and scheduled?
• What problems are expected?

Slide - 45

Step 5: Who Should Lead?

• One person or a team?
• Leader's responsibilities may include:
– Designing the evaluation
– Constructing the instrument
– Collecting data
– Analyzing data
– Writing/delivering reports
– Managing and interacting with personnel

Slide - 46

Step 5: Organizing Tasks

• When a project team, evaluation leader, and the stakeholders meet and ask “What must be done, when and by whom?” the evaluation management
plan begins to take shape. These tasks should be put into some logical organizational structure.

Slide - 47

Step 5: Organizing Tasks

• A management plan should include:
– Evaluation design and a general plan specifying what must be done.
– When each activity needs to be conducted and completed.
– Responsible parties for each activity.
– Available resources.
– Ability to revise or refine the plan as time passes.
– Contingency plans for problems that may arise.

Slide - 48

Step 5: Expected Problems

• Murphy's Law
• Monitor the evaluation's progress
– The earlier you detect a problem, the better able you will be to resolve and continue.
• Develop a contingency plan.

Slide - 49

Conclusion

• Program Evaluation including data collection and analysis can provide an overall focus to the civil rights programs.
• Requires a paradigm shift from daily implementation to program efficiency.
• An effective evaluation plan will help identify the varying types of data that need to be collected in the programs.

Slide - 50

Thank you

• Questions?
• Contact information:

Darren Kaihlanen

FHWA Oklahoma Division

darren.kaihlanen@dot.gov

(405) 254-3312 (voice)

(405) 254-3302 (fax)

FHWA Resource Center logo
yellow circle
CONTACT US
staff / phones

Technical Service Teams
Air Quality

Civil Rights

Construction & Project Mgmt

Environment

Finance Services

Geotechnical

Hydraulics

Operations

Pavement & Materials

Planning

Safety & Design

Structures

FHWA Logo

FHWA Home | Feedback
United States Department of Transportation · Federal Highway Administration