Contents | Next >

Evaluation Framework

A detailed review of the fuel tax attribution methodology was conducted using an evaluation framework that identified four goal areas: legacy system performance, data process quality, risk management, and institutional issues as they relate to the fuel tax attribution process.

• The legacy system performance goal area evaluates the ability, efficiency, and reliability of the fuel tax attribution system to perform its intended functions of accepting, analyzing data, and delivering accurate and consistent state fuel tax attributions.
• The data processing quality goal area evaluates the quality and reliability of data analysis procedures applied in processing of motor fuel tax data from the states. This goal area evaluates the business rules (rules-of-thumb, assumptions, and formulas used in the estimation models) and how these rules affect the results of the attribution process.
• The risk management goal area relates to the policies and practices that ensure consistency and reliability in data quality, acquisition, analysis, and reporting Highway Trust Fund attributions.
• The institutional issues goal area identifies any institutional issues related to data acquisition and handling as well as attribution of fuel tax revenues. This includes institutional arrangements that affect the fuel tax data processing directly or indirectly.

Summary of Findings

The following are major findings of the evaluation presented by goal area.

System Performance

• The use of the Smart Input Tool greatly enhances the quality of fuel tax data by reducing the chances of input errors, improving the efficiency of the attribution process, and saving cost and time associated with data entry at both state and federal levels. Also, by improving the quality of data submitted by the states, the time and effort required to reconcile data discrepancies is reduced. The savings in turn improve efficiency of the fuel tax attribution process.
• The analytical processes, including the assumptions, business rules, and estimation models appear to be consistently applied and yield consistent Highway Trust Fund attributions. These findings suggest that the system is reliable in generating fair and consistent attributions based on the number of gallons reported by the states.
• Although there was insufficient information to draw any definitive conclusions, the available information indicates that the oversight review will potentially improve the quality of data (in terms of accuracy, completeness, and timeliness) submitted to FHWA and enhance efficiency of the fuel tax attribution system as a whole. The oversight review process will potentially save the time and effort needed to reconcile state and FHWA data discrepancies and ultimately reduce reliance on estimation models for imputing missing data.

Data Process Quality

• The review and documentation of the entire fuel tax data analysis process includes detailed descriptions of all steps, assumptions, business rules, and estimation models. The documentation includes flow charts and figures that allow one to understand the sequences of calculations and flow of data elements and outputs.
• The statistical analysis of historical fuel tax attribution data showed that the analytical procedures generally generate consistent and reliable results of HTF attributions that are highly dependent on the fuel consumption information reported. The results support the hypotheses that the business rules and estimation models are consistently applied from year to year and that the data provided by the states are of acceptable quality.
• A statistical measure for assessing reliability of the fuel tax attribution system was developed. This measure allows outliers to be identified and investigated if necessary.
• Some variables used in the estimation models are outdated and/or no longer relevant. These models need to be reviewed and updated with more current data.

Risk Management

• The implementation of the Smart Input Tool is a major improvement directed at reducing the risks associated with states submitting incomplete and inaccurate fuel tax data. The self-error checking feature helps control this risk to a large extent.
• The "Highway Community Exchange" on the Smart Input Tool community website provides a feedback mechanism for FHWA that allows users at the state level to seek clarifications and guidance in using the tool as well as the opportunity to offer suggestions for improvements. This feature is consistent with standard practices of risk management.
• The results of the statistical analysis show that analytical procedures appear to be reliable and consistently applied and that the amount of fuel consumption is highly correlated with the amount of funds attributed to the states. This implies that the fund attributions are reasonably fair and consistent. Departures from the expected results can be easily identified and investigated in conformity with standard risk management procedures.

Institutional Issues

No major institutional issues have been identified that could significantly impact the fuel tax attribution process. It was noted that the FHWA is implementing a number of changes, some of which are institutional in nature, to help improve the fuel tax attribution process. Some institutional arrangements may be necessary to facilitate the operations of the oversight reviews. These institutional changes are expected to promote the efficiency of the fuel tax attribution process rather than impeding the process.

Conclusions

The improvements being implemented by FHWA as part of the continuous improvement model have significant positive impacts on the fuel tax attribution process. These impacts include standardizing and streamlining the data reporting by the states to reduce the risk associated with poor quality data. These improvements greatly enhance the quality and reliability of the entire fuel tax attribution process.

It can be concluded from the evaluation results that the analytical procedures are fairly reliable in generating consistent HTF attributions to the states.

Recommendations

The following are some recommendations to help improve the fuel tax data attribution process.

Conduct periodic evaluations. This evaluation was conducted at a time when the full effects of some of changes implemented had not been completely realized (e.g., Smart Input Tool). In addition, and some improvements had not been fully implemented (e.g., oversight). Thus, the initial indications of the impacts may not be the same in the long term. It is therefore recommended that, as part of the continuous improvement model, periodic evaluations be conducted to help assess the success of the improvements, identify shortcomings and design corrective measures.

Convert the spreadsheets into a simple model. The analysis procedures are currently in a number of spreadsheets that are linked. While the use of spreadsheets provides transparency in the analytical procedure, it can sometimes be cumbersome to execute.

Action items for further improvements: It is recommended that several actions identified to further improve the fuel tax attribution process be reviewed and implemented. These include:

• Development of a structured stand-alone Instructions Manual devoted to providing specific guidance on reporting fuel tax data.
• Introduction of data assessment process tool that enables states to assess the quality of their own prior to submission to FHWA.
• Use Information Exchange website to address issues related to fuel tax data and also serve as a feedback mechanism.
• Establishment of mentoring program to assist states that experience consistent data quality problems.
• Review, enhance, and update of estimation models. Data and assumptions used in developing some of the original models need to be reviewed to determine their validity.
• Enhancement of analytical framework including EVAL, GTA, MF-20 analyses to adapt to improvements in quality of fuel tax data and estimation models.
• Development of a framework for systematic monitoring and evaluation of FHWA's continuous improvement programs.