Data Integration
The Arizona Experience

What Has ADOT Learned?

Organizational Lessons

• Data ownership and accountability play a key role in ADOT's data integration efforts. To this end, AIDW staff have worked to help operational units (e.g., planners and engineers) understand that they own the data they collect and are responsible for its integrity. IT staff are merely custodians of the data, and the data warehouse merely a tool with which to access the data.
  
• Strong partnerships between IT staff and agency practitioners are required for a successful agency-wide IT initiative. In some cases, IT efforts motivate reluctant practitioners to change. In other cases, the practitioners want to move forward quickly and need to be restrained by IT staff to insure consistency with an overall strategy.
  
• A strong mandate for a comprehensive data integration initiative from above is unlikely to happen. Bottom-up desire is usually strong but can be uncoordinated, particularly across divisions, and by itself cannot provide the impetus for moving forward. A critical success factor is a carefully blended mix. Management support is required to ensure that the appropriate tools and resources are available. Bottom-up support is required so that a regression does not occur when the current management leaves.

Process Lessons

• ADOT's success to date is largely due to its focus on delivery. AIDW staff strive to add a data source to the warehouse every three to four months, so they are not seen as only talking about what is possible. It is important to augment a vision and architecture with practical progress.
  
• When beginning a data integration initiative, there may be a strong urge to build the metadata (data about data) and data dictionary layer first. ADOT has learned that this approach does not bring practical value quickly and may cause people to lose interest. Metadata, though important, is a means and not an end. The identification and documentation of data items should be performed in parallel with development of the analytical tool.
  
• Another critical success factor is the ability to match the tools to the users. ADOT's vision is to use business intelligence tools already available in the private sector. However, these tools should not be implemented until they match ADOT's organizational maturity. For example, until the culture is transformed to be information-reliant and the skill set is upgraded to ask and answer business questions and what-ifs, business intelligence and data mining tools are extraneous. ADOT's approach has been to ensure that the proper platform is available to "plug and play" new tools as the organization becomes ready to use them. If an agency implements tools prematurely and turns them over to planners and engineers to use, the success of the overall data warehouse initiative may become incorrectly tied to the success of individual tools.
  
• A solid IT vision and architecture are important, because technology rapidly becomes obsolete, and it is burdensome and inefficient to rewrite code every few years. The IT vision needs to accommodate plug-and-play tools so that the agency does not have to reinvent the wheel. However, it is equally important to be flexible when pulling in data sources, because priorities often shift. Anticipating the next crisis and being ready with the data win every time.

Technical Lessons

• The traditional online analytical process model consists of dimensions and facts. Dimensions are what users slice and dice to answer business questions regarding the facts. This model works well in a sales analysis scenario in private industries. However, it does not work well in a DOT core business, where most data are descriptive (bridge name and type, project description and timeline, etc.) rather than quantitative or additive. For this reason, ADOT uses a mixture of relational (ROLAP) and multidimensional (MOLAP) online analytical processing models.
  
• Data warehousing and GIS remain two separate worlds in the IT field. When staff with different backgrounds talk about data warehousing, they mean different things. Data warehousing is database-intensive. GIS traditionally is flat-file-based and represents one dimension (geography) in answering business questions. It is only recently that the GIS environment has moved towards the spatial database model, and it will take some time before these two areas converge.
  
• One key data warehouse principle is not to alter the original data from the source. At ADOT, data "transformation" is limited to little more than geo-coding—cleansing the geographical information (route, milepost, offset). Much of the data that are not validated at the source end up in "unknown" buckets. ADOT has found that it is critical that data ownership be clear and to build processes that provide feedback on data anomalies to the data owners. It also is critical to create buy-in from data owners so that they understand the importance of changing their systems to validate data at entry.