State of The Practice on Data Access, Sharing, and Integration

CHAPTER 3. EXAMPLE DATA ENVIRONMENTS

UNIVERSITY OF MARYLAND’S REGIONAL INTEGRATED TRANSPORTATION INFORMATION SYSTEM

Effective Practices

The Regional Integrated Transportation Information System (RITIS) is an automated data fusion and dissemination system for transportation-related data. RITIS was originally developed for the National Capital Region. As of 2015, the RITIS platform includes data from 26 State transportation departments and hundreds of agencies. Planning for RITIS began in 2002 with a grant from the Federal Government, and the system was developed in 2006 by the Center for Advanced Transportation Technology (CATT) Laboratory of the University of Maryland. Agency partners originally included the Metropolitan Washington Council of Governments, Maryland Department of Transportation (DOT), Virginia Department of Transportation (VDOT), District of Columbia DOT, Montgomery County Maryland Traffic Management Center, and the Washington Metropolitan Area Transit Authority. Current partners and the 5,000+ users of the system include the following:

• Transportation departments (Federal, State, and local).
  
  
• Transit providers.
  
  
• MPOs.
  
  
• Emergency management agencies.
  
  
• Federal Emergency Management Agency.
  
  
• U.S. Army, Air Force, Navy, and Coast Guard.
  
  
• U.S. Northern Command.
  
  
• U.S. Secret Service.
  
  
• U.S. Capitol & Park Police.
  
  
• Fire and rescue.
  
  
• Law enforcement (State and local).
  
  
• U.S. Joint Forces Headquarters.
  
  
• National Security Agency.
  
  
• U.S. Office of Personnel Management.
  
  
• Third-party travel information providers.
  
  
• University researchers.
  
  
• Consultants working on projects for the transportation departments.
  
  
• Social Security Administration.
  
  
• Pentagon Force Protection Agency.
  
  
• Others.

RITIS has the following two primary capabilities that support data integration and sharing:

• Real-time information exchange: This component collects, filters, standardizes, and disseminates real-time data on transportation conditions in the region. Participating agencies maintain ownership and operation of its field devices for collecting real-time data feeds. Each agency sends its data to RITIS, which translates the disparate data feeds, linear reference systems, and communications protocols into industrywide standardized formats. Data are translated into standardized packets that are transmitted over the Internet to all participating agencies. Once they are received, the data can be integrated back into the receiving agencies’ traffic and incident management systems. Figure 14 shows an overview of the real-time information exchange. For agencies that do not have existing systems or are otherwise not interested in reintegrating data into their native systems, the RITIS platform also includes a Web site that allows users to view and interact with all of the real-time data.
  
  
• Archival and analysis of transportation-related data: RITIS archives all the real-time data feeds and makes them available to participating agencies to support operations, planning, research and development, and performance measurement generation. The archive component includes access to tools that allow users to download historical RITIS data, run reports, assess performance measures, and conduct analysis. These tools allow users to identify accident hot spots, prioritize projects, analyze congestion, perform after action reviews, and evaluate the effectiveness of transportation operations strategies. Data within the archive can also be downloaded or exported so that users can perform their own independent analysis. All data within RITIS are archived indefinitely.⁽²⁸⁾

©University of Maryland CATT Laboratory
Figure 14. Diagram. Data feeds to RITIS, which standardizes the data and presents them to various user groups.⁽²⁹⁾

All data within RITIS are made available to users through a series of tools—some meant for real-time data visualization and analysis and others meant for performance measurement, reporting, and research purposes (see figure 15).

©University of Maryland CATT Laboratory
Figure 15. Diagram. Some of the tools available through RITIS for real-time and historic data visualization, analysis, and sharing.⁽²⁹⁾

The real-time data within RITIS includes traffic volumes and speeds (from probes and sensors), transit, incidents, special events, emergency and planned lane and road closures, recommended detours, CCTV images and device operational status, weather from the Road Weather Information System (RWIS), weather radar and alerts from National Weather Service, DMSs and highway advisory radio messages, AVL, transit service disruptions, managed lane status, evacuation documents, first-responder radio feeds, computer-aided dispatch, and other types of operations data. RITIS also stores static information about roadways and transit service, such as number of lanes, speed limit, location of DMSs, signal timing plans, and transit schedules, routes and stops. As of early 2015, the RITIS platform was processing and archiving approximately 5billion rows of data every day. In the near future, RITIS is expected to be expanded to include the following data elements:

• New York State incident/construction data.
  
  
• Virginia statewide data (not just Northern Virginia).
  
  
• CCTV.
  
  
• Flight-tracking data.
  
  
• Faster, more user-friendly mapping and filtering.
  
  
• North Carolina data.
  
  
• More transit data (real-time bus and rail tracking).
  
  
• Signal systems data.
  
  
• Evacuation routes.

RITIS uses a redundant data architecture in which duplicate data centers are deployed at two or more physical locations. When a user requests data, the request is distributed over several servers residing in one or more locations. The process is automatic and transparent to RITIS users. The data architecture structure is shown in figure 16.

©University of Maryland CATT Laboratory
Figure 16. Diagram. RITIS data architecture.⁽³⁰⁾

RITIS is accessible through a secure Web portal that allows authorized users to view real-time, interactive maps, lists, dashboards, and a variety of visualization and situational awareness tools. For example, users can interact with live events, incidents, weather, sensors, and other data sources and devices through interactive maps, lists, and graphics. Users can apply filters, access contact information, and communicate with other users via chat rooms or integrated Web meeting tools. One interactive graphic is the incident timeline (figure 17), which shows who is at an accident scene, who has been notified, and who has departed. It also shows real-time and historical lane status, traffic queue buildups, CCTV camera feeds, DMS messages, and communication logs between managers. Additional tools are provided for users to query, analyze, and derive performance measures from the RITIS archive.⁽²⁸⁾

©University of Maryland CATT Laboratory
Figure 17. Screenshot. RITIS demonstration snapshot of an incident timeline.⁽²⁸⁾

Success Factors

RITIS is a successful and growing system that has a large collection of meaningful regional transportation information to share in a single location.⁽³¹⁾ By consolidating, disseminating, and archiving transportation-related data from various agencies in the Washington, DC, area, use of RITIS has resulted in the following successful outcomes:

• Regional data standardization, integration, and sharing to allow a comprehensive view of the region’s transit and transportation network in Virginia, Maryland, and the District of Columbia.
  
  
• Demonstration of data fusion and standardization and how this information can be used to support data collection, regional transportation systems management, regional traveler information dissemination, and systems evaluation.
  
  
• Adoption by the I-95 Corridor Coalition partners to power the transportation information applications. In addition to its information sharing features (congestion, incidents, speed, etc.), RITIS provides data visualization and analysis tools that help an agency identify problem areas. RITIS provides information access to a wide range of transportation organizations.
  
  
• One consolidated location for transportation systems data, information gathering, etc., which allows the agencies to better report their achievements to decisionmakers and the public.
  
  
• Overcoming of several obstacles common to enterprise projects, including convincing data owners to share information with project managers and each other; building easy-to-use tools for a wide audience to analyze and visualize data; and expanding the scale of the project from the Washington Beltway to the country. A major success factor in breaking down data silos and getting agencies to share their data has been demonstrating how valuable RITIS tools can be to them. During a major snow event in February 2010, RITIS had more than 600 visitors and logged 1.2 million hits from first responders, management, decisionmakers, politicians, and others seeking to find and share information.

Gaps and Lessons Learned

The following gaps and lessons learned have been documented:

• One of the benefits of RITIS is that it allows cross-agency and cross-modal coordination. Policies relating to this coordination must be established to avoid conflicting data, to support appropriate incident response, and to best manage incidents that cross jurisdictional and modal boundaries.
  
  
• Each agency’s existing system required some modification to view RITIS information within their native systems. This has required close coordination between RITIS developers and agency information technology (IT) staff to identify changes to data elements and systems to accommodate RITIS’ data fusion scheme. Agencies who have not wanted to address reintegration challenges have decided to use the RITIS Web site.
  
  
• Convincing agencies that there is benefit to sharing their data and making them visible to others outside of their own organization was a significant challenge. There seems to be a culture of fear within many organizations: fear of being judged and/or prosecuted for errors in their data or for perceptions of poor performance. RITIS developers had to expend a great deal of resources on education of agencies on their security protocols to convince them that their data would be safe. RITIS developers also found that the visualization tools were helpful in convincing agencies to want to share their information with the RITIS platform. Agencies wanted access to the data visualization tools, and they could not get access unless they decided to share their data. These and other issues have been documented in the National Cooperative Highway Research Program (NCHRP) report NCHRP Synthesis 460: Sharing Operations Data Among Agencies.⁽³²⁾
  
  
• Some agencies have created complex legal documents that have to be signed before allowing access to the data. Out-of-state governmental agencies (such as other State and local transportation departments) often have concerns about signing these legal agreements because they may require the State to agree to the laws of another State.

Applicability to the VDA Framework

RITIS provides important lessons for the development of the VDA Framework, particularly in the area of interagency coordination and data sharing.

For additional information on the RITIS platform, see http://tinyurl.com/lu47c8s.⁽³³⁾

CALTRANS’ PEMS

Effective Practices

Caltrans’ Freeway PeMS is a real-time archived data management system (ADMS) for transportation data from throughout the State. It collects raw detector data in real time, stores and processes these data, and provides a number of Web pages that offer analysis of the performance of the freeway system. Caltrans requires registration to the Web site, and all content can be accessed with the exception of some real-time maps and some pages that are only available to Caltrans employees (i.e., detailed Traffic Accident Surveillance and Analysis System (TASAS) accident analysis). Figure 18 and figure 19 present the PeMS welcome screen and the PeMS home screen, respectively.

©Caltrans
LG = 55–59 mi/h.
Y = 50–54 mi/h.
DY = 45–49 mi/h.
MO = 40–44 mi/h.
O = 35–39 mi/h.
Figure 18. Screenshot. PeMS welcome screen.⁽³⁴⁾

© Caltrans
Figure 19. Screenshot. PeMS main screen.⁽³⁴⁾

Raw freeway detector data are sent from each Caltrans district over the Caltrans wide area network, and CHP incident data are collected by scraping the highway patrol Web site and TASAS incident data from Caltrans directly. Other sources of data include FasTrak® toll tag readers, cities, transit agencies, and third-party data providers.

Some of the information available in PeMS (i.e., plots, tabular, and/or mapped) includes the following:

• Inventory of the freeways and detectors that are in the geographical segment.
  
  
• Inventory of routes, corridors, managed facilities, field elements, arterials, transit agencies, and FSP beats;
  
  
• Performance measures (actual and predicted), including VMT, vehicle hours traveled (VHT), delay, lost productivity (congested lane mile hours), travel times, Q (VMT/VHT), Travel Time Index, congestion pie, bottlenecks, annual average daily traffic (AADT), mobile 6 modeling (measured VMT versus the measured speed), level of service, etc.
  
  
• Detector health.
  
  
• Lane closures.
  
  
• CHP incidents.
  
  
• Photolog images.

Some specific operations planning tools and methods PeMS incorporates include the following:

• Corridor module: Figure 20 shows PeMS data through maps and other graphical representations of all Corridor System Management Plan (CSMP) routes.
  
  
• Departure Time Series: This function can be used to easily generate before and after comparisons. Figure 21 shows an example from August 2007, when Caltrans completed the addition of a lane over the bridge on I-15 at Lake Hodges in district 11. The plot demonstrates a dramatic, sustained drop in travel time.
  
  
• Lane Requirement Chart: This tool allows users to plan work zones by identifying lane requirements by time of day and days of the week based on historical traffic data. Figure 22 presents a sample of this capability.

©Caltrans
Figure 20. Screenshot. PeMS corridor module Alameda I-80 daily animation.⁽³⁴⁾

©Caltrans
Figure 21. Screenshot. PeMS departure time series on I-15 north southbound.⁽³⁴⁾

©Caltrans
Figure 22. Screenshot. PeMS lane requirement chart modeling on I-110 north in district7.⁽³⁴⁾

The data fidelity section in PeMS gives users the ability to see how many points in the dataset were imputed. It also allows users to view the data to see exactly how the imputation decisions were made by the data processing programs.

PeMS includes a data clearinghouse, providing a single access point for downloading PeMS datasets. Users can quickly locate data by district, month, and format. PeMS exports data in a variety of file formats, including HPMS and comma-delimited American Standard Code for Information Interchange (ASCII) text, each with an associated list of data sets that it supports. Figure 23 presents a view of the data clearinghouse.

Success Factors

PeMS had the following success factors:

• Users can view detector health and data fidelity, how many points in the dataset were imputed, and how the imputation decisions were made. This allows the user to assess confidence in the results.
  
  
• PeMS has extensive documentation on the system calculations and common analyses as well as a glossary of terms.
  
  
• Readily available performance measure analysis capabilities and specialized tools
  (e.g., CSMP corridor data, lane requirement chart, before/after comparisons) provide information in a format that is readily usable, understandable, and useful to transportation professionals.

©Caltrans
Figure 23. Screenshot. PeMS data clearinghouse.⁽³⁴⁾

Gaps and Lessons Learned

The following are some of the specific gaps and PeMS challenges:

• Because of processing limitations, the amount of data that can be queried for some Web pages is restricted to limit the load on the server and time required to generate the table orplot.
  
  
• Incident location information may not be at the exact location indicated in PeMS because the staff needs to translate from text on the CHP Web site to an actual geographical location using a relatively basic algorithm.
  
  
• If a user specifies a time range that does not fall directly on a weekly boundary (Sunday to Saturday), the start and end points will be wrong. To fix this, users must make sure the starting point falls on a Sunday and ending point falls on a Saturday (or else they must ignore the first and last point).

Applicability to the VDA Framework

PeMS has several functions and capabilities similar to the framework that will be developed to combine data from multi-agency, multijurisdictional sources to provide a more accurate, real-time understanding of system performance, automation of various capabilities and tools, and sharing to promote cooperation among agencies and for third-party use.

PORTLAND STATE UNIVERSITY’S PORTAL

Effective Practices

PORTAL is the transportation data archive for the Portland-Vancouver metropolitan region with a recent addition of Oregon’s Central Lane County. PORTAL brings together a large variety of transportation data that can be analyzed using PORTAL’s Web site tools to support planning for operations and overall performance measurement. PORTAL implements the U.S. National ITS Architecture’s Archived Data User Service (ADUS) for the Portland-Vancouver metropolitan region. Portland State University (PSU) faculty and students developed and continually enhance PORTAL in collaboration with the Oregon Department of Transportation (ODOT), the City of Portland, TriMet, the Southwest Washington Regional Transportation Council, and other partners.

The current PORTAL system archives traffic and transit data, including freeway loop detector data from the Portland-Vancouver metropolitan region, weather data, freeway incident data, transit AVL/automatic passenger counts data, arterial volume and traffic signal data, and truck weigh-in-motion data. A related project, the Bike-Ped Portal, incorporates bicycle-pedestrian data into the archive. The system focuses on the use of open-source software and open data.

PORTAL began in 2004 with a single source of data, freeway loop detectors from ODOT, and has become a much larger system. PORTAL benefits from a direct fiber optic connection between ODOT and PSU. Freeway loop detector data are obtained from the ODOT Region 1 transportation management operations center, which gathers the data from more than 600 inductive loop detectors comprising the Portland region’s ATMS. These detectors have been installed as part of the comprehensive ramp metering system; therefore, dual mainline loops (also known as speed traps) are located just upstream of on-ramp locations, and the on-ramps themselves are also instrumented. Recently, additional high-definition radar detection devices have been installed at many locations on the region’s freeways, particularly including long segments without ramps. These detector data (i.e., vehicle counts, average speed, and occupancy) are collected at 20-s intervals, and ODOT archives the data at 15-min aggregate intervals for operations control purposes.

PORTAL includes a data archive structure that has capabilities for data storage, data processing, and a user interface. Prior to implementation of the PORTAL data warehouse, the raw 20-s data were retained for only a short time before being discarded. Now, permitted users can access and query the archived data through a Web interface (see figure 24).

©PORTAL
Figure 24. Screenshot. PORTAL snapshot.⁽³⁵⁾

Success Factors

PORTAL has resulted in the following successful outcomes:

• The archive implements a data warehousing strategy that retains large amounts of raw operational data for analysis and decisionmaking processes; these data are stored independently of their operational sources, allowing the execution of time-consuming queries with no impact on critical operational uses.
  
  
• To ensure successful data integration, detector locations within the database have been geocoded for interoperability with GISs.
  
  
• Data security is a key component of the data archive. A working copy of the database maintained on the primary server is replicated in a compressed format at a remote site. These daily backups ensure that the archive service can be rapidly returned to operation with no significant loss of data if a copy of the database is lost. Both the primary database server and the backup storage are located in climate-controlled machine rooms with uninterruptible power supplies and generator backup power, preventing data loss or gaps in data availability due to power outages. The working copy of the database is stored on a redundant array of independent disks device, providing error detection, redundancy, and the ability to rebuild missing data upon device failures. Finally, hardware maintenance and security updates are provided for all computer systems by experienced systems administration personnel.
  
  
• Limited quality control procedures have been applied to identify and mark suspect or erroneous data. Work on additional data quality measures is in process.
  
  
• The data and performance measures in PORTAL have been used in the day-to-day management of the transportation system to identify congested areas and incidents and to dispatch incident response or emergency vehicles to the appropriate locations. The Portland region’s MPO has also successfully demonstrated how archived loop detector data can be used to improve travel demand forecasts for the Portland region.

Gaps and Challenges

The following are some of the specific gaps and PORTAL challenges:

• Geospatial referencing is a common and significant challenge for data integration. PORTAL requires inclusion of latitude and longitude in all data sources for output at minimum as a point-based display, but mismatched segments pose an ongoing challenge for numerous agencies.
  
  
• Although research institutions generally do not discard data, planning and operations agencies must define how much storage can realistically be useful and how to prioritize data archiving to optimize use of storage.

Applicability to the VDA Framework

In developing PORTAL, significant attention was devoted to ensuring that the archive adhered to the recommendations of the National ITS Architecture. The archive fulfills the following five distinct processing functions that are in accordance with the National ITS Architecture:

• Store data in the same format as received from ITS subsystems.
  
  
• Accommodate levels of aggregation and reduction of the data flows, depending on the type of data represented.
  
  
• Sample raw data flows for permanent storage in accordance with user specifications. Permanent storage of the sampled data should be either online, offline, or both.
  
  
• Apply quality control procedures to the data, including the flagging of suspect data and the editing of data.
  
  
• Distinguish among the following data types: unprocessed (raw), edited, aggregated, and transformed.

In Portland, there is also an emphasis on the development of a multimodal arterial performance measurement system. The steps for automating such a system within PORTAL are also applicable to the framework.

UNIVERSITY OF WASHINGTON’S TRAFFIC DATA ACQUISITION AND DISTRIBUTION DATABASE

Effective Practices

Funded by FHWA and WSDOT, the Traffic Data Acquisition and Distribution (TDAD) database was developed by the University of Washington to make the data from the WSDOT Traffic Management System available to researchers and planners. A report published in May 2002 presents a methodology that makes it possible “to create, encode, and decode a self-describing data stream” from a variety of sensors (including loops, probe vehicles, radar, and cameras) using the following elements:⁽³⁶⁾

• Existing data description language standards.
  
  
• Parsers to enforce language compliance.
  
  
• Simple content language that flows from data description language.
  
  
• Architecture-neutral encoders and decoders based on ASN.

TDAD began recording snapshots of data every 20 s in October 1998 and continued through 2007. These data can be queried by sensor type and location for a specified interval to produce a downloadable text file, which can be imported into external programs for additional analysis.⁽³⁸⁾ The TDAD system also houses a high-level application to analyze daily traffic counts, although data are only available through the system from 2003 through 2007. The TDAD architecture diagram is shown in figure 25.

Source: University of Washington, ©Daniel J. Dailey
Figure 25. Diagram. TDAD architecture.⁽³⁸⁾

The TDAD project was intended to provide data sharing capabilities that would not interrupt operations or degrade data quality as new loop stations were deployed and the data were made available to the operating WSDOT Northwest Region Traffic Management System. These loop data were added to TDAD. No new traffic data were added after WSDOT ceased funding the project, but the existing data were left online until the supporting computing equipment failed. The TDAD data elements are listed in table 3.

Success Factors

Documentation of the projects lists the following as successful outcomes of the ITS Data Fusion project:

• Literature review to establish organizational framework.
  
  
• Description of local data fusion application and detailed architecture.
  
  
• Statistically based algorithm for estimating speed from volume/occupancy.

The data fusion techniques shown in table 4 were identified in the development of this system.

Gaps and Lessons Learned

TDAD was developed based on the use of traffic data for reporting, long-term planning, project planning, performance monitoring, and research. The following gaps or challenges have been identified by TDAD stakeholders:

• Limited access to a diversity of data sources (to validate and supplement existing sources or eliminate redundant collection efforts).
  
  
• Need for more vehicle classification data, speed and occupancy data, and 15-min/hourly volume data (versus currently reported averages).

Applicability to the VDA Framework

Sponsored by WSDOT, the ITS Data Fusion project had very similar goals to the VDA Framework that will be developed to combine data from multi-agency, multijurisdictional sources to provide a more accurate, real-time understanding of system performance and to be distributed to promote cooperation between agencies. The project is dated but nonetheless relevant to the VDA Framework project.

CLARUS

Effective Practices

The U.S. Department of Transportation’s (USDOT) Clarus initiative was a research effort to develop an integrated system for surface transportation weather observation, forecasting, and data management. The initiative began in 2005 and ended in June 2013 when the goals and objectives of the Clarus initiative were achieved. The Clarus system, resulting from the initiative, has been incorporated into the National Oceanic and Atmospheric Administration’s (NOAA) Meteorological Assimilation Data Ingest System.

Clarus assimilates, quality checks, and disseminates multisource data to provide transportation managers with reliable information that they can use to proactively manage the impacts of adverse weather on transportation (e.g., incidents, injuries, fatalities, and delays). In addition to weather-responsive traffic management, Clarus can be used to support winter road maintenance, non-winter road maintenance (e.g., spraying, striping, and road repair); traveler information dissemination; safety management, transit vehicle dispatching, and flood control. Private sector and academic organizations can also use data from Clarus to create innovative services and technologies.

Clarus collects near real-time atmospheric and pavement observations from multiple States’ ITSs, including RWIS^[1], environmental sensor stations (ESSs), and mobile observations from AVL-equipped trucks (see figure 26 for a sample Clarus user interface). Since its inception, USDOT has demonstrated Clarus to various States in an effort to encourage ESS data contributions, and more than 75 percent of State transportation departments participated in Clarus. The initiative also advanced the use of weather data collected from passenger vehicles equipped with transceivers—research that is being conducted under the ITS Joint Program Office’s connected vehicle research program.⁽⁴¹⁾

Figure 26. Screenshot. Clarus user interface.

Clarus was designed to meet the following requirements:⁽⁴²⁾

• Acquire, process, and distribute large volumes of observation data in a timely manner.
  
  
• Manage data from thousands of ESSs and other sources.
  
  
• Accommodate data from new sensor technologies.
  
  
• Provide a one-stop Internet portal to report weather observations.
  
  
• Provide data that do not require post-processing.
  
  
• Provide management control for user inputs and extractions.
  
  
• Offer data retrieval tools.

USDOT Clarus-Based Awarded Contracts

In 2010, USDOT issued a Broad Agency Announcement (BAA) to develop Clarus-based tools and applications that could be implemented by agencies that participate in Clarus to improve transportation weather management and operations. USDOT awarded the following eight contracts as a result of the BAA:

• Integrating Clarus Data in Traffic Signal System Operation: A Survivable Real-Time Weather Responsive System (University of Idaho).
  
  
• New Brunswick-Nova Scotia Clarus Integration Plus (commercial contractor).
  
  
• One-Stop Shop for Rural Traveler Information (Western Transportation Institute).
  
  
• The Integration of Multi-State Clarus Data into Real-Time and Archived Regional Integrated Transportation Information System Data Visualization Tools (University of Maryland CATT Laboratory). (This project is discussed in more detail in the next section.)
  
  
• Research on Clarus System Data (integration of Clarus system data with the Data Transmission Network road segment alerting engine) (commercial contractor).
  
  
• Integrating Clarus Weather Station Data and State Crash Data into a Travel Decision Support Tool (Michigan Technological University).
  
  
• Passenger Bus Industry Weather Information Application (commercial contractor).
  
  
• Application of Clarus System Data to the Improvement of Mobile ESS Utilization (University of North Dakota).

Integration of Clarus Into RITIS

One of the contracts awarded as a result of the BAA focused on integrating Clarus data into the University of Maryland’s RITIS. In December 2011, the University of Maryland published a report to document its successful integration of all Clarus data into RITIS for real-time situational awareness and historical safety data analysis.⁽⁴³⁾

RITIS does not collect data directly but rather aggregates data from participating agencies; these data are stored in their native formats but processed for integration. Although the focal objective of RITIS is to support emergency response operations, it has a much broader user base. RITIS automatically generates a wide variety of visualization tools for data display. Figure 27 and figure 28 illustrate the Clarus data integration process. Figure 27 shows the actual Clarus station data superimposed on the RITIS regional map, while figure 28 shows the interpretation of this information to provide useable data for traveler information and traffic management.

Source Michael Pack ©University of Maryland CATT Laboratory
Figure 27. Screenshot. Clarus data on RITIS regional map.

Source Michael Pack ©University of Maryland CATT Laboratory
Figure 28. Screenshot. Graphic interpretation of weather data in RITIS.

Success Factors

Clarus has resulted in the following successful outcomes:⁽⁴⁴⁾

• Development of agency partnerships to connect existing sensors into a nationwide network that generates and delivers Clarus data.
  
  
• Development of innovative methods for robust data assimilation, quality checking, and data dissemination that can provide near real-time atmospheric and pavement observations from existing technologies.⁽⁴⁵⁾
  
  
• Proof-of-concept testing to evaluate the ability for data exchange for surface environmental data and relevant surface transportation conditions.
  
  
• Demonstrations in multiple regions of the innovative use of Clarus data to provide new products and services to support and enhanced transportation agency operations.

Applicability to the VDA Framework

As part of the regional use case project, Integration of Multi-State Clarus Data into RITIS, researchers integrated RWIS data into the RITIS data integration and sharing platform cheaply and quickly as a result of Clarus data integration efforts. The VDA Framework should incorporate similar functionality and include visual analytics tools that allow users to explore the relationships between weather, speed, volume, and incident datasets.

FHWA DATA CAPTURE AND MANAGEMENT PROGRAM TEST DATASETS

Effective Practices

In 2011, the FHWA Real-Time Data Capture and Management (DCM) Program issued a request for procurement for well-documented, quality test datasets available from recent or ongoing operations, field tests, or simulations of emerging technologies that support mobility, environment, transit, freight, weather or other surface transportation research. The test datasets comprised observed data or a combination of observed, derived, and simulated data from the following regions:

• Pasadena, CA: The Pasadena test dataset covers the roadway network in and around Pasadena and includes data collected during September and October 2011. The dataset includes a variety of simulated, real-time, and forecast data. This includes network data (highway network file), demand data (trip tables), network performance data (link volumes, turn volumes, speeds and capacity), work zone data, weather data, CCTV camera data, and changeable message sign data. Data elements were directly collected and processed from these sources, although some data were generated by the Mygistics™ National Traffic Model, which processes live sensor data and fuses them with probe data (AirSage™ mobile telephone call and Internet Protocol (IP) detail records) to produce 15- and 30-min projections of traffic conditions.
  
  
• Seattle, WA: The University of Washington submitted a dataset for I-5 in Seattle that included freeway, arterial, freight, and transit data. Data sources included loop detectors, arterial data from automated license plate readers, Bluetooth® Media Control Access readers, traffic signal timing plans, and freight Global Positioning System (GPS) data. This dataset included 6mo of data.
  
  
• Portland, OR: PSU submitted a multimodal dataset that included freeway, transit, and arterial (second-by-second data from signals) data. The test dataset included freeway data consisting of 2 months of data from dual-loop detectors deployed in the main line and on-ramps of I-205; incident data from the ODOT ATMS database and the planned event data from the ODOT Trip-Check Traveler Information Portal information Web site; weather data from NOAA and remote weather information system station data; arterial volume and occupancy data from four single-loop detectors on 82nd Avenue; signal phase and timing data for 32 signals along the 82d Avenue corridor; limited Bluetooth® travel time data; and transit data from Tri-Met, including schedule, stop event, and passenger counts for both bus and light rail. The data collection period for all datasets is September 15 through November 15, 2011.
  
  
• San Diego, CA: Berkeley Transportation Systems submitted a dataset that included 1year of raw and cleaned data for more than 3,000 traffic detectors deployed along 1,250lane-mi of I-5 in San Diego; cleaned and geographically referenced data for more than 1,500 incidents and lane closures for two sections of I-5 that experienced the greatest number of incidents during 2010; complete trip (origin-to-destination) GPS “breadcrumbs” containing latitude and longitude, vehicle heading and speed data, and time for individual in-vehicle devices updated at 3-s intervals for 10,000 trips taken during 2010; and weather data from 7 weather stations in the San Diego area.

Success Factors

FHWA required the sites to perform the following data cleaning, integration, organization, and documentation activities:

• Clean up data to identify and correct errors, flag suspect data values, and identify periods of missing data.
  
  
• Remove personally identifiable information or sensitive personally identifiable information from datasets.
  
  
• Produce datasets that combine or integrate data from multiple source types (e.g., travelers, vehicles, and infrastructure), multiple source agencies (e.g., transportation departments, commercial freight companies, and transit agencies), or multiple modes (e.g., light vehicles, commercial vehicles, and transit vehicles).
  
  
• Provide documentation of the dataset contents and formats.
  
  
• Provide ASTM-compliant metadata describing data quality, relationship to other data, and information about where and when each value was captured.

The test datasets provide tangible and broadly accessible examples of data sharing and integration that resolved cross-cutting issues such as data integration formats, data standardization, data quality, and data sharing. Sites were required to meet open data principles that allow the Government to share the datasets and corresponding documentation with other researchers, connected vehicle application developers, and the general public under the terms of the Open Data Commons Attribution License. The test datasets and documentation are available through the Research Data Exchange Web site (https://www.its-rde.net).⁽⁴⁶⁾

Gaps and Lessons Learned

The Pasadena, CA, project team documented the following key findings and lessons learned in compiling the Pasadena test dataset:⁽⁴⁷⁾

• Supply of real-time input data: More fixed detector coverage, more probe vehicles, and more roadside infrastructure are needed. The project team must ensure data are synthesized, packaged, and delivered in time for consumption by downstream intelligence and analytics tools.
  
  
• Data standardization and open data structure: Some standards, protocols and open structure for traffic management and traveler information exist, such as Institute of Electrical and Electronic Engineers 1512, Family of Standards for Incident Management Message Sets (https://www.standards.its.dot.gov/Factsheets/Factsheet/12).⁽⁴⁸⁾ However, this is not the case for all data feeds, and it was necessary to develop ad hoc importers for those feeds.
  
  
• The following components are necessary to support high-quality, high-resolution data for real-time traffic operations:
  
  
  • Junction details: Providing junction details such as turn delay, capacities, and lane groupings required significant effort to manually code and synthesize geometric layout and signal control parameters from multiple data sources such as public agencies (e.g., City of Pasadena) and others (e.g., online Google Maps™ and Google Street View™).
    
    
  • Traffic data feeding into the real-time model: The traffic forecast system accounted for the impacts of construction work zones, road closures, and incidents, but could not include adverse weather; this is because inadequate research was performed on modeling weather impacts on road capacity and speed. The researchers found that key parameters such as “number of lanes affected” were missing from coded incident reports, even though the information could be deduced from the incident description. Screening and filtering pre-steps are important for improving data quality.
    
    
  • Geospatial referencing of event locations: The event data feeds (e.g., construction work zones, incidents) each used different referencing schemes, which had to be taken into account when developing the real-time traffic model. Incident data used latitude and longitude referencing and road name and cross street information, while construction work zones used linear referencing by milepost markers. The researchers found that latitude and longitude data were often inaccurate for geocoding to the network and that multiple referencing systems were needed for cross-checking. They suggested the need for a regional or national standard for geospatial referencing event data and location attributes.
    
    
• Development and maintenance of a real-time traffic modeling system requires the following:
  
  
  • Highly efficient data integration: Many automated processes were required to replace time-consuming data collection and quality control work for all system components, including network supply (from navigation network), travel demand (from MobileODTM process) and traffic control (signal control and ramp meter from existing VISUM tools).
    
    
  • Robust computing infrastructure: The data collection system resided on a Windows® 7 Professional Edition virtual machine, with the IP address being authenticated by the live data host servers (e.g., RITIS and Caltrans PeMS systems). Because the virtual machine is within a private cloud computing environment, researchers experienced system downtime during events such as power outages, abnormalities in IP address authentication, and in system time online synchronization. These events affected the archived data as well. The researchers suggested the need for hosting the system via a commercial cloud computing service (e.g., Infrastructure as a Service) when highly reliable and robust operations were necessary.

Applicability to the VDA Framework

The final report for the Pasadena test dataset documents many gaps and lessons learned that are relevant to the framework, including data capture; data integration; geospatial referencing components necessary to support high-quality, high-resolution data; and infrastructure required to support a real-time traffic modeling system.

UTAH DEPARTMENT OF TRANSPORTATION’S UPLAN

Effective Practices

The UPlan system is an interactive, Web-based mapping tool that enables sharing and integration of data from many different sources (see figure 29). The Utah Department of Transportation (UDOT) initiated development of the system in 2007, with an ultimate goal of implementing a data sharing tool that could display information spatially. The system uses ESRI’s ArcGIS™ Online service in a virtual cloud-based data environment.

Figure 29. Screenshot. UPlan user interface.⁽⁴⁹⁾

Data are compiled from a variety of sources and displayed spatially on an interactive map, allowing users to view maps of UDOT and MPO projects and additional contextual information in a project’s area. External agencies, such as MPOs and utility companies, have also contributed data to the site. Key features of UPlan include the following:

• Group creation and user management: Users can form groups both within an agency and with members of other agencies to share data and reports.
  
  
• Rapidly produced reports: Users can quickly generate reports to examine resource details, project impacts, and management or stewardship boundaries.
  
  
• Publicly viewable map presets: UPlan allows agencies to post and distribute publicly accessible maps with predetermined features and symbology. These maps can be easily updated to ensure consistent availability and sharing of information.
  
  
• Incorporation of server-based or local data into preset maps: Users can upload their own datasets or use publicly available Web server data available to overlay on an existing map.
  
  
• Utilization as a development tool: UPlan provides a platform for the development and testing of custom applications. For example, users can view future UDOT projects along with planned utility improvements to see potential conflicts, or they can display critical environmental attributes such as streams, wetlands, and rare plant habitat on top of planned roadway capacity projects for any location in Utah. UDOT also recently put a roadway design project in UPlan and added the electric utility data; State Historic Preservation Office historic and archeological data; UDOT’s safety, pavement, bridges, traffic, environmental, freight, and State Transportation Improvement Plan (historical and current information); and many other data sources.

In terms of recommended protocol and standards for data submittals, the system is flexible enough to accommodate a variety of schemas. UDOT has documented guidance for data management, as well as a user guide for the system.

Success Factors

UPlan has resulted in the following successful outcomes:^(50,51)

• Allows agencies to share data in a common location while viewing and analyzing it for their own needs. This has enabled data silos to be combined and viewed together to allow for better collaboration among agencies.
  
  
• Provides an easy-to-use tool designed for a broader, nontechnical audience. It enables quick access to information and eliminates the need to go through three or four data silo owners.
  
  
• Improves and promotes partnering. UPlan has created new relationships and strengthened existing relationships with other State agencies, Federal agencies, MPOs, the Utah Transit Agency, local governments, utility companies, and among the many divisions within UDOT.
  
  
• Encourages broad participation in the transportation planning process because it presents data in an easy-to-understand format and allows participants to bring their own data to the table. It has also improved the ability to engage public comment and participation in the planning process.
  
  
• Created a catalyst for developing additional planning partnerships as a result of the discovery of data gaps. For example, mapping freight rail corridors and major activity centers has spurred dialogue to begin the planning process for a new freight railway.
  
  
• Allows UDOT to work with other States and organizations to help them implement their own versions of UPlan. There is a relatively low start-up cost ($1,000 to $2,000) for these States to share basic information. Some organizations are making it an enterprise system, while others are adding data as needed and leveraging all of the data in UPlan for their work.

Gaps and Lessons Learned

Several sources have documented the following gaps and lessons learned from UPlan:^(51,52)

• UDOT initially hosted the system on its server, which required a significant amount of time and labor costs to maintain, manage, and upload datasets; modify access rights to the site; and maintain and update the system’s functionality. However, shifting the tool to a Web-based application eliminated many of these management and back-end challenges.
  
  
• With so much information coming into UPlan, UDOT is working to develop better ways to navigate the data and allow users to customize UPlan for their own dataset needs and with a look and feel specific to their organization.
  
  
• UDOT is continuing work to ensure data are displayed as accurately as possible while balancing data quality concerns of individual agencies.
  
  
• UDOT is testing the integration of performance measures into UPlan as part of a prototype project to display UDOT’s strategic goals and compare them with the goals of other partner agencies. UDOT’s strategic goals are zero crashes, injuries and fatalities; preserve infrastructure; and optimize mobility.⁽⁵⁴⁾ It has been able to use UPlan to display performance measures for asset management, safety, and capacity, and it is evaluating measures for efficiency.

Applicability to the VDA Framework

The UPlan system is an example of a successful virtual cloud based data environment that has enhanced data integration, data sharing, and transportation planning activities at UDOT. The success factors and lessons learned are all applicable to the framework.

FLORIDA DEPARTMENT OF TRANSPORTATION’S ITS DATA CAPTURE AND PERFORMANCE MANAGEMENT TOOL

Effective Practices

The ITS Data Capture and Performance Management Tool (ITSDCAP) was developed by the Florida Department of Transportation (FDOT) as a data environment to capture and fuse data from multiple sources to support applications such as performance measurement, transportation system modeling, assessment of ITS application benefits, and data mining and visualization techniques. The tool has recently been updated to become Web-based and is integrated with a real-time decision support environment also developed for FDOT. The tool includes the following functionality:⁽⁵⁴⁾

• Data capture and fusion: Data capture and fusion come from multiple sources, including ITS. If the data come from a system that does not already check for data quality, the tool checks, filters, and repairs missing data by imputation to ensure data quality. Data fusion is accomplished through a common spatial and temporal referencing scheme.
  
  
• Performance measures: ITSDCAP uses performance measures to assess the performance of transportation systems, including mobility, reliability, safety, and environmental measures, as well as to produce performance dashboards.
  
  
• Decision support tools: A number of offline and real-time decision support tools that assess bottleneck breakdowns and event impacts have been incorporated in the tool.
  
  
• Data mining: Data mining techniques allow discovery of patterns and trends in condition data.
  
  
• ITS evaluation: These capabilities generate input parameters for the ITS Data Capture and Performance Management Tool to calculate ITS benefits and benefit/cost ratios at the planning level. In addition, the tool can perform before–after analysis for a number of ITS applications.
  
  
• Real-time data.
  
  
• Model support: Model support is used to generate input parameters for modeling efforts such as traffic demand forecasting, dynamic traffic assignment, analytical traffic analysis procedures, and simulation models. As part of an ongoing FDOT project on multiresolution simulation, utilities are being produced to link ITSDCAP-generated data to NeXTA.
  
  
• Map visualization: Maps of performance measures are used to better understand available data and calculated performance measure values.

ITSDCAP allows data capture and grouping from the following sources:

• SunGuide® ITS sensor data at 20-s aggregation level and lane-by-lane level.
  
  
• Statewide ITS data warehouse data at 5-min aggregation level.
  
  
• SunGuide® DMS activation data.
  
  
• SunGuide® incident management database.
  
  
• Florida Highway Patrol incident database.
  
  
• Private sector travel time data.
  
  
• Automatic vehicle identification (AVI) data (Bluetooth® and Wi-Fi).
  
  
• Inrix® data.
  
  
• Work zone/construction database.
  
  
• Number of 511 calls and Web site hits at an aggregation level of 15 min per corridor per county.
  
  
• Dynamic pricing on managed lanes and ramp metering data.
  
  
• FDOT Crash Analysis Reporting system.
  
  
• Police crash archives.
  
  
• Weather data from RWIS.
  
  
• FDOT statistics database.

A new FDOT project is investigating incorporating connected vehicle data into the tool. A high-level diagram of the ITSDCAP tool is shown in figure 30.

©FDOT
Figure 30. Diagram. High-level design of ITSDCAP.⁽⁵⁴⁾

The data environment includes functionality to support calculating the following performance measures:

• Mobility measures, including speed, density, queue length and location, travel time, delay, VMT, and VHT.
  
  
• Reliability measures, including standard deviation, buffer index, failure and on-time performance, planning time index based on the 95th or 80th percentile, skew statistics, and misery index.
  
  
• Safety measures, including crash frequency by crash type, crash frequency by severity, total crash frequency, crash rate by type, crash rate by severity, and total crash rate.
  
  
• Energy and emission measures, including emission rates using MOBILE6.2 model parameters that are specific to Florida. FDOT plans to update the emission rates based on the more recently released Motor Vehicles Emission Simulator, which appears to be more sensitive to reductions in speed.

An example of the interface for mobility performance estimation is shown in figure 31.

©FDOT
Figure 31. Screenshot. Interface for performance measure estimation.⁽⁵⁵⁾

Success Factors

The report, Integrated Environment for Performance and Assessment of Intelligent Transportation Systems Operations, identifies the following successful outcomes of ITSDCAP:⁽⁵⁴⁾

• FDOT found that to fuse or associate data from different sources, there needs to be a common location reference system. If data sources have different reference systems
  (e.g., one database uses roadway milepost and another uses latitude/longitude coordinates), ITSDCAP translates the locations to mileposts first.
  
  
• ITSDCAP incorporates alternative methods for calculating different performance measures based on point detector data, AVI data, or private sector data. For example, the tool includes four different methods for estimating travel time, including instantaneous travel time at the period of estimation or the experienced travel time as a vehicle progresses from one link to the next.
  
  
• Methods to evaluate benefits of freeway-related ITS implementations, including incident management, ramp metering, managed lanes, smart work zone, and road weather information system, were developed.
  
  
• Decision support tools based on data from different sources were developed. Other fused data can be output for analysis using external dining mining techniques.

Gaps and Lessons Learned

The data environment initially focused on freeway facilities, but FDOT is updating the system to include data from signalized arterials and connected vehicles. The tool has been converted to a Web-based tool that significantly improved its usability and accessibility. There are possibilities of the inclusion of transit and freight information in the tool.

Applicability to the VDA Framework

ITSDCAP has several functions and capabilities that are relevant to the framework, including the development of systems and processes for combining and sharing data from multiple data sources with different location referencing, quality, and aggregation levels; incorporation of alternative methods for calculating performance measures based on the data source (e.g., point-based detector data, AVI data, or private sector data); data mining and visualization capabilities; and adaptation of the tool to generate output for use in external ITS evaluation and modeling applications.

¹ RWISs are typically equipped with cameras and sensors to detect radiation; temperature dew point; precipitation; visibility; snow depth; and road surface, subsurface, flooding, water level, and precipitation accumulated.