“SHOW ME!”

Evaluation Team Original Questions:

1.  Your software development process, techniques, and tools, and their ramifications for the initial deployment and on-going maintenance. 

2.  Your initial ideas for the system's user interfaces, both public (511) and operator (regional display, incident editor, streaming video) - what might they look like, what would they be based on or what options are being considered, what issues need to be resolved? 

3.  How a new node (agency) or changed node (unexpected upgrade or change-out of the software at an existing agency's node) during the contract can be accommodated both technically and contractually.  What will be SACOG’s options for changing or adding nodes after the completion of your contract?  (i.e., ICD’s)

4.  Your plans, options, and issues related to the streaming video components and architecture, including support for multiple existing camera manufacturers and related protocols for pan/tilt/zoom control, and new camera types and protocols in the future (during or after the initial implementation project). 

5.  Your experience with and ideas for the transit-related requirements. 

6.  Your plans for and issues related to other challenging or important requirements. 

7.  Your proposed time schedule, what drives it, the items most at risk of slippage, and items most amenable to advancement. 

Additional Questions:

Do you have an organizational CM Plan?  Can we get a copy?  [Questions below will give a sense of whether they actually follow it or not]

Performance –

Can you show us an incident flow map that portrays BOTH freeway and arterial performance?

How would you ensure that there is uniformity of conditions reporting across systems, facilities and agencies?

Would there be some sort of transportation system performance reporting?  Performance trends?  Historical comparison of performance data?

Are there features in their system to help with assessing and monitoring the accuracy and quality of the data in the system?  [SR4]

Security –

Are there features in their system to possibly prevent a "rogue" operator from posting potentially malicious and damaging information into the system and out to the public?  What about outside hackers?

How can they help the agencies assure that this stuff is secure and robust enough to run on agency IT networks?  Can these guys win over skeptical or uncooperative agency IT management?

You might also want to ask about "maintenance back doors" to the software.  What have they done to make them secure.

Software/Hardware Performance –

Can they tell us about system failure modes?  Then about how easy is it to get it back up and running.  If you experience a major hardware of software crash you don't have to fly the contractor in to fix it or ??????

What might cause a system crash or restart?  If it does crash, how do you know what caused it to crash? What features are there to troubleshoot then repair?

What features does your system have to promote safe or graceful failures?

Then could they tell us what features/functions they have to monitor the performance of the hardware/software system they will provide?  Could it predict potential failures?

What method do you use for software quality control and bug fixes? Configuration Management, Version control, documentation etc.

Requirements Issues -

Ask them about their lack of explanations in Table 5 Compliance with Requirements.   Most of them simply state "full" compliance with no explanation.  How do they intend to show to you that they are in fact fully compliant and/or clarify what it their definition of full compliance to the requirement?  Challenge them and see where they take it and how they respond.

How do they plan to show you in reality that their systems actually comply with the requirements and how do they intend to document it?  How might they react if you had requested a response/explanation to every requirement instead of only those required for partial or no compliance?

If there is no agreement on the meaning of a requirement how would you resolve it?

With a lot of the traffic signal requirements they respond "full compliance" but a lot of the functions listed in the requirement may be proprietary (not NTCIP either) to a particular vendor.  Many of the vendors don't really want anyone else using them or knowing about them.  How do they intend to work with a non cooperative traffic signal (or for that matter any other existing system) software vendor? 

Are there requirements that we overlooked?

What tools do their systems have to help assess the quality of the data being supplied by others?

How much do they need to modify, extend or rewrite their software to meet these requirements?

Other Issues –

How do you plan to involve agency operators?

You have presented alternative user interfaces, but recognize that operator A in Elk Grove may not have same ideas of need than in Sacramento.  Ho do you plan to customize the GUI for everyone’s use?

Streaming video – multiple vendors/protocols/agencies – how to deal with?

What is the stability of development tools/environments?

How to see several authors comments on incident tracker?

Incident Tracker [SR145] – Validate your comment – Show us!