Methods and Metrics of Evaluation of an Automated Real-time Driver Warning System Transportation Research Board Paper No.

Size: px

Start display at page:

Download "Methods and Metrics of Evaluation of an Automated Real-time Driver Warning System Transportation Research Board Paper No."

Edwin Patrick Higgins
5 years ago
Views:

1 Methods and Metrics of Evaluation of an Automated Real-time Driver Warning System Transportation Research Board Paper No. TRB C. Arthur MacCarley California Polytechnic State University San Luis Obispo, California, USA

2 CAWS Study Area Interstate 5 and State Route 120, Near Stockton, California Study in progress through June 30, 2005 Interstate 5 Mossdale Y Route 5

4 CAWS Changeable Message Sign (CMS) No. 1 (Total of 9)

5 Components of the CAWS District 10 Traffic Management Center Meteorological Station Monitoring Computer (Pentium/Win95) Leased and Dedicated V34 BIS Phone Lines 9 Automated Weather Stations CMS Control Computer (486/DOS) 9 Model 500 Changeable Message Signs Traffic (loop detector) Monitoring Computer (486/DOS) 36 Inductive Loop Speed Detectors

6 CAWS Control Computers in District 10 TMC

7 Evaluation Objectives 1. Technical Assessment. Assess quality, innovation, and reliability of deployed system. 2. Operational Assessment. Does the system do what it is supposed to to do? 3. Assessment of impact of system on driver behavior during limited visibility conditions. Observe and assess response of drivers to CAWS warning messages. 4. Assessment of long-term impact of system on accident rates and losses. Compile long-term accident statistics before and after system activation.

8 Fog accidents in study direction 4 year period before CAWS

9 Fog accidents in study direction 4 year period after CAWS

10 High-Risk Area: Mossdale Y

11 All-Weather Accident Rate, Study v.s. Control All-Weather Accident Rate (Study Vs Control) Accident Rate (MVMT) CAWS Activation Jan Oct Jan - Mar Year 2002 Study Control

12 Accidents in Fog, Before and After CAWS Fog Day Accident Rate (Normalized to Absolute VMT) 0.12 Accident Rate (MVMT) CAWS Activation Study Control Jan - Oct Jan - Mar Year 2002

13 Cross-Peak-Period Accident Rates Before and After CAWS Accident Rate (MVMT) Jan AM Study Oct PM Control 1996 CAWS Activation Jan - Years Mar 2002

14 Secondary Accidents, Before and After CAWS Secondary Accident Rate (MVMT) Jan Oct 1996 Study Secondaries Control Secondaries CAWS Activation Jan - Mar Years 2002

15 External Factors Affecting Accident Rates Variable weather number of fog days per year Change in national speed limit, 1998 Changes in commute patterns and vehicle mix Roadway construction / lane closures

16 Direct Assessment of Driver Reaction to CAWS Warning Messages

17 Driver Response Assessment Monitoring Sites Direction of traffic flow

18 6 Loop Detectors (3 lanes) Downing Road Data Acquisition Station Before CMS Evaluation System Components 6 Loop Detectors (3 lanes) French Camp Slough (FCS) Visibility & Illumination Sensors CMS monitor Subsystem Traffic and CMS Video Monitoring Subsystem Data Acquisition Station French Camp CMS Data Acquisition Station CDPD and CDMA to Internet Loragen Evaluation Database and Web Server VPN access through Caltrans D10 Firewall After CMS 6 Loop Detectors (3 lanes) Visibility & Illumination Sensors Mathews Road (MR) Data Acquisition Station CAWS Weather Server in D10 TMC 6 Loop Detectors (3 lanes) El Dorado Undercrossing Data Acquisition Station CAWS Meteorological System Computer in TMC

19 Experimental Apparatus for Monitoring Driver Response to CMS Messages

21 Metrics for Risk or Severity of Collision Aggregated Metrics Mean traffic speed Variance or Sample Standard Deviation (from the mean) of vehicle speeds Measured for individual vehicles over a given period Calculated over a period from aggregated contents of speed categories or buckets

22 Metrics for Risk or Severity of Collision Aggregated Metrics τ gap, i = τ = τ headway off time, avg τ presence = τ period x i 1 v 0, i x i, i ( 1 occupancy) for individual vehicles period average where occupancy isratio of time that loop detector is "on" in presence mode

23 Metrics for Risk or Severity of Collision Individual Vehicle Metrics Time To Collision (TTC) [Hayward 1971, Van der Horst, 1990] TTC i x x l i 1 i i 1 =, vi > vi 1 vi vi 1 Defined at moment that vehicle i initiates braking, holding constant the velocities of both vehicles

24 Metrics for Risk or Severity of Collision Individual Vehicle Metrics TET = Time Exposition Time-to-Collision [Minderhoud and Bovy 2001, Hirst 1997] Fraction of time that a vehicle travels with TTC below a critical value, typically 4.0 seconds TIT = Time Integrated Time-to-Collision [Minderhoud and Bovy 2001] DTS = Deceleration-to-Safety Time [Topp et.al. 1996]

25 v impact,i Metrics for Risk of Collision Enhanced metric for visibility-limited collisions Derived from velocity of impact with an immobile object in roadway as might be encountered in multi-car pile-up v = v 0, i 0, i 1, min{ x min{ x vis vis, x, x x gap, i brake,i gap, i } x } < x react, i react, i, min{ x vis, x gap, i } x react, i x gap, i = ( t0, i t0, i 1) v0, i 1 = separation between vehicle i and i - 1 x x react, i = treact v0, i brake, i = v 2gk 2 0, i friction = reaction distance of vehicle i = braking distance of vehicle i

26 Metrics for Risk of Collision Velocity as a function of distance during aggressive braking event v v = vehicle speed (ft/sec or m/s) v 0 = vehicle speed prior to braking v 0 x = distance traveled (ft. or meters) v impact min{ x vis, x following } x x react x brake

27 Metrics for Risk of Collision Normalized Risk of Collision R i for vehicle i R i = v impact, i v 0, i 100% v v 0, i = impact, i measured velocity of vehicle i at point of = velocity of impact,stationary object detection

28 % 75% Speed Before CMS, MPH Stnd Deviation Before CMS, MPH Speed After CMS, Time Shifted, MPH Stnd Deviation After CMS, Time Shifted, MPH Visibility Before CMS, Extinction Coef. Visibility After CMS, Extinction Coef. Avg. Volume, All Sites, veh/min. CMS Message (blank, 30 MPH, 60 MPH) % Risk of Collision % 80 0% Visibility (Extinction Coef.) 60 Mean Speed Advise 30 MPH Advise 30 MPH 40 Volume 20 Blank Msg Advise 45 MPH Standard Deviation of Speed Blank Msg 0 4:00:00 4:15:00 4:30:00 4:45:00 5:00:00 5:15:00 5:30:00 5:45:00 6:00:00

29 % 75% Period of similar poor visibility Speed Before CMS, MPH Stnd Deviation Before CMS, MPH Speed After CMS, Time Shifted, MPH Stnd Deviation After CMS, Time Shifted, MPH Visibility Before CMS, Extinction Coef. Visibility After CMS, Extinction Coef. Avg. Volume, All Sites, veh/min. CMS Message (blank, 30 MPH, 60 MPH) % 25% Visibility (Extinction Coef.) Risk of Collision 100 0% 80 Mean Speed 60 Volume Standard Deviation of Speed 0 8:00:00 8:15:00 8:30:00 8:45:00 9:00:00 9:15:00 9:30:00 9:45:00 10:00:00

30 Lane-by-lane analysis: Animation of traffic speed distributions by lane during a fog event

31 Conclusions Related to Evaluation Methods Assessment of driver reaction requires the design of an experiment to isolate safety intervention from other factors to maximum extent possible. Accident rates are significantly dependent on external factors: speed limit, construction, annual weather, commute patterns. Difficult to isolate effect of a particular safety intervention. Statistical analysis of traffic accidents is problematic: Long terms (many years) required to amass a statistically significant body of accident data. But long terms introduce dominant changes in experimental conditions and external factors. Data analysis and statistical modeling is sensitive to model design and selection of control areas.

32 Conclusions Related to Evaluation Methods Short period variance observations focus on interactions between proximate vehicles: improved relevance for risk of collision. Aggregated data or long-period averages may conceal useful information in driver reaction patterns. Establishing a reasonable trend line through data based on individual vehicle records involves tradeoffs between visual interpretation and data scatter. Risk of collision metrics based on individual vehicles better reveal response of traffic to transient events or safety interventions. Practical sight distance is an important factor in risk of collision metric for fog accidents.

33 Preliminary Conclusions and Observations Regarding Driver Behavior In cases examined to date: Drivers respond primarily to their perceptions of their immediate situation. Weak or nil influence of dynamic warnings on vehicle speed or separation. Following distances of drivers indicates they consistently underestimate collision risk, especially in fog.

34 Preliminary Conclusions and Observations Regarding the CAWS A complex fully-automated wide-area system one of the first. Visibility is a localized and transient phenomena requiring closelyspaced sensors for accurate effect prediction and warning. CAWS partially limited by operational problems including control strategy design issues and software errors. Improved testing and validation of software recommended prior to system activation, including a formal alpha and beta testing process. Improved documentation of system operation and control strategy needed. Some misconceptions about what the system was doing. Study still in progress. To date, safety enhancement effects not established.

36 All-Weather Accidents, Before and After CAWS Total All Weather Accidents Study direction only Total Travel (MVMT) Study All Weather Accident Rate (Accidents/MV MT) Total All Weather Accidents Control direction only Total Travel (MVMT) Control All Weather Accident Rate (Accidents/M VMT) Jan Oct Oct Dec 96 System Activation Jan Mar

37 All accidents in study direction 4 year period before CAWS

38 All accidents in study direction 4 year period after CAWS

ROAD SAFETY RESEARCH, POLICING AND EDUCATION CONFERENCE, NOV 2001

ROAD SAFETY RESEARCH, POLICING AND EDUCATION CONFERENCE, NOV 2001 Title Young pedestrians and reversing motor vehicles Names of authors Paine M.P. and Henderson M. Name of sponsoring organisation Motor