ICT Technologies for Next Car Generation Christian LAUGIER Research Director at INRIA Deputy Director of the LIG Laboratory (Grenoble France) French / Japanese Workshop on ICT Paris, Nov. 19-20 2009
Socio-Economic & Technical context Because of various Socio-economic and Technical reasons, Transportation systems will drastically change in the next 10-15 years (Driving assistance, V2V & I2V communications, Autonomous driving capabilities, Green technologies ) Human society & Governments feel more and more concerned by Safety, Pollution, and Traffic congestion problems About 31 millions vehicles & 8000 fatalities/year in France in the past years, 1 fatality every 10mn in West Europe (e.g. 140 per day, ~1 plane crash per day) Safety Traffic congestion Pollution & Space Car constructors & Car suppliers are more and more interested in introducing ADAS & Green technologies in commercial cars Thanks to recent progress in Robotics & ICT technologies, ADAS is gradually becoming a reality but cooperative research is still needed for solving some remaining Robustness, Safety, Efficiency, and Driver-Car interaction problems 2
Current & Future car equipments Steering by wire Brake by wire Shift by wire Virtual dash-board Modern wheel Navigation system Wireless Communication Speech Recognition & Synthesis Radar, Cameras, Night Vision, Various sensors. Cost decreasing & Efficiency increasing (future mass prod, SOC, embedded systems )!!!! 3
Autonomous driving : Some large experiments CyberCars Public Experiments (INRIA & EU Partners) Full autonomy is easier than Share Antibes Several successful control large scale experiments in public areas. But some Perception & Control Some CyberCars products in commercial use for private areas (e.g. Robosoft, Frog ) Technologies Will be useful for ICT-Car project Shanghai Public Demo 2007 Floriade 2002 (Amsterdam) 4
Autonomous driving : Some large experiments Urban Challenge 2007 Next step 96 km through an urban environment, 50 manned & unmanned vehicles 35 teams for qualification (NQE during 8 days), 11 selected teams, 6 vehicles finished the race Road map provides a few days before the race, Mission (checkpoints) given 5 mn before the race Big step towards Autonomous Several incident/accidents Vehicles during the event. But Safety is still not guaranteed Too many costly sensors are required Applanix Velodyne Laser SICK LMS Laser INS Riegl Laser Bosch Radar SICK LDLRS Laser IBEO Laser 5
Main required technologies for ICT-Car Car 1. Perceiving the world A world full of Uncertainty Reasoning about driving situations (context...) Dealing with the physical world constraints Reasoning under Uncertainty & Partial information Making Predictions & Risk assessment Traffic scene understanding 6
Main required technologies for ICT-Car Car 1. Perceiving the world A world full of Uncertainty Reasoning about driving situations (context...) Dealing with the physical world constraints Reasoning under Uncertainty & Partial information Making Predictions & Risk assessment 2. Driver-Car interactions & Shared control Human beings are unbeatable in taking decisions in complex situations Technology is better for simple but fast control decisions (ABS, ESP ) Driver himself is a danger factor! => Understanding Driver actions & intentions is mandatory 7
Multi-objects Detection & Tracking PreVent EU project, Versailles demo 2007 (Daimler-Chrysler & Ibeo test vehicle) Mercedes E-Class 350 Grid-Based approach Multiple Hypotheses & Interacting Multiple Models Computational time ~ 10 ms Application: Pre-fire & Braking Pre-fire & Braking Sensors: Two short range radars Two short range radars A laser scanner ALASCA A laser scanner ALASCA Actuators: Electrical belt pre-tensioning Electrical belt pre-tensioning Automatic braking Automatic braking 8
Multi-objects Detection & Tracking PreVent EU project, Versailles demo 2007 (Daimler-Chrysler & Ibeo test vehicle) Mercedes E-Class 350 Grid-Based approach Multiple Hypotheses & Interacting Multiple Models Computational time ~ 10 ms Application: Pre-fire & Braking Pre-fire & Braking Sensors: Two short range radars Two short range radars A laser scanner ALASCA A laser scanner ALASCA Appearance & Dynamic models + Fusion Actuators: Electrical belt pre-tensioning Electrical belt pre-tensioning Automatic braking Automatic braking Next step Reducing false positives & negatives using 9
Robust Perception Dealing with uncertainty Bayesian Occupation Filter paradigm (BOF) Patented by INRIA & Probayes, Commercialized by Probayes BOF Continuous Dynamic environment modelling Grid approach based on Bayesian Filtering Estimates Probability of Occupation & Velocity of each cell in a 4D-grid Application to Obstacle Detection & Tracking + Dynamic Scene Interpretation => More robust to Sensing Errors & Temporary Occultation Successfully tested in real traffic conditions using industrial dataset (e.g. Toyota, Denso, ANR LoVe) Occupancy grid Unobservable space Concealed space ( shadow of the obstacle) Prediction Free space Sensed moving obstacle P( [O c =occ] z c) c = [x, y, 0, 0] and z=(5, 2, 0, 0) Occupied space Estimation [Coué & Laugier IJRR 05]
Robust Perception Dealing with Temporary Occultation Tracking + Conservative anticipation [Coué & al IJRR 05] Autonomous Vehicle Parked Vehicle (occultation) Description Specification Variables : - V k, V k-1 : controlled velocities - Z 0:k : sensor observations - G k : occupancy grid Decomposition : Question Parametric forms : P( G k Z 0:k ) : BOF estimation Inference P( V k V k-1 G k ) : Given or learned Thanks to the prediction capability of the BOF, the Autonomous Vehicle anticipates the behavior of the pedestrian and brakes (even if the pedestrian is temporarily hidden by the parked vehicle) 11
Robust Perception Dealing with Temporary Occultation Tracking + Conservative anticipation [Coué & al IJRR 05] Autonomous Vehicle Parked Vehicle (occultation) Description Specification Variables : SOC - V k implementation, V k-1 & Generic Sensor : controlled velocities - Z 0:k : sensor observations - G k : occupancy grid Decomposition : Next step Fusion Question Parametric forms : P( G k Z 0:k ) : BOF estimation Inference P( V k V k-1 G k ) : Given or learned Thanks to the prediction capability of the BOF, the Autonomous Vehicle anticipates the behavior of the pedestrian and brakes (even if the pedestrian is temporarily hidden by the parked vehicle) 12
Specific detectors & Sensor fusion Pedestrian detector based on Vision & Laser fusion (ANR LoVe, Vislab) ROI Courtesy of A. Broggi (Vislab, Parma University) Technology appearing soon on the market (Volvo)! In 2010, the Volvo S60 will be equipped with automatic braking system for avoiding collisions with pedestrians (below 25km/h) Pedestrian detection is realized by fusing camera and radar data 13
Prediction & Collision Risk Assessment Current world state? Next state? Existing TTC-based crash warning assumes that motion is linear Knowing instantaneous Position & Velocity of obstacles is not sufficient for risk estimation! Consistent Prediction & Risk Assessment also require to reason about Obstacles behaviors (e.g. turning, overtaking...) and Road geometry (e.g. lanes, curves, intersections using GIS) 14
Learn & Predict paradigm Observe & Learn typical motions Continuous Learn & Predict Learn => GHMM & Topological maps (SON) Predict => Exact inference, linear complexity [[Vasquez & Laugier & 06-09]] Experiments using Leeds parking data 15
Ov ertaking TurningLeft TurningRight ContinuingStraightAhead Behaviour Probability + Ov ertaking TurningLeft TurningRight ContinuingStraightAhead Behaviour Probability Probabilistic Collision Risk Patent Inria & Toyota Own vehicle Risk estimation (Gaussian Process) Cooperation INRIA &Toyota & Probayes Own vehicle High-level Behavior prediction for other vehicles (Observations + HMM) + An other vehicle Behavior Prediction (HMM) Observations + 0,4 0,3 0,2 0,1 0 Prediction Behavior models Behavior belief table 0,6 0,5 Risk Assessment (GP) 0,6 0,5 0,4 0,3 0,2 0,1 0 Behavior belief table for each vehicle in the scene Evaluation Road geometry (GIS) + Own vehicle trajectory to evaluate Christian LAUGIER Keynote FSR 09, Boston 16 Collision probability for own vehicle
Monitoring Driver Actions & Intentions Driver inattention When necessary, bring back the driver to the Attentive state Distribution of driver attention status Distraction (visual, auditory, cognitive ) Fatigue (physical, nervous, mental ) Current methods to detect driver inattention Behavior signal processing Speed signal Seat pressure Steering movements Pedal signal Head /Eye Lane position Visual analysis Courtesy Zhencheng James HU, Kumamoto 17 Univ
ICT-Car Car : ICT technologies for next car generation CFP France-Japan ANR-JST Context Previous collaboration on ITS (ICT-Asia projects FACT & CityHome) Japanese Co-mobility project (Keio) Partner Know How (scientific results, patents, industrial results, previous ANR projects) Complementary technical expertise of the French & Japanese partners France : Robust perception, Scene understanding, Risk assessment, Driver model & Learning Japan : Car-Driver interactions, Electric vehicle, Social aspect, Driving simulators France + Japan : V2V and I2V communications (IPV6) Key ideas Focus : Safety & Car personalization & Eco-driving Approach : Fundamental & Applied research, with some expected outputs to industry Main topics Robust & Efficient multi-modal Perception Continuous traffic scene understanding & Risk assessment) Cooperative driving & Communications & Eco driving Driver-Car interaction
ICT Car : ICT technologies for next car generation Main topics & Work-packages Project coordination WP0 Project management WP1 Specification and validation WP2 Environment perception WP3 Continuous traffic scene and driving understanding WP4 Cooperative driving WP5 Car-driver interactions T0.1 Project monitoring T1.1 Use cases definition T2.1 Perception by vision sensors T3.1 Data fusion and integrity issues T4.1 Telecommunications for driving purpose T5.1 Ergonomics and HMI design T0.2 Scientific and technical management T1.2 Sensor & data specification T2.2 Perception by telemetric sensors T3.2 Traffic scene understanding T4.2 Localization T5.2 Car personalization for the usual driver T0.2 Dissemination & communication T1.3 Software specification T2.2 Perception of egomotion T3.3 Driving behavior understanding T4.3 Cooperation for better scene understanding T5.3 Car-driver interactions for safety T1.4 Hardware specification T2.4 Hardware implementation T3.4 Risk assessment T4.4 The sensor-vehicle concept T5.4 Car-driver interactions for eco-driving T1.5 Validation
ICT-Car Car : ICT technologies for next car generation Partnership Academic partners France INRIA (e-motion + Imara) CNRS (Lasmea + Heudiasyc) Japan Kumamoto Univ (Z.J. Hu) Keio Univ (H. Kawashima, Jun Murai?) Tokyo Univ (Yoshio Mita) Contacted industrial partners France Probayes (start-up INRIA, ProBT & BOF libraries) Renault (C. Balle) Japan Nissan (N. Kishi) Renesas Technology s Automobile (SOC division, 80% domestic market in car navigation units)
ICT-Car Car : ICT technologies for next car generation Proposal preparation France leader : INRIA (C. Laugier) Japan leader : Kumamato Univ (Zhencheng James Hu) First partial draft sent to the partners on November 16 State of the Proposal (partnership) INRIA (e-motion & Imara) => OK CNRS (Lasmea & Heudiasyc) => OK Kumamoto Univ => OK Keio Univ & Tokyo Univ => OK Probayes => OK State of the proposal Renault => Interested, final decision next week Nissan => Interested, final decision next week Renesas Technology s Automobile => OK
Thank You! Any questions? http://emotion.inrialpes.fr/laugier christian.laugier@inrialpes.fr 22