AGENT-BASED MODELING, SIMULATION, AND CONTROL SOME APPLICATIONS IN TRANSPORTATION

Transcription

1 AGENT-BASED MODELING, SIMULATION, AND CONTROL SOME APPLICATIONS IN TRANSPORTATION Montasir Abbas, Virginia Tech (with contributions from past and present VT-SCORES students, including: Zain Adam, Sahar Ghanipoor-Machiani, Linsen Chong, and Milos Mladenovic) Workshop III: Traffic Control New Directions in Mathematical Approaches for Traffic Flow Management IPAM October 27,

2 Presentation Outline Agent based modeling what? why? And how? What is the learning framework? What are the techniques? Examples of learning: Controller agents Driver behavior agents Vehicle agents What if we don t incorporate learning? Conclusions 2

3 3 Background

4 Learning Can we predict a condition or a behavior/response from a wealth of data? Can we model and interpret a phenomenon in a state-action framework? The same input data can lead to different performance measures, and we are the reason! 4

5 Motivation Varying Traffic Behavior Maneuvers Naturalistic Data Detailed Behavioral Data Trajectories B C A 45 t= 4 sec 35 D E Y (m) t= 2 sec 5 t= 0 sec (reference time) X (m) F G H I VISSIM Simulation Advanced VISSIM API- Agent Interface Trained Agents 5

6 A Learning Framework State S Diagram Policy P State 1 State 2 State 3 State Other states State 6 State 5 State 4 Action 6

7 Learning Techniques State S Diagram Policy P Machine Learning Q-Learning State 2 State 6 State 1 Other states State 5 State 3 State 4 State Action Reinforcement Learning Etc. 7

8 Q-Learning Acting on environment, receiving rewards, selecting actions to reach a goal 8

9 Application: Dilemma Zone Problem Application of learning to controller and to humans Controllers making decisions Humans learning from mistakes 9

10 To stop or not to stop? That is the question! 10

11 11 To stop

12 12 To go

13 13 Controller Agent Learning the Policy

14 Environment s State Variables: Total number of vehicles in DZ Agent s Actions: - End the Green - Extend the Green Reward: Vehicles caught in DZ Q-learning algorithm parameters: Learning rate: 0.01 Discount rate:

15 Off-line and Online Learning Find P* with simulation Update Q-table with real data Markovian Traffic State Estimation S e Time to max-out (sec) 15

16 Human Learning Model Brain Analogy Semantic Memory Trained Q table Procedural Memory State Q table E-Greedy Action (stop/go) Updated Q table Dataset Memory Decay Propensity Episodic Memory Distractions Working Memory Emotions

17 Dealing with High State Dimensionality (Naturalistic driving behavior study)* Training input: traffic states and actions Training output: acceleration and steering Input variables discretized using fuzzy sets Continuous actions are generated from discrete actions Uses all the safety critical events available in training *Safety and Mobility Agent-based Reinforcement-learning Traffic Simulation Add-on Module (SMART SAM) 17

18 NFACRL Framework S i =the i th input variable (state variable) K =number of input variables NM i =number of fuzzy sets or membership functions for the S i M i a(i) =a(i) th fuzzy set or membership function for the i th input variable R j =the j th fuzzy rule N=number of fuzzy rules λ j =weight between j th fuzzy rule and critic w q j =weight between j th fuzzy rule and action q V =critic value A q =output of q th action Where i = 1, K, a i = 1,.. NM i, j = 1,, N and q = 1,.., P 18

19 Applications and Cross Validation Test the heterogeneity of the drivers Training: Used the data from Agent A in training with its behavioral rules as output Validation: Used the output rule of Agent A and applied it to driver B Heterogeneity of Agent A, B is represented by degree of accuracy in validation 19

20 Agent A: Event Longitidinal Action Estimation Naturalistic Agent Lateral Action Estimation Naturalistic Agent Acceleration (g) Yaw Angle (radius) Time (0.1s) Time (0.1s) 20

21 Agent A: Event Longitidinal Action Estimation Naturalistic Agent Lateral Action Estimation Naturalistic Agent 0.02 Acceleration (g) Yaw Angle (radius) Time (0.1s) Time (0.1s) 21

22 Driver Agent B Longitidinal Action Estimation Naturalistic Agent Lateral Action Estimation Naturalistic Agent Acceleration (g) Yaw Angle (radius) Time (0.1s) Time (0.1s) 22

23 Driver Agent A: Own Behavior Longitidinal Action Estimation Naturalistic Agent Lateral Action Estimation Naturalistic Agent 0.03 Acceleration (g) Yaw Angle (radius) Time (0.1s) Time (0.1s) 23

24 Driver B: Own Behavior Longitidinal Action Estimation Naturalistic Agent Lateral Action Estimation Naturalistic Agent Acceleration (g) Yaw Angle (radius) Time (0.1s) Time (0.1s) 24

25 Driver A: Using Behavior from B Longitidinal Action Estimation Naturalistic Agent Lateral Action Estimation Naturalistic Agent 0.03 Acceleration (g) Yaw Angle (radius) Time (0.1s) Time (0.1s) 25

26 Driver B: Using Behavior from A Heterogeneity is clear Longitidinal Action Estimation Naturalistic Agent Lateral Action Estimation Naturalistic Agent Acceleration (g) Yaw Angle (radius) Time (0.1s) Time (0.1s) 26

27 Mega-Agent Behavior Mega-Agent behaves as Driver B Longitidinal Action Estimation Naturalistic Agent Lateral Action Estimation Naturalistic Agent Acceleration (g) Yaw Angle (radius) Time (0.1s) Time (0.1s) 27

28 Comparison of Mega-Agent to Cross Validation Result Degree of accuracy: R square Event Agent A Agent B Mega long lat long lat long lat Event A Event B

29 But Why NOT Statistical Modeling? Would lead to wrong conclusions! 29

30 Future CV/AV Applications Multi-modal applications: modeling, simulation, and optimization Accounting for different priorities, including emergency vehicles Utilization of the computing capabilities of CV/AV Linking arterial control to freeway management scenarios Characterizing and changing network performance 30

31 Performance Measures Multi-agent System Framework Vehicle Agents User and System Requirements User-Controlled AI-Controlled High-priority Token-based PL selection system AI PL selection system based on performance Pre-set PL based on vehicle type ABM system System Configuration and ABMS Rules Reservation Matrix Revocation-enabled FIFO Trajectory Adjustment Fuel and Emission Optimization Road and Vehicle Characteristics Microscopic simulation framework for system evaluation 31

32 Multi-agent System Framework Distance Time t1, a1, t2, a2 PI RD State Here I Am RD Required Delay for a vehicle after arriving at the intersection until higher priority vehicles clear all conflict tiles Speed t1 a 1 t 2 a 2 Time Time Rather than driving with constant speed, come to a complete stop for a duration of RD before resuming speed, a vehicle follows a modified trajectory to delay its arrival by RD 32

33 33

34 34 Negotiating an Intersection

35 Distance Distance Experiment Setup Simulating high and low priority levels in some approaches Tabulated delay values and vehicle trajectories for different approaches Time-Space Diagram for Phase 2 Time-Space Diagram for Phase Time Time

36 Experiment Results 36 Agents adapt by forming dense platoons to pass through large gaps more efficiently Interesting emergent behavior can be observed from simple interaction rules Low priority agents are sensitive to traffic demand level Frequent EV calls re-synch the EV approach Phase % EV, Ph2 Scenario PL

37 Concluding Remarks Intelligent agents can capture individual learning, and agent-based modeling can capture the emerging system behavior Think state-action framework it can explain a lot of things Win the chess game, not just the next move 37