M. Wellman Nov complex dynamics and uncertainty

Transcription

1 Empirical Game- Theore;c Analysis for Prac;cal Strategic Reasoning Workshop on Reasoning in Adversarial and Noncoopera5ve Environments Michael P. Wellman University of Michigan Real- World Games complex dynamics and uncertainty rich strategy space strategy: obs* ;me ac;on severely incomplete informa;on interdependent types (signals) info par;ally revealed over ;me analy;c game- theore;c solu;ons few and far between two approaches 1. analyze (stylized) approxima;ons one- shot, complete info 2. empirical methods simula;on, sta;s;cs, machine learning, WRANE 1

2 Empirical Game- Theore;c Analysis (EGTA) Game described procedurally, no directly usable analy;cal form Parametrize strategy space based on agent architecture Selec;vely explore strategy/profile space Induce game model (payoff func;on) from simula;on data Empirical game EGTA Process 2. Es;mate empirical game Profile Simulator Payoff Data Empirical Game Profile Space Game Analysis (NE) 3. Solve empirical game Strategy Set 1. Parametrize strategy space WRANE 2

3 TAC Supply Chain Mgmt Game Hard Disk Memory Motherboard CPU suppliers Pintel IMD Basus Macrostar Mec Queenmax Watergate Mintor component RFQs supplier offers component orders manufacturers Manufacturer 1 Manufacturer 2 Manufacturer 3 Manufacturer 4 Manufacturer 5 Manufacturer 6 PC RFQs PC bids PC orders customer 10 component types 16 PC types 220 simulation days 15 seconds per day Two- Strategy Game (Unpreempted) WRANE 3

4 Two- Strategy Game (Unpreempted) Three- Strategy Game: Devia;ons WRANE 4

5 TAC/SCM- 06 Devia;on Graph concentric regret regions CDA Devia;on Graph [4GDX] [3GD, 1GDX] 4 strategies: GD, GDX, ZI, Kap [1GD, 3Kap] WRANE 5

6 Ranking Strategies: TAC/SCM- 07 SCM- 07 Tournament SCM- 07 EGTA from PR Jordan PhD Thesis, 2009 Strategy Ranking (TAC Travel) Strategies ranked with respect to the final equilibrium context from LJ Schvartzman PhD Thesis, 2009! $!$!' "!!" ''!!"##!!$##!!###!%##!&##!"##!$## # $## +,-./0.1234/.2 ') (# "* "$ $" ( "' ") $# "" '! "#!) * ' $( )!% '* '#!& $& " '$ $% $$ % & $) $*!* "( "& $'!# '( '& '"!( $! '%!! WRANE 6

7 DeepMaize- 08 Design Explora;on from PR Jordan PhD Thesis, 2009 JVW, AAMAS- 08 Sampling Control Problem Profile Select Profile Space Itera;ve EGTA Process Simulator Payoff Data Game Model Induc;on Problem Empirical Game Game Analysis (NE) JW, AAMAS- 09 Strategy Set Add Strategy Strategy Space More Strategies Refine? More Samples Strategy Explora;on Problem JSW, AAMAS- 10 N End WRANE 7

8 Sampling Control Problem Revealed payoff model sample provides exact payoff minimum- regret- first search (MRFS) ajempts to refute best current candidate Noisy payoff model sample drawn from payoff distribu;on informa;on gain search (IGS) sample profile maximizing entropy difference wrt probability of being min- regret profile Min- Regret- First Search c1 c2 c3 c4 Profile ε-bound (r1,c1) 0 r1 9,5 3,3 2,5 4,8 r2 6,4 8,8 3,0 5,3 r3 2,2 2,1 3,2 4,6 r4 4,4 2,0 2,2 9,3 WRANE 8

9 Min- Regret Search c1 c2 c3 c4 r1 9,5 3,3 2,5 4,8 Profile ε-bound (r1,c1) 0 (r1,c2) 2 r2 6,4 8,8 3,0 5,3 r3 2,2 2,1 3,2 4,6 r4 4,4 2,0 2,2 9,3 Select random devia;on from current best profile Min- Regret Search c1 c2 c3 c4 r1 9,5 3,3 2,5 4,8 Profile ε-bound (r1,c1) 0 (r1,c2) 2 (r2,c1) 3 r2 6,4 8,8 3,0 5,3 r3 2,2 2,1 3,2 4,6 r4 4,4 2,0 2,2 9,3 WRANE 9

10 Min- Regret Search c1 c2 c3 c4 r1 9,5 3,3 2,5 4,8 Profile ε-bound (r1,c1) 0 (r1,c2) 2 (r2,c1) 3 (r3,c1) 7 r2 6,4 8,8 3,0 5,3 r3 2,2 2,1 3,2 4,6 r4 4,4 2,0 2,2 9,3 Min- Regret Search c1 c2 c3 c4 r1 9,5 3,3 2,5 4,8 r2 6,4 8,8 3,0 5,3 Profile ε-bound (r1,c1) 3 (r1,c2) 5 (r2,c1) 3 (r3,c1) 7 (r1,c4) 0 r3 2,2 2,1 3,2 4,6 r4 4,4 2,0 2,2 9,3 WRANE 10

11 Min- Regret Search c1 c2 c3 c4 r1 9,5 3,3 2,5 4,8 r2 6,4 8,8 3,0 5,3 Profile ε-bound (r1,c1) 3 (r1,c2) 5 (r2,c1) 3 (r3,c1) 7 (r1,c4) 1 (r2,c4) 1 r3 2,2 2,1 3,2 4,6 r4 4,4 2,0 2,2 9,3 Min- Regret Search c1 c2 c3 c4 r1 9,5 3,3 2,5 4,8 r2 6,4 8,8 3,0 5,3 r3 2,2 2,1 3,2 4,6 Profile ε-bound (r1,c1) 3 (r1,c2) 5 (r2,c1) 4 (r3,c1) 7 (r1,c4) 1 (r2,c4) 5 (r2,c2) 0 r4 4,4 2,0 2,2 9,3 WRANE 11

12 Min- Regret Search c1 c2 c3 c4 r1 9,5 3,3 2,5 4,8 r2 6,4 8,8 3,0 5,3 r3 2,2 2,1 3,2 4,6 Profile ε-bound (r1,c1) 3 (r1,c2) 5 (r2,c1) 4 (r3,c1) 7 (r1,c4) 1 (r2,c4) 5 (r2,c2) 0 (r2,c3) 8 r4 4,4 2,0 2,2 9,3 Min- Regret Search c1 c2 c3 c4 r1 9,5 3,3 2,5 4,8 r2 6,4 8,8 3,0 5,3 r3 2,2 2,1 3,2 4,6 Profile ε-bound (r1,c1) 3 (r1,c2) 5 (r2,c1) 4 (r3,c1) 7 (r1,c4) 1 (r2,c4) 5 (r2,c2) 0 (r2,c3) 8 (r3,c2) 6 r4 4,4 2,0 2,2 9,3 WRANE 12

13 Min- Regret Search c1 c2 c3 c4 r1 9,5 3,3 2,5 4,8 r2 6,4 8,8 3,0 5,3 r3 2,2 2,1 3,2 4,6 r4 4,4 2,0 2,2 9,3 Profile ε-bound (r1,c1) 3 (r1,c2) 5 (r2,c1) 4 (r3,c1) 7 (r1,c4) 1 (r2,c4) 5 (r2,c2) 0* (r2,c3) 8 (r3,c2) 6 (r4,c2) 6 Finding Approximate PSNE WRANE 13

14 ... Sampling Control Problem Profile Select Profile Space Itera;ve EGTA Process Simulator Payoff Data Game Model Induc;on Problem Empirical Game Game Analysis (NE) JW, AAMAS- 09 Strategy Set Add Strategy Strategy Space More Strategies Refine? More Samples Strategy Explora;on Problem JSW, AAMAS- 10 N End Construct Empirical Game Simplest approach: direct es;ma;on employ control variates and other variance reduc;on techniques Empirical Game (s 1,u(s 1 )) (s L,u(s L ))? u( ) Payoff data from selected profiles WRANE 14

15 Payoff Func;on Regression S i = [0,1] 0 generate data (simula;ons) ,3 1,4 1,1 FPSB2 Example ,1 1,1 2,2 4,1 1,0 3,3 learn regression solve learned game eq = (0.32,0.32) Vorobeychik et al., ML 2007 Generaliza;on Risk Approach Model varia;ons func;onal forms, rela;onship structures, parameters strategy granularity Approach: Treat candidate game model as a predictor for payoff data Adopt loss func;on for predictor Select model candidate minimizing expected loss Cross Valida4on Observa5on Data Fold 1 Fold 2 Fold 3 Training Valida5on WRANE 15

16 1 Sensi;vity Analysis Frequency twostrategy mixtures [49]0.275 [50]0.725 ( , 0.00, 33.01,100.0%) [49]1.000 ( , 16.14,115.89,100.0%) Regret Bound Itera;ve EGTA Process Sampling Control Problem Profile Select Simulator Payoff Data Game Model Induc;on Problem Empirical Game Profile Space Game Analysis (NE) Strategy Set Add Strategy Strategy Space More Strategies Refine? More Samples Strategy Explora;on Problem JSW, AAMAS- 10 N End WRANE 16

17 Learning New Strategies: EGTA+RL Profile Select Simulator Payoff Data Empirical Game Profile Space Online Learning Game Analysis (NE) Strategy Set New Strategy RL: Best response to NE More Strategies Refine? More Samples Add new Strategy Y N Y Deviates? N Improve RL Model? N End CDA Learning Problem Setup History of recent trades H1: Moving average H2: Frequency weighted ra;o, threshold= V H3: Frequency weighted ra;o, threshold= A Ac;ons A: Offset from V State Space Quotes Time Pending Trades Q1: Opposite role Q2: Same role T1: Total T2: Since last trade U: Number of trades leq V: Value of next unit to be traded Rewards R: Difference between unit valua;on and trade price WRANE 17

18 EGTA/RL Round 1 Strategies Payoff NE Learning Kaplan ZI ZIbtq Strategy Dev. Payoff ZI L L L1 EGTA/RL Round 2 Strategies Payoff NE Learning Kaplan ZI ZIbtq Strategy Dev. Payoff ZI L L L1 ZIP ZIP GD GD L GD L9 L2- L8 L L WRANE 18

19 EGTA/RL Rounds 3+ Strategies Payoff NE Learning Strategy Dev. Payoff L GD L10 L L11 GDX GDX L11 L L L12 L L L13 RB L L13 L L L L Final champion Strategy Explora;on Problem Premise: Limited ability to cover profile space Expecta;on to reasonably evaluate all considered strategies Need deliberate policy to decide which strategies to introduce RL for strategy explora;on ajempt at best response to current equilibrium is this a good heuris;c (even assuming ideal BR calc?) WRANE 19

20 Example" Introduce strategies in order: A1, A2, A3, A4 Regret may increase over subsequent steps!" A1 A2 A3 A4 A1 1, 1 1, 2 1, 3 1, 4 A2 2, 1 2, 2 2, 3 2, 6 A3 3, 1 3, 2 3, 3 3, 8 A4 4, 1 6, 2 8, 3 4, 4 Strategy Set Candidate Eq. Regret wrt True Game {A1} (A1,A1) 3 {A1,A2} (A2,A2) 4 {A1,A2,A3} (A3,A3) 5 {A1,A2,A3,A4} (A4,A4) 0 FPSB2 Regret Surface 0.14 BR E(DEV) DEV [MESH] !(k j ) k j k i WRANE 20

21 Explora;on Policies RND: Random (uniform) selec;on Devia;on- Based DEV: Uniform among strategies that deviate from current equilibrium BR: Best response to current equilibrium BR+DEV: Alternate on successive itera;ons ST(τ): Soqmax selec;on among deviators, propor;onal to gain MEMT: Select strategy that maximizes the gain (regret) from devia;ng to a strategy outside the set from any mixture over the set. CDA 4" Expected Regret 10 3 MEMT DEV 10 2 RND BR 10 1 ST 10 ST ST !1 10!2 10! Step WRANE 21

22 EGTA Applica;ons Market games TAC: Travel, Supply Chain, Ad Auc;on Canonical auc;ons: SAAs, CDAs Equity premium in financial trading Networking games privacy ajacks, rou;ng, wireless access point selec;on Mechanism design Conclusion: EGTA Methodology Extends scope of GT to procedurally defined scenarios Embraces sta;s;cal underpinnings of strategic reasoning Search process: GT for establishing salient strategic context Strategy explora;on: e.g., RL to search for best response to that context Principled approach to evaluate complex strategy spaces Growing toolbox of EGTA techniques WRANE 22