Elements of Applied Stochastic Processes Third Edition U. NARAYAN BHAT Southern Methodist University GREGORY K. MILLER Stephen E Austin State University,WILEY- INTERSCIENCE A JOHN WILEY & SONS, INC., PUBLICATION
Contents Preface ix 1 STOCHASTIC PROCESSES: Description and Definition 1 1.1 Introduction 1 1.2 Description and Definition 5 1.3 Probability Distributions 8 1.4 The Markov Process 9 1.5 The Renewal Process 12 1.6 The Stationary Process 13 1.7 A Plan for the Remaining Chapters 14 References 14 Exercises 16 Elementary Review Exercises 18 Advanced Review Exercises 22 2 MARKOV CHAINS 27 2.1 Introduction 27 2.2 The n-step Transition Probability Matrix 31 2.3 Classification of States ' 32 2.4 A Canonical Representation of the Transition Probability Matrix 40 2.5 Classification of States in Practice 43 2.6 Finite Markov Chains with Transient States 45
vl References 54 Exercises 54 3 IRREDUCIBLE MARKOV CHAINS WITH ERGODIC STATES 63 3.1 Transient Behavior 63 3.2 Limiting Behavior 69 3.3 First Passage and Related Results 86 References 96 Exercises 97 4 BRANCHING PROCESSES AND OTHER SPECIAL TOPICS 111 4.1 Branching Processes 111 4.2 Markov Chains of Order Higher than 1 116 4.3 Lumpable Markov Chains 118 4.4 Reversed Markov Chains 123 References 125 Exercises ; ' 125 5 STATISTICAL INFERENCE FOR MARKOV CHAINS 129 5.1 Estimation of the Elements in a Transition Probability Matrix 129 5.2 Hypothesis Testing Issues for Markov Chains 133 5.3 Inference From Partially Observable Markov Chains 141 5.4 Statistical Inference for Branching Processes 144 5.5 Additional Comments 146 References 147 Exercises 149 6 APPLIED MARKOV CHAINS 157 6.1 Queueing Models 157 6.2 Inventory Systems 167 6.3 Storage Models 169 6.4 Industrial Mobility of Labor 172 6.5 Educational Advancement 176 6.6 Human Resource Management 179 6.7 Term Structure 181 6.8 Income Determination under Uncertainty 184 6.9 A Markov Decision Process 185 References 190 7 SIMPLE MARKOV PROCESSES ' 193 7.1 Examples 193 7.2 Markov Processes: General Properties 195 7.3 The Poisson Process 200 7.4 The Pure Birth Process 209
Vii 7.5 The Pure Death Process 212 7.6 Birth and Death Processes 213 7.7 Limiting Distributions 221 7.8 Markovian Networks 224 7.9 Additional Examples 231 References 235 Exercises 235 8 STATISTICAL INFERENCE FOR SIMPLE MARKOV PROCESSES 247 8.1 Estimation of Parameters 247 8.2 Hypothesis Testing for Simple Markov Processes 254 8.3 Statistical Inference for Queues 256 8.4 Additional Examples 263 References 266 Exercises 268 9 APPLIED MARKOV PROCESSES 271 9.1 Queueing Models jl 271 9.2 The Machine Interference Problem 277 9.3 Queueing Networks 278 9.4 Flexible Manufacturing Systems 285 9.5 Inventory Systems 287 9.6 Reliability Models 293 9.7 Markovian Combat Models 304 9.8 Stochastic Models for Social Networks 317 9.9 Recovery, Relapse, and Death Due to Disease 320 References 323 10 RENEWAL PROCESSES 327 10.1 Introduction 327 10.2 Renewal Processes when Time is Discrete 328 10.3 Renewal Processes when Time is Continuous 335 10.4 Alternating Renewal Processes 344 10.5 Markov Renewal Processes (Semi-Markov Processes) 346 10.6 Renewal Reward Processes 350 10.7 Statistical Inference for Renewal Processes 351 10.8 Additional Examples 353 References 357 Exercises 358 11 STATIONARY PROCESSES AND TIME SERIES ANALYSIS 365 11.1 Definition 365 11.2 Some Examples 370 11.3 Ergodic Theorems 378
Viii 11.4 Covariance Stationary Processes in the Frequency Domain 379 11.5 Time Series Analysis: Introduction 388 11.6 Stochastic Models for Time Series 391 11.7 The Autoregressive Process 396 11.8 The Moving Average Process 401 11.9 A Mixed Autoregressive Moving Average Process 403 11.10 Autoregressive Integrated Moving Average Processes 405 11.11 Time Series Analysis in the Time Domain 408 11.12 Spectral Analysis of Time Series Data 413 References 417 Exercises 418 12 SIMULATION AND MARKOV CHAIN MONTE CARLO 421 12.1 Introduction 421 12.2 Simulation 422 12.3 Markov Chain Monte Carlo 426 References y 430 Answers to Selected Exercises 433 Appendix 443 Author Index 451 Subject Index 455