Markov Chains and Stochastic Stability, 1993
Communications and Control Engineering Series

Markov Chains and Stochastic Stability is part of the Communications and Control Engineering Series (CCES) edited by Professors B.W. Dickinson, E.D. Sontag, M. Thoma, A. Fettweis, J.L. Massey and J.W. Modestino. The area of Markov chain theory and application has matured over the past 20 years into something more accessible and complete. It is of increasing interest and importance. This publication deals with the action of Markov chains on general state spaces. It discusses the theories and the use to be gained, concentrating on the areas of engineering, operations research and control theory. Throughout, the theme of stochastic stability and the search for practical methods of verifying such stability, provide a new and powerful technique. This does not only affect applications but also the development of the theory itself. The impact of the theory on specific models is discussed in detail, in order to provide examples as well as to demonstrate the importance of these models. Markov Chains and Stochastic Stability can be used as a textbook on applied Markov chain theory, provided that one concentrates on the main aspects only. It is also of benefit to graduate students with a standard background in countable space stochastic models. Finally, the book can serve as a research resource and active tool for practitioners.

I Communication and Regeneration.- 1 Heuristics.- 1.1 A Range of Markovian Environments.- 1.2 Basic Models in Practice.- 1.3 Stochastic Stability For Markov Models.- 1.4 Commentary.- 2 Markov Models.- 2.1 Markov Models in Time Series.- 2.2 Nonlinear State Space Models.- 2.3 Models in Control And Systems Theory.- 2.4 Markov Models with Regeneration Times.- 2.5 Commentary.- 3 Transition Probabilities.- 3.1 Defining a Markovian Process.- 3.2 Foundations on a Countable Space.- 3.3 Specific Transition Matrices.- 3.4 Foundations for General State Space Chains.- 3.5 Building Transition Kernels for Specific Models.- 3.6 Commentary.- 4 Irreducibility.- 4.1 Communication and Irreducibility: Countable Spaces.- 4.2 ?-Irreducibility.- 4.3 ?-Irreducibility for Random Walk Models.- 4.4 ?-Irreducible Linear Models.- 4.5 Commentary.- 5 Pseudo-atoms.- 5.1 Splitting ?-Irreducible Chains.- 5.2 Small Sets.- 5.3 Small Sets for Specific Models.- 5.4 Cyclic Behavior.- 5.5 Petite Sets and Sampled Chains.- 5.6 Commentary.- 6 Topology and Continuity.- 6.1 Feller Properties and Forms of Stability.- 6.2 T-chains.- 6.3 Continuous Components for Specific Models.- 6.4 e-Chains.- 6.5 Commentary.- 7 The Nonlinear State Space Model.- 7.1 Forward Accessibility and Continuous Components.- 7.2 Minimal Sets and Irreducibility.- 7.3 Periodicity for nonlinear state space models.- 7.4 Forward Accessible Examples.- 7.5 Equicontinuity and the nonlinear state space model.- 7.6 Commentary.- II Stability Structures.- 8 Transience and Recurrence.- 8.1 Classifying chains on countable spaces.- 8.2 Classifying ?-irreducible chains.- 8.3 Recurrence and transience relationships.- 8.4 Classification using drift criteria.- 8.5 Classifying random walk on IR+.- 8.6 Commentary.- 9 Harris and Topological Recurrence.- 9.1 Harris recurrence.- 9.2 Non-evanescent and recurrent chains.- 9.3 Topologically recurrent and transient states.- 9.4 Criteria for stability on a topological space.- 9.5 Stochastic comparison and increment analysis.- 9.6 Commentary.- 10 The Existence of ?.- 10.1 Stationarity and Invariance.- 10.2 The existence of ?: chains with atoms.- 10.3 Invariant measures: countable space models.- 10.4 The existence of ?: ?-irreducible chains.- 10.5 Invariant Measures: General Models.- 10.6 Commentary.- 11 Drift and Regularity.- 11.1 Regular chains.- 11.2 Drift, hitting times and deterministic models.- 11.3 Drift criteria for regularity.- 11.4 Using the regularity criteria.- 11.5 Evaluating non-positivity.- 11.6 Commentary.- 12 Invariance and Tightness.- 12.1 Chains bounded in probability.- 12.2 Generalized sampling and invariant measures.- 12.3 The existence of A ?-finite invariant measure.- 12.4 Invariant Measures for e-Chains..- 12.5 Establishing boundedness in probability.- 12.6 Commentary.- III Convergence.- 13 Ergodicity.- 13.1 Ergodic chains on countable spaces.- 13.2 Renewal and regeneration.- 13.3 Ergodicity of positive Harris chains.- 13.4 Sums of transition probabilities.- 13.5 Commentary.- 14 ƒ-Ergodicity and ƒ-Regularity.- 14.1 ƒ-Properties: chains with atoms.- 14.2 ƒ-Regularity and drift.- 14.3 ƒ-Ergodicity for general chains.- 14.4 ƒ-Ergodicity of specific models.- 14.5 A Key Renewal Theorem.- 14.6 Commentary.- 15 Geometric Ergodicity.- 15.1 Geometric properties: chains with atoms.- 15.2 Kendall sets and drift criteria.- 15.3 ƒ-Geometric regularity of ?and ?n.- 15.4 ƒ-Geometric ergodicity for general chains.- 15.5 Simple random walk and linear models.- 15.6 Commentary.- 16 V-Uniform Ergodicity.- 16.1 Operator norm convergence.- 16.2 Uniform ergodicity.- 16.3 Geometric ergodicity and increment analysis.- 16.4 Models from queueing theory.- 16.5 Autoregressive and state space models.- 16.6 Commentary.- 17 Sample Paths and Limit Theorems.- 17.1 Invariant ?-Fields and the LLN.- 17.2 Ergodic Theorems for Chains Possessing an Atom.- 17.3 General Harris Chains.- 17.4 The Functional CLT.- 17.5 Criteria for the CLT and the LIL.- 17.6 Applications.- 17.7 Commentary.- 18 Positivity.- 18.1 Null recurrent chains.- 18.2 Characterizing positivity using Pn.- 18.3 Positivity and T-chains.- 18.4 Positivity and e-Chains.- 18.5 The LLN for e-Chains.- 18.6 Commentary.- 19 Generalized Classification Criteria.- 19.1 State-dependent drifts.- 19.2 History-dependent drift criteria.- 19.3 Mixed drift conditions.- 19.4 Commentary.- IV Appendices.- A Mud Maps.- A.l Recurrence versus transience.- A.2 Positivity versus nullity.- A.3 Convergence Properties.- B Testing for Stability.- B.l A Glossary of Drift Conditions.- B.2 The scalar SETAR Model: a complete classification.- C A Glossary of Model Assumptions..- C.l Regenerative Models.- C.2 State Space Models.- D Some Mathematical Background.- D.l Some Measure Theory.- D.2 Some Probability Theory.- D.3 Some Topology.- D.4 Some Real Analysis.- D.5 Some Convergence Concepts for Measures.- D.6 Some Martingale Theory.- D.7 Some Results on Sequences and Numbers.- References.- Symbols Index.