Reinforcement Learning and Approximate Dynamic Programming for Feedback Control
IEEE Press Series on Computational Intelligence Series

Reinforcement learning (RL) and adaptive dynamic programming (ADP) has been one of the most critical research fields in science and engineering for modern complex systems. This book describes the latest RL and ADP techniques for decision and control in human engineered systems, covering both single player decision and control and multi-player games. Edited by the pioneers of RL and ADP research, the book brings together ideas and methods from many fields and provides an important and timely guidance on controlling a wide variety of systems, such as robots, industrial processes, and economic decision-making.

PREFACE xix

CONTRIBUTORS xxiii

PART I FEEDBACK CONTROL USING RL AND ADP

1. Reinforcement Learning and Approximate Dynamic Programming (RLADP)—Foundations, Common Misconceptions, and the Challenges Ahead 3
Paul J. Werbos

1.1 Introduction 3

1.2 What is RLADP? 4

1.3 Some Basic Challenges in Implementing ADP 14

2. Stable Adaptive Neural Control of Partially Observable Dynamic Systems 31
J. Nate Knight and Charles W. Anderson

2.1 Introduction 31

2.2 Background 32

2.3 Stability Bias 35

2.4 Example Application 38

3. Optimal Control of Unknown Nonlinear Discrete-Time Systems Using the Iterative Globalized Dual Heuristic Programming Algorithm 52
Derong Liu and Ding Wang

3.1 Background Material 53

3.2 Neuro-Optimal Control Scheme Based on the Iterative ADP Algorithm 55

3.3 Generalization 67

3.4 Simulation Studies 68

3.5 Summary 74

4. Learning and Optimization in Hierarchical Adaptive Critic Design 78
Haibo He, Zhen Ni, and Dongbin Zhao

4.1 Introduction 78

4.2 Hierarchical ADP Architecture with Multiple-Goal Representation 80

4.3 Case Study: The Ball-and-Beam System 87

4.4 Conclusions and Future Work 94

5. Single Network Adaptive Critics Networks—Development, Analysis, and Applications 98
Jie Ding, Ali Heydari, and S.N. Balakrishnan

5.1 Introduction 98

5.2 Approximate Dynamic Programing 100

5.3 SNAC 102

5.4 J-SNAC 104

5.5 Finite-SNAC 108

5.6 Conclusions 116

6. Linearly Solvable Optimal Control 119
K. Dvijotham and E. Todorov

6.1 Introduction 119

6.2 Linearly Solvable Optimal Control Problems 123

6.3 Extension to Risk-Sensitive Control and Game Theory 130

6.4 Properties and Algorithms 134

6.5 Conclusions and Future Work 139

7. Approximating Optimal Control with Value Gradient Learning 142
Michael Fairbank, Danil Prokhorov, and Eduardo Alonso

7.1 Introduction 142

7.2 Value Gradient Learning and BPTT Algorithms 144

7.3 A Convergence Proof for VGL(1) for Control with Function Approximation 148

7.4 Vertical Lander Experiment 154

7.5 Conclusions 159

8. A Constrained Backpropagation Approach to Function Approximation and Approximate Dynamic Programming 162
Silvia Ferrari, Keith Rudd, and Gianluca Di Muro

8.1 Background 163

8.2 Constrained Backpropagation (CPROP) Approach 163

8.3 Solution of Partial Differential Equations in Nonstationary Environments 170

8.4 Preserving Prior Knowledge in Exploratory Adaptive Critic Designs 174

8.5 Summary 179

9. Toward Design of Nonlinear ADP Learning Controllers with Performance Assurance 182
Jennie Si, Lei Yang, Chao Lu, Kostas S. Tsakalis, and Armando A. Rodriguez

9.1 Introduction 183

9.2 Direct Heuristic Dynamic Programming 184

9.3 A Control Theoretic View on the Direct HDP 186

9.4 Direct HDP Design with Improved Performance Case 1—Design Guided by a Priori LQR Information 193

9.5 Direct HDP Design with Improved Performance Case 2—Direct HDP for Coorindated Damping Control of Low-Frequency Oscillation 198

9.6 Summary 201

10. Reinforcement Learning Control with Time-Dependent Agent Dynamics 203
Kenton Kirkpatrick and John Valasek

10.1 Introduction 203

10.2 Q-Learning 205

10.3 Sampled Data Q-Learning 209

10.4 System Dynamics Approximation 213

10.5 Closing Remarks 218

11. Online Optimal Control of Nonaffine Nonlinear Discrete-Time Systems without Using Value and Policy Iterations 221
Hassan Zargarzadeh, Qinmin Yang, and S. Jagannathan

11.1 Introduction 221

11.2 Background 224

11.3 Reinforcement Learning Based Control 225

11.4 Time-Based Adaptive Dynamic Programming-Based Optimal Control 234

11.5 Simulation Result 247

12. An Actor–Critic–Identifier Architecture for Adaptive Approximate Optimal Control 258
S. Bhasin, R. Kamalapurkar, M. Johnson, K.G. Vamvoudakis, F.L. Lewis, and W.E. Dixon

12.1 Introduction 259

12.2 Actor–Critic–Identifier Architecture for HJB Approximation 260

12.3 Actor–Critic Design 263

12.4 Identifier Design 264

12.5 Convergence and Stability Analysis 270

12.6 Simulation 274

12.7 Conclusion 275

13. Robust Adaptive Dynamic Programming 281
Yu Jiang and Zhong-Ping Jiang

13.1 Introduction 281

13.2 Optimality Versus Robustness 283

13.3 Robust-ADP Design for Disturbance Attenuation 288

13.4 Robust-ADP for Partial-State Feedback Control 292

13.5 Applications 296

13.6 Summary 300

PART II LEARNING AND CONTROL IN MULTIAGENT GAMES

14. Hybrid Learning in Stochastic Games and Its Application in Network Security 305
Quanyan Zhu, Hamidou Tembine, and Tamer Basar

14.1 Introduction 305

14.2 Two-Person Game 308

14.3 Learning in NZSGs 310

14.4 Main Results 314

14.5 Security Application 322

14.6 Conclusions and Future Works 326

15. Integral Reinforcement Learning for Online Computation of Nash Strategies of Nonzero-Sum Differential Games 330
Draguna Vrabie and F.L. Lewis

15.1 Introduction 331

15.2 Two-Player Games and Integral Reinforcement Learning 333

15.3 Continuous-Time Value Iteration to Solve the Riccati Equation 337

15.4 Online Algorithm to Solve Nonzero-Sum Games 339

15.5 Analysis of the Online Learning Algorithm for NZS Games 342

15.6 Simulation Result for the Online Game Algorithm 345

15.7 Conclusion 347

16. Online Learning Algorithms for Optimal Control and Dynamic Games 350
Kyriakos G. Vamvoudakis and Frank L. Lewis

16.1 Introduction 350

16.2 Optimal Control and the Continuous Time Hamilton–Jacobi–Bellman Equation 352

16.3 Online Solution of Nonlinear Two-Player Zero-Sum Games and Hamilton–Jacobi–Isaacs Equation 360

16.4 Online Solution of Nonlinear Nonzero-Sum Games and Coupled Hamilton–Jacobi Equations 366

PART III FOUNDATIONS IN MDP AND RL

17. Lambda-Policy Iteration: A Review and a New Implementation 381
Dimitri P. Bertsekas

17.1 Introduction 381

17.2 Lambda-Policy Iteration without Cost Function Approximation 386

17.3 Approximate Policy Evaluation Using Projected Equations 388

17.4 Lambda-Policy Iteration with Cost Function Approximation 395

17.5 Conclusions 406

18. Optimal Learning and Approximate Dynamic Programming 410
Warren B. Powell and Ilya O. Ryzhov

18.1 Introduction 410

18.2 Modeling 411

18.3 The Four Classes of Policies 412

18.4 Basic Learning Policies for Policy Search 416

18.5 Optimal Learning Policies for Policy Search 421

18.6 Learning with a Physical State 427

19. An Introduction to Event-Based Optimization: Theory and Applications 432
Xi-Ren Cao, Yanjia Zhao, Qing-Shan Jia, and Qianchuan Zhao

19.1 Introduction 432

19.2 Literature Review 433

19.3 Problem Formulation 434

19.4 Policy Iteration for EBO 435

19.5 Example: Material Handling Problem 441

19.6 Conclusions 448

20. Bounds for Markov Decision Processes 452
Vijay V. Desai, Vivek F. Farias, and Ciamac C. Moallemi

20.1 Introduction 452

20.2 Problem Formulation 455

20.3 The Linear Programming Approach 456

20.4 The Martingale Duality Approach 458

20.5 The Pathwise Optimization Method 461

20.6 Applications 463

20.7 Conclusion 470

21. Approximate Dynamic Programming and Backpropagation on Timescales 474
John Seiffertt and Donald Wunsch

21.1 Introduction: Timescales Fundamentals 474

21.2 Dynamic Programming 479

21.3 Backpropagation 485

21.4 Conclusions 492

22. A Survey of Optimistic Planning in Markov Decision Processes 494
Lucian Busoniu, Remi Munos, and Robert Babu¡ska

22.1 Introduction 494

22.2 Optimistic Online Optimization 497

22.3 Optimistic Planning Algorithms 500

22.4 Related Planning Algorithms 509

22.5 Numerical Example 510

23. Adaptive Feature Pursuit: Online Adaptation of Features in Reinforcement Learning 517
Shalabh Bhatnagar, Vivek S. Borkar, and L.A. Prashanth

23.1 Introduction 517

23.2 The Framework 520

23.3 The Feature Adaptation Scheme 522

23.4 Convergence Analysis 525

23.5 Application to Traffic Signal Control 527

23.6 Conclusions 532

24. Feature Selection for Neuro-Dynamic Programming 535
Dayu Huang, W. Chen, P. Mehta, S. Meyn, and A. Surana

24.1 Introduction 535

24.2 Optimality Equations 536

24.3 Neuro-Dynamic Algorithms 542

24.4 Fluid Models 551

24.5 Diffusion Models 554

24.6 Mean Field Games 556

24.7 Conclusions 557

25. Approximate Dynamic Programming for Optimizing Oil Production 560
Zheng Wen, Louis J. Durlofsky, Benjamin Van Roy, and Khalid Aziz

25.1 Introduction 560

25.2 Petroleum Reservoir Production Optimization Problem 562

25.3 Review of Dynamic Programming and Approximate Dynamic Programming 564

25.4 Approximate Dynamic Programming Algorithm for Reservoir Production Optimization 566

25.5 Simulation Results 573

25.6 Concluding Remarks 578

23.6 Conclusions 532

24. Feature Selection for Neuro-Dynamic Programming 535
Dayu Huang, W. Chen, P. Mehta, S. Meyn, and A. Surana

24.1 Introduction 535

24.2 Optimality Equations 536

24.3 Neuro-Dynamic Algorithms 542

24.4 Fluid Models 551

24.5 Diffusion Models 554

24.6 Mean Field Games 556

24.7 Conclusions 557

25. Approximate Dynamic Programming for Optimizing Oil Production 560
Zheng Wen, Louis J. Durlofsky, Benjamin Van Roy, and Khalid Aziz

25.1 Introduction 560

25.2 Petroleum Reservoir Production Optimization Problem 562

25.3 Review of Dynamic Programming and Approximate Dynamic Programming 564

25.4 Approximate Dynamic Programming Algorithm for Reservoir Production Optimization 566

25.5 Simulation Results 573

25.6 Concluding Remarks 578

26. A Learning Strategy for Source Tracking in Unstructured Environments 582
Titus Appel, Rafael Fierro, Brandon Rohrer, Ron Lumia, and John Wood

26.1 Introduction 582

26.2 Reinforcement Learning 583

26.3 Light-Following Robot 589

26.4 Simulation Results 592

26.5 Experimental Results 595

26.6 Conclusions and Future Work 599

References 599

INDEX 601

Dr. Frank Lewis is a Professor of Electrical Engineering at The University of Texas at Arlington, where he was awarded the Moncrief-O'Donnell Endowed Chair in 1990 at the Automation & Robotics Research Institute. He has served as Visiting Professor at Democritus University in Greece, Hong Kong University of Science and Technology, Chinese University of Hong Kong, City University of Hong Kong, National University of Singapore, Nanyang Technological University Singapore. Elected Guest Consulting Professor at Shanghai Jiao Tong University and South China University of Technology.

Derong Liu received the B.S. degree in mechanical engineering from the East China Institute of Technology (now Nanjing University of Science and Technology), Nanjing, China, in 1982, the M.S. degree in automatic control theory and applications from the Institute of Automation, Chinese Academy of Sciences, Beijing, China, in 1987, and the Ph.D. degree in electrical engineering from the University of Notre Dame, Notre Dame, IN, in 1994.