RL & Safe RL
A structured path from reinforcement-learning foundations to safety under nonstationarity and adaptive AI safety.
How to Use This Guide
This is a curated learning path for students and researchers moving from reinforcement-learning foundations toward safe reinforcement learning, continual adaptation, and safety under nonstationarity. It is not an exhaustive bibliography: the emphasis is on what to read, why it matters, and where it fits.
Where a strong overview exists, each stage begins with Start with a survey. The remaining readings are divided into essential works and deeper or more specialized material. Books are collected separately because they are best used as durable foundations and references rather than read in the same way as research papers.
Recommended Books
Core texts and references
Sutton & Barto — Reinforcement Learning: An Introduction, 2nd ed.
The standard starting point for RL: MDPs, value functions, dynamic programming, temporal-difference learning, policy methods, and function approximation.
Eitan Altman — Constrained Markov Decision Processes
The mathematical foundation for constrained sequential decision-making and an important bridge from classical MDPs to constrained and safe RL.
Martin L. Puterman — Markov Decision Processes: Discrete Stochastic Dynamic Programming
A deeper theoretical reference for Markov decision processes, optimality, policies, and dynamic programming.
Dimitri P. Bertsekas — Dynamic Programming and Optimal Control
A rigorous bridge among sequential optimization, control, dynamic programming, and ideas underlying reinforcement learning.
Bertsekas & Tsitsiklis — Neuro-Dynamic Programming
A classic treatment of approximation, dynamic programming, and learning that helps connect classical control ideas to modern RL.
Boyd & Vandenberghe — Convex Optimization
A useful mathematical reference for constrained optimization, duality, and optimization concepts that recur throughout constrained policy learning.
The labels “read substantially,” “read relevant chapters,” and “reference” are intentional: the goal is not to require every book to be read cover-to-cover.
Foundations of Reinforcement Learning
Build the vocabulary needed for everything that follows: MDPs, value functions, Bellman equations, temporal-difference learning, policy optimization, deep RL, and probabilistic views of control.
Begin with Sutton & Barto
Use the core book above as the principal orientation source for this section; then move to representative deep-RL and policy-optimization papers.
Mnih et al. — Human-level Control through Deep Reinforcement Learning
The DQN paper: a natural bridge from classical value-based RL to deep function approximation.
Schulman et al. — Proximal Policy Optimization Algorithms
A compact introduction to one of the most influential modern policy-optimization approaches.
Haarnoja et al. — Soft Actor-Critic
An important off-policy actor-critic method built around maximum-entropy reinforcement learning.
Levine — Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
A valuable alternative perspective connecting reinforcement learning, control, and probabilistic inference.
From RL to Safe RL
Introduce the central change in perspective: reward maximization is no longer sufficient when behavior must also satisfy explicit safety requirements.
García & Fernández — A Comprehensive Survey on Safe Reinforcement Learning
A foundational taxonomy of Safe RL and a strong first map of the field.
Gu et al. — modern review of Safe Reinforcement Learning
A newer complement emphasizing major Safe-RL problems, methods, theory, and practical applications.
Achiam et al. — Constrained Policy Optimization
A central paper for understanding policy optimization subject to safety constraints.
Perkins & Barto — Lyapunov Design for Safe Reinforcement Learning
An early and influential connection between Lyapunov reasoning and safety in reinforcement learning.
Dalal et al. — Safe Exploration in Continuous Action Spaces
Introduces a safety-layer perspective for enforcing constraints while an RL agent explores.
Wachi & Sui — Safe RL in Constrained Markov Decision Processes
A useful CMDP-centered treatment connecting formal constraints to learning.
Safe Reinforcement Learning
Go deeper into how safety is represented and enforced: constraint formulations, policy optimization, safe exploration, shielding, runtime intervention, and control-theoretic approaches.
Wachi et al. — survey of constraint formulations in Safe RL
Particularly useful before comparing algorithms because it focuses attention on a prior question: what exactly is the safety constraint?
Liu et al. — survey of constrained policy learning in model-free RL
A focused orientation to constrained policy-optimization methods.
Brunke et al. — Safe Learning in Robotics
A complementary control and robotics perspective on learning-based safety.
Berkenkamp et al. — Safe Model-based Reinforcement Learning with Stability Guarantees
Representative work on learning while explicitly restricting exploration to regions considered safe.
Chow et al. — Lyapunov-based Safe Policy Optimization
Extends Lyapunov ideas into modern constrained policy optimization.
Feasible Actor-Critic — constrained RL for state-wise safety
Useful for distinguishing cumulative constraints from safety requirements that must hold at individual states or decisions.
Carr et al. — shielding under partial observability
Connects runtime shielding with uncertainty about the true system state.
Khattar et al. — meta-safe reinforcement learning
A bridge toward adaptation: how safety can be considered when an agent must adapt across tasks.
Continual RL & Nonstationarity
Temporarily separate adaptation from safety. First understand what changing tasks, dynamics, observations, and contexts do to a learning system.
Padakandla — survey of RL algorithms for dynamically varying environments
An orientation to reinforcement learning when the environment itself changes over time.
Khetarpal et al. — Towards Continual Reinforcement Learning: A Review and Perspectives
A broad conceptual map of continual reinforcement learning and the challenges of persistent adaptation.
Gama et al. — A Survey on Concept Drift Adaptation
Broader than RL, but valuable for understanding nonstationarity and changing data-generating processes.
Al-Shedivat et al. — Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments
Shows how meta-learning can support ongoing adaptation in changing environments.
Rakelly et al. — PEARL
Probabilistic context variables provide an important conceptual bridge to context inference and adaptation.
Nagabandi et al. — meta-RL for adaptation in dynamic environments
Useful for understanding adaptation of learned dynamics and control under environmental change.
Nagabandi, Finn & Levine — continual adaptation for model-based RL
Connects online learning, meta-learning, and persistent model adaptation.
Eysenbach et al. — off-dynamics reinforcement learning
A useful perspective on transfer when deployment dynamics differ from training dynamics.
Safety under Nonstationarity
Bring the two threads together. The key question is no longer only whether the policy can adapt, but whether the safety specification and safety mechanism remain valid while adaptation occurs.
Tomashevskiy — Safe Continual Reinforcement Learning Methods for Nonstationary Environments. Towards a Survey of the State of the Art
A synthesis of Safe RL, continual learning, and control perspectives, with particular attention to static, adaptive, and context-dependent safety constraints under changing conditions.
Chen et al. — context-aware Safe RL for nonstationary environments
Representative work connecting environmental context with safety adaptation.
Chandak et al. — safe policy improvement for nonstationary MDPs
Addresses policy improvement when the decision process changes over time.
Ding et al. — Safe RL in episodic CMDPs with nonstationary objectives and constraints
Important for understanding explicitly changing constrained decision problems.
Wei et al. — model-free algorithms for nonstationary CMDPs
A theoretical perspective on efficient learning when both the environment and constraints may vary.
Zeng et al. — online primal-dual natural actor-critic for CMDPs
Useful for readers interested in finite-time analysis and online constrained optimization.
Wen et al. — robustness and safety for pre-adaptation in meta-RL
Connects safety with the period before and during adaptation.
Yuan et al. — all-time safety for online safe meta-RL
Examines safety throughout online adaptation rather than only after convergence.
Luo et al. — MESA
An offline meta-RL approach to safe adaptation and fault tolerance.
From adaptation to adaptive safety assurance
For readers continuing into my current research, the next topics are context-based safety constraints, adaptation-speed safety, soft-to-hard runtime control, proactive context forecasting, and benchmarking safety under controlled nonstationarity.
See related publications →Broader AI & Agent Safety
A selective bridge beyond Safe RL. These readings connect sequential decision-making to broader questions of runtime assurance, monitoring, robustness, agent safety, and multi-agent systems.
Amodei et al. — Concrete Problems in AI Safety
An influential framing of practical safety problems that arise as learning systems become more capable and autonomous.
Everitt — Towards Safe Artificial General Intelligence
A thesis-level treatment useful for connecting RL-style agents to broader AI-safety questions.
SOTER — runtime assurance for safe robotic systems
A useful runtime-safety perspective: learned behavior can be wrapped by an independent assurance mechanism.
Bartocci et al. — Introduction to Runtime Verification
Background for readers interested in monitoring properties during execution rather than relying only on design-time verification.
Gronauer & Diepold — multi-agent deep reinforcement learning survey
A bridge toward endogenous nonstationarity created by other learning agents.
Schwalbe & Schels — safety assurance of machine-learning-based systems
Connects learning-system safety with the broader assurance and safety-engineering perspective.
A Note on Scope
The guide intentionally becomes more selective as it moves outward from RL and Safe RL. It is designed as a path into adaptive AI safety, not as a comprehensive catalogue of reinforcement learning or AI-safety literature. The selections will evolve as the literature develops and as stronger pedagogical routes become clear.