Publications
Selected work
Journal articles, conference papers and workshop contributions, newest first.
2026
Journal
TMLR
Do Not Imitate, Reinforce: Iterative Classification via Belief Refinement
M. Kallel, J. Tölle, A. Hendawy, C. D’Eramo
Transactions on Machine Learning Research (TMLR)
Details
Standard supervised classification trains models to imitate the exact labels provided by a perfect oracle. This imitation happens in a single pass, restricting the model to a fixed compute budget even when inputs vary in complexity. Moreover, the rigid training objective forces the model to express absolute certainty on its training data, resulting in overconfident predictions during evaluation. We propose Reinforced Iterative Classification (RIC), which replaces the imitative objective with Reinforcement Learning (RL). RIC deploys a recurrent agent that iteratively updates a predictive distribution over classes, receiving reward for stepwise improvement in prediction quality. The value function provides a natural halting criterion by estimating the remaining scope for improvement. We prove that the iterative formulation recovers the same optimal predictions as cross-entropy while yielding an anytime classifier. On image classification benchmarks, RIC matches the accuracy of supervised baselines with improved calibration and learns to allocate computation adaptively across inputs.
Journal
RA-L
Rationality Randomization with Maximum Entropy for Robust Dynamic Obstacle Avoidance on Legged Robots
J. L. Holgado-Alvarez, A. Reddi, G. Tiboni, C. D’Eramo
IEEE Robotics and Automation Letters (RA-L)
Details
Overview of RAMEN. (a) The performance-constrained curriculum starts near α ≈ 1, where obstacles move mostly randomly, and gradually introduces lower-α adversarial behaviors. (b) The high-level policy is trained on this diverse set of obstacle behaviors. (c) Deployment of the learned policy on a Unitree Go1 navigating among moving and static obstacles. Reinforcement learning (RL) for legged robots has recently gained significant attention due to its ability to enable traversal of complex terrains and navigation in cluttered environments. However, most existing approaches focus on static or predictably moving obstacles and rely on handcrafted curricula or manual task design, which may limit robustness to diverse obstacle behaviors. In this letter, we propose a method to automatically generate rich and dynamic obstacle-avoidance scenarios for learning robust navigation policies. Inspired by adversarial RL, we model obstacles as agents that induce challenging interactions for a protagonist policy. To improve robustness, we vary obstacle behavior by modulating their rationality, defined as the level of stochasticity in their actions. This is controlled via a temperature parameter that interpolates between deterministic, goal-directed motion and random behavior. We implement this through a domain randomization framework that samples rationality coefficients from a probability distribution, exposing the policy to a diverse range of obstacle behaviors during training. Crucially, this distribution is adapted online via a maximum entropy objective, which expands behavioral diversity while maintaining task performance. We demonstrate that the proposed method, RAtionality randomization with Maximum ENtropy (RAMEN), enables the learning of navigation policies that generalize across a wide range of simulated dynamic environments. Furthermore, we show successful zero-shot deployment on a legged robot operating in dynamic real-world scenarios using only onboard sensory inputs.
Journal
RLJ
PPO+: Enhancing Proximal Policy Optimization
M. Kallel, J. L. Holgado-Alvarez, S. Tosatto, C. D’Eramo
Reinforcement Learning Journal (RLJ)
Details
On-policy Reinforcement Learning (RL) offers desirable features such as stable learning, fewer policy updates, and the ability to evaluate a policy’s return during training. While recent efforts have focused on off-policy methods, achieving significant advancements, PPO remains the go-to algorithm for on-policy RL due to its apparent simplicity and effectiveness. Nonetheless, PPO relies on subtle, often poorly documented adjustments that can critically affect its performance. Our proposed approach, PPO+, is a principled adaptation of the PPO algorithm that strengthens its adherence to the on-policy objective, enhancing stability and efficiency. PPO+ demonstrates significantly improved asymptotic performance over PPO, and a substantially reduced performance gap with off-policy algorithms in several challenging continuous control tasks. Beyond just performance, our findings offer a fresh perspective on on-policy RL that can serve as guidance for future research endeavors.
Journal
RLJ
Gradient Iterated Temporal-Difference Learning
T. Vincent, K. Gerhardt, Y. Tripathi, H. Maraqten, A. White, M. White, J. Peters, C. D’Eramo
Reinforcement Learning Journal (RLJ)
Outstanding Paper Award for Empirical Reinforcement Learning ResearchDetails
Figure 1: Representation of bootstrapping methods in the space of action-value functions. We represent the projection on the space of function approximation (in green) with dashed grey lines. Left: TD learning uses a target estimator Q¯0\bar{Q}_{0} to construct a regression target estimating the Bel Temporal-difference (TD) learning is highly effective at controlling and evaluating an agent's long-term outcomes. Most approaches in this paradigm implement a semi-gradient update to boost the learning speed, which consists of ignoring the gradient of the bootstrapped estimate. While popular, this type of update is prone to divergence, as Baird's counterexample illustrates. Gradient TD methods were introduced to overcome this issue, but have not been widely used, potentially due to issues with learning speed compared to semi-gradient methods. Recently, iterated TD learning was developed to increase the learning speed of TD methods. For that, it learns a sequence of action-value functions in parallel, where each function is optimized to represent the application of the Bellman operator over the previous function in the sequence. While promising, this algorithm can be unstable due to its semi-gradient nature, as each function tracks a moving target. In this work, we modify iterated TD learning by computing the gradients over those moving targets, aiming to build a powerful gradient TD method that competes with semi-gradient methods. Our evaluation reveals that this algorithm, called Gradient Iterated Temporal-Difference learning, has a competitive learning speed against semi-gradient methods across various benchmarks, including Atari games, a result that no prior work on gradient TD methods has demonstrated.
Conference
CoRL
HARBOR: A Harness Framework for Agentic Robot Reinforcement Learning
Z. Li, Y. Jin, X. J. Liu, P. Liu, V. Prasad, C. D’Eramo, G. Chalvatzaki
Conference on Robot Learning (CoRL)
Details
Figure 1: Overview of HARBOR. (1) HARBOR supports arbitrary simulator × task × algorithm combinations across diverse benchmarks. (2) HARBOR treats robot RL automation as an RL harness that coordinates specialized agents, executable commands, mutable artifacts, verifiable gates, and reusable knowledg Reinforcement learning (RL) has become a powerful paradigm for robot learning, particularly in sim-to-real settings, but its broader adoption remains limited by the engineering pipeline surrounding the algorithms. Building tasks, shaping rewards, and tuning hyperparameters require substantial expert effort, making RL workflows costly and difficult to scale. We introduce HARBOR, an agentic framework that frames robot RL automation as a harness-engineering problem: given a simulator codebase and a task specification, it automates the workflow from environment setup to policy training in simulation. HARBOR decomposes such high-level objectives into bounded stages executed by specialized agents through standardized commands, persistent artifacts, executable gates, and reusable knowledge, and scales iteration via decentralized parallel trials and experience learning across runs. We evaluate HARBOR across 6 benchmarks and 16 tasks in total, spanning manipulation, locomotion, and bimanual dexterous control. We demonstrate that HARBOR automates the simulation RL workflow end-to-end, designs rewards, tunes algorithms to match or improve over default configurations, and reduces engineering effort at practical token and wall-clock cost; the resulting policies can also be transferred to real robots.
Conference
IJCAI
Adaptive Reward Design in Reinforcement Learning: A Taxonomy and Survey
R. Baybas, C. D’Eramo, P. Brune
International Joint Conference on Artificial Intelligence (IJCAI), Survey Track
Details
Adaptive Reward Design (ARD) is becoming a fundamental component for Reinforcement Learning (RL) agents, as they are deployed in increasingly complex settings where a single static reward across all phases of learning is rarely sufficient. Yet, ARD is rarely studied as a coherent topic: Relevant ideas are dispersed across reward shaping, curriculum learning, intrinsic motivation, non-stationary objectives, and preference- or feedback-based learning, which obscures conceptual connections and complicates method selection. This survey provides a unified view of ARD in RL by introducing a taxonomy, organized by the primary driver of the reward variation. The taxonomy distinguishes external-feedback-driven reward updates from reward adaptations driven by endogenous within-run signals and those conditioned on exogenous context signals. Using explicit assignment rules, we place work published between 2010 and 2025 within this taxonomy. By synthesizing typical RL settings and domains at the driver level, we simplify the method selection in ARD. Further, we describe the evolution and current trends of ARD and conclude by outlining promising future research directions.
Conference
ICLR
Bridging the Performance Gap Between Target-Free and Target-Based Reinforcement Learning with Iterated Q-Learning
T. Vincent, Y. Tripathi, T. Faust, Y. Oren, J. Peters, C. D’Eramo
International Conference on Learning Representations (ICLR)
Details
The use of target networks in deep reinforcement learning is a widely popular solution to mitigate the brittleness of semi-gradient approaches and stabilize learning. However, target networks notoriously require additional memory and delay the propagation of Bellman updates compared to an ideal target-free approach. In this work, we step out of the binary choice between target-free and target-based algorithms. We introduce a new method that uses a copy of the last linear layer of the online network as a target network, while sharing the remaining parameters with the up-to-date online network. This simple modification enables us to keep the target-free's low-memory footprint while leveraging the target-based literature. We find that combining our approach with the concept of iterated Q-learning, which consists of learning consecutive Bellman updates in parallel, helps improve the sample-efficiency of target-free approaches. Our proposed method, iterated Shared Q-Learning (iS-QL), bridges the performance gap between target-free and target-based approaches across various problems while using a single Q-network, thus stepping towards resource-efficient reinforcement learning algorithms.
Conference
ICLR
Use the Online Network If You Can: Towards Fast and Stable Reinforcement Learning
A. Hendawy, H. Metternich, T. Vincent, M. Kallel, J. Peters, C. D’Eramo
International Conference on Learning Representations (ICLR)
Details
The use of target networks is a popular approach for estimating value functions in deep Reinforcement Learning (RL). While effective, the target network remains a compromise solution that preserves stability at the cost of slowly moving targets, thus delaying learning. Conversely, using the online network as a bootstrapped target is intuitively appealing, albeit well-known to lead to unstable learning. In this work, we aim to obtain the best out of both worlds by introducing a novel update rule that computes the target using the MINimum estimate between the Target and Online network, giving rise to our method, MINTO. Through this simple, yet effective modification, we show that MINTO enables faster and stable value function learning, by mitigating the potential overestimation bias of using the online network for bootstrapping. Notably, MINTO can be seamlessly integrated into a wide range of value-based and actor-critic algorithms with a negligible cost. We evaluate MINTO extensively across diverse benchmarks, spanning online and offline RL, as well as discrete and continuous action spaces. Across all benchmarks, MINTO consistently improves performance, demonstrating its broad applicability and effectiveness.
Workshop
NeurIPS
AXIOM: Foundations of Efficient Deep Learning
M. Kallel, N. Jamoussi, J. Tölle, M. Kountouris, C. D’Eramo
Workshop at Advances in Neural Information Processing Systems (NeurIPS)
Workshop
EWRL
Gradient Iterated Temporal-Difference Learning
T. Vincent, K. Gerhardt, Y. Tripathi, H. Maraqten, A. White, M. White, J. Peters, C. D’Eramo
European Workshop on Reinforcement Learning (EWRL)
Details
Figure 1: Representation of bootstrapping methods in the space of action-value functions. We represent the projection on the space of function approximation (in green) with dashed grey lines. Left: TD learning uses a target estimator Q¯0\bar{Q}_{0} to construct a regression target estimating the Bel Temporal-difference (TD) learning is highly effective at controlling and evaluating an agent's long-term outcomes. Most approaches in this paradigm implement a semi-gradient update to boost the learning speed, which consists of ignoring the gradient of the bootstrapped estimate. While popular, this type of update is prone to divergence, as Baird's counterexample illustrates. Gradient TD methods were introduced to overcome this issue, but have not been widely used, potentially due to issues with learning speed compared to semi-gradient methods. Recently, iterated TD learning was developed to increase the learning speed of TD methods. For that, it learns a sequence of action-value functions in parallel, where each function is optimized to represent the application of the Bellman operator over the previous function in the sequence. While promising, this algorithm can be unstable due to its semi-gradient nature, as each function tracks a moving target. In this work, we modify iterated TD learning by computing the gradients over those moving targets, aiming to build a powerful gradient TD method that competes with semi-gradient methods. Our evaluation reveals that this algorithm, called Gradient Iterated Temporal-Difference learning, has a competitive learning speed against semi-gradient methods across various benchmarks, including Atari games, a result that no prior work on gradient TD methods has demonstrated.
Workshop
EWRL
Use the Online Network If You Can: Towards Fast and Stable Reinforcement Learning
A. Hendawy, H. Metternich, T. Vincent, M. Kallel, J. Peters, C. D’Eramo
European Workshop on Reinforcement Learning (EWRL)
Details
The use of target networks is a popular approach for estimating value functions in deep Reinforcement Learning (RL). While effective, the target network remains a compromise solution that preserves stability at the cost of slowly moving targets, thus delaying learning. Conversely, using the online network as a bootstrapped target is intuitively appealing, albeit well-known to lead to unstable learning. In this work, we aim to obtain the best out of both worlds by introducing a novel update rule that computes the target using the MINimum estimate between the Target and Online network, giving rise to our method, MINTO. Through this simple, yet effective modification, we show that MINTO enables faster and stable value function learning, by mitigating the potential overestimation bias of using the online network for bootstrapping. Notably, MINTO can be seamlessly integrated into a wide range of value-based and actor-critic algorithms with a negligible cost. We evaluate MINTO extensively across diverse benchmarks, spanning online and offline RL, as well as discrete and continuous action spaces. Across all benchmarks, MINTO consistently improves performance, demonstrating its broad applicability and effectiveness.
Workshop
EWRL
BluffJAX: Adversarial Imperfect Information Games in JAX
A. Reddi, J. Peters, C. D’Eramo
European Workshop on Reinforcement Learning (EWRL)
Details
(a) Kuhn Poker We introduce BluffJAX: an open-source suite of adversarial imperfect information games in JAX. We provide canonical implementations of games designed for high simulation throughputs and parallelization on GPU accelerators. Our suite consists of well-studied benchmarks such as Texas Hold'Em Poker and Kuhn Poker, as well as games that have not been previously studied in reinforcement learning research, such as Bluff, Stud Poker, and Kemps. We hope that implementing a variety of game mechanics and difficulties will introduce new challenges and foster novel research directions in game-theoretic methods for RL. We benchmark the throughput performance and memory usage of our environments in single and multi-GPU settings, demonstrating scaling of up to hundreds of millions of samples per second, and motivating the usage of BluffJAX over related GPU and CPU-based libraries. We benchmark reinforcement learning, tree search, and game-solving algorithms in JAX in order to provide users with baseline results and facilitate future comparisons.
Workshop
EWRL
Rationality Randomization with Maximum Entropy for Robust Dynamic Obstacle Avoidance on Legged Robots
J. L. Holgado-Alvarez, G. Tiboni, A. Reddi, C. D’Eramo
European Workshop on Reinforcement Learning (EWRL)
Details
Overview of RAMEN. (a) The performance-constrained curriculum starts near α ≈ 1, where obstacles move mostly randomly, and gradually introduces lower-α adversarial behaviors. (b) The high-level policy is trained on this diverse set of obstacle behaviors. (c) Deployment of the learned policy on a Unitree Go1 navigating among moving and static obstacles. Reinforcement learning (RL) for legged robots has recently gained significant attention due to its ability to enable traversal of complex terrains and navigation in cluttered environments. However, most existing approaches focus on static or predictably moving obstacles and rely on handcrafted curricula or manual task design, which may limit robustness to diverse obstacle behaviors. In this letter, we propose a method to automatically generate rich and dynamic obstacle-avoidance scenarios for learning robust navigation policies. Inspired by adversarial RL, we model obstacles as agents that induce challenging interactions for a protagonist policy. To improve robustness, we vary obstacle behavior by modulating their rationality, defined as the level of stochasticity in their actions. This is controlled via a temperature parameter that interpolates between deterministic, goal-directed motion and random behavior. We implement this through a domain randomization framework that samples rationality coefficients from a probability distribution, exposing the policy to a diverse range of obstacle behaviors during training. Crucially, this distribution is adapted online via a maximum entropy objective, which expands behavioral diversity while maintaining task performance. We demonstrate that the proposed method, RAtionality randomization with Maximum ENtropy (RAMEN), enables the learning of navigation policies that generalize across a wide range of simulated dynamic environments. Furthermore, we show successful zero-shot deployment on a legged robot operating in dynamic real-world scenarios using only onboard sensory inputs.
Workshop
EWRL
Streaming Reinforcement Learning under Partial Observability with Real-Time Recurrent Learning
N. Farr, A. Reddi, J. Peters, C. D’Eramo
European Workshop on Reinforcement Learning (EWRL)
Best Paper AwardDetails
Streaming reinforcement learning has emerged as an online learning paradigm that conforms to the restrictions of natural learning agents that process data incrementally, i.e. with a batch size of 1 and no replay buffer. While streaming RL has recently been shown to scale with deep function approximation with full observability, partially observable settings have remained out of reach. Truncated backpropagation through time collapses to a one-step gradient horizon under the streaming setting, and exact real-time recurrent learning is prohibitively expensive. We close this gap using recurrent trace units, a diagonal recurrent architecture that enables exact RTRL with linear time and memory complexity in the parameter count, and show that they integrate cleanly into existing streaming algorithms across both discrete and continuous control. On a MemoryChain diagnostic with chain lengths from 2 to 128, our method sustains performance where streaming TBPTT(1) baselines using feedforward, GRU, and RTU networks collapse. On five POPGym tasks and on partially observable MuJoCo continuous control, the streaming approach is competitive with batched PPO on POPGym and recovers a substantial fraction of batched performance on masked MuJoCo, despite using no replay buffer or batched updates.
Workshop
EWRL
Do Not Imitate, Reinforce: Iterative Classification via Belief Refinement
M. Kallel, J. Tölle, A. Hendawy, C. D’Eramo
European Workshop on Reinforcement Learning (EWRL)
Details
Standard supervised classification trains models to imitate the exact labels provided by a perfect oracle. This imitation happens in a single pass, restricting the model to a fixed compute budget even when inputs vary in complexity. Moreover, the rigid training objective forces the model to express absolute certainty on its training data, resulting in overconfident predictions during evaluation. We propose Reinforced Iterative Classification (RIC), which replaces the imitative objective with Reinforcement Learning (RL). RIC deploys a recurrent agent that iteratively updates a predictive distribution over classes, receiving reward for stepwise improvement in prediction quality. The value function provides a natural halting criterion by estimating the remaining scope for improvement. We prove that the iterative formulation recovers the same optimal predictions as cross-entropy while yielding an anytime classifier. On image classification benchmarks, RIC matches the accuracy of supervised baselines with improved calibration and learns to allocate computation adaptively across inputs.
Workshop
EWRL
Analyzing Value-Based Reinforcement Learning through the Lens of Mode Connectivity
A. Hendawy, H. Metternich, J. Peters, C. D’Eramo
European Workshop on Reinforcement Learning (EWRL)
Details
Ensemble methods are a common strategy for improving the stability and performance of value-based reinforcement learning (RL) algorithms. However, conventional ensembles operate in the output space and require maintaining multiple models during inference, resulting in an inference cost that grows linearly with the ensemble size. In this work, we study the geometric structure of the objective landscape in value-based RL through the lens of mode connectivity and analyze how independently trained Q-networks relate in parameter space. Our analysis reveals conditions under which weight-space ensembling becomes effective, namely when Q-networks lie within flat and linearly connected regions of the weight space. Building on these insights, we propose Q-Soups, a weight-averaging approach that merges multiple Q-networks into a single model. Empirical results on Atari benchmarks show that Q-Soups achieves performance comparable to traditional output-space ensembles while maintaining a single model at inference time. To the best of our knowledge, this work presents the first systematic investigation of mode connectivity and weight-space ensembling in value-based RL.
Workshop
RLC
Do Not Imitate, Reinforce: Iterative Classification via Belief Refinement
M. Kallel, J. Tölle, A. Hendawy, C. D’Eramo
Finding the Frame Workshop, Reinforcement Learning Conference (RLC)
Details
Standard supervised classification trains models to imitate the exact labels provided by a perfect oracle. This imitation happens in a single pass, restricting the model to a fixed compute budget even when inputs vary in complexity. Moreover, the rigid training objective forces the model to express absolute certainty on its training data, resulting in overconfident predictions during evaluation. We propose Reinforced Iterative Classification (RIC), which replaces the imitative objective with Reinforcement Learning (RL). RIC deploys a recurrent agent that iteratively updates a predictive distribution over classes, receiving reward for stepwise improvement in prediction quality. The value function provides a natural halting criterion by estimating the remaining scope for improvement. We prove that the iterative formulation recovers the same optimal predictions as cross-entropy while yielding an anytime classifier. On image classification benchmarks, RIC matches the accuracy of supervised baselines with improved calibration and learns to allocate computation adaptively across inputs.
Workshop
RLC
Streaming Reinforcement Learning under Partial Observability with Real-Time Recurrent Learning
N. Farr, A. Reddi, J. Peters, C. D’Eramo
RL in Big Worlds Workshop, Reinforcement Learning Conference (RLC)
Details
Streaming reinforcement learning has emerged as an online learning paradigm that conforms to the restrictions of natural learning agents that process data incrementally, i.e. with a batch size of 1 and no replay buffer. While streaming RL has recently been shown to scale with deep function approximation with full observability, partially observable settings have remained out of reach. Truncated backpropagation through time collapses to a one-step gradient horizon under the streaming setting, and exact real-time recurrent learning is prohibitively expensive. We close this gap using recurrent trace units, a diagonal recurrent architecture that enables exact RTRL with linear time and memory complexity in the parameter count, and show that they integrate cleanly into existing streaming algorithms across both discrete and continuous control. On a MemoryChain diagnostic with chain lengths from 2 to 128, our method sustains performance where streaming TBPTT(1) baselines using feedforward, GRU, and RTU networks collapse. On five POPGym tasks and on partially observable MuJoCo continuous control, the streaming approach is competitive with batched PPO on POPGym and recovers a substantial fraction of batched performance on masked MuJoCo, despite using no replay buffer or batched updates.
Workshop
ICRA
Self-Paced Curriculum Reinforcement Learning for Autonomous Superbike Racing in Simulation
L. Ghisi, J. Essenziale, C. D’Eramo, M. Luperto
Autonomous Driving Workshop, IEEE International Conference on Robotics and Automation (ICRA)
Details
Fig. 1: An example of the view from the VRider SBK game (top) and a rendering of the game from a third-person point of view (bottom), showing the motorbike leaning through a curve at high speed. Autonomous Racing has seen remarkable progress through deep Reinforcement Learning (RL), primarily for four-wheeled vehicles. However, motorbikes introduce substantially greater complexity due to the need to manage balance and lean angle, in addition to more reactive steering and throttle control, and a smaller weight. In this work, we present a framework for training an autonomous agent to race a superbike in VRider SBK, a physics-accurate Unity-based motorbike simulator. Our approach integrates Soft Actor-Critic (SAC) with Self-Paced curriculum Deep reinforcement Learning (SPDL), which dynamically generates progressively more challenging tasks based on the agent's performance, without requiring manual curriculum design. The agent's state space comprises proprioceptive features extended with lean-angle history, along with global track features via course points. The reward signal is shaped to encourage progress along the track while penalizing instability-inducing behaviors specific to two-wheeled dynamics. Preliminary experimental results demonstrate that SPDL outperforms SAC alone in training efficiency, lap time, and driving stability across multiple tracks and motorbike models, establishing a first baseline for RL-based autonomous motorbike racing.
2025
Journal
TMLR
Machine Learning with Physics Knowledge for Prediction: A Survey
J. Watson et al.
Transactions on Machine Learning Research (TMLR)
Survey CertificationDetails
This survey examines the broad suite of methods and models for combining machine learning with physics knowledge for prediction and forecast, with a focus on partial differential equations. These methods have attracted significant interest due to their potential impact on advancing scientific research and industrial practices by improving predictive models with small- or large-scale datasets and expressive predictive models with useful inductive biases. The survey has two parts. The first considers incorporating physics knowledge on an architectural level through objective functions, structured predictive models, and data augmentation. The second considers data as physics knowledge, which motivates looking at multi-task, meta, and contextual learning as an alternative approach to incorporating physics knowledge in a data-driven fashion. Finally, we also provide an industrial perspective on the application of these methods and a survey of the open-source ecosystem for physics-informed machine learning.
Journal
RLJ
Learning to Explore in Diverse Reward Settings via Temporal-Difference-Error Maximization
S. Griesbach, C. D’Eramo
Reinforcement Learning Journal (RLJ)
Details
(a) Four sampled trajectories of a mixed policy. Numerous heuristics and advanced approaches have been proposed for exploration in different settings for deep reinforcement learning. Noise-based exploration generally fares well with dense-shaped rewards and bonus-based exploration with sparse rewards. However, these methods usually require additional tuning to deal with undesirable reward settings by adjusting hyperparameters and noise distributions. Rewards that actively discourage exploration, i.e., with an action cost and no other dense signal to follow, can pose a major challenge. We propose a novel exploration method, Stable Error-seeking Exploration (SEE), that is robust across dense, sparse, and exploration-adverse reward settings. To this endeavor, we revisit the idea of maximizing the TD-error as a separate objective. Our method introduces three design choices to mitigate instability caused by far-off-policy learning, the conflict of interest of maximizing the cumulative TD-error in an episodic setting, and the non-stationary nature of TD-errors. SEE can be combined with off-policy algorithms without modifying the optimization pipeline of the original objective. In our experimental analysis, we show that a Soft-Actor Critic agent with the addition of SEE performs robustly across three diverse reward settings in a variety of tasks without hyperparameter adjustments.
Journal
RLJ
Eau De Q-Network: Adaptive Distillation of Neural Networks in Deep Reinforcement Learning
T. Vincent, T. Faust, Y. Tripathi, J. Peters, C. D’Eramo
Reinforcement Learning Journal (RLJ)
Details
Figure 13: We evaluate EauDeSAC on 66 MuJoCo environments for 11M with the small network. Left: PolyPruneSAC requires tuning as its performance depends on its hard-coded final sparsity level. Conversely, EauDeSAC avoids unsafe final sparsity levels by discovering its final sparsity level, therefore Recent works have successfully demonstrated that sparse deep reinforcement learning agents can be competitive against their dense counterparts. This opens up opportunities for reinforcement learning applications in fields where inference time and memory requirements are cost-sensitive or limited by hardware. Until now, dense-to-sparse methods have relied on hand-designed sparsity schedules that are not synchronized with the agent's learning pace. Crucially, the final sparsity level is chosen as a hyperparameter, which requires careful tuning as setting it too high might lead to poor performances. In this work, we address these shortcomings by crafting a dense-to-sparse algorithm that we name Eau De Q-Network (EauDeQN). To increase sparsity at the agent's learning pace, we consider multiple online networks with different sparsity levels, where each online network is trained from a shared target network. At each target update, the online network with the smallest loss is chosen as the next target network, while the other networks are replaced by a pruned version of the chosen network. We evaluate the proposed approach on the Atari 2600 benchmark and the MuJoCo physics simulator, showing that EauDeQN reaches high sparsity levels while keeping performances high.
Journal
TMLR
Iterated Q-Network: Beyond One-Step Bellman Updates in Deep Reinforcement Learning
T. Vincent, D. Palenicek, B. Belousov, J. Peters, C. D’Eramo
Transactions on Machine Learning Research (TMLR)
J2C Certification (top 10% of accepted papers)Details
Figure 11: Losses and neural networks architectures of DQN and i-DQN. The dotted lines link the outputs of the neural networks to the objects they represent. The flash signs stress how each projection step is being learned, where Γ^\hat{\Gamma} is the empirical Bellman operator. The target networks The vast majority of Reinforcement Learning methods is largely impacted by the computation effort and data requirements needed to obtain effective estimates of action-value functions, which in turn determine the quality of the overall performance and the sample-efficiency of the learning procedure. Typically, action-value functions are estimated through an iterative scheme that alternates the application of an empirical approximation of the Bellman operator and a subsequent projection step onto a considered function space. It has been observed that this scheme can be potentially generalized to carry out multiple iterations of the Bellman operator at once, benefiting the underlying learning algorithm. However, till now, it has been challenging to effectively implement this idea, especially in high-dimensional problems. In this paper, we introduce iterated Q-Network (i-QN), a novel principled approach that enables multiple consecutive Bellman updates by learning a tailored sequence of action-value functions where each serves as the target for the next. We show that i-QN is theoretically grounded and that it can be seamlessly used in value-based and actor-critic methods. We empirically demonstrate the advantages of i-QN in Atari 2600 games and MuJoCo continuous control problems.
Conference
ICML
Monte-Carlo Tree Search with Uncertainty Propagation via Optimal Transport
T. Dam, P. Stenger, L. Schneider, J. Pajarinen, C. D’Eramo, O. Maillard
International Conference on Machine Learning (ICML)
Spotlight Poster (top 2% of submitted papers)Details
This paper introduces a novel backup strategy for Monte-Carlo Tree Search (MCTS) designed for highly stochastic and partially observable Markov decision processes. We adopt a probabilistic approach, modeling both value and action-value nodes as Gaussian distributions. We introduce a novel backup operator that computes value nodes as the Wasserstein barycenter of their action-value children nodes; thus, propagating the uncertainty of the estimate across the tree to the root node. We study our novel backup operator when using a novel combination of L^1-Wasserstein barycenter with α-divergence, by drawing a notable connection to the generalized mean backup operator. We complement our probabilistic backup operator with two sampling strategies, based on optimistic selection and Thompson sampling, obtaining our Wasserstein MCTS algorithm. We provide theoretical guarantees of asymptotic convergence to the optimal policy, and an empirical evaluation on several stochastic and partially observable environments, where our approach outperforms well-known related baselines.
Conference
ICLR
Adaptive Q-Network: On-the-Fly Target Selection for Deep Reinforcement Learning
T. Vincent, F. Wahren, J. Peters, B. Belousov, C. D’Eramo
International Conference on Learning Representations (ICLR)
Details
Deep Reinforcement Learning (RL) is well known for being highly sensitive to hyperparameters, requiring practitioners substantial efforts to optimize them for the problem at hand. This also limits the applicability of RL in real-world scenarios. In recent years, the field of automated Reinforcement Learning (AutoRL) has grown in popularity by trying to address this issue. However, these approaches typically hinge on additional samples to select well-performing hyperparameters, hindering sample-efficiency and practicality. Furthermore, most AutoRL methods are heavily based on already existing AutoML methods, which were originally developed neglecting the additional challenges inherent to RL due to its non-stationarities. In this work, we propose a new approach for AutoRL, called Adaptive Q-Network (AdaQN), that is tailored to RL to take into account the non-stationarity of the optimization procedure without requiring additional samples. AdaQN learns several Q-functions, each one trained with different hyperparameters, which are updated online using the Q-function with the smallest approximation error as a shared target. Our selection scheme simultaneously handles different hyperparameters while coping with the non-stationarity induced by the RL optimization procedure and being orthogonal to any critic-based RL algorithm. We demonstrate that AdaQN is theoretically sound and empirically validate it in MuJoCo control problems and Atari 2600 games, showing benefits in sample-efficiency, overall performance, robustness to stochasticity and training stability.
Conference
AAMAS
Real-World Testing Matters in Reinforcement Learning for Education
A. Riedmann, C. D’Eramo, B. Lugrin
International Conference on Autonomous Agents and Multiagent Systems (AAMAS)
Details
Deep Reinforcement Learning (DRL) has proven its usefulness across various fields, sparking growing interest in applying it to education. However, most research on DRL in educational applications utilizes methods in simulation, with little evaluation involving real learners, resulting in limited evidence of their effectiveness in real-world contexts. Arguably, we consider real-world applications and in-situ experiments with users as essential for a thorough evaluation. We thus propose ResUli-RL, a novel DRL approach rooted in educational psychology, designed to provide adaptive feedback to young learners in the form of a pedagogical agent in a mobile educational app. To investigate its effectiveness, we conducted a five-week real-world evaluation with 56 primary school students, comparing ResUli-RL to an expert-designed baseline. Both groups significantly improved in reading competence, with no significant differences between them and a notable decrease in motivation in both conditions. In our aim to further improve the children's reading competence using DRL, our approach did, however, not yield the expected results. Our findings provide guidance for future work and highlight the need for real-world evaluations in education to assess the value of an educational DRL approach.
Workshop
EWRL
Learning to Explore in Diverse Reward Settings via Temporal-Difference-Error Maximization
S. Griesbach, C. D’Eramo
European Workshop on Reinforcement Learning (EWRL)
Details
(a) Four sampled trajectories of a mixed policy. Numerous heuristics and advanced approaches have been proposed for exploration in different settings for deep reinforcement learning. Noise-based exploration generally fares well with dense-shaped rewards and bonus-based exploration with sparse rewards. However, these methods usually require additional tuning to deal with undesirable reward settings by adjusting hyperparameters and noise distributions. Rewards that actively discourage exploration, i.e., with an action cost and no other dense signal to follow, can pose a major challenge. We propose a novel exploration method, Stable Error-seeking Exploration (SEE), that is robust across dense, sparse, and exploration-adverse reward settings. To this endeavor, we revisit the idea of maximizing the TD-error as a separate objective. Our method introduces three design choices to mitigate instability caused by far-off-policy learning, the conflict of interest of maximizing the cumulative TD-error in an episodic setting, and the non-stationary nature of TD-errors. SEE can be combined with off-policy algorithms without modifying the optimization pipeline of the original objective. In our experimental analysis, we show that a Soft-Actor Critic agent with the addition of SEE performs robustly across three diverse reward settings in a variety of tasks without hyperparameter adjustments.
Workshop
EWRL
K-Level Policy Gradients for Multi-Agent Reinforcement Learning
A. Reddi, G. Tiboni, J. Peters, C. D’Eramo
European Workshop on Reinforcement Learning (EWRL)
Details
Figure 1: Schematic illustration of the KK-level Policy Gradient (KPG) update for two agents. Actor-critic algorithms for deep multi-agent reinforcement learning (MARL) typically employ a policy update that responds to the current strategies of other agents. While being straightforward, this approach does not account for the updates of other agents at the same update step, resulting in miscoordination. In this paper, we introduce the K-Level Policy Gradient (KPG), a method that recursively updates each agent against the updated policies of other agents, speeding up the discovery of effective coordinated policies. We theoretically prove that KPG with finite iterates achieves monotonic convergence to a local Nash equilibrium under certain conditions. We provide principled implementations of KPG by applying it to the deep MARL algorithms MAPPO, MADDPG, and FACMAC. Empirically, we demonstrate superior performance over existing deep MARL algorithms in StarCraft II and multi-agent MuJoCo.
Workshop
EWRL
It Is All Connected: Multi-Task Reinforcement Learning via Mode Connectivity
A. Hendawy, H. Metternich, J. Peters, G. Tiboni, C. D’Eramo
European Workshop on Reinforcement Learning (EWRL)
Details
Acquiring a universal policy that performs multiple tasks is a crucial building block in endowing agents with generalized capabilities. To this end, the field of Multi-Task Reinforcement Learning (MTRL) proposes sharing parameters and representations among tasks during the learning process. Still, optimizing for a single solution that is able to perform various skills remains challenging. Recent works attempt to address these challenges using a mixture of experts, though this comes at the cost of additional inference-time complexity. In this paper, we introduce STAR, a novel MTRL algorithm that leverages mode connectivity to share knowledge across single skills, while remaining parameter-efficient at deployment time. Particularly, we show that single-task policies can be linearly connected in policy parameter space to the multi-task policy, i.e., task performance is maintained throughout the linear path connecting the two policies. Our experimental evaluation demonstrates that mode connectivity at training time induces implicit regularization in the multi-task policy, surpassing related baselines on MTRL benchmarks MuJoCo and Metaworld. Furthermore, STAR achieves competitive performance even with methods that retain multiple models at inference time.
Workshop
EWRL
Revisiting Proximal Policy Optimization
M. Kallel, J. L. Holgado-Alvarez, S. Tosatto, C. D’Eramo
European Workshop on Reinforcement Learning (EWRL)
Details
On-policy Reinforcement Learning (RL) offers desirable features such as stable learning, fewer policy updates, and the ability to evaluate a policy’s return during training. While recent efforts have focused on off-policy methods, achieving significant advancements, PPO remains the go-to algorithm for on-policy RL due to its apparent simplicity and effectiveness. Nonetheless, PPO is highly sensitive to hyperparameters and relies on subtle, often poorly documented adjustments that can critically affect its performance, thereby limiting its utility in complex scenarios. In this paper, we revisit the PPO algorithm by introducing principled enhancements that improve performance while eliminating the need for extensive hyperparameter tuning and implementation-specific optimizations. Our proposed approach, PPO+, is a principled adaptation of the PPO algorithm that strengthens its adherence to the on-policy objective, enhancing stability and efficiency. PPO+ sets a new state-of-the-art for deep on-policy RL on MuJoCo control problems while maintaining a straightforward implementation. PPO+ demonstrates significantly improved asymptotic performance over PPO, and a substantially reduced performance gap with off-policy algorithms in several challenging continuous control tasks. Beyond just performance, our findings offer a fresh perspective on on-policy RL.
Workshop
EWRL
Bridging the Performance Gap Between Target-Free and Target-Based Reinforcement Learning with Iterated Q-Learning
T. Vincent, Y. Tripathi, T. Faust, Y. Oren, J. Peters, C. D’Eramo
European Workshop on Reinforcement Learning (EWRL)
Details
The use of target networks in deep reinforcement learning is a widely popular solution to mitigate the brittleness of semi-gradient approaches and stabilize learning. However, target networks notoriously require additional memory and delay the propagation of Bellman updates compared to an ideal target-free approach. In this work, we step out of the binary choice between target-free and target-based algorithms. We introduce a new method that uses a copy of the last linear layer of the online network as a target network, while sharing the remaining parameters with the up-to-date online network. This simple modification enables us to keep the target-free's low-memory footprint while leveraging the target-based literature. We find that combining our approach with the concept of iterated Q-learning, which consists of learning consecutive Bellman updates in parallel, helps improve the sample-efficiency of target-free approaches. Our proposed method, iterated Shared Q-Learning (iS-QL), bridges the performance gap between target-free and target-based approaches across various problems while using a single Q-network, thus stepping towards resource-efficient reinforcement learning algorithms.
Workshop
RLC
Bridging the Performance Gap Between Target-Free and Target-Based Reinforcement Learning with Iterated Q-Learning
T. Vincent, Y. Tripathi, T. Faust, Y. Oren, J. Peters, C. D’Eramo
Finding the Frame Workshop, Reinforcement Learning Conference (RLC)
Details
The use of target networks in deep reinforcement learning is a widely popular solution to mitigate the brittleness of semi-gradient approaches and stabilize learning. However, target networks notoriously require additional memory and delay the propagation of Bellman updates compared to an ideal target-free approach. In this work, we step out of the binary choice between target-free and target-based algorithms. We introduce a new method that uses a copy of the last linear layer of the online network as a target network, while sharing the remaining parameters with the up-to-date online network. This simple modification enables us to keep the target-free's low-memory footprint while leveraging the target-based literature. We find that combining our approach with the concept of iterated Q-learning, which consists of learning consecutive Bellman updates in parallel, helps improve the sample-efficiency of target-free approaches. Our proposed method, iterated Shared Q-Learning (iS-QL), bridges the performance gap between target-free and target-based approaches across various problems while using a single Q-network, thus stepping towards resource-efficient reinforcement learning algorithms.
Workshop
RLDM
Eau De Q-Network: Adaptive Distillation of Neural Networks in Deep Reinforcement Learning
T. Vincent, J. Peters, C. D’Eramo
Multidisciplinary Conference on Reinforcement Learning and Decision Making (RLDM)
Details
Figure 13: We evaluate EauDeSAC on 66 MuJoCo environments for 11M with the small network. Left: PolyPruneSAC requires tuning as its performance depends on its hard-coded final sparsity level. Conversely, EauDeSAC avoids unsafe final sparsity levels by discovering its final sparsity level, therefore Recent works have successfully demonstrated that sparse deep reinforcement learning agents can be competitive against their dense counterparts. This opens up opportunities for reinforcement learning applications in fields where inference time and memory requirements are cost-sensitive or limited by hardware. Until now, dense-to-sparse methods have relied on hand-designed sparsity schedules that are not synchronized with the agent's learning pace. Crucially, the final sparsity level is chosen as a hyperparameter, which requires careful tuning as setting it too high might lead to poor performances. In this work, we address these shortcomings by crafting a dense-to-sparse algorithm that we name Eau De Q-Network (EauDeQN). To increase sparsity at the agent's learning pace, we consider multiple online networks with different sparsity levels, where each online network is trained from a shared target network. At each target update, the online network with the smallest loss is chosen as the next target network, while the other networks are replaced by a pruned version of the chosen network. We evaluate the proposed approach on the Atari 2600 benchmark and the MuJoCo physics simulator, showing that EauDeQN reaches high sparsity levels while keeping performances high.
2024
Journal
JAIR
A Unified Perspective on Value Backup and Exploration in Monte-Carlo Tree Search
T. Dam, C. D’Eramo, J. Peters, J. Pajarinen
Journal of Artificial Intelligence Research (JAIR)
Details
Figure 5: For different branching factor kk (rows) and depth dd (columns), the heatmaps show: the absolute error of the value estimate at the root node after the last simulation of each algorithm w.r.t. the respective optimal value (a), and w.r.t. the optimal value of UCT (b); regret at the root nod Monte-Carlo Tree Search (MCTS) is a class of methods for solving complex decision-making problems through the synergy of Monte-Carlo planning and Reinforcement Learning (RL). The highly combinatorial nature of the problems commonly addressed by MCTS requires the use of efficient exploration strategies for navigating the planning tree and quickly convergent value backup methods. These crucial problems are particularly evident in recent advances that combine MCTS with deep neural networks for function approximation. In this work, we propose two methods for improving the convergence rate and exploration based on a newly introduced backup operator and entropy regularization. We provide strong theoretical guarantees to bound convergence rate, approximation error, and regret of our methods. Moreover, we introduce a mathematical framework based on the use of the α-divergence for backup and exploration in MCTS. We show that this theoretical formulation unifies different approaches, including our newly introduced ones, under the same mathematical framework, allowing to obtain different methods by simply changing the value of α. In practice, our unified perspective offers a flexible way to balance between exploration and exploitation by tuning the single α parameter according to the problem at hand. We validate our methods through a rigorous empirical study from basic toy problems to the complex Atari games, and including both MDP and POMDP problems.
Journal
RLJ
Cyclicity-Regularized Coordination Graphs
O. Järnefelt, M. Kallel, C. D’Eramo
Reinforcement Learning Journal (RLJ)
Details
In parallel with the rise of the successful value function factorization approach, numerous recent studies on Cooperative Multi-Agent Reinforcement Learning (MARL) have explored the application of Coordination Graphs (CG) to model the communication requirements among the agent population. These coordination problems often exhibit structural sparsity, which facilitates accurate joint value function learning with CGs. Value-based methods necessitate the computation of argmaxes over the exponentially large joint action space, leading to the adoption of the max-sum method from the distributed constraint optimization (DCOP) literature. However, it has been empirically observed that the performance of max-sum deteriorates with an increase in the number of agents, attributed to the increased cyclicity of the graph. While previous works have tackled this issue by sparsifying the graph based on a metric of edge importance, thereby demonstrating improved performance, we argue that neglecting topological considerations during the sparsification procedure can adversely affect action selection. Consequently, we advocate for the explicit consideration of graph cyclicity alongside edge importances. We demonstrate that this approach results in superior performance across various challenging coordination problems.
Journal
TPAMI
On the Benefit of Optimal Transport for Curriculum Reinforcement Learning
P. Klink, C. D’Eramo, J. Peters, J. Pajarinen
IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)
Details
Fig. 1: Our approach (currot) addresses problems of existing curriculum RL methods, such as sprl, which create curricula between a distribution of initial tasks (blue) and a distribution of target tasks (green). In this example, the curriculum can change the task via two parameters c1c_{1} and c2c_{ Curriculum reinforcement learning (CRL) allows solving complex tasks by generating a tailored sequence of learning tasks, starting from easy ones and subsequently increasing their difficulty. Although the potential of curricula in RL has been clearly shown in various works, it is less clear how to generate them for a given learning environment, resulting in various methods aiming to automate this task. In this work, we focus on framing curricula as interpolations between task distributions, which has previously been shown to be a viable approach to CRL. Identifying key issues of existing methods, we frame the generation of a curriculum as a constrained optimal transport problem between task distributions. Benchmarks show that this way of curriculum generation can improve upon existing CRL methods, yielding high performance in various tasks with different characteristics.
Conference
KI
Uli-RL: A Real-World Deep Reinforcement Learning Pedagogical Agent for Children
A. Riedmann, J. Götz, C. D’Eramo, B. Lugrin
German Conference on Artificial Intelligence (KI)
Details
Deep Reinforcement Learning (DRL) has proven its usefulness in various fields, such as robotic control systems, recommendation algorithms, and natural language dialogue interfaces. Recently, we have been witnessing a growing interest in applying DRL in education, with early results suggesting beneficial effects. However, the majority of research in educational applications apply methods in simulation without evaluation with real learners, thus providing scarce evidence of its effectiveness on real-world problems. Arguably, real-world applications are crucial to properly assess the validity of DRL methods. To this end, we present an approach for integrating DRL into an empirically validated digital reading application for second graders in the form of an adaptive pedagogical agent. We use DRL to tailor the agent’s feedback behavior to each child’s individual learning needs. We evaluate our approach with second graders, investigating their performance and overall motivation, and compare it to a control version of the app. Through this work, we contribute an innovative approach to the use of DRL within the context of primary education, showcasing promising results in a real-world evaluation.
Conference
ICRA
Contact Energy Based Hindsight Experience Prioritization
E. Sayar, Z. Bing, C. D’Eramo, S. O. Oguz, A. Knoll
IEEE International Conference on Robotics and Automation (ICRA)
Details
(a) Touch sensors on the left and right side of the gripper. Multi-goal robot manipulation tasks with sparse rewards are difficult for reinforcement learning (RL) algorithms due to the inefficiency in collecting successful experiences. Recent algorithms such as Hindsight Experience Replay (HER) expedite learning by taking advantage of failed trajectories and replacing the desired goal with one of the achieved states so that any failed trajectory can be utilized as a contribution to learning. However, HER uniformly chooses failed trajectories, without taking into account which ones might be the most valuable for learning. In this paper, we address this problem and propose a novel approach Contact Energy Based Prioritization~(CEBP) to select the samples from the replay buffer based on rich information due to contact, leveraging the touch sensors in the gripper of the robot and object displacement. Our prioritization scheme favors sampling of contact-rich experiences, which are arguably the ones providing the largest amount of information. We evaluate our proposed approach on various sparse reward robotic tasks and compare them with the state-of-the-art methods. We show that our method surpasses or performs on par with those methods on robot manipulation tasks. Finally, we deploy the trained policy from our method to a real Franka robot for a pick-and-place task. We observe that the robot can solve the task successfully. The videos and code are publicly available at: https://erdiphd.github.io/HER_force
Conference
ICLR
Robust Adversarial Reinforcement Learning via Bounded Rationality Curricula
A. Reddi, M. Tölle, J. Peters, G. Chalvatzaki, C. D’Eramo
International Conference on Learning Representations (ICLR)
Spotlight Presentation (top 15% of accepted papers)Details
Figure 2: Robustness analysis of QARL and related baselines on MuJoCo swingup and locomotion problems (see title of heatmaps). Each heatmap shows the performance obtained after training for varying properties of the environment, described in the x−yx-y axes. The number next to the name of each algor Robustness against adversarial attacks and distribution shifts is a long-standing goal of Reinforcement Learning (RL). To this end, Robust Adversarial Reinforcement Learning (RARL) trains a protagonist against destabilizing forces exercised by an adversary in a competitive zero-sum Markov game, whose optimal solution, i.e., rational strategy, corresponds to a Nash equilibrium. However, finding Nash equilibria requires facing complex saddle point optimization problems, which can be prohibitive to solve, especially for high-dimensional control. In this paper, we propose a novel approach for adversarial RL based on entropy regularization to ease the complexity of the saddle point optimization problem. We show that the solution of this entropy-regularized problem corresponds to a Quantal Response Equilibrium (QRE), a generalization of Nash equilibria that accounts for bounded rationality, i.e., agents sometimes play random actions instead of optimal ones. Crucially, the connection between the entropy-regularized objective and QRE enables free modulation of the rationality of the agents by simply tuning the temperature coefficient. We leverage this insight to propose our novel algorithm, Quantal Adversarial RL (QARL), which gradually increases the rationality of the adversary in a curriculum fashion until it is fully rational, easing the complexity of the optimization problem while retaining robustness. We provide extensive evidence of QARL outperforming RARL and recent baselines across several MuJoCo locomotion and navigation problems in overall performance and robustness.
Conference
ICLR
Multi-Task Reinforcement Learning with Mixture of Orthogonal Experts
A. Hendawy, J. Peters, C. D’Eramo
International Conference on Learning Representations (ICLR)
Details
Figure 1: MOORE illustrative diagram. A state ss is encoded as a set of representations using a mixture of experts. The Gram-Schmidt process orthogonalizes the representations to encourage diversity. Then, the output head processes the representations VsV_{s} by interpolating the task-specific repre Multi-Task Reinforcement Learning (MTRL) tackles the long-standing problem of endowing agents with skills that generalize across a variety of problems. To this end, sharing representations plays a fundamental role in capturing both unique and common characteristics of the tasks. Tasks may exhibit similarities in terms of skills, objects, or physical properties while leveraging their representations eases the achievement of a universal policy. Nevertheless, the pursuit of learning a shared set of diverse representations is still an open challenge. In this paper, we introduce a novel approach for representation learning in MTRL that encapsulates common structures among the tasks using orthogonal representations to promote diversity. Our method, named Mixture Of Orthogonal Experts (MOORE), leverages a Gram-Schmidt process to shape a shared subspace of representations generated by a mixture of experts. When task-specific information is provided, MOORE generates relevant representations from this shared subspace. We assess the effectiveness of our approach on two MTRL benchmarks, namely MiniGrid and MetaWorld, showing that MOORE surpasses related baselines and establishes a new state-of-the-art result on MetaWorld.
Conference
ICLR
Domain Randomization via Entropy Maximization
G. Tiboni, P. Klink, T. Tommasi, J. Peters, C. D’Eramo, G. Chalvatzaki
International Conference on Learning Representations (ICLR)
Details
(a) Task illustration Varying dynamics parameters in simulation is a popular Domain Randomization (DR) approach for overcoming the reality gap in Reinforcement Learning (RL). Nevertheless, DR heavily hinges on the choice of the sampling distribution of the dynamics parameters, since high variability is crucial to regularize the agent's behavior but notoriously leads to overly conservative policies when randomizing excessively. In this paper, we propose a novel approach to address sim-to-real transfer, which automatically shapes dynamics distributions during training in simulation without requiring real-world data. We introduce DOmain RAndomization via Entropy MaximizatiON (DORAEMON), a constrained optimization problem that directly maximizes the entropy of the training distribution while retaining generalization capabilities. In achieving this, DORAEMON gradually increases the diversity of sampled dynamics parameters as long as the probability of success of the current policy is sufficiently high. We empirically validate the consistent benefits of DORAEMON in obtaining highly adaptive and generalizable policies, i.e. solving the task at hand across the widest range of dynamics parameters, as opposed to representative baselines from the DR literature. Notably, we also demonstrate the Sim2Real applicability of DORAEMON through its successful zero-shot transfer in a robotic manipulation setup under unknown real-world parameters.
Conference
ICLR
Augmented Bayesian Policy Search
M. Kallel, D. Basu, R. Akrour, C. D’Eramo
International Conference on Learning Representations (ICLR)
Details
(a) t=0t=0: Observed only θ\theta (star) Deterministic policies are often preferred over stochastic ones when implemented on physical systems. They can prevent erratic and harmful behaviors while being easier to implement and interpret. However, in practice, exploration is largely performed by stochastic policies. First-order Bayesian Optimization (BO) methods offer a principled way of performing exploration using deterministic policies. This is done through a learned probabilistic model of the objective function and its gradient. Nonetheless, such approaches treat policy search as a black-box problem, and thus, neglect the reinforcement learning nature of the problem. In this work, we leverage the performance difference lemma to introduce a novel mean function for the probabilistic model. This results in augmenting BO methods with the action-value function. Hence, we call our method Augmented Bayesian Search~(ABS). Interestingly, this new mean function enhances the posterior gradient with the deterministic policy gradient, effectively bridging the gap between BO and policy gradient methods. The resulting algorithm combines the convenience of the direct policy search with the scalability of reinforcement learning. We validate ABS on high-dimensional locomotion problems and demonstrate competitive performance compared to existing direct policy search schemes.
Conference
AAAI
Parameterized Projected Bellman Operator
T. Vincent, A. Metelli, B. Belousov, J. Peters, M. Restelli, C. D’Eramo
AAAI Conference on Artificial Intelligence (AAAI)
Details
Figure 1: PBO Λ\Lambda (left) operates on value function parameters as opposed to AVI (right), that uses the empirical Bellman operator Γ\Gamma followed by the projection operator Πproj\Pi_{\text{proj}}. Approximate value iteration (AVI) is a family of algorithms for reinforcement learning (RL) that aims to obtain an approximation of the optimal value function. Generally, AVI algorithms implement an iterated procedure where each step consists of (i) an application of the Bellman operator and (ii) a projection step into a considered function space. Notoriously, the Bellman operator leverages transition samples, which strongly determine its behavior, as uninformative samples can result in negligible updates or long detours, whose detrimental effects are further exacerbated by the computationally intensive projection step. To address these issues, we propose a novel alternative approach based on learning an approximate version of the Bellman operator rather than estimating it through samples as in AVI approaches. This way, we are able to (i) generalize across transition samples and (ii) avoid the computationally intensive projection step. For this reason, we call our novel operator projected Bellman operator (PBO). We formulate an optimization problem to learn PBO for generic sequential decision-making problems, and we theoretically analyze its properties in two representative classes of RL problems. Furthermore, we theoretically study our approach under the lens of AVI and devise algorithmic implementations to learn PBO in offline and online settings by leveraging neural network parameterizations. Finally, we empirically showcase the benefits of PBO w.r.t. the regular Bellman operator on several RL problems.
Workshop
CoRL
Dynamic Obstacle Avoidance with Bounded Rationality Adversarial Reinforcement Learning
J. L. Holgado-Alvarez, A. Reddi, C. D’Eramo
LocoLearn Workshop: From Bioinspired Gait Generation to Active Perception, Conference on Robot Learning (CoRL)
Details
(a) Approaching the obstacle. Reinforcement Learning (RL) has proven largely effective in obtaining stable locomotion gaits for legged robots. However, designing control algorithms which can robustly navigate unseen environments with obstacles remains an ongoing problem within quadruped locomotion. To tackle this, it is convenient to solve navigation tasks by means of a hierarchical approach with a low-level locomotion policy and a high-level navigation policy. Crucially, the high-level policy needs to be robust to dynamic obstacles along the path of the agent. In this work, we propose a novel way to endow navigation policies with robustness by a training process that models obstacles as adversarial agents, following the adversarial RL paradigm. Importantly, to improve the reliability of the training process, we bound the rationality of the adversarial agent resorting to quantal response equilibria, and place a curriculum over its rationality. We called this method Hierarchical policies via Quantal response Adversarial Reinforcement Learning (Hi-QARL). We demonstrate the robustness of our method by benchmarking it in unseen randomized mazes with multiple obstacles. To prove its applicability in real scenarios, our method is applied on a Unitree GO1 robot in simulation.
Workshop
EWRL
Multi-Task Reinforcement Learning with Mixture of Orthogonal Experts
A. Hendawy, J. Peters, C. D’Eramo
European Workshop on Reinforcement Learning (EWRL)
Details
Figure 1: MOORE illustrative diagram. A state ss is encoded as a set of representations using a mixture of experts. The Gram-Schmidt process orthogonalizes the representations to encourage diversity. Then, the output head processes the representations VsV_{s} by interpolating the task-specific repre Multi-Task Reinforcement Learning (MTRL) tackles the long-standing problem of endowing agents with skills that generalize across a variety of problems. To this end, sharing representations plays a fundamental role in capturing both unique and common characteristics of the tasks. Tasks may exhibit similarities in terms of skills, objects, or physical properties while leveraging their representations eases the achievement of a universal policy. Nevertheless, the pursuit of learning a shared set of diverse representations is still an open challenge. In this paper, we introduce a novel approach for representation learning in MTRL that encapsulates common structures among the tasks using orthogonal representations to promote diversity. Our method, named Mixture Of Orthogonal Experts (MOORE), leverages a Gram-Schmidt process to shape a shared subspace of representations generated by a mixture of experts. When task-specific information is provided, MOORE generates relevant representations from this shared subspace. We assess the effectiveness of our approach on two MTRL benchmarks, namely MiniGrid and MetaWorld, showing that MOORE surpasses related baselines and establishes a new state-of-the-art result on MetaWorld.
Workshop
EWRL
Augmented Bayesian Policy Search
M. Kallel, D. Basu, R. Akrour, C. D’Eramo
European Workshop on Reinforcement Learning (EWRL)
Details
(a) t=0t=0: Observed only θ\theta (star) Deterministic policies are often preferred over stochastic ones when implemented on physical systems. They can prevent erratic and harmful behaviors while being easier to implement and interpret. However, in practice, exploration is largely performed by stochastic policies. First-order Bayesian Optimization (BO) methods offer a principled way of performing exploration using deterministic policies. This is done through a learned probabilistic model of the objective function and its gradient. Nonetheless, such approaches treat policy search as a black-box problem, and thus, neglect the reinforcement learning nature of the problem. In this work, we leverage the performance difference lemma to introduce a novel mean function for the probabilistic model. This results in augmenting BO methods with the action-value function. Hence, we call our method Augmented Bayesian Search~(ABS). Interestingly, this new mean function enhances the posterior gradient with the deterministic policy gradient, effectively bridging the gap between BO and policy gradient methods. The resulting algorithm combines the convenience of the direct policy search with the scalability of reinforcement learning. We validate ABS on high-dimensional locomotion problems and demonstrate competitive performance compared to existing direct policy search schemes.
Workshop
EWRL
Cyclicity-Regularized Coordination Graphs
O. Järnefelt, M. Kallel, C. D’Eramo
European Workshop on Reinforcement Learning (EWRL)
Details
In parallel with the rise of the successful value function factorization approach, numerous recent studies on Cooperative Multi-Agent Reinforcement Learning (MARL) have explored the application of Coordination Graphs (CG) to model the communication requirements among the agent population. These coordination problems often exhibit structural sparsity, which facilitates accurate joint value function learning with CGs. Value-based methods necessitate the computation of argmaxes over the exponentially large joint action space, leading to the adoption of the max-sum method from the distributed constraint optimization (DCOP) literature. However, it has been empirically observed that the performance of max-sum deteriorates with an increase in the number of agents, attributed to the increased cyclicity of the graph. While previous works have tackled this issue by sparsifying the graph based on a metric of edge importance, thereby demonstrating improved performance, we argue that neglecting topological considerations during the sparsification procedure can adversely affect action selection. Consequently, we advocate for the explicit consideration of graph cyclicity alongside edge importances. We demonstrate that this approach results in superior performance across various challenging coordination problems.
Workshop
EWRL
Deterministic Exploration via Stationary Bellman Error Maximization
S. Griesbach, C. D’Eramo
European Workshop on Reinforcement Learning (EWRL)
Details
Figure 1: Comparison between ε\varepsilon-greedy and SEE, the x-axis in 10,00010,000 steps. The line shows the mean performance across 5050 runs, the shaded area shows the standard error of the mean. Exploration is a crucial and distinctive aspect of reinforcement learning (RL) that remains a fundamental open problem. Several methods have been proposed to tackle this challenge. Commonly used methods inject random noise directly into the actions, indirectly via entropy maximization, or add intrinsic rewards that encourage the agent to steer to novel regions of the state space. Another previously seen idea is to use the Bellman error as a separate optimization objective for exploration. In this paper, we introduce three modifications to stabilize the latter and arrive at a deterministic exploration policy. Our separate exploration agent is informed about the state of the exploitation, thus enabling it to account for previous experiences. Further components are introduced to make the exploration objective agnostic toward the episode length and to mitigate instability introduced by far-off-policy learning. Our experimental results show that our approach can outperform varepsilon-greedy in dense and sparse reward settings.
Workshop
EWRL
Adaptive Q-Network: On-the-Fly Target Selection for Deep Reinforcement Learning
T. Vincent, F. Wahren, J. Peters, B. Belousov, C. D’Eramo
European Workshop on Reinforcement Learning (EWRL)
Details
Deep Reinforcement Learning (RL) is well known for being highly sensitive to hyperparameters, requiring practitioners substantial efforts to optimize them for the problem at hand. This also limits the applicability of RL in real-world scenarios. In recent years, the field of automated Reinforcement Learning (AutoRL) has grown in popularity by trying to address this issue. However, these approaches typically hinge on additional samples to select well-performing hyperparameters, hindering sample-efficiency and practicality. Furthermore, most AutoRL methods are heavily based on already existing AutoML methods, which were originally developed neglecting the additional challenges inherent to RL due to its non-stationarities. In this work, we propose a new approach for AutoRL, called Adaptive Q-Network (AdaQN), that is tailored to RL to take into account the non-stationarity of the optimization procedure without requiring additional samples. AdaQN learns several Q-functions, each one trained with different hyperparameters, which are updated online using the Q-function with the smallest approximation error as a shared target. Our selection scheme simultaneously handles different hyperparameters while coping with the non-stationarity induced by the RL optimization procedure and being orthogonal to any critic-based RL algorithm. We demonstrate that AdaQN is theoretically sound and empirically validate it in MuJoCo control problems and Atari 2600 games, showing benefits in sample-efficiency, overall performance, robustness to stochasticity and training stability.
Workshop
EWRL
Revisiting On-Policy Deep Reinforcement Learning
M. Kallel, S. Tosatto, C. D’Eramo
European Workshop on Reinforcement Learning (EWRL)
Details
On-policy Reinforcement Learning (RL) offers desirable features such as stable learning, fewer policy updates, and the ability to evaluate a policy’s return during training. While recent efforts have focused on off-policy methods, achieving significant advancements, Proximal Policy Optimization (PPO) remains the go-to algorithm for on-policy RL due to its apparent simplicity and effectiveness. However, despite its apparent simplicity, PPO is highly sensitive to hyperparameters and depends on subtle and poorly documented tweaks that can make or break its success--hindering its applicability in complex problems. In this paper, we revisit on-policy deep RL with a focus on improving PPO, by introducing principled solutions that enhance its performance while eliminating the need for extensive hyperparameter tuning and implementation-level optimizations. Our effort leads to PPO+, a methodical adaptation of the PPO algorithm that adheres closer to its theoretical foundations. PPO+ sets a new state-of-the-art for on-policy RL on MuJoCo control problems while maintaining a straightforward trick-free implementation. Beyond just performance, our findings offer a fresh perspective on on-policy RL that could reignite interest in these approaches.
Workshop
ICML
Adaptive Q-Network: On-the-Fly Target Selection for Deep Reinforcement Learning
T. Vincent, F. Wahren, J. Peters, B. Belousov, C. D’Eramo
AutoRL Workshop: Exploring Meta-Learning, AutoML, and LLMs, International Conference on Machine Learning (ICML)
Details
Deep Reinforcement Learning (RL) is well known for being highly sensitive to hyperparameters, requiring practitioners substantial efforts to optimize them for the problem at hand. This also limits the applicability of RL in real-world scenarios. In recent years, the field of automated Reinforcement Learning (AutoRL) has grown in popularity by trying to address this issue. However, these approaches typically hinge on additional samples to select well-performing hyperparameters, hindering sample-efficiency and practicality. Furthermore, most AutoRL methods are heavily based on already existing AutoML methods, which were originally developed neglecting the additional challenges inherent to RL due to its non-stationarities. In this work, we propose a new approach for AutoRL, called Adaptive Q-Network (AdaQN), that is tailored to RL to take into account the non-stationarity of the optimization procedure without requiring additional samples. AdaQN learns several Q-functions, each one trained with different hyperparameters, which are updated online using the Q-function with the smallest approximation error as a shared target. Our selection scheme simultaneously handles different hyperparameters while coping with the non-stationarity induced by the RL optimization procedure and being orthogonal to any critic-based RL algorithm. We demonstrate that AdaQN is theoretically sound and empirically validate it in MuJoCo control problems and Atari 2600 games, showing benefits in sample-efficiency, overall performance, robustness to stochasticity and training stability.
2023
Journal
IJRR
Composable Energy Policies for Reactive Motion Generation and Reinforcement Learning
J. Urain, A. Li, P. Liu, C. D’Eramo, J. Peters
International Journal of Robotics Research (IJRR)
Details
Fig. 1: Composable Energy Policies generate trajectories reactively in a wide set of environments, as the ones in the figure. The generated trajectories are attracted to a target pose, visualized by the green sphere, and avoid a set of obstacles, shown as red spheres. Reactive motion generation problems are usually solved by computing actions as a sum of policies. However, these policies are independent of each other and thus, they can have conflicting behaviors when summing their contributions together. We introduce Composable Energy Policies (CEP), a novel framework for modular reactive motion generation. CEP computes the control action by optimization over the product of a set of stochastic policies. This product of policies will provide a high probability to those actions that satisfy all the components and low probability to the others. Optimizing over the product of the policies avoids the detrimental effect of conflicting behaviors between policies choosing an action that satisfies all the objectives. Besides, we show that CEP naturally adapts to the Reinforcement Learning problem allowing us to integrate, in a hierarchical fashion, any distribution as prior, from multimodal distributions to non-smooth distributions and learn a new policy given them.
Workshop
NeurIPS
Using Proto-Value Functions for Curriculum Generation in Goal-Conditioned RL
H. Metternich, A. Hendawy, P. Klink, C. D’Eramo
Goal-Conditioned Reinforcement Learning Workshop, Advances in Neural Information Processing Systems (NeurIPS)
Details
In this paper, we investigate the use of Proto Value Functions (PVFs) for measuring the similarity between tasks in the context of Curriculum Learning (CL). PVFs serve as a mathematical framework for generating basis functions for the state space of a Markov Decision Process (MDP). They capture the structure of the state space manifold and have been shown to be useful for value function approximation in Reinforcement Learning (RL). We show that even a few PVFs allow us to estimate the similarity between tasks. Based on this observation, we introduce a new algorithm called Curriculum Representation Policy Iteration (CRPI) that uses PVFs for CL, and we provide a proof of concept in a Goal-Conditioned Reinforcement Learning (GCRL) setting.
Workshop
CoRL
Parameter-Efficient Tuning of Pretrained Visual-Language Models in Multitask Robot Learning
M. Mittenbuehler, A. Hendawy, C. D’Eramo, G. Chalvatzaki
Workshop on Learning Effective Abstractions for Planning (LEAP), Conference on Robot Learning (CoRL)
Details
Multimodal pretrained visual-language models (pVLMs) have showcased excellence across several applications, like visual question-answering. Their recent application for policy learning manifested promising avenues for augmenting robotic capabilities in the real world. This paper delves into the problem of parameter-efficient tuning of pVLMs for adapting them to robotic manipulation tasks with low-resource data. We showcase how Low-Rank Adapters (LoRA) can be injected into behavioral cloning temporal transformers to fuse language, multi-view images, and proprioception for multitask robot learning, even for long-horizon tasks. Preliminary results indicate our approach vastly outperforms baseline architectures and tuning methods, paving the way toward parameter-efficient adaptation of pretrained large multimodal transformers for robot learning with only a handful of demonstrations.
Workshop
EWRL
A Unified Perspective on Value Backup and Exploration in Monte-Carlo Tree Search
T. Dam, C. D’Eramo, J. Pajarinen, J. Peters
European Workshop on Reinforcement Learning (EWRL)
Details
Figure 5: For different branching factor kk (rows) and depth dd (columns), the heatmaps show: the absolute error of the value estimate at the root node after the last simulation of each algorithm w.r.t. the respective optimal value (a), and w.r.t. the optimal value of UCT (b); regret at the root nod Monte-Carlo Tree Search (MCTS) is a class of methods for solving complex decision-making problems through the synergy of Monte-Carlo planning and Reinforcement Learning (RL). The highly combinatorial nature of the problems commonly addressed by MCTS requires the use of efficient exploration strategies for navigating the planning tree and quickly convergent value backup methods. These crucial problems are particularly evident in recent advances that combine MCTS with deep neural networks for function approximation. In this work, we propose two methods for improving the convergence rate and exploration based on a newly introduced backup operator and entropy regularization. We provide strong theoretical guarantees to bound convergence rate, approximation error, and regret of our methods. Moreover, we introduce a mathematical framework based on the use of the α-divergence for backup and exploration in MCTS. We show that this theoretical formulation unifies different approaches, including our newly introduced ones, under the same mathematical framework, allowing to obtain different methods by simply changing the value of α. In practice, our unified perspective offers a flexible way to balance between exploration and exploitation by tuning the single α parameter according to the problem at hand. We validate our methods through a rigorous empirical study from basic toy problems to the complex Atari games, and including both MDP and POMDP problems.
Workshop
EWRL
Iterated Deep Q-Network: Efficient Learning of Bellman Iterations for Deep Reinforcement Learning
T. Vincent, B. Belousov, C. D’Eramo, J. Peters
European Workshop on Reinforcement Learning (EWRL)
Details
Value-based Reinforcement Learning (RL) methods hinge on the application of the Bellman operator, which needs to be approximated from samples. Most approaches consist of an iterative scheme alternating the application of a Bellman iteration and a subsequent projection step in the considered function space. In this paper, we propose a new perspective by introducing iterated Deep Q-Network (iDQN), a novel DQN-based algorithm that aims to obtain an approximation of several consecutive Bellman iterations at once. To this end, iDQN leverages the online network of DQN to build a target for a second online network, which in turn serves as a target for a third online network, and so forth, thereby taking into account future Bellman iterations. This entails that iDQN allows for better learning of the Bellman iterations than DQN, while using the same number of gradient steps. We theoretically prove the benefit of iDQN in terms of error propagation under the lens of approximate value iteration. Then, we evaluate iDQN against relevant baselines on 54 Atari 2600 games, showing that iDQN outperforms DQN while being orthogonal to more advanced DQN-based approaches.
2022
Conference
ICML
Curriculum Reinforcement Learning via Constrained Optimal Transport
P. Klink, H. Yang, C. D’Eramo, J. Peters, J. Pajarinen
International Conference on Machine Learning (ICML)
Details
Curriculum reinforcement learning (CRL) allows solving complex tasks by generating a tailored sequence of learning tasks, starting from easy ones and subsequently increasing their difficulty. Although the potential of curricula in RL has been clearly shown in a variety of works, it is less clear how to generate them for a given learning environment, resulting in a variety of methods aiming to automate this task. In this work, we focus on the idea of framing curricula as interpolations between task distributions, which has previously been shown to be a viable approach to CRL. Identifying key issues of existing methods, we frame the generation of a curriculum as a constrained optimal transport problem between task distributions. Benchmarks show that this way of curriculum generation can improve upon existing CRL methods, yielding high performance in a variety of tasks with different characteristics.
Conference
IJCNN
Prioritized Sampling with Intrinsic Motivation in Multi-Task Reinforcement Learning
C. D’Eramo, G. Chalvatzaki
IEEE International Joint Conference on Neural Networks (IJCNN)
Details
Deep Reinforcement Learning (RL) promises to lead the next advances towards the development of coveted future intelligent agents. However, the unprecedented representational power of deep function approximators, e.g. deep neural networks, comes at the cost of demanding a huge amount of experience, making deep RL impractical for applications requiring interactions with the real world. We study the problem of making use of samples in deep RL more efficiently, exploiting the desirable properties of knowledge generalization resulting from learning multiple tasks together. The outcome of our work is the coupling of multi-task RL algorithms with a task-sampling policy based on the well-known intrinsic motivation paradigm. In particular, we leverage on the notion of TD-error of Bellman updates, as an effective measure of learning progress, to prioritize sampling from the tasks contributing the most to the learning of the agent. This sampling strategy speeds up the learning of tasks for which the agent is showing progress, and postpones the learning of the remaining ones, resulting in an optimized collection of samples. Our method is supported by experimental evaluations on well-known RL control tasks, for which our approach shows superior sample-efficiency and performance compared to representative baselines. We eventually evaluate our approach on simulated control tasks based on Quanser robotics systems, confirming the advantages over the baselines also in more realistic applications.
Conference
ICLR
Boosted Curriculum Reinforcement Learning
P. Klink, C. D’Eramo, J. Peters, J. Pajarinen
International Conference on Learning Representations (ICLR)
Details
Curriculum value-based reinforcement learning (RL) solves a complex target task by reusing action-values across a tailored sequence of related tasks of increasing difficulty. However, finding an exact way of reusing action-values in this setting is still a poorly understood problem. In this paper, we introduce the concept of boosting to curriculum value-based RL, by approximating the action-value function as a sum of residuals trained on each task. This approach, which we refer to as boosted curriculum reinforcement learning (BCRL), has the benefit of naturally increasing the representativeness of the functional space by adding a new residual each time a new task is presented. This procedure allows reusing previous action-values while promoting expressiveness of the action-value function. We theoretically study BCRL as an approximate value iteration algorithm, discussing advantages over regular curriculum RL in terms of approximation accuracy and convergence to the optimal action-value function. Finally, we provide detailed empirical evidence of the benefits of BCRL in problems requiring curricula for accurate action-value estimation and targeted exploration.
Workshop
ICML
Parameterized Projected Bellman Operator
T. Vincent, A. Metelli, J. Peters, M. Restelli, C. D’Eramo
New Frontiers in Learning, Control, and Dynamical Systems Workshop, International Conference on Machine Learning (ICML)
Details
Figure 1: PBO Λ\Lambda (left) operates on value function parameters as opposed to AVI (right), that uses the empirical Bellman operator Γ\Gamma followed by the projection operator Πproj\Pi_{\text{proj}}. Approximate value iteration (AVI) is a family of algorithms for reinforcement learning (RL) that aims to obtain an approximation of the optimal value function. Generally, AVI algorithms implement an iterated procedure where each step consists of (i) an application of the Bellman operator and (ii) a projection step into a considered function space. Notoriously, the Bellman operator leverages transition samples, which strongly determine its behavior, as uninformative samples can result in negligible updates or long detours, whose detrimental effects are further exacerbated by the computationally intensive projection step. To address these issues, we propose a novel alternative approach based on learning an approximate version of the Bellman operator rather than estimating it through samples as in AVI approaches. This way, we are able to (i) generalize across transition samples and (ii) avoid the computationally intensive projection step. For this reason, we call our novel operator projected Bellman operator (PBO). We formulate an optimization problem to learn PBO for generic sequential decision-making problems, and we theoretically analyze its properties in two representative classes of RL problems. Furthermore, we theoretically study our approach under the lens of AVI and devise algorithmic implementations to learn PBO in offline and online settings by leveraging neural network parameterizations. Finally, we empirically showcase the benefits of PBO w.r.t. the regular Bellman operator on several RL problems.
Workshop
IROS
Modular Value Function Factorization in Multi-Agent Reinforcement Learning
O. Järnefelt, C. D’Eramo
Decision Making in Multi-Agent Systems Workshop, IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
Oral Presentation (5 of 16 papers)Workshop
RLDM
Efficient Task Sampling and Shared Knowledge in Multi-Task Reinforcement Learning
C. D’Eramo, F. Wahren, G. Chalvatzaki
Multidisciplinary Conference on Reinforcement Learning and Decision Making (RLDM)
Workshop
EWRL
Curriculum Reinforcement Learning via Constrained Optimal Transport
P. Klink, H. Yang, C. D’Eramo, J. Peters, J. Pajarinen
European Workshop on Reinforcement Learning (EWRL)
Details
Curriculum reinforcement learning (CRL) allows solving complex tasks by generating a tailored sequence of learning tasks, starting from easy ones and subsequently increasing their difficulty. Although the potential of curricula in RL has been clearly shown in a variety of works, it is less clear how to generate them for a given learning environment, resulting in a variety of methods aiming to automate this task. In this work, we focus on the idea of framing curricula as interpolations between task distributions, which has previously been shown to be a viable approach to CRL. Identifying key issues of existing methods, we frame the generation of a curriculum as a constrained optimal transport problem between task distributions. Benchmarks show that this way of curriculum generation can improve upon existing CRL methods, yielding high performance in a variety of tasks with different characteristics.
2021
Journal
JMLR
Gaussian Approximation for Bias Reduction in Q-Learning
C. D’Eramo, A. Cini, A. Nuara, M. Pirotta, C. Alippi, J. Peters, M. Restelli
Journal of Machine Learning Research (JMLR)
Details
Temporal-Difference off-policy algorithms are among the building blocks of reinforcement learning (RL). Within this family, Q-Learning is arguably the most famous one, which has been widely studied and extended. The update rule of Q-learning involves the use of the maximum operator to estimate the maximum expected value of the return. However, this estimate is positively biased, and may hinder the learning process, especially in stochastic environments and when function approximation is used. We introduce the Weighted Estimator as an effective solution to mitigate the negative effects of overestimation in Q-Learning. The Weighted Estimator estimates the maximum expected value as a weighted sum of the action values, with the weights being the probabilities that each action value is the maximum. In this work, we study the problem from the statistical perspective of estimating the maximum expected value of a set of random variables and provide bounds to the bias and the variance of the Weighted Estimator, showing its advantages over other estimators present in literature. Then, we derive algorithms to enable the use of the Weighted Estimator, in place of the Maximum Estimator, in online and batch RL, and we introduce a novel algorithm for deep RL. Finally, we empirically evaluate our algorithms in a large set of heterogeneous problems, encompassing discrete and continuous, low and high dimensional, deterministic and stochastic environments. Experimental results show the effectiveness of the Weighted Estimator in controlling the bias of the estimate, resulting in better performance than representative baselines and robust learning w.r.t. a large set of diverse environments.
Journal
JMLR
MushroomRL: Simplifying Reinforcement Learning Research
C. D’Eramo, D. Tateo, A. Bonarini, M. Restelli, J. Peters
Journal of Machine Learning Research (JMLR)
Details
(a) Rendering of the game MushroomRL is an open-source Python library developed to simplify the process of implementing and running Reinforcement Learning (RL) experiments. Compared to other available libraries, MushroomRL has been created with the purpose of providing a comprehensive and flexible framework to minimize the effort in implementing and testing novel RL methodologies. Indeed, the architecture of MushroomRL is built in such a way that every component of an RL problem is already provided, and most of the time users can only focus on the implementation of their own algorithms and experiments. The result is a library from which RL researchers can significantly benefit in the critical phase of the empirical analysis of their works. MushroomRL stable code, tutorials and documentation can be found at https://github.com/MushroomRL/mushroom-rl.
Journal
JMLR
A Probabilistic Interpretation of Self-Paced Learning with Applications to Reinforcement Learning
P. Klink, H. Abdulsamad, B. Belousov, C. D’Eramo, J. Peters, J. Pajarinen
Journal of Machine Learning Research (JMLR)
Details
Across machine learning, the use of curricula has shown strong empirical potential to improve learning from data by avoiding local optima of training objectives. For reinforcement learning (RL), curricula are especially interesting, as the underlying optimization has a strong tendency to get stuck in local optima due to the exploration-exploitation trade-off. Recently, a number of approaches for an automatic generation of curricula for RL have been shown to increase performance while requiring less expert knowledge compared to manually designed curricula. However, these approaches are seldomly investigated from a theoretical perspective, preventing a deeper understanding of their mechanics. In this paper, we present an approach for automated curriculum generation in RL with a clear theoretical underpinning. More precisely, we formalize the well-known self-paced learning paradigm as inducing a distribution over training tasks, which trades off between task complexity and the objective to match a desired task distribution. Experiments show that training on this induced distribution helps to avoid poor local optima across RL algorithms in different tasks with uninformative rewards and challenging exploration requirements.
Conference
ICML
Convex Regularization in Monte-Carlo Tree Search
T. Dam, C. D’Eramo, J. Peters, J. Pajarinen
International Conference on Machine Learning (ICML)
Details
Figure 1: For each algorithm, we show the convergence of the value estimate at the root node to the respective optimal value (top), to the UCT optimal value (middle), and the regret (bottom). Monte-Carlo planning and Reinforcement Learning (RL) are essential to sequential decision making. The recent AlphaGo and AlphaZero algorithms have shown how to successfully combine these two paradigms in order to solve large scale sequential decision problems. These methodologies exploit a variant of the well-known UCT algorithm to trade off exploitation of good actions and exploration of unvisited states, but their empirical success comes at the cost of poor sample-efficiency and high computation time. In this paper, we overcome these limitations by considering convex regularization in Monte-Carlo Tree Search (MCTS), which has been successfully used in RL to efficiently drive exploration. First, we introduce a unifying theory on the use of generic convex regularizers in MCTS, deriving the regret analysis and providing guarantees of exponential convergence rate. Second, we exploit our theoretical framework to introduce novel regularized backup operators for MCTS, based on the relative entropy of the policy update, and on the Tsallis entropy of the policy. Finally, we empirically evaluate the proposed operators in AlphaGo and AlphaZero on problems of increasing dimensionality and branching factor, from a toy problem to several Atari games, showing their superiority w.r.t. representative baselines.
Conference
RSS
Composable Energy Policies for Reactive Motion Generation and Reinforcement Learning
J. Urain, A. Li, P. Liu, C. D’Eramo, J. Peters
Robotics: Science and Systems (RSS)
Details
Fig. 1: Composable Energy Policies generate trajectories reactively in a wide set of environments, as the ones in the figure. The generated trajectories are attracted to a target pose, visualized by the green sphere, and avoid a set of obstacles, shown as red spheres. Reactive motion generation problems are usually solved by computing actions as a sum of policies. However, these policies are independent of each other and thus, they can have conflicting behaviors when summing their contributions together. We introduce Composable Energy Policies (CEP), a novel framework for modular reactive motion generation. CEP computes the control action by optimization over the product of a set of stochastic policies. This product of policies will provide a high probability to those actions that satisfy all the components and low probability to the others. Optimizing over the product of the policies avoids the detrimental effect of conflicting behaviors between policies choosing an action that satisfies all the objectives. Besides, we show that CEP naturally adapts to the Reinforcement Learning problem allowing us to integrate, in a hierarchical fashion, any distribution as prior, from multimodal distributions to non-smooth distributions and learn a new policy given them.
Conference
ICRA
Model Predictive Actor-Critic: Accelerating Robot Skill Acquisition with Deep Reinforcement Learning
A. S. Morgan, D. Nandha, G. Chalvatzaki, C. D’Eramo, A. M. Dollar, J. Peters
IEEE International Conference on Robotics and Automation (ICRA)
Details
Fig. 1: (a) Complex robot skills, like in-hand manipulation, are difficult to acquire and perform with conventional methods. (b) MoPAC is suitable for training an underactuated hand to efficiently learn such tasks, like finger gaiting, compared to other state-of-the-art techniques. Substantial advancements to model-based reinforcement learning algorithms have been impeded by the model-bias induced by the collected data, which generally hurts performance. Meanwhile, their inherent sample efficiency warrants utility for most robot applications, limiting potential damage to the robot and its environment during training. Inspired by information theoretic model predictive control and advances in deep reinforcement learning, we introduce Model Predictive Actor-Critic (MoPAC), a hybrid model-based/model-free method that combines model predictive rollouts with policy optimization as to mitigate model bias. MoPAC leverages optimal trajectories to guide policy learning, but explores via its model-free method, allowing the algorithm to learn more expressive dynamics models. This combination guarantees optimal skill learning up to an approximation error and reduces necessary physical interaction with the environment, making it suitable for real-robot training. We provide extensive results showcasing how our proposed method generally outperforms current state-of-the-art and conclude by evaluating MoPAC for learning on a physical robotic hand performing valve rotation and finger gaiting--a task that requires grasping, manipulation, and then regrasping of an object.
2020
Journal
Frontiers
Multi-Channel Interactive Reinforcement Learning for Sequential Tasks
D. Koert, M. Kircher, V. Salikutluk, C. D’Eramo, J. Peters
Frontiers in Robotics and AI
Details
The ability to learn new tasks by sequencing already known skills is an important requirement for future robots. Reinforcement learning is a powerful tool for this as it allows for a robot to learn and improve on how to combine skills for sequential tasks. However, in real robotic applications, the cost of sample collection and exploration prevent the application of reinforcement learning for a variety of tasks. To overcome these limitations, human input during reinforcement can be beneficial to speed up learning, guide the exploration and prevent the choice of disastrous actions. Nevertheless, there is a lack of experimental evaluations of multi-channel interactive reinforcement learning systems solving robotic tasks with input from inexperienced human users, in particular for cases where human input might be partially wrong. Therefore, in this paper, we present an approach that incorporates multiple human input channels for interactive reinforcement learning in a unified framework and evaluate it on two robotic tasks with 20 inexperienced human subjects. To enable the robot to also handle potentially incorrect human input we incorporate a novel concept for self-confidence, which allows the robot to question human input after an initial learning phase. The second robotic task is specifically designed to investigate if this self-confidence can enable the robot to achieve learning progress even if the human input is partially incorrect. Further, we evaluate how humans react to suggestions of the robot, once the robot notices human input might be wrong. Our experimental evaluations show that our approach can successfully incorporate human input to accelerate the learning process in both robotic tasks even if it is partially wrong. However, not all humans were willing to accept the robot's suggestions or its questioning of their input, particularly if they do not understand the learning process and the reasons behind the robot's suggestions. We believe that the findings from this experimental evaluation can be beneficial for the future design of algorithms and interfaces of interactive reinforcement learning systems used by inexperienced users.
Conference
ICLR
Sharing Knowledge in Multi-Task Deep Reinforcement Learning
C. D’Eramo, D. Tateo, A. Bonarini, M. Restelli, J. Peters
International Conference on Learning Representations (ICLR)
Details
We study the benefit of sharing representations among tasks to enable the effective use of deep neural networks in Multi-Task Reinforcement Learning. We leverage the assumption that learning from different tasks, sharing common properties, is helpful to generalize the knowledge of them resulting in a more effective feature extraction compared to learning a single task. Intuitively, the resulting set of features offers performance benefits when used by Reinforcement Learning algorithms. We prove this by providing theoretical guarantees that highlight the conditions for which is convenient to share representations among tasks, extending the well-known finite-time bounds of Approximate Value-Iteration to the multi-task setting. In addition, we complement our analysis by proposing multi-task extensions of three Reinforcement Learning algorithms that we empirically evaluate on widely used Reinforcement Learning benchmarks showing significant improvements over the single-task counterparts in terms of sample efficiency and performance.
Conference
NeurIPS
Self-Paced Deep Reinforcement Learning
P. Klink, C. D’Eramo, J. Peters, J. Pajarinen
Advances in Neural Information Processing Systems (NeurIPS)
Oral Presentation (top 2% of accepted papers)Details
Figure 1: Environments used for experimental evaluation. For the point mass environment (a), the upper plot shows the target task. The shaded areas in picture (c) visualize the target distribution of ball positions (green) as well as the ball positions for which the initial policy succeeds (blue). Curriculum reinforcement learning (CRL) improves the learning speed and stability of an agent by exposing it to a tailored series of tasks throughout learning. Despite empirical successes, an open question in CRL is how to automatically generate a curriculum for a given reinforcement learning (RL) agent, avoiding manual design. In this paper, we propose an answer by interpreting the curriculum generation as an inference problem, where distributions over tasks are progressively learned to approach the target task. This approach leads to an automatic curriculum generation, whose pace is controlled by the agent, with solid theoretical motivation and easily integrated with deep RL algorithms. In the conducted experiments, the curricula generated with the proposed algorithm significantly improve learning performance across several environments and deep RL algorithms, matching or outperforming state-of-the-art existing CRL algorithms.
Conference
IJCAI
Generalized Mean Estimation in Monte-Carlo Tree Search
T. Dam, P. Klink, C. D’Eramo, J. Peters, J. Pajarinen
International Joint Conference on Artificial Intelligence (IJCAI)
Details
Figure 2: Performance of Power-UCT compared to UCT in rocksample. The mean of total discounted reward over 10001000 evaluation runs is shown by thick lines while the shaded area shows standard error. We consider Monte-Carlo Tree Search (MCTS) applied to Markov Decision Processes (MDPs) and Partially Observable MDPs (POMDPs), and the well-known Upper Confidence bound for Trees (UCT) algorithm. In UCT, a tree with nodes (states) and edges (actions) is incrementally built by the expansion of nodes, and the values of nodes are updated through a backup strategy based on the average value of child nodes. However, it has been shown that with enough samples the maximum operator yields more accurate node value estimates than averaging. Instead of settling for one of these value estimates, we go a step further proposing a novel backup strategy which uses the power mean operator, which computes a value between the average and maximum value. We call our new approach Power-UCT, and argue how the use of the power mean operator helps to speed up the learning in MCTS. We theoretically analyze our method providing guarantees of convergence to the optimum. Finally, we empirically demonstrate the effectiveness of our method in well-known MDP and POMDP benchmarks, showing significant improvement in performance and convergence speed w.r.t. state of the art algorithms.
2019
Conference
IJCNN
Exploration Driven by an Optimistic Bellman Equation
S. Tosatto, C. D’Eramo, J. Pajarinen, M. Restelli, J. Peters
IEEE International Joint Conference on Neural Networks (IJCNN)
Details
Exploring high-dimensional state spaces and finding sparse rewards are central problems in reinforcement learning. Exploration strategies are frequently either naïve (e.g., simplistic-greedy or Boltzmann policies), intractable (i.e., full Bayesian treatment of reinforcement learning) or rely heavily on heuristics. The lack of a tractable but principled exploration approach unnecessarily complicates the application of reinforcement learning to a broader range of problems. Efficient exploration can be accomplished by relying on the uncertainty of the state-action value function. To obtain the uncertainty, we maintain an ensemble of value function estimates and present an optimistic Bellman equation (OBE) for such ensembles. This OBE is derived from a relative entropy maximization principle and yields an implicit exploration bonus resulting in improved exploration during action selection. The implied exploration bonus can be seen as a well-principled type of intrinsic motivation and exhibits favorable theoretical properties. OBE can be applied to a wide range of algorithms. We propose two algorithms as an application of the principle: Optimistic Q-learning and Optimistic DQN which outperform comparison methods on standard benchmarks.
Conference
IJCNN
Exploiting Action-Value Uncertainty to Drive Exploration in Reinforcement Learning
C. D’Eramo, A. Cini, M. Restelli
IEEE International Joint Conference on Neural Networks (IJCNN)
Details
Most of the research in Reinforcement Learning (RL) focuses on balancing exploration and exploitation. Indeed, the reasons for the success or failure of an RL algorithm often deal with the choice between the execution of exploratory actions and the exploitation of actions that are known to be good. In the context of Multi-Armed Bandits (MABs), many algorithms have addressed this dilemma. In particular, Thompson Sampling (TS) is a solution that, besides having good theoretical properties, usually works very well in practice. Unfortunately, the success of TS in MAB problems has not been replicated in RL, where it has shown to scale very poorly w.r.t. the dimensionality of the problem. Nevertheless, the application of TS in RL, instead of more myopic strategies such as ε-greedy, remains a promising solution. This paper addresses such issue proposing several algorithms to use TS in RL and deep RL in a feasible way. We present these algorithms explaining the intuitions and theoretical considerations behind them and discussing their advantages and drawbacks. Furthermore, we provide an empirical evaluation on an increasingly complex set of RL problems, showing the benefit of TS w.r.t. other sampling strategies available in classical and more recent RL literature.
2017
Conference
ICML
Boosted Fitted Q-Iteration
S. Tosatto, M. Pirotta, C. D’Eramo, M. Restelli
International Conference on Machine Learning (ICML)
Details
This paper is about the study of B-FQI, an Approximated Value Iteration (AVI) algorithm that exploits a boosting procedure to estimate the action-value function in reinforcement learning problems. ...
Conference
AAAI
Estimating the Maximum Expected Value in Continuous Reinforcement Learning Problems
C. D’Eramo, A. Nuara, M. Pirotta, M. Restelli
AAAI Conference on Artificial Intelligence (AAAI)
Details
This paper is about the estimation of the maximum expected value of an infinite set of random variables.This estimation problem is relevant in many fields, like the Reinforcement Learning (RL) one.In RL it is well known that, in some stochastic environments, a bias in the estimation error can increase step-by-step the approximation error leading to large overestimates of the true action values. Recently, some approaches have been proposed to reduce such bias in order to get better action-value estimates, but are limited to finite problems.In this paper, we leverage on the recently proposed weighted estimator and on Gaussian process regression to derive a new method that is able to natively handle infinitely many random variables.We show how these techniques can be used to face both continuous state and continuous actions RL problems.To evaluate the effectiveness of the proposed approach we perform empirical comparisons with related approaches.
Conference
SSCI
Exploiting Structure and Uncertainty of Bellman Updates in Markov Decision Processes
D. Tateo, C. D’Eramo, A. Nuara, M. Restelli, A. Bonarini
IEEE Symposium Series on Computational Intelligence (SSCI)
Details
In many real-world problems stochasticity is a critical issue for the learning process. The sources of stochasticity come from the transition model, the explorative component of the policy or, even worse, from noisy observations of the reward function. For a finite number of samples, traditional Reinforcement Learning (RL) methods provide biased estimates of the action-value function possibly leading to poor estimates, then propagated by the application of the Bellman operator. While some approaches assume that the estimation bias is the key problem in the learning process, we show that in some cases this assumption does not necessarily hold. We propose a method that exploits the structure of the Bellman update and the uncertainty of the estimation in order to better use the amount of information provided by the samples. We show theoretical considerations about this method and its relation w.r.t. Q-Learning. Moreover, we test it in environments available in literature in order to demonstrate its effectiveness against other algorithms that focus on bias and sample-efficiency.
2016
Conference
ICML
Estimating Maximum Expected Value through Gaussian Approximation
C. D’Eramo, A. Nuara, M. Restelli
International Conference on Machine Learning (ICML)
Details
This paper is about the estimation of the maximum expected value of a set of independent random variables. The performance of several learning algorithms (e.g., Q-learning) is affected by the accur...