Publications
Selected work
Journal articles, conference papers and workshop contributions, newest first.
2026
Journal
TMLR
Do Not Imitate, Reinforce: Iterative Classification via Belief Refinement
M. Kallel, J. Tölle, A. Hendawy, C. D’Eramo
Transactions on Machine Learning Research (TMLR)
Details
Standard supervised classification trains models to imitate the exact labels provided by a perfect oracle. This imitation happens in a single pass, restricting the model to a fixed compute budget even when inputs vary in complexity. Moreover, the rigid training objective forces the model to express absolute certainty on its training data, resulting in overconfident predictions during evaluation. We propose Reinforced Iterative Classification (RIC), which replaces the imitative objective with Reinforcement Learning (RL). RIC deploys a recurrent agent that iteratively updates a predictive distribution over classes, receiving reward for stepwise improvement in prediction quality. The value function provides a natural halting criterion by estimating the remaining scope for improvement. We prove that the iterative formulation recovers the same optimal predictions as cross-entropy while yielding an anytime classifier. On image classification benchmarks, RIC matches the accuracy of supervised baselines with improved calibration and learns to allocate computation adaptively across inputs.
Journal
RA-L
Rationality Randomization with Maximum Entropy for Robust Dynamic Obstacle Avoidance on Legged Robots
J. L. Holgado-Alvarez, A. Reddi, G. Tiboni, C. D’Eramo
IEEE Robotics and Automation Letters (RA-L)
Details
Overview of RAMEN. (a) The performance-constrained curriculum starts near α ≈ 1, where obstacles move mostly randomly, and gradually introduces lower-α adversarial behaviors. (b) The high-level policy is trained on this diverse set of obstacle behaviors. (c) Deployment of the learned policy on a Unitree Go1 navigating among moving and static obstacles. Reinforcement learning (RL) for legged robots has recently gained significant attention due to its ability to enable traversal of complex terrains and navigation in cluttered environments. However, most existing approaches focus on static or predictably moving obstacles and rely on handcrafted curricula or manual task design, which may limit robustness to diverse obstacle behaviors. In this letter, we propose a method to automatically generate rich and dynamic obstacle-avoidance scenarios for learning robust navigation policies. Inspired by adversarial RL, we model obstacles as agents that induce challenging interactions for a protagonist policy. To improve robustness, we vary obstacle behavior by modulating their rationality, defined as the level of stochasticity in their actions. This is controlled via a temperature parameter that interpolates between deterministic, goal-directed motion and random behavior. We implement this through a domain randomization framework that samples rationality coefficients from a probability distribution, exposing the policy to a diverse range of obstacle behaviors during training. Crucially, this distribution is adapted online via a maximum entropy objective, which expands behavioral diversity while maintaining task performance. We demonstrate that the proposed method, RAtionality randomization with Maximum ENtropy (RAMEN), enables the learning of navigation policies that generalize across a wide range of simulated dynamic environments. Furthermore, we show successful zero-shot deployment on a legged robot operating in dynamic real-world scenarios using only onboard sensory inputs.
Journal
RLJ
PPO+: Enhancing Proximal Policy Optimization
M. Kallel, J. L. Holgado-Alvarez, S. Tosatto, C. D’Eramo
Reinforcement Learning Journal (RLJ)
Details
On-policy Reinforcement Learning (RL) offers desirable features such as stable learning, fewer policy updates, and the ability to evaluate a policy’s return during training. While recent efforts have focused on off-policy methods, achieving significant advancements, PPO remains the go-to algorithm for on-policy RL due to its apparent simplicity and effectiveness. Nonetheless, PPO relies on subtle, often poorly documented adjustments that can critically affect its performance. Our proposed approach, PPO+, is a principled adaptation of the PPO algorithm that strengthens its adherence to the on-policy objective, enhancing stability and efficiency. PPO+ demonstrates significantly improved asymptotic performance over PPO, and a substantially reduced performance gap with off-policy algorithms in several challenging continuous control tasks. Beyond just performance, our findings offer a fresh perspective on on-policy RL that can serve as guidance for future research endeavors.
Journal
RLJ
Gradient Iterated Temporal-Difference Learning
T. Vincent, K. Gerhardt, Y. Tripathi, H. Maraqten, A. White, M. White, J. Peters, C. D’Eramo
Reinforcement Learning Journal (RLJ)
Outstanding Paper Award for Empirical Reinforcement Learning ResearchDetails
Figure 1: Representation of bootstrapping methods in the space of action-value functions. We represent the projection on the space of function approximation (in green) with dashed grey lines. Left: TD learning uses a target estimator Q¯0\bar{Q}_{0} to construct a regression target estimating the Bel Temporal-difference (TD) learning is highly effective at controlling and evaluating an agent's long-term outcomes. Most approaches in this paradigm implement a semi-gradient update to boost the learning speed, which consists of ignoring the gradient of the bootstrapped estimate. While popular, this type of update is prone to divergence, as Baird's counterexample illustrates. Gradient TD methods were introduced to overcome this issue, but have not been widely used, potentially due to issues with learning speed compared to semi-gradient methods. Recently, iterated TD learning was developed to increase the learning speed of TD methods. For that, it learns a sequence of action-value functions in parallel, where each function is optimized to represent the application of the Bellman operator over the previous function in the sequence. While promising, this algorithm can be unstable due to its semi-gradient nature, as each function tracks a moving target. In this work, we modify iterated TD learning by computing the gradients over those moving targets, aiming to build a powerful gradient TD method that competes with semi-gradient methods. Our evaluation reveals that this algorithm, called Gradient Iterated Temporal-Difference learning, has a competitive learning speed against semi-gradient methods across various benchmarks, including Atari games, a result that no prior work on gradient TD methods has demonstrated.
2025
Journal
TMLR
Machine Learning with Physics Knowledge for Prediction: A Survey
J. Watson et al.
Transactions on Machine Learning Research (TMLR)
Survey CertificationDetails
This survey examines the broad suite of methods and models for combining machine learning with physics knowledge for prediction and forecast, with a focus on partial differential equations. These methods have attracted significant interest due to their potential impact on advancing scientific research and industrial practices by improving predictive models with small- or large-scale datasets and expressive predictive models with useful inductive biases. The survey has two parts. The first considers incorporating physics knowledge on an architectural level through objective functions, structured predictive models, and data augmentation. The second considers data as physics knowledge, which motivates looking at multi-task, meta, and contextual learning as an alternative approach to incorporating physics knowledge in a data-driven fashion. Finally, we also provide an industrial perspective on the application of these methods and a survey of the open-source ecosystem for physics-informed machine learning.
Journal
RLJ
Learning to Explore in Diverse Reward Settings via Temporal-Difference-Error Maximization
S. Griesbach, C. D’Eramo
Reinforcement Learning Journal (RLJ)
Details
(a) Four sampled trajectories of a mixed policy. Numerous heuristics and advanced approaches have been proposed for exploration in different settings for deep reinforcement learning. Noise-based exploration generally fares well with dense-shaped rewards and bonus-based exploration with sparse rewards. However, these methods usually require additional tuning to deal with undesirable reward settings by adjusting hyperparameters and noise distributions. Rewards that actively discourage exploration, i.e., with an action cost and no other dense signal to follow, can pose a major challenge. We propose a novel exploration method, Stable Error-seeking Exploration (SEE), that is robust across dense, sparse, and exploration-adverse reward settings. To this endeavor, we revisit the idea of maximizing the TD-error as a separate objective. Our method introduces three design choices to mitigate instability caused by far-off-policy learning, the conflict of interest of maximizing the cumulative TD-error in an episodic setting, and the non-stationary nature of TD-errors. SEE can be combined with off-policy algorithms without modifying the optimization pipeline of the original objective. In our experimental analysis, we show that a Soft-Actor Critic agent with the addition of SEE performs robustly across three diverse reward settings in a variety of tasks without hyperparameter adjustments.
Journal
RLJ
Eau De Q-Network: Adaptive Distillation of Neural Networks in Deep Reinforcement Learning
T. Vincent, T. Faust, Y. Tripathi, J. Peters, C. D’Eramo
Reinforcement Learning Journal (RLJ)
Details
Figure 13: We evaluate EauDeSAC on 66 MuJoCo environments for 11M with the small network. Left: PolyPruneSAC requires tuning as its performance depends on its hard-coded final sparsity level. Conversely, EauDeSAC avoids unsafe final sparsity levels by discovering its final sparsity level, therefore Recent works have successfully demonstrated that sparse deep reinforcement learning agents can be competitive against their dense counterparts. This opens up opportunities for reinforcement learning applications in fields where inference time and memory requirements are cost-sensitive or limited by hardware. Until now, dense-to-sparse methods have relied on hand-designed sparsity schedules that are not synchronized with the agent's learning pace. Crucially, the final sparsity level is chosen as a hyperparameter, which requires careful tuning as setting it too high might lead to poor performances. In this work, we address these shortcomings by crafting a dense-to-sparse algorithm that we name Eau De Q-Network (EauDeQN). To increase sparsity at the agent's learning pace, we consider multiple online networks with different sparsity levels, where each online network is trained from a shared target network. At each target update, the online network with the smallest loss is chosen as the next target network, while the other networks are replaced by a pruned version of the chosen network. We evaluate the proposed approach on the Atari 2600 benchmark and the MuJoCo physics simulator, showing that EauDeQN reaches high sparsity levels while keeping performances high.
Journal
TMLR
Iterated Q-Network: Beyond One-Step Bellman Updates in Deep Reinforcement Learning
T. Vincent, D. Palenicek, B. Belousov, J. Peters, C. D’Eramo
Transactions on Machine Learning Research (TMLR)
J2C Certification (top 10% of accepted papers)Details
Figure 11: Losses and neural networks architectures of DQN and i-DQN. The dotted lines link the outputs of the neural networks to the objects they represent. The flash signs stress how each projection step is being learned, where Γ^\hat{\Gamma} is the empirical Bellman operator. The target networks The vast majority of Reinforcement Learning methods is largely impacted by the computation effort and data requirements needed to obtain effective estimates of action-value functions, which in turn determine the quality of the overall performance and the sample-efficiency of the learning procedure. Typically, action-value functions are estimated through an iterative scheme that alternates the application of an empirical approximation of the Bellman operator and a subsequent projection step onto a considered function space. It has been observed that this scheme can be potentially generalized to carry out multiple iterations of the Bellman operator at once, benefiting the underlying learning algorithm. However, till now, it has been challenging to effectively implement this idea, especially in high-dimensional problems. In this paper, we introduce iterated Q-Network (i-QN), a novel principled approach that enables multiple consecutive Bellman updates by learning a tailored sequence of action-value functions where each serves as the target for the next. We show that i-QN is theoretically grounded and that it can be seamlessly used in value-based and actor-critic methods. We empirically demonstrate the advantages of i-QN in Atari 2600 games and MuJoCo continuous control problems.
2024
Journal
JAIR
A Unified Perspective on Value Backup and Exploration in Monte-Carlo Tree Search
T. Dam, C. D’Eramo, J. Peters, J. Pajarinen
Journal of Artificial Intelligence Research (JAIR)
Details
Figure 5: For different branching factor kk (rows) and depth dd (columns), the heatmaps show: the absolute error of the value estimate at the root node after the last simulation of each algorithm w.r.t. the respective optimal value (a), and w.r.t. the optimal value of UCT (b); regret at the root nod Monte-Carlo Tree Search (MCTS) is a class of methods for solving complex decision-making problems through the synergy of Monte-Carlo planning and Reinforcement Learning (RL). The highly combinatorial nature of the problems commonly addressed by MCTS requires the use of efficient exploration strategies for navigating the planning tree and quickly convergent value backup methods. These crucial problems are particularly evident in recent advances that combine MCTS with deep neural networks for function approximation. In this work, we propose two methods for improving the convergence rate and exploration based on a newly introduced backup operator and entropy regularization. We provide strong theoretical guarantees to bound convergence rate, approximation error, and regret of our methods. Moreover, we introduce a mathematical framework based on the use of the α-divergence for backup and exploration in MCTS. We show that this theoretical formulation unifies different approaches, including our newly introduced ones, under the same mathematical framework, allowing to obtain different methods by simply changing the value of α. In practice, our unified perspective offers a flexible way to balance between exploration and exploitation by tuning the single α parameter according to the problem at hand. We validate our methods through a rigorous empirical study from basic toy problems to the complex Atari games, and including both MDP and POMDP problems.
Journal
RLJ
Cyclicity-Regularized Coordination Graphs
O. Järnefelt, M. Kallel, C. D’Eramo
Reinforcement Learning Journal (RLJ)
Details
In parallel with the rise of the successful value function factorization approach, numerous recent studies on Cooperative Multi-Agent Reinforcement Learning (MARL) have explored the application of Coordination Graphs (CG) to model the communication requirements among the agent population. These coordination problems often exhibit structural sparsity, which facilitates accurate joint value function learning with CGs. Value-based methods necessitate the computation of argmaxes over the exponentially large joint action space, leading to the adoption of the max-sum method from the distributed constraint optimization (DCOP) literature. However, it has been empirically observed that the performance of max-sum deteriorates with an increase in the number of agents, attributed to the increased cyclicity of the graph. While previous works have tackled this issue by sparsifying the graph based on a metric of edge importance, thereby demonstrating improved performance, we argue that neglecting topological considerations during the sparsification procedure can adversely affect action selection. Consequently, we advocate for the explicit consideration of graph cyclicity alongside edge importances. We demonstrate that this approach results in superior performance across various challenging coordination problems.
Journal
TPAMI
On the Benefit of Optimal Transport for Curriculum Reinforcement Learning
P. Klink, C. D’Eramo, J. Peters, J. Pajarinen
IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)
Details
Fig. 1: Our approach (currot) addresses problems of existing curriculum RL methods, such as sprl, which create curricula between a distribution of initial tasks (blue) and a distribution of target tasks (green). In this example, the curriculum can change the task via two parameters c1c_{1} and c2c_{ Curriculum reinforcement learning (CRL) allows solving complex tasks by generating a tailored sequence of learning tasks, starting from easy ones and subsequently increasing their difficulty. Although the potential of curricula in RL has been clearly shown in various works, it is less clear how to generate them for a given learning environment, resulting in various methods aiming to automate this task. In this work, we focus on framing curricula as interpolations between task distributions, which has previously been shown to be a viable approach to CRL. Identifying key issues of existing methods, we frame the generation of a curriculum as a constrained optimal transport problem between task distributions. Benchmarks show that this way of curriculum generation can improve upon existing CRL methods, yielding high performance in various tasks with different characteristics.
2023
Journal
IJRR
Composable Energy Policies for Reactive Motion Generation and Reinforcement Learning
J. Urain, A. Li, P. Liu, C. D’Eramo, J. Peters
International Journal of Robotics Research (IJRR)
Details
Fig. 1: Composable Energy Policies generate trajectories reactively in a wide set of environments, as the ones in the figure. The generated trajectories are attracted to a target pose, visualized by the green sphere, and avoid a set of obstacles, shown as red spheres. Reactive motion generation problems are usually solved by computing actions as a sum of policies. However, these policies are independent of each other and thus, they can have conflicting behaviors when summing their contributions together. We introduce Composable Energy Policies (CEP), a novel framework for modular reactive motion generation. CEP computes the control action by optimization over the product of a set of stochastic policies. This product of policies will provide a high probability to those actions that satisfy all the components and low probability to the others. Optimizing over the product of the policies avoids the detrimental effect of conflicting behaviors between policies choosing an action that satisfies all the objectives. Besides, we show that CEP naturally adapts to the Reinforcement Learning problem allowing us to integrate, in a hierarchical fashion, any distribution as prior, from multimodal distributions to non-smooth distributions and learn a new policy given them.
2021
Journal
JMLR
Gaussian Approximation for Bias Reduction in Q-Learning
C. D’Eramo, A. Cini, A. Nuara, M. Pirotta, C. Alippi, J. Peters, M. Restelli
Journal of Machine Learning Research (JMLR)
Details
Temporal-Difference off-policy algorithms are among the building blocks of reinforcement learning (RL). Within this family, Q-Learning is arguably the most famous one, which has been widely studied and extended. The update rule of Q-learning involves the use of the maximum operator to estimate the maximum expected value of the return. However, this estimate is positively biased, and may hinder the learning process, especially in stochastic environments and when function approximation is used. We introduce the Weighted Estimator as an effective solution to mitigate the negative effects of overestimation in Q-Learning. The Weighted Estimator estimates the maximum expected value as a weighted sum of the action values, with the weights being the probabilities that each action value is the maximum. In this work, we study the problem from the statistical perspective of estimating the maximum expected value of a set of random variables and provide bounds to the bias and the variance of the Weighted Estimator, showing its advantages over other estimators present in literature. Then, we derive algorithms to enable the use of the Weighted Estimator, in place of the Maximum Estimator, in online and batch RL, and we introduce a novel algorithm for deep RL. Finally, we empirically evaluate our algorithms in a large set of heterogeneous problems, encompassing discrete and continuous, low and high dimensional, deterministic and stochastic environments. Experimental results show the effectiveness of the Weighted Estimator in controlling the bias of the estimate, resulting in better performance than representative baselines and robust learning w.r.t. a large set of diverse environments.
Journal
JMLR
MushroomRL: Simplifying Reinforcement Learning Research
C. D’Eramo, D. Tateo, A. Bonarini, M. Restelli, J. Peters
Journal of Machine Learning Research (JMLR)
Details
(a) Rendering of the game MushroomRL is an open-source Python library developed to simplify the process of implementing and running Reinforcement Learning (RL) experiments. Compared to other available libraries, MushroomRL has been created with the purpose of providing a comprehensive and flexible framework to minimize the effort in implementing and testing novel RL methodologies. Indeed, the architecture of MushroomRL is built in such a way that every component of an RL problem is already provided, and most of the time users can only focus on the implementation of their own algorithms and experiments. The result is a library from which RL researchers can significantly benefit in the critical phase of the empirical analysis of their works. MushroomRL stable code, tutorials and documentation can be found at https://github.com/MushroomRL/mushroom-rl.
Journal
JMLR
A Probabilistic Interpretation of Self-Paced Learning with Applications to Reinforcement Learning
P. Klink, H. Abdulsamad, B. Belousov, C. D’Eramo, J. Peters, J. Pajarinen
Journal of Machine Learning Research (JMLR)
Details
Across machine learning, the use of curricula has shown strong empirical potential to improve learning from data by avoiding local optima of training objectives. For reinforcement learning (RL), curricula are especially interesting, as the underlying optimization has a strong tendency to get stuck in local optima due to the exploration-exploitation trade-off. Recently, a number of approaches for an automatic generation of curricula for RL have been shown to increase performance while requiring less expert knowledge compared to manually designed curricula. However, these approaches are seldomly investigated from a theoretical perspective, preventing a deeper understanding of their mechanics. In this paper, we present an approach for automated curriculum generation in RL with a clear theoretical underpinning. More precisely, we formalize the well-known self-paced learning paradigm as inducing a distribution over training tasks, which trades off between task complexity and the objective to match a desired task distribution. Experiments show that training on this induced distribution helps to avoid poor local optima across RL algorithms in different tasks with uninformative rewards and challenging exploration requirements.
2020
Journal
Frontiers
Multi-Channel Interactive Reinforcement Learning for Sequential Tasks
D. Koert, M. Kircher, V. Salikutluk, C. D’Eramo, J. Peters
Frontiers in Robotics and AI
Details
The ability to learn new tasks by sequencing already known skills is an important requirement for future robots. Reinforcement learning is a powerful tool for this as it allows for a robot to learn and improve on how to combine skills for sequential tasks. However, in real robotic applications, the cost of sample collection and exploration prevent the application of reinforcement learning for a variety of tasks. To overcome these limitations, human input during reinforcement can be beneficial to speed up learning, guide the exploration and prevent the choice of disastrous actions. Nevertheless, there is a lack of experimental evaluations of multi-channel interactive reinforcement learning systems solving robotic tasks with input from inexperienced human users, in particular for cases where human input might be partially wrong. Therefore, in this paper, we present an approach that incorporates multiple human input channels for interactive reinforcement learning in a unified framework and evaluate it on two robotic tasks with 20 inexperienced human subjects. To enable the robot to also handle potentially incorrect human input we incorporate a novel concept for self-confidence, which allows the robot to question human input after an initial learning phase. The second robotic task is specifically designed to investigate if this self-confidence can enable the robot to achieve learning progress even if the human input is partially incorrect. Further, we evaluate how humans react to suggestions of the robot, once the robot notices human input might be wrong. Our experimental evaluations show that our approach can successfully incorporate human input to accelerate the learning process in both robotic tasks even if it is partially wrong. However, not all humans were willing to accept the robot's suggestions or its questioning of their input, particularly if they do not understand the learning process and the reasons behind the robot's suggestions. We believe that the findings from this experimental evaluation can be beneficial for the future design of algorithms and interfaces of interactive reinforcement learning systems used by inexperienced users.