Publications
Selected work
Journal articles, conference papers and workshop contributions, newest first.
2026
Journal
RLJ
Gradient Iterated Temporal-Difference Learning
T. Vincent, K. Gerhardt, Y. Tripathi, H. Maraqten, A. White, M. White, J. Peters, C. D’Eramo
Reinforcement Learning Journal (RLJ)
Outstanding Paper Award for Empirical Reinforcement Learning ResearchDetails
Figure 1: Representation of bootstrapping methods in the space of action-value functions. We represent the projection on the space of function approximation (in green) with dashed grey lines. Left: TD learning uses a target estimator Q¯0\bar{Q}_{0} to construct a regression target estimating the Bel Temporal-difference (TD) learning is highly effective at controlling and evaluating an agent's long-term outcomes. Most approaches in this paradigm implement a semi-gradient update to boost the learning speed, which consists of ignoring the gradient of the bootstrapped estimate. While popular, this type of update is prone to divergence, as Baird's counterexample illustrates. Gradient TD methods were introduced to overcome this issue, but have not been widely used, potentially due to issues with learning speed compared to semi-gradient methods. Recently, iterated TD learning was developed to increase the learning speed of TD methods. For that, it learns a sequence of action-value functions in parallel, where each function is optimized to represent the application of the Bellman operator over the previous function in the sequence. While promising, this algorithm can be unstable due to its semi-gradient nature, as each function tracks a moving target. In this work, we modify iterated TD learning by computing the gradients over those moving targets, aiming to build a powerful gradient TD method that competes with semi-gradient methods. Our evaluation reveals that this algorithm, called Gradient Iterated Temporal-Difference learning, has a competitive learning speed against semi-gradient methods across various benchmarks, including Atari games, a result that no prior work on gradient TD methods has demonstrated.
Workshop
EWRL
Streaming Reinforcement Learning under Partial Observability with Real-Time Recurrent Learning
N. Farr, A. Reddi, J. Peters, C. D’Eramo
European Workshop on Reinforcement Learning (EWRL)
Best Paper AwardDetails
Streaming reinforcement learning has emerged as an online learning paradigm that conforms to the restrictions of natural learning agents that process data incrementally, i.e. with a batch size of 1 and no replay buffer. While streaming RL has recently been shown to scale with deep function approximation with full observability, partially observable settings have remained out of reach. Truncated backpropagation through time collapses to a one-step gradient horizon under the streaming setting, and exact real-time recurrent learning is prohibitively expensive. We close this gap using recurrent trace units, a diagonal recurrent architecture that enables exact RTRL with linear time and memory complexity in the parameter count, and show that they integrate cleanly into existing streaming algorithms across both discrete and continuous control. On a MemoryChain diagnostic with chain lengths from 2 to 128, our method sustains performance where streaming TBPTT(1) baselines using feedforward, GRU, and RTU networks collapse. On five POPGym tasks and on partially observable MuJoCo continuous control, the streaming approach is competitive with batched PPO on POPGym and recovers a substantial fraction of batched performance on masked MuJoCo, despite using no replay buffer or batched updates.
2025
Journal
TMLR
Machine Learning with Physics Knowledge for Prediction: A Survey
J. Watson et al.
Transactions on Machine Learning Research (TMLR)
Survey CertificationDetails
This survey examines the broad suite of methods and models for combining machine learning with physics knowledge for prediction and forecast, with a focus on partial differential equations. These methods have attracted significant interest due to their potential impact on advancing scientific research and industrial practices by improving predictive models with small- or large-scale datasets and expressive predictive models with useful inductive biases. The survey has two parts. The first considers incorporating physics knowledge on an architectural level through objective functions, structured predictive models, and data augmentation. The second considers data as physics knowledge, which motivates looking at multi-task, meta, and contextual learning as an alternative approach to incorporating physics knowledge in a data-driven fashion. Finally, we also provide an industrial perspective on the application of these methods and a survey of the open-source ecosystem for physics-informed machine learning.
Journal
TMLR
Iterated Q-Network: Beyond One-Step Bellman Updates in Deep Reinforcement Learning
T. Vincent, D. Palenicek, B. Belousov, J. Peters, C. D’Eramo
Transactions on Machine Learning Research (TMLR)
J2C Certification (top 10% of accepted papers)Details
Figure 11: Losses and neural networks architectures of DQN and i-DQN. The dotted lines link the outputs of the neural networks to the objects they represent. The flash signs stress how each projection step is being learned, where Γ^\hat{\Gamma} is the empirical Bellman operator. The target networks The vast majority of Reinforcement Learning methods is largely impacted by the computation effort and data requirements needed to obtain effective estimates of action-value functions, which in turn determine the quality of the overall performance and the sample-efficiency of the learning procedure. Typically, action-value functions are estimated through an iterative scheme that alternates the application of an empirical approximation of the Bellman operator and a subsequent projection step onto a considered function space. It has been observed that this scheme can be potentially generalized to carry out multiple iterations of the Bellman operator at once, benefiting the underlying learning algorithm. However, till now, it has been challenging to effectively implement this idea, especially in high-dimensional problems. In this paper, we introduce iterated Q-Network (i-QN), a novel principled approach that enables multiple consecutive Bellman updates by learning a tailored sequence of action-value functions where each serves as the target for the next. We show that i-QN is theoretically grounded and that it can be seamlessly used in value-based and actor-critic methods. We empirically demonstrate the advantages of i-QN in Atari 2600 games and MuJoCo continuous control problems.
Conference
ICML
Monte-Carlo Tree Search with Uncertainty Propagation via Optimal Transport
T. Dam, P. Stenger, L. Schneider, J. Pajarinen, C. D’Eramo, O. Maillard
International Conference on Machine Learning (ICML)
Spotlight Poster (top 2% of submitted papers)Details
This paper introduces a novel backup strategy for Monte-Carlo Tree Search (MCTS) designed for highly stochastic and partially observable Markov decision processes. We adopt a probabilistic approach, modeling both value and action-value nodes as Gaussian distributions. We introduce a novel backup operator that computes value nodes as the Wasserstein barycenter of their action-value children nodes; thus, propagating the uncertainty of the estimate across the tree to the root node. We study our novel backup operator when using a novel combination of L^1-Wasserstein barycenter with α-divergence, by drawing a notable connection to the generalized mean backup operator. We complement our probabilistic backup operator with two sampling strategies, based on optimistic selection and Thompson sampling, obtaining our Wasserstein MCTS algorithm. We provide theoretical guarantees of asymptotic convergence to the optimal policy, and an empirical evaluation on several stochastic and partially observable environments, where our approach outperforms well-known related baselines.
2024
Conference
ICLR
Robust Adversarial Reinforcement Learning via Bounded Rationality Curricula
A. Reddi, M. Tölle, J. Peters, G. Chalvatzaki, C. D’Eramo
International Conference on Learning Representations (ICLR)
Spotlight Presentation (top 15% of accepted papers)Details
Figure 2: Robustness analysis of QARL and related baselines on MuJoCo swingup and locomotion problems (see title of heatmaps). Each heatmap shows the performance obtained after training for varying properties of the environment, described in the x−yx-y axes. The number next to the name of each algor Robustness against adversarial attacks and distribution shifts is a long-standing goal of Reinforcement Learning (RL). To this end, Robust Adversarial Reinforcement Learning (RARL) trains a protagonist against destabilizing forces exercised by an adversary in a competitive zero-sum Markov game, whose optimal solution, i.e., rational strategy, corresponds to a Nash equilibrium. However, finding Nash equilibria requires facing complex saddle point optimization problems, which can be prohibitive to solve, especially for high-dimensional control. In this paper, we propose a novel approach for adversarial RL based on entropy regularization to ease the complexity of the saddle point optimization problem. We show that the solution of this entropy-regularized problem corresponds to a Quantal Response Equilibrium (QRE), a generalization of Nash equilibria that accounts for bounded rationality, i.e., agents sometimes play random actions instead of optimal ones. Crucially, the connection between the entropy-regularized objective and QRE enables free modulation of the rationality of the agents by simply tuning the temperature coefficient. We leverage this insight to propose our novel algorithm, Quantal Adversarial RL (QARL), which gradually increases the rationality of the adversary in a curriculum fashion until it is fully rational, easing the complexity of the optimization problem while retaining robustness. We provide extensive evidence of QARL outperforming RARL and recent baselines across several MuJoCo locomotion and navigation problems in overall performance and robustness.
2022
Workshop
IROS
Modular Value Function Factorization in Multi-Agent Reinforcement Learning
O. Järnefelt, C. D’Eramo
Decision Making in Multi-Agent Systems Workshop, IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
Oral Presentation (5 of 16 papers)
2020
Conference
NeurIPS
Self-Paced Deep Reinforcement Learning
P. Klink, C. D’Eramo, J. Peters, J. Pajarinen
Advances in Neural Information Processing Systems (NeurIPS)
Oral Presentation (top 2% of accepted papers)Details
Figure 1: Environments used for experimental evaluation. For the point mass environment (a), the upper plot shows the target task. The shaded areas in picture (c) visualize the target distribution of ball positions (green) as well as the ball positions for which the initial policy succeeds (blue). Curriculum reinforcement learning (CRL) improves the learning speed and stability of an agent by exposing it to a tailored series of tasks throughout learning. Despite empirical successes, an open question in CRL is how to automatically generate a curriculum for a given reinforcement learning (RL) agent, avoiding manual design. In this paper, we propose an answer by interpreting the curriculum generation as an inference problem, where distributions over tasks are progressively learned to approach the target task. This approach leads to an automatic curriculum generation, whose pace is controlled by the agent, with solid theoretical motivation and easily integrated with deep RL algorithms. In the conducted experiments, the curricula generated with the proposed algorithm significantly improve learning performance across several environments and deep RL algorithms, matching or outperforming state-of-the-art existing CRL algorithms.