Research highlight · Reinforcement Learning Journal (RLJ) 2026 · 🏆 Outstanding Paper Award for Empirical Reinforcement Learning Research
The problem
Temporal-difference (TD) learning is the workhorse of reinforcement learning. Almost every popular method uses a semi-gradient update: it ignores how the bootstrapped target depends on the parameters being learned. This makes learning fast, but it can also make it diverge. Gradient TD methods fix the divergence, but they have rarely been used in practice because they learn too slowly.
Our idea
Iterated TD learning speeds up learning by training a chain of action-value functions in parallel, each one learning the Bellman update of the previous one. However, each function chases a moving target, which can make it unstable. We take the gradient through those moving targets, turning iterated TD into a true gradient TD method.
What we found
Gradient Iterated TD learns as fast as semi-gradient methods across a range of benchmarks, including Atari games. No previous gradient TD method had shown this.
Paper: Gradient Iterated Temporal-Difference Learning
T. Vincent, K. Gerhardt, Y. Tripathi, H. Maraqten, A. White, M. White, J. Peters, C. D’Eramo
Read the paper →