Research highlight · European Workshop on Reinforcement Learning (EWRL) 2026 · 🏆 Best Paper Award
The problem
Streaming reinforcement learning processes experience the way natural learners do: one sample at a time, with a batch size of 1 and no replay buffer. It has recently been shown to work with deep networks, but only when the agent can observe everything it needs. When the agent must remember past observations, it needs recurrent networks, and the standard way of training them breaks down in this setting. Truncated backpropagation through time collapses to a single step, and exact real-time recurrent learning (RTRL) is far too expensive.
Our idea
We use recurrent trace units, a diagonal recurrent architecture for which exact RTRL costs only linear time and memory in the number of parameters. They slot cleanly into existing streaming algorithms for both discrete and continuous control.
What we found
On a memory diagnostic with chains of up to 128 steps, our method keeps working where streaming baselines with feedforward, GRU and RTU networks collapse. On POPGym it is competitive with batched PPO, and on partially observable MuJoCo it recovers a substantial part of batched performance, all without a replay buffer.
Paper: Streaming Reinforcement Learning under Partial Observability with Real-Time Recurrent Learning
N. Farr, A. Reddi, J. Peters, C. D’Eramo
Read the paper →