Our paper “Iterated Q-Network: Beyond One-Step Bellman Updates in Deep Reinforcement Learning” has been accepted in Transactions on Machine Learning Research (TMLR). It also received the J2C Certification, awarded to the top 10% of accepted papers.
T. Vincent, D. Palenicek, B. Belousov, J. Peters, C. D’Eramo
Most value-based methods apply the Bellman operator one step at a time. i-QN instead learns a sequence of action-value functions, each serving as the target for the next, so the agent effectively performs several Bellman updates at once. The approach is theoretically grounded, works with both value-based and actor-critic methods, and improves results on Atari 2600 and MuJoCo.