GC-TTT - Test-time Offline Reinforcement Learning on Goal-related Experience

Marco Bagatella1,2$\ast$, Mert Albaba1, 2$\ast$, Jonas Hübotter1, Georg Martius2,3,
Andreas Krause1,
$\ast$ Equal contribution.
1ETH Zürich    2Max Planck Institute - Intelligent Systems    3University of Tübingen
Paper
We introduce test-time training in the context of offline goal-conditioned reinforcement learning. The same data used for pre-training is filtered and leveraged to improve the policy locally during evaluation. This results in significant performance gains in standard benchmarks (left) when combined with common offline RL backbones, GC-BC, GC-IQL, and SAW.

Abstract

Foundation models compress a large amount of information in a single, large neural network, which can then be queried for individual tasks. There are strong parallels between this widespread framework and offline goal-conditioned reinforcement learning algorithms: a universal value function is trained on a large number of goals, and the policy is evaluated on a single goal in each test episode. Extensive research in foundation models has shown that performance can be substantially improved through test-time training, specializing the model to the current goal. We find similarly that test-time offline reinforcement learning on experience related to the test goal can lead to substantially better policies at minimal compute costs. We propose a novel self-supervised data selection criterion, which selects transitions from an offline dataset according to their relevance to the current state and quality with respect to the evaluation goal. We demonstrate across a wide range of high-dimensional loco-navigation and manipulation tasks that fine-tuning a policy on the selected data for a few gradient steps leads to significant performance gains over standard offline pre-training. Our goal-conditioned test-time training (GC-TTT) algorithm applies this routine in a receding-horizon fashion during evaluation, adapting the policy to the current trajectory as it is being rolled out. Finally, we study compute allocation at inference, demonstrating that, at comparable costs, GC-TTT induces performance gains that are not achievable by scaling model size.

GC-TTT specializes the agent to the next steps for achieving its target goal.

BibTeX

@misc{bagatella2025testtime,
      title={Test-time Offline Reinforcement Learning on Goal-related Experience}, 
      author={Mert Albaba and Marco Bagatella and Jonas Hübotter and Georg Martius and Andreas Krause},
      year={2025},
      eprint={2507.18809},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2507.18809}, 
    }