StraTA: Incentivizing Agentic Reinforcement Learning with Strategic Trajectory Abstraction
Abstract
Strategic Trajectory Abstraction framework enhances long-horizon decision making in large language models by introducing trajectory-level strategies that improve sample efficiency and performance across interactive environments.
Large language models (LLMs) are increasingly used as interactive agents, but optimizing them for long-horizon decision making remains difficult because current methods are largely purely reactive, which weakens both exploration and credit assignment over extended trajectories. In this work, we present Strategic Trajectory Abstraction (StraTA), a simple framework that introduces an explicit trajectory-level strategy into agentic reinforcement learning (RL). StraTA samples a compact strategy from the initial task state, conditions subsequent actions on that strategy, and trains strategy generation and action execution jointly with a hierarchical GRPO-style rollout design, further enhanced by diverse strategy rollout and critical self-judgment. Experiments on ALFWorld, WebShop, and SciWorld show that StraTA consistently improves both sample efficiency and final performance over strong baselines. StraTA reaches success rates of 93.1% on ALFWorld and 84.2% on WebShop. On SciWorld, StraTA attains a 63.5% overall score, outperforming frontier closed-source models.
Community
Hi everyone! 👋 Sharing our latest work: StraTA: Incentivizing Agentic Reinforcement Learning with Strategic Trajectory Abstraction.
Should long-horizon LLM agents stay purely reactive? We argue they shouldn't. When the agent has to decide both the immediate action and the overall course of action from the current state alone, planning and execution get tangled — and exploration and credit assignment suffer.
Our fix: StraTA samples a compact natural-language strategy from the initial state and conditions all subsequent actions on it. We jointly train both levels with a hierarchical GRPO-style rollout, enhanced by diverse strategy sampling and critical self-judgment.
Does it work? StraTA improves both sample efficiency and final performance, outperforming both frontier closed-source models and prior RL baselines.
We believe StraTA highlights the value of explicit trajectory-level abstraction for more structured and effective long-horizon agentic RL.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Hierarchical Reinforcement Learning with Augmented Step-Level Transitions for LLM Agents (2026)
- DPEPO: Diverse Parallel Exploration Policy Optimization for LLM-based Agents (2026)
- Co-Evolution of Policy and Internal Reward for Language Agents (2026)
- SLEA-RL: Step-Level Experience Augmented Reinforcement Learning for Multi-Turn Agentic Training (2026)
- Dynamic Dual-Granularity Skill Bank for Agentic RL (2026)
- T$^2$PO: Uncertainty-Guided Exploration Control for Stable Multi-Turn Agentic Reinforcement Learning (2026)
- RewardFlow: Topology-Aware Reward Propagation on State Graphs for Agentic RL with Large Language Models (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper