Listen to this Post
Introduction: Why Agentic RL Is the Real Test for Open Models
Agentic reinforcement learning (RL) represents a major shift in how large language models are trained and evaluated. Instead of optimizing a single response in isolation, agentic RL teaches models to plan, act, observe outcomes, and adapt over multi-step interactions with tools and environments. This paradigm is essential for real-world AI agents that must reason under uncertainty, call external services, and pursue goals across long horizons. Until now, whether open-source backbone models like GPT-OSS could reliably support this training regime remained an open—and risky—question.
What Agentic RL Really Changes in LLM Training
Traditional RL or preference-based fine-tuning relies on static datasets and short-term rewards. Agentic RL, by contrast, operates in a closed loop: the model collects its own data by acting in an environment, receives delayed rewards, and updates its policy based on entire trajectories. Decisions like tool selection, query reformulation, and execution order suddenly matter as much as the final answer. This makes credit assignment harder—but also far more powerful.
Why LinkedIn’s Use Case Raises the Stakes
As an AI-first company, LinkedIn builds agents that must handle incomplete information, evolving user intent, and structured tools. Recruiters, job seekers, and learners don’t want a single clever answer; they need systems that can coordinate actions across steps. Agentic RL provides the mathematical and practical foundation for this kind of reliability, making it a natural fit for large-scale professional AI products.
GPT-OSS Enters the Arena
GPT-OSS has already demonstrated performance comparable to OpenAI’s o3-mini and o4-mini in standard benchmarks. However, most prior work focused on supervised fine-tuning without tool use. This raised a critical question: can GPT-OSS survive the instability, long horizons, and tool-driven complexity of agentic RL? This work set out to answer that by pushing GPT-OSS directly into on-policy reinforcement learning.
Experimental Setup and Benchmarks
The team used the open-source verl framework, a popular choice for agentic RL experimentation. Training tasks included GSM8K for math reasoning, ReTool for tool-assisted problem solving, and verifiable instruction-following benchmarks. Experiments focused primarily on GPT-OSS-20B, with GPT-OSS-120B also validated for key fixes. Qwen-2.5-32B served as a comparative baseline for training dynamics.
Early Warning Signs: When Training Goes Off the Rails
Initial runs revealed severe instability. Rewards failed to increase, KL divergence and entropy exploded, and gradient norms spiraled out of control. Compared to Qwen-32B, GPT-OSS-20B showed dramatically worse learning behavior. These symptoms pointed to a fundamental breakdown in on-policy PPO assumptions rather than simple hyperparameter issues.
The Core Problem: PPO On-Policy Integrity Was Broken
Pure on-policy PPO requires the importance sampling ratio to be exactly one. However, in GPT-OSS training, non-zero clipping appeared even in strictly on-policy settings. The root cause was a mismatch between current and old log-probabilities—fatal for PPO stability.
Why MoE Architectures Made It Worse
GPT-OSS uses a Mixture of Experts (MoE) architecture, where a gating network routes tokens to different experts. Before verl 0.3.0, log-probabilities were computed via two separate forward passes. Small numerical differences or stochastic routing caused different experts to be selected across passes, breaking the assumption that π(a|s) equals πold(a|s). The result: false PPO clipping and unstable updates.
The Critical Fix: Forcing the Ratio Back to One
The solution was deceptively simple but mathematically precise. In known on-policy settings, the old log-probability is replaced with the current log-probability (detached from gradients). This enforces an importance ratio of exactly one, restoring PPO’s theoretical guarantees and neutralizing MoE routing nondeterminism.
Why Stability Still Didn’t Fully Return
Even after fixing PPO clipping, gradients continued to explode and rewards stagnated. Simplifying the task to GSM8K—without tools or multi-step trajectories—did not resolve the issue. This indicated a deeper mismatch between training-time and inference-time execution.
Training–Inference Mismatch: The Silent Killer
Inference engines like vLLM and SGLang aggressively optimize for throughput, while training under FSDP prioritizes numerical stability. These differences can subtly convert on-policy RL into off-policy optimization. Once rollout correction was applied to align training with inference behavior, gradient norms stabilized—but learning was still slower than expected.
Attention Becomes the Prime Suspect
Freezing attention layers produced nearly identical reward curves, suggesting that learning was dominated by MoE layers. At the same time, large token-level probability mismatches appeared between inference and training stacks using different attention kernels. This shifted focus squarely onto GPT-OSS’s attention mechanism.
Attention Sinks: Powerful but Poorly Supported
GPT-OSS relies on attention sinks—learnable scalar parameters that act as virtual tokens in softmax normalization. They improve stability for long-context and streaming scenarios. However, FlashAttention v2 did not support them, and even FlashAttention v3 lacked a proper backward pass. This created a catastrophic training–inference mismatch.
The Breakthrough: Full Attention Sink Support in FlashAttention v3
By adapting the forward pass from vLLM’s FlashAttention fork and implementing a correct backward pass, the team finally aligned training and inference. Attention sinks now participated correctly in both passes, eliminating hidden inconsistencies that had crippled learning.
Results: GPT-OSS Finally Learns Like an Agent
With the fix applied, GPT-OSS-20B showed dramatically faster convergence across tasks. GSM8K rewards rose quickly, instruction-following no longer collapsed, and ReTool training stabilized with steady gains and no gradient explosions. Validation accuracy increased meaningfully, confirming real learning rather than numerical luck.
Memory Bottlenecks Nearly Killed Training
Another major obstacle was memory blow-ups under FSDP. A Hugging Face inference path duplicated hidden states across all experts, triggering massive GPU allocations—up to 180 GiB—during log-probability computation. This made GPT-OSS-20B training on high-end H200 clusters unexpectedly fragile.
Fixing MoE Memory Explosion
The team patched the Transformers implementation to use the more memory-efficient training forward path, processing experts sequentially rather than materializing them in bulk. This eliminated repeated out-of-memory failures and made large-context training feasible.
Sequence Parallelism: Scaling Context Without Breaking GPUs
Agentic RL requires ever-growing context windows. By combining FSDP with sequence parallelism and FlashAttention v3, the team reduced per-GPU activation memory dramatically. Tokens were repartitioned across GPUs, with minimal communication overhead confined to the attention layer.
Why This Matters for Real Agents
These fixes aren’t academic. Multi-step agents with tool use demand long contexts, stable gradients, and strict on-policy guarantees. Without addressing attention sinks, MoE nondeterminism, and memory efficiency, open models simply can’t compete in agentic settings.
Conclusion: GPT-OSS Is No Longer Just “Promising”
This work shows that GPT-OSS can serve as a scalable, stable backbone for agentic reinforcement learning—but only after deep, surgical engineering. By restoring PPO integrity, fixing attention sinks, and taming memory usage, the team turned a fragile model into a viable agentic platform.
What Undercode Say:
The most important takeaway isn’t that GPT-OSS “can” do agentic RL—it’s that open models require far more infrastructure honesty than closed ones. Proprietary stacks quietly co-design training, inference, kernels, and hardware. Open-source ecosystems expose every mismatch, and GPT-OSS paid that price in public. What’s impressive here is not just the fix, but the methodology: isolate, simplify, falsify assumptions, and rebuild the stack until theory and practice finally agree.
This also reframes the MoE debate. MoE isn’t just a scaling trick; it fundamentally changes RL dynamics. Without careful handling, MoE routing noise turns on-policy algorithms into something else entirely. The enforced log-probability substitution is a warning sign for anyone training MoE agents with PPO.
Attention sinks are the sleeper issue. They look like a minor architectural detail, but here they were the difference between collapse and convergence. This suggests many reported “RL instabilities” in large models may actually be kernel-level inconsistencies, not algorithmic failures.
Finally, memory efficiency is no longer optional. Agentic RL plus long context plus MoE is a worst-case scenario for GPUs. Sequence parallelism combined with attention-aware kernels may become the default recipe for next-generation agent training.
🔍 Fact Checker Results
✅ GPT-OSS showed unstable PPO behavior due to MoE log-probability mismatch.
✅ FlashAttention v2 lacked proper attention sink support.
❌ No evidence suggests these issues were solvable by hyperparameter tuning alone.
📊 Prediction
Agentic RL will become the defining benchmark for open models in 2026. Projects that fail to align training and inference stacks at the kernel level will quietly stall, while models like GPT-OSS—now battle-tested—will emerge as serious alternatives to closed-agent systems.
🕵️📝✔️Let’s dive deep and fact‑check.
References:
Reported By: huggingface.co
Extra Source Hub (Possible Sources for article):
https://www.digitaltrends.com
Wikipedia
OpenAi & Undercode AI
Image Source:
Unsplash
Undercode AI DI v2
Bing
🔐JOIN OUR CYBER WORLD [ CVE News • HackMonitor • UndercodeNews ]
📢 Follow UndercodeNews & Stay Tuned:
𝕏 formerly Twitter 🐦 | @ Threads | 🔗 Linkedin | 🦋BlueSky | 🐘Mastodon
