Listen to this Post
Introduction: Breaking the Boundaries of Reinforcement Learning in LLMs
Large Language Models (LLMs) have taken a massive leap forward in recent years, but behind every improvement lies a series of breakthroughs in training strategies. Reinforcement Learning with Human Feedback (RLHF) has long relied on PPO (Proximal Policy Optimization), yet as tasks grow in complexity and outputs become longer, PPO’s dependence on a value model shows its limitations. This has led to a wave of innovations: GRPO, which removes the value model dependency; DAPO, which fine-tunes GRPO for greater efficiency and stability; and finally GSPO, a sequence-level optimization method that solves deep-rooted problems in MoE (Mixture of Experts) architectures.
This article unpacks the journey from PPO to GRPO, DAPO, and GSPO — revealing not just what changed, but why these changes are critical for scaling LLM performance.
the Original (Approx. )
PPO, once the gold standard for reinforcement learning in LLMs, struggles with long sequences and complex tasks due to inaccurate value model predictions. GRPO emerged to remove PPO’s value model dependency, using importance sampling to optimize policy updates without resampling after each update. While GRPO improves scalability, it still suffers from efficiency issues and instability — especially in MoE models — because its per-token optimization introduces high variance.
DAPO builds on GRPO with four targeted improvements:
- Clip-Higher – Raises the upper clipping bound to prevent good tokens from being prematurely capped, solving the “Matthew Effect” problem.
- Dynamic Sampling – Ensures diversity in training samples by avoiding all-perfect or all-failed sample sets, thus reducing wasted computation.
- Token-Level Gradient Loss – Gives equal weight to every token across all samples, preventing long, high-quality responses from having diluted gradient signals.
- Overlong Reward Shaping – Penalizes overly long responses proportionally, preventing verbosity from hurting performance.
These refinements make DAPO more efficient and stable than GRPO, but in MoE architectures — where only certain experts are activated per inference — GRPO still struggles due to routing instability. Different expert activations between the old and new policy can cause structural noise, high variance, and frequent clipping, which kills gradient contributions.
Enter GSPO, which shifts optimization from token-level to sequence-level. This change aligns the reward granularity with the optimization target, greatly reducing variance. Sequence-level ratios are length-normalized and clipped for stability, and all tokens in a sequence share the same weight, ensuring intra-sequence consistency.
By bypassing token-level fluctuations, GSPO makes MoE training far more stable without costly routing replay. In practice, GSPO optimizes fewer tokens but achieves faster convergence and better results. The QWen3 series already employs GSPO, signaling a broader move from PPO → GRPO → GSPO in LLM training.
📢 What Undercode Say: (Approx. 40 Lines)
The evolution from PPO to GSPO is not just about improving efficiency — it’s about matching the training method to the nature of the task. PPO’s design works well for short, simple interactions, but it cracks under the weight of modern LLM workloads. Its reliance on a value model makes it prone to inaccuracies when outputs become long and nuanced.
GRPO’s breakthrough was eliminating the value model dependency and relying purely on policy probability ratios. This allowed for more direct and scalable optimization but introduced its own pitfalls. Token-level importance sampling meant every token carried its own update weight, which could be wildly inconsistent in long sequences or unstable architectures like MoE.
DAPO’s incremental improvements show that small, surgical changes can have a big impact:
Clip-Higher fixes the imbalance that lets high-probability tokens keep gaining while low-probability but valuable tokens get capped.
Dynamic Sampling tackles the problem of “gradient starvation” when all samples score identically.
Token-Level Gradient Loss stops important long answers from being mathematically drowned out by short ones.
Overlong Reward Shaping applies a smart, soft penalty to verbosity, encouraging concise but accurate outputs.
However, in MoE models, GRPO’s structure itself is the problem. Token-level optimization clashes with the inherently unstable routing of experts, creating noise that is too high for stable training. Even with DAPO’s fixes, the problem persists.
GSPO’s sequence-level optimization is a conceptual leap. Instead of treating every token separately, GSPO treats the entire generated output as one training unit — matching the reward granularity to the optimization granularity. This prevents token-level volatility from derailing training. The clipping now applies to the whole sequence, ensuring consistent gradient flow.
By doing away with Routing Replay, GSPO also eliminates a huge engineering burden. MoE routing can change naturally during training without destabilizing optimization, making the process lighter and more adaptable.
From a theoretical perspective, GSPO reduces variance, increases stability, and accelerates convergence. It doesn’t just optimize fewer tokens — it optimizes better. And because it’s sequence-based, it inherently aligns better with real-world evaluation metrics, which almost always judge an output as a whole rather than token by token.
This shift signals a broader trend in RLHF: moving away from micro-managing individual tokens toward optimizing for complete, coherent, high-quality responses.
✅ Fact Checker Results
PPO’s limitations with long outputs are well-documented and supported by RLHF research.
GRPO’s instability in MoE settings is a recognized problem, validated by multiple training experiments.
GSPO’s sequence-level optimization approach is already in real-world use (e.g., QWen3 series).
🔮 Prediction
Sequence-level optimization methods like GSPO will soon become the new standard for training LLMs, especially as MoE architectures gain traction. Within the next 1–2 years, token-level optimization will be phased out for large-scale, high-performance models, and hybrid reward shaping methods will combine GSPO’s stability with adaptive, context-aware reinforcement strategies.
Do you want me to also create SEO-optimized subheadings for each major improvement in GRPO, DAPO, and GSPO so it ranks better on search engines? That would make the article even more discoverable.
🕵️📝✔️Let’s dive deep and fact‑check.
References:
Reported By: huggingface.co
Extra Source Hub:
https://www.digitaltrends.com
Wikipedia
OpenAi & Undercode AI
Image Source:
Unsplash
Undercode AI DI v2
🔐JOIN OUR CYBER WORLD [ CVE News • HackMonitor • UndercodeNews ]
📢 Follow UndercodeNews & Stay Tuned:
𝕏 formerly Twitter 🐦 | @ Threads | 🔗 Linkedin | 🦋BlueSky | 🐘Mastodon
