Listen to this Post
Introduction: The New Era of AI Requires More Than Bigger Models
The artificial intelligence race is no longer defined only by building larger models. The real challenge has shifted toward making enormous AI systems practical, affordable, and fast enough for real-world deployment. As open-weight models continue to grow into hundreds of billions of parameters, the gap between theoretical capability and usable performance has become one of the biggest engineering challenges in the AI industry.
The deployment of a model such as Qwen3.5-397B demonstrates this reality. With nearly 400 billion parameters, advanced multimodal capabilities, and a sophisticated Mixture-of-Experts (MoE) architecture, the model represents the future direction of large-scale AI. However, running such a model efficiently requires much more than simply loading weights onto powerful hardware.
Engineers must carefully redesign the relationship between software, compilers, memory systems, networking, and specialized AI accelerators. Every bottleneck matters. A small inefficiency in memory access, routing communication, or kernel execution can dramatically reduce performance when operating at hundreds of billions of parameters.
Google’s TPU optimization team approached this challenge using a first-principles engineering strategy rather than traditional trial-and-error optimization. Instead of rebuilding solutions for every new AI architecture, the team created reusable optimization modules that could adapt to future models.
Through this approach, Qwen3.5-397B achieved major performance improvements on Google’s Ironwood TPU platform, delivering approximately 3.1x faster decode performance and 4.7x faster prefill performance under heavy workloads.
The achievement represents a broader shift in AI infrastructure development: the future of artificial intelligence depends not only on smarter models, but also on smarter systems engineering.
Qwen3.5-397B: A 400 Billion Parameter Model That Behaves Like a Smaller AI System
The Power of Sparse Mixture-of-Experts Architecture
At first glance, Qwen3.5-397B appears to be an extremely demanding model because it contains approximately 397 billion parameters. However, its Mixture-of-Experts architecture changes how computation is performed.
Unlike traditional dense models where every parameter participates in every token generation step, Qwen3.5 activates only a small portion of its network.
For every token:
Total parameters: 397 billion
Active parameters: 17 billion
Activation ratio: approximately 4.3%
This allows the model to provide the intelligence capacity of a 400B-class system while maintaining an inference workload closer to a much smaller model.
The architecture uses:
512 specialized experts
Top-10 expert routing per token
One shared expert pathway
Hybrid attention mechanisms
Multimodal training across text, image, and video
This design represents the direction modern AI models are moving toward: increasing intelligence without requiring proportional increases in computational cost.
The Hybrid Architecture Behind Qwen3.5
A Combination of Traditional Attention and Gated DeltaNet
One of Qwen3.5’s most important innovations is its hybrid attention system.
Instead of relying entirely on traditional Transformer attention, the model combines:
Gated DeltaNet (GDN) linear attention
Grouped Query Attention (GQA)
The network contains:
60 layers
Hidden dimension: 4096
Vocabulary size: 248,320 tokens
Native context length: 262,144 tokens
Extended context capability: over 1 million tokens using YaRN scaling
The model follows a repeating block structure where approximately 75% of layers use Gated DeltaNet while 25% use traditional attention.
This combination attempts to solve one of the biggest problems in modern AI:
Long-context memory efficiency.
Gated DeltaNet: Reducing the Cost of Long Context AI
Replacing Quadratic Attention With Linear Memory Scaling
Traditional Transformer attention has a major weakness.
The computational complexity grows according to:
O(S²)
where S represents sequence length.
As context windows increase from thousands to hundreds of thousands of tokens, this becomes extremely expensive.
Gated DeltaNet solves this problem by using recurrent state updates.
Instead of creating a massive attention matrix, the system maintains a compact hidden memory state.
Its complexity becomes:
O(S)
This dramatically reduces memory requirements during long-context generation.
The GDN system uses:
64 value heads
16 query/key heads
Head dimension of 128
Learned gating mechanism
Causal convolution preprocessing
The result is a more efficient attention mechanism designed for future AI applications requiring extremely long conversations, documents, and multimodal reasoning.
Grouped Query Attention: Optimizing Memory Usage
Reducing KV Cache Pressure During Generation
While GDN handles most of the network, Qwen3.5 still uses standard attention components through Grouped Query Attention.
The GQA configuration includes:
32 query heads
2 key-value heads
Head dimension of 256
RoPE dimension of 64
The advantage is a significantly smaller KV cache.
However, this created an unexpected hardware challenge.
Traditional tensor parallelism assumes attention heads can be evenly divided between devices.
But Qwen3.5 only has:
2 KV heads
Trying to split this across:
8 devices
creates:
2 ÷ 8 = 0.25 heads per device
which is impossible.
This forced engineers to redesign the entire hardware distribution strategy.
The Major Engineering Challenge: Scaling MoE Models on TPU
Why Traditional Parallelism Failed
Large AI models normally rely on tensor parallelism.
The model is divided across multiple accelerators, allowing each device to process a portion of the workload.
However, Qwen3.5’s architecture broke traditional assumptions.
The problems included:
Extremely small KV head count
Hundreds of experts
Massive communication requirements
Large memory footprint
Replicating attention weights across devices increased memory consumption.
Sharding attention caused impossible divisions.
The result was poor scalability.
Early testing showed the system could support only around:
200 concurrent requests
instead of the expected:
512 concurrent requests
A completely different approach was required.
The Solution: Attention Data Parallelism + Expert Parallelism
Creating a Hybrid Sharding Strategy
Engineers developed a new architecture:
Attention:
8-way Data Parallelism
MoE Layers:
8-way Expert Parallelism
This approach allowed:
Each device to maintain complete attention capability
KV cache consistency
Reduced communication overhead
Efficient distribution of experts
The 512 experts were distributed:
512 experts ÷ 8 devices = 64 experts per device
This avoided duplicating the massive parameter footprint while maintaining efficient execution.
The result was a balanced system designed specifically around Qwen3.5’s architecture.
Communication Optimization: Removing Hidden Latency
Making Expert Routing Faster
Mixture-of-Experts models require constant communication between devices.
Tokens must be:
Routed to experts
Processed
Returned to the correct device
Initially, this required multiple collective communication operations.
Engineers optimized this by combining routing metadata:
Expert indexes
Top-k weights
into a single packed structure.
This reduced communication operations and cut routing overhead.
The team also replaced standard All-Reduce operations with a custom:
Hierarchical Reduce-Scatter
implemented using:
Pallas
Mosaic
TPU-native communication pathways
The system used micro-batching and asynchronous transfers to overlap communication with computation.
Deep Analysis: The Hardware-Level Optimization Strategy
Custom Kernels With JAX and Pallas
The biggest performance improvements came from replacing generic compiler paths with manually optimized kernels.
Engineers used:
JAX custom kernels
Pallas programming framework
TPU SparseCore acceleration
Hardware-aware memory scheduling
This allowed direct control over:
Virtual memory layout
Registers
TensorCore execution
DMA transfers
Ragged Page Attention Optimization
The KV cache management system originally suffered from indexing overhead.
The previous block size:
–block-size=16
created excessive indexing operations.
Engineers increased it:
–block-size=256
The result:
Decode latency improvement:
428µs → 283µs
Performance gain:
33.8%
Batched RPA Kernel Optimization
Instead of processing requests individually, engineers grouped multiple decode streams.
Example configuration:
--max-num-batched-tokens=1024 --max-num-seqs=64
This improved:
Memory alignment
Instruction efficiency
TPU utilization
SparseCore Token Routing Optimization
MoE routing creates irregular memory access patterns.
Traditional TensorCore execution caused:
Padding waste
Unaligned memory writes
Increased HBM bandwidth usage
The solution:
Move routing operations to TPU SparseCore.
Benefits:
Direct HBM gathering
Reduced intermediate tensors
Lower memory traffic
Memory operations improved:
Before:
HBM Reads: 20
HBM Writes: 15
After:
HBM Reads: 10
HBM Writes: 5
Gated DeltaNet Kernel Fusion
The GDN system previously required multiple memory transfers.
Engineers fused:
Causal convolution
Recurrent state updates
into a single execution unit.
This eliminated unnecessary HBM communication.
Additional improvements included:
FP32 → BF16 state conversion
Register-level caching
Algebraic optimization
The result was higher throughput without reducing accuracy.
Benchmark Results: A Major Performance Leap
From April to June 2026
After optimization, Qwen3.5-397B achieved:
Decode-heavy workloads:
Performance improvement:
3.1x
Prefill-heavy workloads:
Performance improvement:
4.7x
at:
512 concurrency
The results demonstrated that software optimization can unlock hardware potential without requiring new chips.
The TPU platform approached its theoretical roofline performance limits, showing that the remaining inefficiencies were minimized.
What Undercode Say:
AI Performance Is Becoming a Systems Engineering War
The Qwen3.5 optimization story reveals an important reality about modern artificial intelligence.
The future AI race will not be won only by companies creating larger models.
It will be won by organizations that can deploy those models efficiently.
A 400 billion parameter model sounds impossible for normal deployment.
However, MoE architecture changes the equation.
Sparse activation allows enormous intelligence while reducing computational demand.
The biggest lesson is that AI scaling has reached a new stage.
Hardware alone is no longer enough.
A powerful accelerator running inefficient software can still perform poorly.
The difference between average and exceptional AI infrastructure comes from optimization layers hidden below the model.
Memory management has become as important as raw compute power.
HBM capacity is now one of the biggest limitations in AI systems.
Communication between accelerators is another major bottleneck.
As models become more distributed, network efficiency becomes equally important.
The Qwen3.5 case demonstrates why generic parallelism approaches are reaching their limits.
Future AI systems will require architecture-specific optimization.
The development of reusable optimization modules is especially significant.
Instead of spending months optimizing each new model, engineers can build a universal performance framework.
This creates a software ecosystem where new AI architectures can be accelerated faster.
The importance of open-source optimization should not be underestimated.
Making advanced serving frameworks compatible with vLLM and SGLang allows enterprises worldwide to benefit from these improvements.
AI infrastructure is becoming similar to operating systems.
The winning platforms will be those capable of supporting many different models efficiently.
The move toward hybrid attention mechanisms is also a major trend.
Traditional Transformers are not disappearing, but they are being combined with new memory-efficient techniques.
Long-context AI will require these innovations.
Models capable of processing millions of tokens cannot depend entirely on quadratic attention.
The engineering techniques developed here will likely influence future AI accelerators.
Custom kernels, hardware-aware compilers, and specialized memory strategies will become standard.
The next generation of AI breakthroughs may come not from increasing model size, but from making existing models dramatically faster.
Qwen3.5 optimization represents a blueprint for the future of scalable artificial intelligence.
✅ Qwen3.5-397B uses Mixture-of-Experts architecture:
The model design described follows modern sparse MoE principles where only selected experts activate per token, reducing inference cost compared with dense models.
✅ Hardware-aware optimization improves AI inference performance:
Custom kernels, memory optimization, and communication improvements are proven techniques used across major AI hardware platforms.
❌ A 400B model does not literally run like a 20B model in every scenario:
Sparse activation reduces active computation, but total memory requirements, routing complexity, and system communication remain significant challenges.
✅ Gated DeltaNet represents a move toward efficient long-context processing:
Linear attention approaches aim to reduce the quadratic cost associated with traditional Transformer attention.
Prediction
(+1) Future AI deployments will rely heavily on modular optimization frameworks
As models continue to evolve rapidly, companies will move away from rebuilding infrastructure for every architecture. Reusable optimization layers will become a competitive advantage.
(+1) Mixture-of-Experts models will dominate enterprise AI
Sparse architectures provide a practical path toward larger and smarter models without unlimited hardware requirements.
(+1) Custom accelerator programming will become mainstream
Frameworks like Pallas, Triton, and specialized kernel languages will become essential skills for AI engineers.
(-1) AI hardware demand will continue increasing dramatically
Even with better efficiency, larger context windows, multimodal models, and agentic AI systems will continue consuming enormous computational resources.
(-1) Software complexity will become a major barrier
Future AI systems will require increasingly specialized engineering knowledge, making deployment harder for smaller organizations.
(+1) Open-source AI infrastructure will become more important
Projects supporting optimized inference will determine how widely advanced AI models can be adopted.
▶️ Related Video (76% Match):
🕵️📝Let’s dive deep and fact‑check.
🎓 Live Courses & Certifications:
Join Undercode Academy for Verified Certifications
🚀 Request a Custom Project:
Secure, high-velocity infrastructure and disruptive technological engineering. Contact our engineering team for high-tier development and proprietary systems:
[email protected]
💎 Smart Architecture | 🛡️ Secure by Design | ⭐ Trusted by Thousands
References:
Reported By: developers.googleblog.com
Extra Source Hub (Possible Sources for article):
https://www.reddit.com/r/AskReddit
Wikipedia
OpenAi & Undercode AI
Image Source:
Unsplash
Undercode AI DI v2
🔐JOIN OUR CYBER WORLD [ CVE News • HackMonitor • UndercodeNews ]
📢 Follow UndercodeNews & Stay Tuned:
𝕏 formerly Twitter 🐦 | @ Threads | 🔗 Linkedin | 🦋BlueSky | 🐘Mastodon | 📺Youtube
