Eight unchanged AMD Instinct MI355X GPUs raised GPT-OSS-120B throughput by 28 percent Offline and 38 percent server-to-server between MLPerf 6.0 and 6.1. The silicon did not get a respin, more memory, or a higher GPU count during that comparison.
Placed beside the latest MLPerf inference hardware results, the jump is easy to mistake for a footnote. It is actually a useful reminder that an accelerator ships with a performance ceiling nobody reaches on day one.
Fresh models expose rough edges in kernels, schedulers, memory paths, and communication code. Months of profiling can turn the same box into a materially faster inference server without touching the motherboard.
Attention got a more surgical treatment. AMD replaced one unified attention kernel with separate paths for prefill and decode, because prefill is heavily compute-bound while decode spends far more time moving through a paged KV cache.
The split improved attention performance by 25 to 28 percent, but not total application throughput by the same amount. The distinction is important because a faster kernel only accelerates the slice of runtime that actually uses it.
Decode received a new paged-attention implementation with coalesced KV-cache loads and instruction scheduling designed to overlap memory movement with matrix work. Prefill went through a variable-length flash-attention path that handles masking and related operations inside the kernel instead of bouncing through extra work between launches.
Kernel fusion trims another kind of waste. Folding operations together can remove intermediate reads, writes, launches, or standalone quantization passes, which is valuable when the arithmetic itself is already extremely fast.
AMD changed its vLLM scheduling policy so the Server run could target MLPerf's time-to-first-token and per-output-token limits directly. The company attributes an 8 percent Server throughput improvement to that scheduler work, separate from the underlying attention changes.
The same tension appears in KV-cache-aware online scheduling for LLM inference, where request arrivals and cache growth are treated as scheduling constraints rather than background details. Once a model is serving many users, memory occupancy and queue decisions become part of performance just as surely as matrix throughput.
Graph capture and warmup matter for similar reasons. Extending captured execution paths across the decode batch sizes reduces repeated launch overhead and keeps the GPU on a better-prepared path as batch shape changes.
None of these tricks adds compute units. They reduce the amount of time existing compute units spend stalled, waiting, launching tiny operations, moving redundant data, or sitting behind a badly balanced queue.
For an operator, fewer GPUs for the same throughput can affect rack space, power, networking, and how quickly new capacity has to be purchased. Software improvements can therefore change the useful lifetime of an installed fleet even when the hardware invoice is already paid.
AMD is not the only vendor showing the pattern in this round. Intel reported 36 percent higher GPT-OSS-120B Server throughput and 27 percent higher Offline throughput on the same four Arc Pro B70 GPUs used in MLPerf 6.0, while its Xeon 6980P result also rose substantially on unchanged processor hardware.
The important caveat is workload specificity. A 38 percent gain on GPT-OSS-120B Server does not mean every model, prompt length, batch size, or production service becomes 38 percent faster after a software update.
Llama 2 70B on the same MI355X platform improved only about 3 percent between those two MLPerf rounds. Different models stress different kernels and memory paths, so mature workloads may have less slack left to recover.
A production deployment can also miss a benchmark gain when its traffic shape differs from the tuned case. Longer prompts, lower concurrency, different output lengths, or a stricter latency target can move the bottleneck elsewhere, so the useful test is the same model and serving pattern your users actually generate.
Placed beside the latest MLPerf inference hardware results, the jump is easy to mistake for a footnote. It is actually a useful reminder that an accelerator ships with a performance ceiling nobody reaches on day one.
Fresh models expose rough edges in kernels, schedulers, memory paths, and communication code. Months of profiling can turn the same box into a materially faster inference server without touching the motherboard.
The benchmark exposes what software leaves unused
GPT-OSS-120B is a mixture-of-experts model, so much of its compute lands in MoE layers rather than one dense path. AMD tuned its low-precision MoE kernels for the token distributions produced by each MLPerf scenario, reporting an 8 percent end-to-end gain Offline and 5 percent in Server from that work alone.Attention got a more surgical treatment. AMD replaced one unified attention kernel with separate paths for prefill and decode, because prefill is heavily compute-bound while decode spends far more time moving through a paged KV cache.
The split improved attention performance by 25 to 28 percent, but not total application throughput by the same amount. The distinction is important because a faster kernel only accelerates the slice of runtime that actually uses it.
Decode received a new paged-attention implementation with coalesced KV-cache loads and instruction scheduling designed to overlap memory movement with matrix work. Prefill went through a variable-length flash-attention path that handles masking and related operations inside the kernel instead of bouncing through extra work between launches.
Kernel fusion trims another kind of waste. Folding operations together can remove intermediate reads, writes, launches, or standalone quantization passes, which is valuable when the arithmetic itself is already extremely fast.
Serving software decides whether fast kernels stay busy
A benchmark server can own excellent kernels and still waste expensive GPU time through poor admission control. Requests arrive with different prompt lengths and output lengths, while the scheduler has to keep batches full without blowing through memory or violating latency limits.AMD changed its vLLM scheduling policy so the Server run could target MLPerf's time-to-first-token and per-output-token limits directly. The company attributes an 8 percent Server throughput improvement to that scheduler work, separate from the underlying attention changes.
The same tension appears in KV-cache-aware online scheduling for LLM inference, where request arrivals and cache growth are treated as scheduling constraints rather than background details. Once a model is serving many users, memory occupancy and queue decisions become part of performance just as surely as matrix throughput.
Graph capture and warmup matter for similar reasons. Extending captured execution paths across the decode batch sizes reduces repeated launch overhead and keeps the GPU on a better-prepared path as batch shape changes.
None of these tricks adds compute units. They reduce the amount of time existing compute units spend stalled, waiting, launching tiny operations, moving redundant data, or sitting behind a badly balanced queue.
Same-hardware gains change capacity planning
The cluster-level effect is larger than a prettier benchmark bar. AMD reports that 72 MI355X GPUs in MLPerf 6.1 delivered more GPT-OSS-120B throughput than 94 of the same GPUs managed in round 6.0.For an operator, fewer GPUs for the same throughput can affect rack space, power, networking, and how quickly new capacity has to be purchased. Software improvements can therefore change the useful lifetime of an installed fleet even when the hardware invoice is already paid.
AMD is not the only vendor showing the pattern in this round. Intel reported 36 percent higher GPT-OSS-120B Server throughput and 27 percent higher Offline throughput on the same four Arc Pro B70 GPUs used in MLPerf 6.0, while its Xeon 6980P result also rose substantially on unchanged processor hardware.
The important caveat is workload specificity. A 38 percent gain on GPT-OSS-120B Server does not mean every model, prompt length, batch size, or production service becomes 38 percent faster after a software update.
Llama 2 70B on the same MI355X platform improved only about 3 percent between those two MLPerf rounds. Different models stress different kernels and memory paths, so mature workloads may have less slack left to recover.
A production deployment can also miss a benchmark gain when its traffic shape differs from the tuned case. Longer prompts, lower concurrency, different output lengths, or a stricter latency target can move the bottleneck elsewhere, so the useful test is the same model and serving pattern your users actually generate.
