MLPerf Inference v6.1 includes the benchmark’s first cross-vendor heterogeneous submission, combining eight NVIDIA H200s with eight AMD Instinct MI350X GPUs. Cisco tied the two accelerator families into one inference pool over a Silicon One G200 network rather than treating them as separate islands.
The result sits outside the split MLPerf 6.1 hardware results because the interesting part is no longer which individual GPU wins. Mixed hardware forces the serving layer to decide where work belongs, how much each node should receive, and whether the network can move state without eating the gain.
Older and newer accelerators already coexist in plenty of real fleets. Replacing every useful GPU at once is expensive, so a heterogeneous inference cluster becomes attractive if the scheduler can turn mismatched capacity into one usable service instead of leaving half the rack stranded.
Splitting those phases also makes the network part of the inference path. KV-cache state has to reach the hardware responsible for decode, and slow transfers can erase whatever benefit came from assigning each phase to a better-suited resource.
A mixed GPU pool adds another complication because equal request counts do not mean equal work. Different accelerators can have different throughput, memory capacity, precision support, software maturity, and response under load, so a naive round-robin scheduler can turn the slower side into a queue.
The broader idea already appears in phase-aware serving on heterogeneous GPU clusters, where model placement, precision, and workload phases are treated as scheduling variables rather than fixed assumptions. MLPerf v6.1 matters because heterogeneous serving has now moved from an academic systems problem into a standardized industry benchmark.
The practical win is not that every GPU behaves identically. It is that a serving layer can expose one endpoint while routing work around the differences instead of forcing every device to march in lockstep.
Unlike MangoBoost’s separate prefill and decode experiments, this multi-region configuration kept both phases colocated on each node. Its scheduler sent requests according to the capacity each node could absorb, which is important because the MI300X and MI355X do not deliver identical performance.
Offline throughput reached 285,454 tokens per second, equal to 97 percent of the sum of the four nodes running independently. Server reached 253,501 tokens per second at 94 percent scaling efficiency, where stricter latency requirements leave less room to hide network delay behind queued work.
MangoBoost attributed the remaining Offline loss largely to the trans-Pacific round trip. The number is more interesting than the record label because it puts a visible price on geographic distribution instead of treating remote capacity as free.
Distance also changes what good scheduling means. A request sent to an idle GPU thousands of miles away can still be a bad decision if network latency pushes the response beyond a service target, so raw accelerator utilization cannot be the only goal.
Heterogeneous LLM inference gives those leftovers a possible second life. The scheduler can treat each node as capacity with a particular speed and latency profile rather than pretending every accelerator is interchangeable, allowing older hardware to keep serving work it can finish efficiently.
The catch is software complexity. Operators need model formats and runtimes that function across different accelerator stacks, routing that reacts to changing load, enough observability to spot a weak node, and networking that does not turn remote capacity into expensive idle time.
MLPerf’s two hybrid v6.1 submissions do not prove every mixed fleet will scale cleanly. They do show two distinct ways to avoid the weakest-device trap, with Cisco separating work across different vendors and MangoBoost assigning requests proportionally across different generations and locations.
For buyers, this changes the infrastructure question. A future inference cluster may not need one accelerator generation everywhere, but it does need a serving layer smart enough to know which hardware should handle each piece of work and when a distant GPU is no longer worth using.
The result sits outside the split MLPerf 6.1 hardware results because the interesting part is no longer which individual GPU wins. Mixed hardware forces the serving layer to decide where work belongs, how much each node should receive, and whether the network can move state without eating the gain.
Older and newer accelerators already coexist in plenty of real fleets. Replacing every useful GPU at once is expensive, so a heterogeneous inference cluster becomes attractive if the scheduler can turn mismatched capacity into one usable service instead of leaving half the rack stranded.
Mixed accelerators make scheduling the real contest
Cisco’s v6.1 entry used two nodes and disaggregated prefill from decode while pooling H200 and MI350X hardware. Prefill processes the prompt and builds the KV cache, while decode repeatedly generates new tokens, so the two phases place different pressure on compute, memory bandwidth, and communication.Splitting those phases also makes the network part of the inference path. KV-cache state has to reach the hardware responsible for decode, and slow transfers can erase whatever benefit came from assigning each phase to a better-suited resource.
A mixed GPU pool adds another complication because equal request counts do not mean equal work. Different accelerators can have different throughput, memory capacity, precision support, software maturity, and response under load, so a naive round-robin scheduler can turn the slower side into a queue.
The broader idea already appears in phase-aware serving on heterogeneous GPU clusters, where model placement, precision, and workload phases are treated as scheduling variables rather than fixed assumptions. MLPerf v6.1 matters because heterogeneous serving has now moved from an academic systems problem into a standardized industry benchmark.
The practical win is not that every GPU behaves identically. It is that a serving layer can expose one endpoint while routing work around the differences instead of forcing every device to march in lockstep.
A four-site cluster shows what distance costs
MangoBoost pushed the same idea much farther with 32 AMD accelerators spread across four sites on two continents. The pool combined 16 MI300X GPUs with 16 newer MI355X GPUs operated by MangoBoost, Dell, TensorWave, and Microsoft Azure, then presented them as one GPT-OSS-120B endpoint.Unlike MangoBoost’s separate prefill and decode experiments, this multi-region configuration kept both phases colocated on each node. Its scheduler sent requests according to the capacity each node could absorb, which is important because the MI300X and MI355X do not deliver identical performance.
Offline throughput reached 285,454 tokens per second, equal to 97 percent of the sum of the four nodes running independently. Server reached 253,501 tokens per second at 94 percent scaling efficiency, where stricter latency requirements leave less room to hide network delay behind queued work.
MangoBoost attributed the remaining Offline loss largely to the trans-Pacific round trip. The number is more interesting than the record label because it puts a visible price on geographic distribution instead of treating remote capacity as free.
Distance also changes what good scheduling means. A request sent to an idle GPU thousands of miles away can still be a bad decision if network latency pushes the response beyond a service target, so raw accelerator utilization cannot be the only goal.
Mixed fleets change how spare GPUs can be valued
Homogeneous clusters are easier to tune because every worker has roughly the same capabilities. Real infrastructure ages unevenly, though, and organizations can end up with several GPU generations, cloud capacity from different providers, and machines bought under completely different procurement cycles.Heterogeneous LLM inference gives those leftovers a possible second life. The scheduler can treat each node as capacity with a particular speed and latency profile rather than pretending every accelerator is interchangeable, allowing older hardware to keep serving work it can finish efficiently.
The catch is software complexity. Operators need model formats and runtimes that function across different accelerator stacks, routing that reacts to changing load, enough observability to spot a weak node, and networking that does not turn remote capacity into expensive idle time.
MLPerf’s two hybrid v6.1 submissions do not prove every mixed fleet will scale cleanly. They do show two distinct ways to avoid the weakest-device trap, with Cisco separating work across different vendors and MangoBoost assigning requests proportionally across different generations and locations.
For buyers, this changes the infrastructure question. A future inference cluster may not need one accelerator generation everywhere, but it does need a serving layer smart enough to know which hardware should handle each piece of work and when a distant GPU is no longer worth using.
