MLPerf Inference v6.1 added an Edge Agentic workload built around 1,007 turns across 20 recorded coding conversations on a single accelerator. The test uses Qwen3.6-27B and keeps only one request in flight, so a fast result cannot hide behind datacenter-scale batching.
Seen beside the wider MLPerf 6.1 hardware race, this benchmark measures a different kind of pressure. Your model has to stay responsive while the conversation gets longer, tool outputs pile up, and previous turns remain relevant to whatever happens next.
A fixed 32K served context keeps the playing field controlled rather than letting each device stretch the window to suit its memory capacity. The recorded trajectories peak at roughly 23.5K input tokens, which is long enough for context handling to become part of the performance problem without forcing valid turns to be dropped.
MLCommons tracks time to first token, time per output token, and end-to-end turn latency for this workload. A machine can therefore generate quickly once it starts yet still feel sluggish if rebuilding the prompt takes too long at every turn.
The performance side is a deterministic replay rather than a fresh coding agent improvising its way through every run. Recorded tool calls drive the conversations forward, while an inline multiset comparison checks the executed calls during the same pass, keeping the timing workload reproducible without throwing correctness completely out of the window.
Accuracy is not left floating either. The benchmark uses BFCL v4 single-turn cases as a gate, drawing on function-calling evaluation across real tool-use settings rather than accepting speed from a model that stops calling tools correctly. The reference run scored 86.23 percent overall and 87.96 percent on the category-balanced normalized measure, with submissions required to clear 97 percent of both reference scores.
Those two gates matter because one large category cannot simply swamp weaker behavior elsewhere. A system has to remain close to the reference on both the sample-weighted result and the normalized result before its performance numbers count.
The reference setup fixes Qwen3.6-27B at a Q4_K_M quantization with a 32,768-token context and a single parallel slot. Quantization gets the 27-billion-parameter model into an edge-friendly memory footprint, but the benchmark does not let vendors win simply by shrinking the model until its tool use falls apart because the accuracy gates still apply.
The 32K limit is a benchmark parameter, not a claim about the maximum context a particular device can support. None of the 20 reference trajectories overflow it, which means every recorded turn can finish and differences in truncation policy do not quietly decide the result.
Context reuse becomes especially important here. NVIDIA’s Jetson AGX Thor submission completed the 1,007-turn replay in 24 minutes and 36 seconds, while its llama.cpp reference run on the same board took about two hours and 37 minutes. NVIDIA attributed much of the gain to NVFP4 execution, FP8 KV cache, multi-token prediction, and reuse of cached state across turns.
The cache detail is the interesting part. NVIDIA reported that roughly 96 percent of prompt tokens could be served from hot cache during its run, which shows how much repeated prefill work exists inside a long agent trajectory. Raw token-generation speed still matters, but avoiding needless reconstruction of old context can change the shape of the result.
The replay’s inline tool-call score barely moved in return, rising from 0.6335 to 0.6374. Spending more compute therefore produced a slower run without a meaningful tool-use payoff in this particular workload, a useful warning when vendors market longer reasoning as an automatic upgrade.
Agentic edge inference is consequently not a miniature throughput contest. It tests whether one device can keep a long, tool-using session accurate and responsive while context keeps growing, cache strategy starts to dominate prompt handling, and extra reasoning can become dead weight instead of useful work.
Seen beside the wider MLPerf 6.1 hardware race, this benchmark measures a different kind of pressure. Your model has to stay responsive while the conversation gets longer, tool outputs pile up, and previous turns remain relevant to whatever happens next.
A fixed 32K served context keeps the playing field controlled rather than letting each device stretch the window to suit its memory capacity. The recorded trajectories peak at roughly 23.5K input tokens, which is long enough for context handling to become part of the performance problem without forcing valid turns to be dropped.
Agentic edge inference changes what the benchmark measures
Conventional inference tests can reward a system for pushing enormous batches through a server. Edge Agentic instead targets one interactive user, one accelerator, and one live conversation, which makes first-response delay and per-token delay much harder to bury.MLCommons tracks time to first token, time per output token, and end-to-end turn latency for this workload. A machine can therefore generate quickly once it starts yet still feel sluggish if rebuilding the prompt takes too long at every turn.
The performance side is a deterministic replay rather than a fresh coding agent improvising its way through every run. Recorded tool calls drive the conversations forward, while an inline multiset comparison checks the executed calls during the same pass, keeping the timing workload reproducible without throwing correctness completely out of the window.
Accuracy is not left floating either. The benchmark uses BFCL v4 single-turn cases as a gate, drawing on function-calling evaluation across real tool-use settings rather than accepting speed from a model that stops calling tools correctly. The reference run scored 86.23 percent overall and 87.96 percent on the category-balanced normalized measure, with submissions required to clear 97 percent of both reference scores.
Those two gates matter because one large category cannot simply swamp weaker behavior elsewhere. A system has to remain close to the reference on both the sample-weighted result and the normalized result before its performance numbers count.
Growing context turns memory behavior into performance
Every new agent turn carries baggage. Earlier prompts, model responses, and tool results expand the next input, so the accelerator repeatedly faces work that a short single-shot benchmark barely exposes.The reference setup fixes Qwen3.6-27B at a Q4_K_M quantization with a 32,768-token context and a single parallel slot. Quantization gets the 27-billion-parameter model into an edge-friendly memory footprint, but the benchmark does not let vendors win simply by shrinking the model until its tool use falls apart because the accuracy gates still apply.
The 32K limit is a benchmark parameter, not a claim about the maximum context a particular device can support. None of the 20 reference trajectories overflow it, which means every recorded turn can finish and differences in truncation policy do not quietly decide the result.
Context reuse becomes especially important here. NVIDIA’s Jetson AGX Thor submission completed the 1,007-turn replay in 24 minutes and 36 seconds, while its llama.cpp reference run on the same board took about two hours and 37 minutes. NVIDIA attributed much of the gain to NVFP4 execution, FP8 KV cache, multi-token prediction, and reuse of cached state across turns.
The cache detail is the interesting part. NVIDIA reported that roughly 96 percent of prompt tokens could be served from hot cache during its run, which shows how much repeated prefill work exists inside a long agent trajectory. Raw token-generation speed still matters, but avoiding needless reconstruction of old context can change the shape of the result.
Reasoning overhead can make an agent worse
More internal reasoning sounds useful for an agent, yet the reference configuration deliberately disables it. MLCommons reported 86.23 percent accuracy with reasoning off against 78.19 percent with reasoning on for the accuracy gate, while the reasoning-enabled performance replay took about 60 percent longer.The replay’s inline tool-call score barely moved in return, rising from 0.6335 to 0.6374. Spending more compute therefore produced a slower run without a meaningful tool-use payoff in this particular workload, a useful warning when vendors market longer reasoning as an automatic upgrade.
Agentic edge inference is consequently not a miniature throughput contest. It tests whether one device can keep a long, tool-using session accurate and responsive while context keeps growing, cache strategy starts to dominate prompt handling, and extra reasoning can become dead weight instead of useful work.
