Reply to thread

Message: [QUOTE="Bombastus, post: 92386, member: 2178"] MLPerf Endpoints v0.7 requires every published system curve to contain at least seven measured operating points, with each performance point running for at least 600 seconds. Instead of handing you one flattering throughput number, it shows how the same serving stack behaves as concurrent load rises. The shift matters beyond [B][URL='https://goldmidi.com/community/threads/amd-nvidia-split-mlperf-6-1-ai-benchmark-leads.77754/']the current MLPerf 6.1 inference results[/URL][/B] because traditional datacenter results still organize performance around fixed scenarios such as Server and Interactive. Endpoints moves the measurement boundary outward and treats the deployed service, reached through an API, as the thing being tested. The original [B][URL='https://arxiv.org/abs/1911.02549']MLPerf Inference benchmark framework[/URL][/B] was built to make hardware and software systems comparable under controlled workloads. Endpoints keeps that discipline but changes what gets exposed to the load generator, which makes routers, queues, serving engines, batching, cache policy, and model placement part of the visible result. [HEADING=2]Endpoints move the benchmark to the API boundary[/HEADING] The system under test can be addressed as an endpoint rather than wired directly into a benchmark-specific backend. MLCommons built the client to send concurrent HTTP requests, consume streaming responses, and drive different serving APIs while keeping the benchmark harness from becoming the obvious bottleneck. For a buyer, this changes the object being compared. A bare-metal rack, a cloud instance, or a managed inference service can all be measured from the request boundary, provided the submission follows the workload and disclosure rules that apply to its division. A URL does not make unlike systems magically identical. Hardware count, software stack, model, precision, availability, and submission status still matter, but the API boundary gets closer to what an application actually touches when it sends production traffic. Endpoints also does not simply rename the old Server scenario. MLCommons describes it as a fuller characterization that effectively replaces Server and Interactive for this style of datacenter serving while leaving the benchmark focused on how the complete system behaves under load. [HEADING=2]A curve shows where throughput starts hurting users[/HEADING] Each operating point fixes a concurrency level and records system throughput, interactivity, and time to first token at the 95th percentile. System throughput measures total output tokens per second, while interactivity tracks output tokens per second per user. Those numbers pull in opposite directions once a system gets busy. Adding concurrent requests can improve accelerator utilization and total token output, yet each individual user may see slower streaming and a longer wait before the first token appears. The useful result is therefore a curve rather than a podium number. One deployment may be stronger at low concurrency where responsiveness matters, while another may carry far more simultaneous work before its per-user speed falls below an acceptable level. Submitters cannot build that curve from two convenient samples. A submission needs at least seven and no more than 32 operating points, and every point runs for at least ten minutes, which makes short-lived burst behavior less useful. There is another wrinkle hiding inside the curve. The model, endpoint, and software stack stay consistent across its points, but the system configuration may change between runs, including how resources are divided between prefill and decode. Accuracy is checked again for each distinct configuration. A vendor can tune the serving system for different load levels, but it cannot freely change the configuration at the busy end of the curve and assume the earlier accuracy result still covers it. [HEADING=2]The best operating point depends on your service[/HEADING] A chatbot with people waiting on every response may care about a tight time-to-first-token limit and healthy per-user streaming speed. An internal batch-heavy service may accept slower individual responses if higher concurrency produces much better total throughput from the same hardware. This is where a single maximum tokens-per-second claim becomes slippery. The highest-throughput operating point may sit in a region where first-token delay or per-user speed would be unacceptable for the application you actually plan to run. Endpoints makes that trade-off visible, but it does not choose the threshold for you. Your latency target, expected concurrency, model, traffic shape, and budget still determine which part of the curve is useful. The current framework also distinguishes verified, provisional, and unverified results, so the shape of a published curve is not the only thing worth checking. A procurement comparison should keep verification status and the exact system configuration in view before turning a graph into a capacity plan. MLCommons is also building Endpoints around rolling submissions rather than relying only on large benchmark rounds. The important change is structural. Generative AI serving now moves too quickly for one fixed operating point to describe how a system behaves from a handful of users through saturation. [/QUOTE]

Name