Reply to thread

Message: [QUOTE="Bombastus, post: 92377, member: 2178"] MLPerf Inference v6.1 turns retrieval-augmented generation into two measured workloads, one for database ingestion and one for end-to-end question answering. The split matters because a production RAG system can lose time before a language model even sees the final prompt. The test starts with 2,515 frozen Wikipedia HTML pages and turns them into roughly 107,000 searchable passages. Each passage is 768 characters long with 32 characters of overlap, which keeps boundary facts from vanishing while limiting how much unrelated text rides along. This is a different problem from [B][URL='https://goldmidi.com/community/threads/amd-nvidia-split-mlperf-6-1-ai-benchmark-leads.77754/']the broader MLPerf 6.1 hardware results[/URL][/B] because the score belongs to a chain of components rather than one model call. Fast accelerators help, but retrieval, reranking, CPU work, memory placement, and repeated language-model passes can all become the slow part. [HEADING=2]RAG performance starts before the first question[/HEADING] The ingestion workload parses each source page, flattens tables and lists into text, chunks the result, creates embeddings, and builds a FAISS HNSW vector index. MLPerf reports this side in documents per second, so indexing speed is measured separately from how quickly the finished database can answer questions. The separation is useful because ingestion and serving stress different parts of a machine. A system that builds an index quickly may still stumble during repeated retrieval and reasoning, while a slower indexing setup could serve the question workload efficiently once the database exists. The question set comes from [B][URL='https://aclanthology.org/2025.naacl-long.243/']the FRAMES multi-hop evaluation[/URL][/B], which contains 824 questions requiring information to be combined across multiple documents. MLPerf freezes the document snapshot instead of pulling live Wikipedia pages, removing a moving target that would otherwise make one submission subtly different from another. The question pipeline is deliberately busy. A query rewriter can produce up to three focused subqueries, the embedder searches the vector database, ColBERTv2 reranks candidates, a document grader filters them, and a sufficiency check decides whether the evidence is enough. [HEADING=2]Five retrieval hops turn throughput into a systems problem[/HEADING] If the evidence is weak, the system can rewrite the search and go around again for as many as five hops. One question can therefore trigger many model calls before final answer generation, which is why ordinary tokens per second would hide too much of the work. MLPerf uses tasks per second for the question workload instead. One completed task represents one question pushed through the full retrieval and reasoning path, giving schedulers, model placement, accelerator communication, and cache behavior room to affect the score. The first version runs in Offline mode rather than a live Server scenario. All 824 tasks are available together, so a submitter can batch and schedule aggressively without having to satisfy the arrival pattern and latency pressure of an interactive production service. Performance runs also trade some realism for repeatability. Recorded reference inputs fix the retrieval path and hop count for each stage, while the models still execute their computational work and their generated outputs are discarded for the replayed path. This prevents random wording or retrieval changes from turning one timing run into a different workload. It also means the number is best read as pipeline execution throughput under a controlled path, not a complete measurement of how a free-running RAG service behaves with unpredictable user traffic. [HEADING=2]Accuracy exposes where the pipeline still breaks[/HEADING] The reference pipeline reaches about 35 percent final-answer accuracy across the full 824-query set, and valid submissions must reach at least 97 percent of that reference score. Numerical and tabular questions sit near the bottom at roughly 31 percent, while multiple-constraint questions reach about 38 percent. Those numbers expose a problem that raw retrieval speed cannot fix. A pipeline can retrieve plausible passages and still fail when the answer depends on arithmetic, table structure, date logic, or a chain of facts split across several pages. Retrieval quality itself is not the headline accuracy metric in this first version. MLPerf uses database checks to keep independently built indexes close to the reference behavior, while final-answer accuracy determines whether a submission remains valid. A separate compliance test also checks the mean output length of the answer generator so a system cannot gain speed by quietly producing much shorter answers. The check guards one visible stage, not every retrieval, reranking, sufficiency, or intermediate reasoning step, leaving room for future versions to tighten pipeline-wide compliance. For anyone comparing systems, the practical detail is where the work lands. A better score can come from faster embedding, smarter CPU scheduling, tighter GPU partitioning, lower communication overhead, better KV-cache handling, different model precision, or some combination of all six rather than a single faster generator. [/QUOTE]

Name