Tech innovators at MangoBoost have blown past industry benchmarks with their latest AI performance breakthrough. Using 32 AMD Instinct MI300X GPUs spread across four server nodes, the company shattered previous MLPerf Inference records for the Llama2-70B model.
Their breakthrough MLOps software, Mango LLMBoost, crushed competitor performance metrics by delivering 103,182 tokens per second in offline scenarios. More impressively, they achieved this at a dramatically lower cost compared to NVIDIA's top-tier H100 GPUs. AMD chips cost $15,000-$17,000 versus NVIDIA's $32,000-$40,000 price tag.
Cost-conscious tech departments will appreciate the massive savings. Compared to alternative systems, MangoBoost delivers approximately 2.8 times more inference throughput per thousand dollars spent. Their software supports over 50 open models and works seamlessly across cloud platforms like AWS, Microsoft Azure, and Google Cloud.
The company's success stems from tight collaboration with AMD using the ROCm software stack. Beyond MLPerf achievements, they've proven lightning-fast performance across various configurations. On AWS, an 8-GPU setup demonstrated 138 times faster inference compared to competing platforms like Ollama and HuggingFace TGI.
Their breakthrough MLOps software, Mango LLMBoost, crushed competitor performance metrics by delivering 103,182 tokens per second in offline scenarios. More impressively, they achieved this at a dramatically lower cost compared to NVIDIA's top-tier H100 GPUs. AMD chips cost $15,000-$17,000 versus NVIDIA's $32,000-$40,000 price tag.
Cost-conscious tech departments will appreciate the massive savings. Compared to alternative systems, MangoBoost delivers approximately 2.8 times more inference throughput per thousand dollars spent. Their software supports over 50 open models and works seamlessly across cloud platforms like AWS, Microsoft Azure, and Google Cloud.
The company's success stems from tight collaboration with AMD using the ROCm software stack. Beyond MLPerf achievements, they've proven lightning-fast performance across various configurations. On AWS, an 8-GPU setup demonstrated 138 times faster inference compared to competing platforms like Ollama and HuggingFace TGI.
