Reply to thread

Message: [QUOTE="Bombastus, post: 91514, member: 2178"] Minisforum reports 16 tokens per second for Qwen3.5-397B running across two MS-S1 MAX workstations with 192GB of memory each. The figure doesn't tell you when an answer starts. For a [B][URL='https://goldmidi.com/community/threads/minisforum-has-introduced-the-ms-s1-max-p495-workstation.76847/']two-machine MS-S1 MAX-P495 setup[/URL][/B], measure the delay from submitting a request to receiving answer text. Keep startup time separate from requests made while the model is already loaded. A llama.cpp result describes your setup, not necessarily the one Minisforum tested. [HEADING=2]A generation test leaves out parts of your request[/HEADING] llama-bench separates prompt processing from text generation. Its measurements exclude tokenization, which turns text into tokens, and sampling, which selects the next token. A tg128 result measures generation of 128 tokens. It isn't a 128-token conversation timed from your first click. Even a combined prompt-and-generation test leaves those processing steps outside its measurements. Measure time to first token through a streaming request from the client you'll actually use. Start the timer before sending the request and stop it when generated text arrives. Record the completion time separately. If the service queues your request, that wait belongs in the client result. A prompt-processing timer inside the server can help diagnose the delay, but it isn't the whole experience. Use the same GGUF model file, runtime build, prompt, and context allocation for each comparison. Start with one request at a time. llama.cpp's distributed RPC mode divides model weights and context memory according to each device's available memory. Other GPU applications running during loading can change those proportions. Two identical machines can still receive different amounts of work. A 32,768-token context setting doesn't prove that a test supplied 32,768 tokens of text. A short instruction and a long report can use that same setting while requiring different amounts of prompt processing. Repeat each condition several times and retain the individual timings. Report the median alongside the slowest observed response. A handful of runs won't support confident claims about rare delays. [HEADING=2]A restart can leave the worker cache intact[/HEADING] With llama.cpp RPC, remote workers can save large tensors, the model's numerical arrays, in a local disk cache. Enabling -c on the worker avoids resending matching cached tensors over the network during later model loads. The default directory is $HOME/.cache/llama.cpp/rpc. Restarting your main application doesn't, by itself, remove files from that directory on the other machine. A restart can therefore benefit from an earlier run. Measure loading with an empty worker cache separately from loading with that cache populated. Then time requests with the model already loaded. These are three distinct starting conditions, not three attempts at the same test. The tensor cache stores model data, not the computed state of your document. A faster cached launch isn't evidence of faster prompt processing. Prompt reuse is a separate mechanism. In llama-server's chat response, two counters show how much input work was reused. cache_n counts reused prompt tokens, while prompt_n counts tokens processed for the current request. Record both alongside the elapsed time. A request can contain a long document without processing all of it again when the required cached state is reusable. Read those counters before calling a repeated document test a fresh-input result. Changing only the question at the end doesn't necessarily remove the cached document prefix. A new chat window isn't proof of an uncached request either. Check what the server actually reused. [HEADING=2]Thinking mode adds a wait before answer text[/HEADING] Qwen3.5-397B-A17B produces reasoning before its answer by default. When the server returns this separately in reasoning_content, a first generated token can belong to that reasoning rather than the answer. If your interface hides that text, a fast first token can still leave the answer area blank. Measure [B]the wait until answer text arrives[/B] separately from the first reasoning output. Neither timing tells you when the response finishes. At a hypothetical steady 16 tokens per second, 800 reasoning tokens would take about 50 seconds to generate. This is arithmetic, not a latency measurement from the P495 demonstration. Record the first non-empty reasoning text and the first non-empty answer text for the same request. Exclude empty stream messages from both measurements. After measuring one request, repeat the document task with two requests submitted together. Keep their timings separate rather than reporting only their combined output rate. Log any failed requests too. Keep thinking mode unchanged during those comparisons. Qwen3.5 doesn't officially support disabling it by appending /nothink to the prompt. In llama-server, --reasoning off controls thinking through the chat template. Label a non-thinking result separately, even when the model file and question are unchanged. [/QUOTE]

Name