Summary
First-party measurements of dense Qwen3.8-27B Q8_0 on an M3 Ultra workstation with 512 GB unified memory found warm decode near 16 tokens/s for short inputs, falling to roughly 6 tokens/s in a 190k-input probe. Long-context cold starts were much slower than warm requests. These are historical, configuration-specific results, not measurements of a newer model or a current service.
Method and collection date
The August 2026 dataset contains 451 successful logged requests over roughly 14 hours, supplemented by direct streaming probes. The runtime was llama.cpp llama-server [1], using GGUF Q8_0 weights, F16 KV cache, a 524,288-token context allocation across two parallel slots and a 65,536 MiB reusable prompt cache. The source describes an effective limit of 262,144 tokens per active slot.
Logged speed summaries use request-level output-token rates, favoring medians because very short responses distort means and maxima. Synthetic probe sizes were estimated from measured character-to-token ratios: about 1.78 for hexadecimal checksum lines and 4.3 for a natural-language calibration. These are not interchangeable workloads.
Findings
Warm performance deteriorated at long context
| Input size | Observed output rate | Warm time to first token |
|---|---|---|
| Below 5k | About 16 tokens/s median; interquartile range 14–18 | About 0.8 s median |
| 20–50k | About 15 tokens/s | About 2.7 s median |
| 50–100k | Log median 16.3 tokens/s; direct 100k probes 10.4–12.4 | About 3.9 s median |
| 137k | About 12.8 output tokens/s averaged over end-to-end duration | Not separately reported |
| 190k | About 6 tokens/s in a direct streaming probe | About 22.2 s |
The 137k observation produced 791 output tokens in roughly 62 seconds. Its end-to-end rate should not be treated as a separately instrumented decode-only rate. The 190k estimate came from 60 streamed chunks over 9.8 seconds after the initial wait; chunk timing is a weaker basis for token throughput than explicit token accounting.
Acceptance and startup cost are separate questions
A roughly 466k-token prompt was rejected with HTTP 400 and a reported context limit of 262,144. A 190k-input streaming probe returned HTTP 200. These observations establish tested acceptance and rejection points, not successful operation at every intermediate length.
The source attributes the same native context specification, and a possible 1M extension through YaRN, to the public model card [2]. That extension was not tested. The largest logged successful input was 137,471 tokens; the longer successful probe was separate from the log dataset.
A cold 190k request took about 1,294 seconds, or 21 minutes, to reach its first token. The source reports that this interval included model loading and memory allocation. Warm and cold requests therefore need separate latency budgets; a short warm-request timeout is not a reliable cold-start budget.
Limitations
This is one hardware, quantization and serving configuration, with mixed interactive and synthetic traffic rather than a controlled benchmark suite. Runtime revision, per-bucket sample counts and complete raw measurements are not provided. The source's reported maximum total-token count conflicts with its stated per-slot limit, so that total is not used here; token-accounting details remain unverified.
Longer prompts coincided with slower output, but KV-cache pressure was not isolated experimentally as the cause. Cold-start attribution was likewise not decomposed into separately timed components. No quality evaluation, 1M-context test or service-level guarantee follows from these results. Model availability and specifications are reported as of August 2026, not independently revalidated here.
References
- llama.cpp serving runtime: https://github.com/ggml-org/llama.cpp
- Qwen3.8-27B model card: https://huggingface.co/Qwen/Qwen3.8-27B