On 11 September we published a load of Kimi-K3 on one Azure VM and called the decode number provisional, because it came off a pre-merge build and the coherence gate had not been run. We said both gaps would close the next time we could get a box, and that getting a box was the problem.
On 18 September, after 68 refused deployments across 17 days, the box allocated first try. This is what three hours and about $35 of spot bought.
The box
Eight AMD Instinct MI300X, 205.6 GB of memory per card, 1,644.6 GB in total. Host: 96 Xeon cores, 1.8 TB of RAM, eight 3.5 TB NVMe drives. France Central, spot, about $11 an hour. Same hardware class as the 11 September piece, this time on the shipped appliance image (0.15.1, ROCm 7.2.4) rather than a lab build.
Every number below was measured on that box, on that day, on that image. Nothing is carried over from August.
Two models, every number
Both models passed the three-prompt coherence gate, and both ran the uncapped long-form probe before any timed run, which is the order that catches an engine producing fluent nonsense.
| Qwen3.8-2.4T-A95B | Kimi-K3 | |
|---|---|---|
| Parameters | 2.4 trillion, 95 billion active | 2.78 trillion |
| Quantisation | UD-IQ4_XS | UD-Q4_K_XL |
| Bytes on the cards | 1,318 GB | 1,539 GB |
| Headroom | 327 GB | 106 GB |
| Prefill, median of 3 | 540 tokens/sec | 263 tokens/sec |
| Decode, median of 3 | 21.87 tokens/sec | 14.31 tokens/sec |
| Decode spread across reps | 5.1% | 7.1% |
| Natural decode in the uncapped probe | 23.08 tokens/sec | 14.30 tokens/sec |
| Cold load | 257 seconds | 1,213 seconds |
| Warm load, same file | 206 seconds | 204 seconds |
| Coherence gate | 3 of 3 | 3 of 3 |
Two of those rows deserve a sentence each.
Kimi-K3 at 1,539 GB is the largest thing this project has ever held in memory, with 106 GB of the box left over. The 11 September figure of 1,405 GiB was the weight file; this is what the cards actually reported holding once the model was placed.
The cold-to-warm load is a 5.9x difference on an identical file: 20 minutes the first time, 3 minutes 24 seconds the second. The host's 1.8 TB of RAM caches the weights after the first read. If you are going to restart the server on a box like this, do it before the box goes away, not after.
Why one request only uses half the box
That screenshot is the finding. Eight cards, one request in flight, and the utilisation is not a flat line across them. It is a staircase.
We logged all eight cards from sysfs at about 4 Hz for the whole session, 167,592 rows. During steady-state Qwen decode the per-card busy means run 12.9%, 25.1%, 37.5%, 48.4%, 60.7%, 73.0%, 85.3%, 95.6%, first card to last. Summed, that is 438% of a possible 800%: one request keeps about 4.4 cards' worth of silicon busy. On Kimi-K3 the same measure is 169%, less than two cards' worth.
The reason is the engine, not the hardware. These models only fit on the box because they are quantised to GGUF, and the runtime that serves GGUF splits a model across cards by layer and works through them in sequence. Each token passes down the chain, card to card, and the late cards wait on the early ones. The staircase is a pipeline draining.
The hardware is capable of more. During prefill, when the engine has a whole prompt to chew through at once, all eight cards hit 100% at the same sample, and the box draws 4.4 kW. Idle with no model loaded it draws 1.1 kW. Single-user decode sits at 2.4 kW. So the machine will do the work if it is given work in a shape that lets all eight cards act together. A single conversation is not that shape.
We did not measure concurrent requests on this box, so we are not going to guess at how much of the remaining silicon a second, fourth or eighth user would recover. That is the obvious next run, and it needs another window.
The context ceiling
Both models load cleanly and then die on the first token if the KV cache asks for more than the weights leave behind. The server binds its port and reports ready before it finds out.
| Model | Context | KV cache | Result |
|---|---|---|---|
| Qwen3.8-2.4T | 16,384 | f16 | Loaded, then ROCm error: an illegal memory access was encountered on the first token |
| Qwen3.8-2.4T | 4,096 | f16 | Healthy, 23 tokens/sec |
| Qwen3.8-2.4T | 49,152 | q8_0 | Healthy, the brief below ran here |
| Kimi-K3 | 49,152 | q8_0 | cudaMalloc failed: out of memory, 282 GB requested on card 0 |
| Kimi-K3 | 16,384 | q8_0 | Healthy |
The operational rule that falls out: quantise the KV cache before you enlarge the context on a box this full, and treat a server that has bound its port as not ready until the health endpoint stops reporting a loading model. A crashed server can also survive its own abort while still holding 1.3 TB of memory and the port. Kill it explicitly.
What the models did with a real brief
We gave each model a 6,600 to 7,000-token working brief and let it run. This is not a benchmark score, it is a record of behaviour.
Qwen3.8-2.4T prefilled 6,983 tokens in 8.2 seconds, thought for 95,224 characters, wrote 28,604 characters of answer at 20.7 tokens a second, and was cut off mid-sentence at the 32,000-token output cap we had set. Twenty-six minutes of wall time. A 2.4 trillion parameter model needed more room than we gave it, and that is a harness setting, not a hardware limit.
Kimi-K3 on its first run wrote nothing at all. It spent the entire 9,000-token allowance thinking and returned an empty answer. Given a 3,000-token reasoning budget on the second run it wrote a 23,881-character answer in 11 minutes at 14 tokens a second.
The reasoning budget is the lever when a model cannot fit both its thinking and its answer in the context you can afford. On a box where the weights take 94% of the memory, that is most of the time.
What this is for
A server in this class costs about what a typical UK house costs. Here it is for $35 an afternoon, on a card, in your own subscription.
Two uses follow from that. The first is deciding whether a frontier-scale open model is actually good enough for your work before you commit real money to hardware, and doing that decision on your own data rather than a leaderboard. The second is asking the model something you cannot send to an API. The weights, the prompt and the answer never leave your subscription, and you can prove it from the network logs.
The appliance deploys this stack onto the eight-card box from the Azure Marketplace, with the same image these numbers came from. The sizes and regions page lists where the hardware genuinely exists and what actually deploys today.
The warning
Getting the box took 68 attempts over 17 days, in both regions that offer it, on two subscriptions, while the portal reported it at 81% off and in the lowest eviction band the whole time. Both of those readings were true. Neither can see an allocator saying no.
Receipts: all figures from one three-hour session on an eight-card AMD MI300X VM in France Central on 18 September 2026, on the shipped appliance image 0.15.1 with ROCm 7.2.4. Method: three-prompt coherence gate, uncapped long-form probe first, then pp2048 and tg128 three times each with exact-token prompts; medians reported. Per-card telemetry from sysfs at about 4 Hz. Spot price is a reading at the time of the session, not a quote. Concurrent-request throughput was not measured on this box and no figure for it appears here. Decode figures are from one box on one night; a second allocation would be needed to call them repeatable.