DeepSeek-V4-Flash on a $0.70/hour Azure CPU: Reading Went From 33 to 202 Tokens a Second

You do not need a GPU allocation to ingest a long document.

Same 145 GB model file, same 96-core Arm VM on Azure, one user, no GPU anywhere: prompt processing went from 33 to 202 tokens a second, and the reply from 15 to 20. What changed was the engine, and a 21-line fix that let the faster engine run DeepSeek on Arm at all.

Every number below is reproducible on any Azure subscription. The flags and the measurement method are at the end.

The headline pair

Azure Cobalt 100 VM	Cores	llama.cpp: reply / prompt (tokens/s)	ik_llama.cpp with the fix: reply / prompt	10,000-token paste
D96ps_v6, ~$0.70/hour spot	96	14.8 / 33	19.8 / 202	5 minutes to 50 seconds
D64ps_v6	64	not measured	18.5 / 152	70 seconds
E32ps_v6, $0.31/hour spot	32	13.0 / 16	17.4 / 84	10 minutes to 2 minutes

DeepSeek-V4-Flash 0731, MXFP4, 145 GB. 2,048-token prompt, 128-token reply, median of three runs, single user.

Two engines, one file. llama.cpp is the default open-source engine for running these models on a CPU. ik_llama.cpp is a performance fork of it, by Iwan Kawrakow, that rewrites the CPU matrix kernels and the mixture-of-experts path. On x86 it has been the fastest way to run large mixture-of-experts models on CPUs for some time. On Arm, with DeepSeek-V4, it did not run at all until this week.

The fix

On Arm, the fork aborted the moment a prompt passed 2,048 tokens. That is not a random number. It is the length of the dense prefix in DeepSeek's sparse attention: the point where the model starts selecting which earlier tokens each new token should look at, using a component called the top-k indexer. Every prompt short enough for a chat window worked. Every document did not.

We traced it to two CPU-side defects that only execute on non-x86 chips:

1. A missing kernel. The indexer multiplies a half-precision key matrix by a single-precision query. The Arm build had no kernel for that combination, so it aborted. The fix converts the query through the existing conversion hook and sizes two work buffers to match.
2. A counter that was never zeroed. The bucketed top-k selection skipped the reset on the non-AVX2 code path, so it produced garbage indices and wrote out of range. Dead code on x86, live on Arm. One line.

We filed it as issue 2450 just after midnight UTC on Monday 15 September. The maintainer replied that morning and asked for a pull request. PR 2456 went up at 11:40 UTC, 21 lines added in one file, and was approved and merged at 12:18. Report to merged fix: about 12 hours. The fix is in ik_llama.cpp main. We verified it afterwards on 32, 64 and 96-core machines with prompts up to 16,384 tokens, a fact retrieved from the very start of a 2,600-token prompt, and correct arithmetic after 2,400 tokens of filler.

Why reading speed is the number that matters

Two speeds decide whether a private model is usable at a desk:

• Reply speed (decode) is how fast it writes the answer. Above roughly 15 tokens a second you stop waiting on it.
• Reading speed (prefill) is how fast it takes in what you gave it. Paste a 40-page contract, a log file or a code repository, and prefill decides whether you wait ten seconds or five minutes before the first word appears.

Reply speed on a CPU is capped by memory bandwidth. The active weights stream through RAM once per token, and no software makes DDR5 faster. That is why the reply only moved 1.2x to 2.4x across everything we measured. Reading is arithmetic, and arithmetic is where software still had headroom. The useful change is reading the brief.

The 96-core box: 3.0x to 6.1x faster reading, up to 806 tokens a second

Model (quantisation), 96 cores	llama.cpp: reply / prompt	ik_llama.cpp: reply / prompt	Reading gain
DeepSeek-V4-Flash 0731 (MXFP4, 145 GB), D96ps_v6	14.8 / 33	19.8 / 202	6.1x
Qwen3-Coder-30B-A3B (Q4_K_M)	58.7 / 145	84.7 / 806	5.6x
Qwen3-Next-80B-A3B (Q4_K_M)	29.5 / 133	69.7 / 665	5.0x
Qwen3-235B-A22B-Instruct (Q4_K_M)	14.6 / 36	17.5 / 130	3.6x
gpt-oss-120b (Q4_K_M)	56.0 / 187	67.7 / 559	3.0x

E96ps_v6 (96 cores, two NUMA nodes, 672 GB). The DeepSeek row is the D96ps_v6: same cores, half the memory.

Qwen3-Coder-30B reads a 2,048-token prompt in 2.5 seconds and a 10,000-token file in 12, then writes at 85 tokens a second. That is coding-assistant territory on a VM with no GPU. The gain over llama.cpp is larger on 96 cores than on 32 because the fork's batched kernels put the second NUMA node's cores to work on prompt processing.

For one user, the extra cores are reading money: reply speed is one NUMA node's memory bandwidth per instance. For a team, the box splits. Two instances, one per node, ran at 55.7 + 59.6 = 115 tokens a second combined, with 864 tokens a second of combined reading. A 96-core Cobalt VM is two 32-core VMs in one, with a 660 GB ceiling.

The 32-core box: eleven models, 2.6x to 5.2x faster reading

Model (quantisation), E32ps_v6, $0.31/hour	llama.cpp: reply / prompt	ik_llama.cpp: reply / prompt	Reading gain
DeepSeek-V4-Flash 0731 (MXFP4, 145 GB)	13 / 16	17.4 / 84	5.2x
Qwen3-Coder-30B-A3B (Q4_K_M)	46 / 89	74 / 337	3.8x
Qwen3-30B-A3B-Instruct (Q4_K_M)	48 / 89	71 / 337	3.8x
GLM-4.5-Air (Q4_K_M)	17 / 26	23 / 87	3.3x
ERNIE-4.5-21B-A3B (Q4_K_M)	58 / 117	74 / 388	3.3x
Qwen3-Coder-Next-80B (UD-Q4_K_XL)	28 / 108	57 / 327	3.0x
gpt-oss-20b (Q4_K_M)	63 / 128	80 / 380	3.0x
Qwen3-235B-A22B-Instruct (Q4_K_M)	11 / 16	13 / 48	3.0x
gpt-oss-120b (Q4_K_M)	43 / 84	55 / 232	2.8x
Qwen3-Next-80B-A3B (Q4_K_M)	29 / 124	60 / 321	2.6x
Qwen3.6-35B-A3B (UD-Q4_K_M)	33 / 174	66 / 445	2.6x

Every model is faster on both counts, and none reads less than 2.6x faster. The models that gain most on reply speed (Qwen3-Next, Qwen3-Coder-Next and Qwen3.6, all around 2x) are the ones with the most software between the weights and the output: hybrid attention and unusual expert routing. The plain mixture-of-experts models gain 15 to 60%, then hit the memory wall.

A dense model: Qwen3.8-27B

Every model above is a mixture-of-experts model, where a 145 GB file only touches a few GB of weights per token. A reader asked how a dense model does. A dense model touches all of its weights for every token, so the memory wall arrives much sooner. We measured Qwen3.8-27B (UD-Q4_K_XL, 17.6 GB) on the 64-core and 96-core Cobalt VMs the same day, on the current appliance image, three runs each after the same correctness checks.

Qwen3.8-27B	llama.cpp: reply / prompt	ik_llama.cpp: reply / prompt	ik_llama.cpp with -rtr: reply / prompt
D64ps_v6 (64 cores, one NUMA node)	11.7 / 54	17.3 / 122	18.7 / 111
D96ps_v6 (96 cores, two NUMA nodes)	11.4 / 70	15.1 / 160	17.2 / 148

Tokens per second, 2,048-token prompt, 128-token reply, median of three runs, single user.

Three things stand out:

• The fork helps a dense model on both counts. Reading is 2.2x to 2.3x faster than llama.cpp on the same VM, and replies are 1.3x to 1.6x faster, as much as or more than most of the mixture-of-experts models gain. It also runs in about 22 GB of memory, against llama.cpp's 35 GB.
• More cores help reading, not replies. The 96-core VM reads fastest (160 tokens a second), but the 64-core VM replies fastest (18.7), because on the 96-core VM the reply threads stay on one node's 48 cores.
• -rtr trades reading for replies. Repacking the weights at load time adds 8 to 14% to reply speed and costs about 8% of reading speed, with no extra memory.

For a dense 27B on a CPU, the 64-core VM with -rtr is the pick for chat, and the 96-core VM is the pick for long documents.

The thread rule for two-node machines

If you run a two-socket or two-NUMA-node CPU machine, never set the thread count to all cores. On the 96-core E96ps_v6, -t 96 collapsed reply speed to 5 tokens a second on the fork and 2 on llama.cpp. -t 48 -tb 96 (reply threads on one node's cores, prompt threads on all of them, no CPU pinning) gave 70 reply and 665 reading on Qwen3-Next-80B. Every 96-core figure above uses that rule. Pinning the whole server to one node is also wrong: it squeezes 96 prompt threads onto 48 cores, and reading speed fell from 665 to 42.

What this changes for a private AI deployment

• A CPU-only VM is now a document machine, not just a chat machine. 84 to 445 tokens a second of reading on 32 cores, and 130 to 806 on 96. "Paste the whole thing" is a normal interaction on a VM with no GPU and no GPU quota.
• The 145 GB frontier-class model is the interesting one. DeepSeek-V4-Flash at 17 to 20 tokens a second, with a 50-second read of a 10,000-token document on a $0.70/hour VM, is a sovereign-AI entry point that did not exist in July.
• Engine choice is a bigger lever than VM size for reading speed. Moving from llama.cpp to the fork on the 32-core VM (16 to 84) is worth more than moving from 32 to 96 cores on llama.cpp (16 to 33). Do both and it is 16 to 202.

Caveats

• Single-user numbers. Concurrency is a separate measurement.
• Spot prices at the time of the runs. The 32-core price is checked against our invoice; the 96-core price is the portal figure and has not been checked against a bill yet.
• A 2,048-token prompt and a 128-token reply. The 21 tokens a second we have quoted elsewhere is a live chat reply on the 96-core VM after a 10,000-token paste; the table figure is 19.8.
• Same model file, same VM and same method for every pair. We never compare numbers across measurement methods, because reading speed can differ by up to 3x between them. The 32-core pairs were run through our appliance image's serving path (three of the eleven in an earlier sweep with two runs each); the 96-core pairs are direct launches of the same image binaries.
• The fix is in ik_llama.cpp main but not yet in a tagged release, and it reaches our appliance in its next update, not the version on the Azure Marketplace today.
• ik_llama.cpp is a fork. Model loading, the GGUF format and the server come from llama.cpp; the fork replaces the CPU compute path.

Reproduce it

On any Azure subscription:

1. Create an E32ps_v6 or D96ps_v6 spot VM (Ubuntu 24.04 arm64).
2. Download the ggml-org DeepSeek-V4-Flash-0731 MXFP4 GGUF, or the unsloth GGUF of the other models, from Hugging Face.
3. Build either engine from source with -DGGML_NATIVE=ON. For DeepSeek on Arm, use ik_llama.cpp at or after commit fdb4599.
4. Serve with -t 32 -fa 1 --jinja -c 32768 --parallel 1 on 32 cores (add -rtr for the Qwen and ERNIE rows on the fork). On the 96-core, two-node VMs use -t 48 -tb 96 in place of -t 32. For DeepSeek on llama.cpp on Arm, add --no-repack (the repacked copy does not fit in memory).
5. Measure with a 2,048-token prompt and a 128-token reply through the server API. Run it three times and take the median.

Projects: ik_llama.cpp and llama.cpp.

Caleta.io sells an Azure Marketplace appliance that packages this. Everything above is reproducible without it.