I ran a 744 billion parameter AI model with no GPU, for under a dollar an hour

The claim

Everyone building AI right now is fighting for GPUs. I loaded GLM-5.2, one of the most capable open weight models in the world at 744 billion parameters, entirely into the memory of a single Azure server with no GPU, and it generates working code at around 10 tokens per second. The box costs 94 cents an hour on Azure spot.

This is the first in a series where I throw every kind of box Azure has at one question: how cheaply can you run a serious model inside your own tenancy? Everything here is measured or clearly labelled as a projection, and I will be honest about the numbers, including the parts that are slow.

Why everyone thinks this is impossible

The received wisdom is that a model this large needs a cluster of expensive GPUs, because it does not fit in a single GPU's memory and GPUs are what make AI fast. Both halves of that are only half true.

A mixture of experts model like GLM-5.2 has 744 billion total parameters but only activates about 40 billion of them per token. And the thing that limits how fast those weights get read is not raw compute, it is memory bandwidth: how many gigabytes per second the machine can pull from RAM. GPUs win on AI mostly because their memory is very fast, not because of some special sauce.

So the real question is not "do you have a GPU". It is "how much memory bandwidth can you get, and can you fit the model in it".

The bandwidth nobody measured

Across every Azure CPU server I have benchmarked, the most memory bandwidth I ever got to a single workload was about 120 GB/s. Azure slices its standard x86 machines in a way that hands each virtual machine only a fraction of the underlying memory channels. That fraction capped every big model at around 8 tokens per second, no matter how many cores I threw at it.

Then I tested Microsoft's own Arm chip, the Cobalt 100, which shows up on Azure as the Epsv6 family. Microsoft builds it for inference and pitches it for models at modest scale, and the one detailed public benchmark I could find, by the Microsoft MVP Thomas Van Laere last October, stops at dense models around 15 billion parameters. Nobody had published a memory-bandwidth figure for the chip, and nobody had put a frontier-scale model on it. So I measured the bandwidth, 435 GB/s to a single machine as one shared pool, roughly 3 and a half times anything the x86 boxes gave me, and then I ran the model that number should allow: GLM-5.2, 744 billion parameters.

But doesn't a Mac already do this?

Fair question, and worth answering head on. Yes, a maxed out Mac Studio with 512GB of fast unified memory runs large models too, and the Apple Silicon community does exactly this. AWS and others have their own Arm chips that do it as well. The difference is what you are buying, and where. A Mac is several thousand pounds of hardware you own, maintain, and physically keep. And the reason this particular box matters to a business already on Microsoft is simple: it is in Azure, in the same tenancy, under the same governance as everything else you run. It is spare cloud capacity at 94 cents an hour, spins up in minutes where your data already lives, and switches off when you are done. Same physics, very different economics for a business.

The honest part: this is bleeding edge, and slow to start

Here is the number the headline hides. Generating text, once it gets going, runs at around 10 tokens per second for a single user, which is faster than you read. But reading your prompt in the first place, what the machine calls prefill, is slow: a long prompt takes real time to ingest before the model says a word, and deep into a long conversation the generation itself drops toward 7 tokens per second.

Two things make this bleeding edge rather than broken. Running frontier scale models on Arm CPUs is new enough that the software is still catching up with the hardware, and the prefill bottleneck specifically has active fixes landing in llama.cpp right now. The headroom is measurable, not hopeful: during generation the model is pulling about 260 GB/s of the 435 GB/s the chip can deliver, so the software is leaving roughly 40 percent of the machine on the table. These numbers are a snapshot of a moving frontier, and the direction of travel is up.

So be clear about what this is not, today. It is not a sub second inline autocomplete, and it is not the tool for pasting in a huge document and expecting an instant answer. If that is your use case, you want a GPU.

The thought experiment: what if you ran it as a token factory?

Now the part I want to be careful to label for what it is: early numbers plus arithmetic, not a production benchmark.

That slow single user speed exists because one request cannot keep the memory pool busy. Give the box many requests at once and it shares the expensive step, reading the model, across all of them. Here is the curve from short structured runs, and the cost per million tokens of output it implies at the real 94 cents an hour spot price:

At 32 concurrent jobs the curve was still climbing, and I have seen the combined rate touch 30 tokens per second, so there is headroom here even before the Arm software matures. The implied cost is about $9.60 per million tokens, from a machine with no GPU in it.

The caveats, stated plainly. I have not yet run this configuration hard for days at a time, and short runs flatter every system; that sustained test is coming, and I will publish whatever it shows. And each individual lane is slow, so even if the numbers hold, this shape suits batch and agent fleets, overnight document processing, bulk drafting, agent runs that grind while you sleep, not a person waiting at a keyboard.

And one honest comparison, because a cost-minded reader will ask it. If all you want is cheap tokens and you do not care where they run, a hosted API for one of these open models is cheaper still, by a wide margin. This is not the cheapest way to get tokens. It is the cheapest way to get them without your data ever leaving your own cloud. The moment privacy, data residency, or fine-tuning on your own material becomes a hard requirement, the API is off the table, and this is what is left: a frontier model, in your tenancy, producing tokens for single-digit dollars per million.

What the price buys

The 96-core box is 94 cents an hour. The 64-core box runs the same model at the same generation speed for 63 cents; the extra cores only buy memory headroom for long context, not speed. Below that the box can no longer hold the full model in memory. So you size the machine to your model, not to a spec sheet.

Those are spot prices, measured in Sweden Central at an 80 percent discount to pay-as-you-go, so the same machines cost roughly five times more on-demand. Spot is interruptible capacity that Azure can reclaim, and that discount is part of the story, not a footnote. For batch work an interruption is an inconvenience rather than a disaster, the job resumes and carries on, and I publish a monthly report on exactly where spot is cheap and where it is calm. This is a measured trade, not a hopeful one.

There is a smaller, faster tier underneath this for coding-size models that costs pennies an hour, and I will get into what it costs to run a whole team on it next week.

What I am not putting in this post

Getting from "it runs" to these numbers took months of tuning: the right box, the right build, a re-pack of the model that bought 20 percent for free, and a set of kernel and threading settings that together roughly doubled what the naive setup delivered. I am not going to reproduce that recipe step by step, because doing that work for you, and keeping it current as models and hardware change, is exactly what we are building at Caleta.

Why this matters

For less than a dollar an hour, you can run a frontier class open weight model entirely inside your own tenancy. Your data never leaves your network. There is no per seat licence. And because it is an open weight model, you can fine tune it on your own data. That is the sovereign AI thesis in one line: you no longer have to choose between capable and private.

Run it yourself, soon

I am packaging this exact setup as a one click Azure image, so you can deploy it into your own tenancy in minutes instead of reproducing the work above. If you want to know the day it is live, get in touch and mention the one click image, and I will tell you personally.

And if your organisation wants help running AI in your own environment right now, that is what we do at Caleta. Start the conversation here.