Caleta AI Appliance / Documentation
Caleta AI Appliance user guide
Deploy the VM, SSH in, type llm, and serve an open-weight model privately in your own Azure subscription. This guide covers the menu, the chat UI and API, publishing over HTTPS, and locking the box down.
The appliance listens on loopback only. Nothing is reachable from the network until you deliberately expose it.
Sections 1 to 11 apply to both architectures, with x86 sizes and models in sections 10 and 11. Section 12 covers Azure Cobalt (ARM64).
Section 1
What the appliance is
The Caleta AI Appliance is one Azure Marketplace VM image that runs open-weight large language models privately in your own Azure subscription. No data leaves the VM during inference.
A single image serves both CPU and GPU sizes. It detects a GPU at run time and uses the CUDA engine automatically, otherwise it runs on CPU.
The engine is llama.cpp (only). The chat interface is Open WebUI. There is nothing to install, it is all baked into the image.
Section 2
Deploy and first login
Deploy the offer from the Azure Marketplace into your own subscription. We recommend Trusted Launch (the image supports it) and one of the sizes in the supported sizes table.
Connect with an SSH key
ssh <admin-user>@<vm-public-ip>Connect with Microsoft Entra ID (recommended)
Entra brings MFA and conditional access with no SSH keys to manage. Add the "Login with Entra ID" extension at deploy time, assign the signed-in user the Virtual Machine Administrator Login role on the VM, then connect:
az ssh vm -g <resource-group> -n <vm-name>The menu launches automatically
On interactive login the llm menu starts by itself. A short boot banner appears first (the Caleta cloud logo and a few hints). For a plain shell, choose quit in the menu, or connect with the environment variable CALETA_NO_MENU=1 set.
Section 3
The llm menu
Every time it opens, the menu shows a live status block at the top, then the list of actions.
What the status block reports
- Engine state (UP, STARTING or DOWN) and the model being served
- Auto-restore (ON or off)
- Chat UI state
- GPU memory, or "none" on CPU sizes
- RAM
- Disk usage and mount point
- HTTPS exposure state
- Firewall (ufw) state
Actions
- serve / swap model
- Start (or switch to) a downloaded model.
- download a catalogue model
- Fetch a model from Hugging Face.
- spot placement (cheapest regions)
- Advisory: cheapest low-eviction Azure regions for the VM size (Marketplace-deployed boxes only).
- test prompt
- Send a quick prompt and stream the answer in the terminal.
- tail server log
- Watch the engine log live.
- show API key
- Print this boot’s inference API key and example commands.
- register for update notifications
- Optional email opt-in for image and security update notices.
- expose on :443 (TLS) / stop exposing
- Publish the chat UI and API over HTTPS.
- toggle auto-restore
- Make the box re-serve its last model automatically after a restart or deallocate.
- firewall (ufw)
- Manage the host firewall.
- stop server
- Stop serving.
- quit
- Leave the menu to a shell.
Section 4
Download a model, then serve it
These are two separate steps. download a catalogue model puts the weights on disk. serve / swap model loads them into the engine. Downloading does not serve.
Downloading
Downloads pull direct from Hugging Face using the fast Xet transfer path. The progress bar shows size in GB and speed in Gbit/s, with peak transfer on completion. No model weights ship inside the image, and each model downloads under its own Hugging Face licence.
Weights are cached at /mnt/nvme/models on sizes with a local NVMe disk. On sizes with no local disk they fall back to the OS disk, so size the OS disk for the model you want.
Serving
Serving uses a measured preset of engine flags per catalogue model, which you can edit at the prompt (press Enter to keep the preset).
Presets serve a single user with the full context window. For a team, raise --parallel to the number of concurrent users. The context is split evenly between them, so --ctx-size 65536 --parallel 16 gives each user 4096 tokens.
Fit guards
- On CPU sizes, an oversized model warns and asks before serving.
- On GPU sizes, the menu declines a model that does not fit in GPU memory. A spilled GPU model fails outright rather than just running slowly.
Use the supported sizes and model catalogue tables to choose.
Section 5
Chat UI and API
The chat UI and API bind to loopback only (127.0.0.1). Nothing is reachable from the network until you expose it (section 6).
Open an SSH tunnel from your laptop
With key auth:
ssh -L 3000:localhost:3000 -L 8080:localhost:8080 <admin-user>@<vm-public-ip>With Entra:
az ssh vm -g <resource-group> -n <vm-name> -- -L 3000:localhost:3000 -L 8080:localhost:8080Chat UI
Open http://localhost:3000 for Open WebUI. The first account you create becomes the local admin. It is a local account on the VM, not a Caleta or Microsoft login.
API
The API is at http://localhost:8080 and is OpenAI-compatible (/v1/chat/completions, /v1/models). Every request needs the per-boot key as an Authorization: Bearer <key> header.
The per-boot API key
A fresh key is minted on every boot, stored root-only on the VM, never baked into the image and never reused across boots. Print it with the menu's show API key option. The chat UI is wired to the current boot's key automatically.
Section 6
Publish over HTTPS with Let's Encrypt
The appliance uses Caddy to obtain and renew a real Let's Encrypt certificate automatically. This is the expose on :443 (TLS) menu option.
Do these two things first
These are the top support issue. Exposure will not work without them.
- Create a DNS A record for a domain you own, pointing at the VM's public IP.
- Open ports 80 and 443 inbound in the VM's Azure Network Security Group, in the portal. The menu only manages the host firewall. The NSG is cloud-side and separate.
Then expose
- Choose expose on :443 (TLS), enter the domain, and optionally restrict access to a source CIDR.
- Caddy obtains the certificate over ports 80 and 443. The menu verifies the domain actually resolves to this box and warns if not.
- Chat is then at
https://<your-domain>/and the API athttps://<your-domain>/v1/.... The API key is still required on/v1.
Watch issuance and look for "certificate obtained":
sudo journalctl -u caddy -fTo stop publishing, run the same menu option and enter off.
Section 7
Lock the box down
The host firewall (ufw) is on by default: deny inbound, SSH allowed.
Recommended first line: restrict SSH at the NSG
Restrict SSH to your own IP in the Azure Network Security Group, in the portal, not in the VM. A ufw SSH lock can shut you out if your IP changes, and the serial console is then the only way back.
The firewall (ufw) menu option
- Allow a CIDR to reach 443 and 80 (for expose)
- Remove a rule
- Restrict SSH to a CIDR, with a lockout warning and a typed confirmation
- Re-open SSH
Remember that the chat UI (port 3000) and the API (port 8080) are loopback-bound, so opening them in ufw does nothing on its own. Reach them by SSH tunnel, or by exposing on :443.
Section 8
Extras
Auto-restore
Toggle it on and, after a restart or deallocate, the box re-downloads the last model to the local disk and serves it again by itself. Useful for parking the box overnight. Note that the ephemeral NVMe at /mnt/nvme is wiped on deallocate, which is why auto-restore re-downloads.
Spot placement
On a Marketplace-deployed appliance this shows the cheapest Azure regions with acceptable eviction rates for the VM size. On a test or gallery VM, which is not a real Marketplace deployment, this feature shows nothing. That is by design.
Update notifications
Optional. Registering sends only the email address to Caleta, rate limited, with no other data. See the privacy policy for what is stored. Unsubscribe via support@caleta.io.
Offline and sovereign operation
Once a model is downloaded, the appliance serves fully offline. Block all outbound traffic and it keeps working after a reboot. The only outbound calls are model downloads from Hugging Face, Let's Encrypt when you expose, and the disclosed Caleta update and attestation channel. All three are set out in the privacy policy.
Section 9
Built-in tools
Baked into the image, nothing to install.
- llama.cpp
- The inference engine, CPU and CUDA builds, pinned versions.
- Open WebUI
- The private chat UI (unbranded), loopback only.
- Hugging Face CLI (hf)
- Fast model downloads (Xet transfer).
- Caddy
- Automatic HTTPS and Let’s Encrypt reverse proxy (off until you expose).
- btop and nvtop
- System and GPU monitors.
- ufw
- Host firewall.
- gum
- The interactive menu UI (the menu falls back to plain prompts if absent).
Section 10
Supported and recommended VM sizes
The appliance runs on many sizes, but we quote only configurations we have measured. These are the recommended sizes.
CPU tier (measured, recommended)
| Size | vCPU / physical cores | RAM | What fits |
|---|---|---|---|
| Standard_E16ads_v7 | 16 / 8 | 128 GB | Entry. Models up to ~100 GB (Qwen3-Coder-30B, gpt-oss-20b). 145 GB DeepSeek-V4-Flash does not fit. |
| Standard_E32ads_v7 | 32 / 16 | 256 GB | DeepSeek-V4-Flash Q4 fits. ~35 tokens/sec on Qwen3-Coder. |
| Standard_E64ads_v7 | 64 / 32 | 512 GB | Flagship CPU box. Frontier mixture-of-experts models at usable speed. |
CPU serving is beta. It genuinely works at single-user reading speed and is a striking capability, but GPU sizes are the recommended production experience.
GPU tier (tested on this image)
| Size | GPUs | Tested model | Measured (single-stream) |
|---|---|---|---|
| Standard_NC24ads_A100_v4 | 1x A100 80GB | gpt-oss-120b | ~140 tokens/sec |
| Standard_NC40ads_H100_v5 | 1x H100 NVL | (Trusted Launch verified) | GPU tier |
| Standard_NC80adis_H100_v5 | 2x H100 NVL | Qwen3-235B 4-bit | ~64 tokens/sec |
| Standard_NC96ads_A100_v4 | 4x A100 80GB | GLM-5.2 IQ2_M | ~27 tokens/sec |
The image supports Trusted Launch (Secure Boot and vTPM), NVMe disk, accelerated networking and Microsoft Entra login.
These are the x86 sizes. For Azure Cobalt (ARM64) sizes, see section 12.
Section 11
Model catalogue
"Caleta Score" is our in-house infrastructure-as-code benchmark, where higher is better. CPU speeds are the beta lane, GPU is recommended. This is the x86 catalogue. Azure Cobalt (ARM64) has its own curated list in section 12.
| Model | Type | Download | Caleta Score | Measured GPU | Measured CPU (beta) |
|---|---|---|---|---|---|
| Qwen3.6-27B (Q4) | dense 27B | 17 GB | 90 | NC80 2xH100: 67 t/s | - |
| Qwen3-Coder-30B (Q4), the trial | MoE, 3B active | 18 GB | 80 | NC80: 240 t/s | E32: 35 t/s |
| Qwen3-Coder-Next-80B (Q4) | MoE, 3B active | 46 GB | 78 | NC80: 143 t/s | - |
| Devstral-Small-2 (Q4) | dense 24B | 14 GB | 84 | NC80: 90 t/s | - |
| Qwen2.5-Coder-32B (Q4) | dense 32B | 19 GB | 86 | NC80: 66 t/s | - |
| Phi-4 (Q4) | dense 14B | 9 GB | 74 | NC80: 137 t/s | - |
| gpt-oss-20b (MXFP4) | MoE, 4B active | 12 GB | 60 | NC80: 307 t/s | - |
| Gemma-4-26B-A4B (Q4) | MoE, 4B active | 15 GB | 82 | NC80: 195 t/s | - |
| Qwen3.6-35B-A3B (Q4) | MoE, 3B active | 21 GB | 76 | NC80: 179 t/s | - |
| GLM-4.7-Flash (Q4) | MoE, 3B active | 18 GB | 56 | NC80: 156 t/s | - |
| gpt-oss-120b (MXFP4) | MoE, 5B active | 61 GB | unscored | NC24 1xA100: 140 t/s | E64: 48 t/s |
| Qwen3-235B (Q4) | MoE, 22B active | 126 GB | 86 | NC80: 75 t/s | E64: 13 t/s |
| DeepSeek-V4-Flash (Q4) | MoE, 13B active | 145 GB | 90 | NC80: 24 t/s | E64: 9.7 t/s |
| GLM-5.2 IQ2_M | MoE, 40B active | 239 GB | 90 | NC96 4xA100: 28 t/s | E64: 7 t/s |
| GLM-5.2 IQ3_S | MoE, 40B active | 288 GB | 98 | NC96: 26 t/s | - |
| Kimi-K2.7-Code (Q2) | MoE, 32B active | 339 GB | unscored | - | E64: 9.4 t/s |
| DeepSeek-R1-0528 (Q2) | MoE, 37B active | 234 GB | unscored | - | E64: 9 t/s |
Licences are permissive (Apache 2.0, MIT or equivalent) except Kimi, which is modified MIT: products over roughly 100 million monthly active users must display the model name. That is not a constraint for appliance use.
Section 12
ARM / Azure Cobalt
The appliance also ships as an ARM64 build for Azure Cobalt 100. It is a separate plan in the same Marketplace offer, and you choose x86 or Azure Cobalt at deploy.
What is the same
The llm menu, the download and serve flow, the chat UI, the SSH tunnel, expose with Let's Encrypt, the firewall, the per-boot API key and Entra login are all identical to x86. They are shared code. Trusted Launch (Secure Boot and vTPM), NVMe disk, accelerated networking and Entra login are supported on ARM exactly as on x86, verified on the shipping ARM image.
The menu looks the same on ARM. It simply reports gpu: none, because Cobalt is CPU-only.
What is different
- CPU-only. Cobalt is an ARM CPU platform, so there is no GPU tier on ARM. The engine is llama.cpp. There is no vLLM path, and none is offered on ARM. All ARM figures are CPU inference.
- Separate, curated model list. ARM has its own measured catalogue, because not every model is built and verified for ARM. It is a focused list of models proven on Cobalt.
- Spot placement. The spot-placement advisory does not yet cover Cobalt sizes, so on an ARM box that menu item may show nothing for now. It is on the roadmap.
CPU serving is beta on both architectures. It genuinely works at single-user reading speed, and the x86 GPU sizes remain the recommended production experience.
Supported ARM sizes
Azure Cobalt Epsv6.
| Size | Notes |
|---|---|
| Standard_E8ps_v6 | Fits the small models (gpt-oss-20b, ERNIE-4.5-21B-A3B). |
| Standard_E16ps_v6 | Minimum for GLM-4.5-Air. |
| Standard_E32ps_v6 | The default, about USD 0.31/hour on spot. Fits most of the list. |
| Standard_E64ps_v6 | About USD 0.63/hour on spot. For the largest models, including GLM-5.2 (744B). |
ARM model catalogue
Measured on Azure Cobalt Epsv6 with llama.cpp on CPU. Measured on Standard_E32ps_v6 unless noted. CPU inference, single-user speeds.
| Model | Type | Download | Measured (Cobalt CPU) | Notes |
|---|---|---|---|---|
| gpt-oss-120b | MoE reasoner, 5B active | 63 GB | ~53 t/s | 120B brain at 5B-active speed; value pick |
| Qwen3-Coder-30B-A3B | MoE coder, 3B active | 19 GB | ~56-64 t/s | fast mature coder; safe default |
| Qwen3.6-35B-A3B | MoE coder, 3B active | 22 GB | ~34 t/s | best coder quality (SWE-bench 73.4) |
| Qwen3-30B-A3B-Instruct | MoE general, 3B active | 19 GB | ~63 t/s | fast general daily-driver |
| gpt-oss-20b | MoE reasoner, 4B active | 12 GB | ~75 t/s | fastest strong reasoner; fits E8ps_v6 |
| ERNIE-4.5-21B-A3B | MoE general, 3B active | 13 GB | ~67 t/s | tiny, Apache-2.0; fits E8ps_v6 |
| Qwen3-Next-80B-A3B | MoE general, 3B active | 49 GB | ~28 t/s | smart, not a speed pick |
| GLM-4.5-Air | MoE mid, 12B active | 73 GB | ~25 t/s | best mid-tier, MIT; needs E16ps_v6 or larger |
| Qwen3-235B-A22B | MoE big, 22B active | 142 GB | ~15 t/s | big brain, RAM-heavy |
| GLM-5.2 (744B) | MoE frontier, 40B active | 467 GB | ~9.5-11.7 t/s | frontier flagship; E64ps_v6 only |
Licences are Apache 2.0, MIT or equivalent permissive, the same as x86. Models download from Hugging Face under each model's own licence.
Related pages
- Caleta AI Appliance overview
- Privacy policy, covering what stays in your subscription and every outbound connection
- Terms of service, covering billing, model licensing, warranties and support
- Security, covering the security posture and vulnerability disclosure