Monthly Archives: April 2026

Running Kimi K3 model with 4GB VRAM – using AirWrapperLLM and AirLLM

Etc

Introduction

Running a model too large for your GPU is a problem most people solve by giving up. AirLLM takes the opposite approach: it streams the model’s layers (or expert sublayers, for MoE) from disk into the GPU one at a time, so even a 2.8-trillion-parameter model like Kimi K3 can run on a single 16 GB card. The catch is that AirLLM is a Python library, not a server — there is no HTTP endpoint, no API key, no streaming, no tool-call parsing. You have to write that yourself.

That’s what AirWrapperLLM is: an OpenAI-compatible FastAPI server that wraps AirLLM and gives it the surface every modern agent expects. This guide walks through the complete install + first-run, including the hardware you’ll actually need and the speed you’ll actually get — both of which AirLLM’s marketing sometimes glosses over.

What AirWrapperLLM gives you

After one pip install and one server start, you have:

  • POST /v1/chat/completions — full OpenAI schema, both streaming and non-streaming responses.
  • GET /v1/models — model listing.
  • GET /health — liveness check, reports server version + uptime.
  • Bearer-token API-key auth — the key is auto-generated on first start and persisted with chmod 600.
  • Multi-turn conversations — pass message history, the tokenizer’s chat template renders it.
  • Thinking / reasoning mode — the model’s <think> trace is split out into message.reasoning.
  • Tool / function calling — pass tools=[...] in the standard OpenAI format; AirWrapperLLM parses the model’s XTML <tools> channel and returns structured tool_calls.
  • Thinking-effort control — reasoning={"effort": "low" | "medium" | "high" | "max"} in the request body, or lower-level chat_template_kwargs.

The XTML parser is a standalone, dependency-free module (airwrapper_xtml.py) with 7 unit tests. If you want to build your own service around AirLLM, you can lift the parser straight out.

Use it as a backend for your agent

AirWrapperLLM speaks the standard OpenAI HTTP API, so any agent that supports an OpenAI-compatible base_url works against it without code changes. Just point the agent at http://host:port/v1 with the Bearer token:

Agent Configuration
Hermes Agent Edit ~/.hermes/config.yaml: set model.base_url: "http://<host>:20002/v1" and model.api_key: "<your air-... key>".
OpenClaw / openclaw-agent Set LLM_BASE_URL=http://<host>:20002/v1 and LLM_API_KEY=<your air-... key>.
OpenCode Set OPENAI_API_BASE=http://<host>:20002/v1 and OPENAI_API_KEY=<your air-... key>.
Aider / Cline / Continue / LangChain / LiteLLM Wherever the agent asks for an OpenAI base_url and api_key, supply http://<host>:20002/v1 and your air-... key.

What works through the agent: multi-turn, tool calling, reasoning/thinking. What to watch out for: latency is high (first-token takes tens of seconds to minutes; tokens then trickle out as layers stream from disk), there is no live token-by-token streaming (the full sequence arrives in one batch), and the server is single-tenant — one generation at a time, period.

Hardware requirements (be honest with yourself)

AirLLM’s tagline is “run any model on any GPU.” That’s true in the sense that the model will load and produce output. It is not true in the sense that it will be fast. Before you commit, measure against this table:

Scenario GPU you need Disk space Realistic throughput
Kimi K3 (2.8T params, FP8) 1× any CUDA GPU with 8–16 GB ~1.5 TB during layer-split ~0.5–1 tok/s
DeepSeek-V3 (671B) 1× RTX 5090 / 6000 Ada (32 GB) ~340 GB ~2–5 tok/s
Llama 3.1 405B 1× A100 (80 GB) ~240 GB ~5–10 tok/s
Qwen3 32B 1× any 8 GB+ GPU ~65 GB ~10–25 tok/s

Three hardware rules that aren’t optional:

  1. Disk speed matters as much as GPU speed. The model is read off NVMe for every token. A SATA SSD will make 0.5 tok/s into 0.05 tok/s. Get NVMe with at least 1 GB/s sequential read.
  2. RAM matters too. AirLLM can pin host memory for layer prefetching; 16 GB is a minimum, 64 GB is comfortable for the 100B+ class.
  3. You need 2–3× the model size in free disk during the initial layer-split transform (the original Hugging Face snapshot plus the per-shard directory).

If you need vLLM-class speed (dozens to hundreds of tok/s), AirLLM is the wrong tool. AirLLM is the right tool when the model is too big to fit any other way and you’re willing to wait.

Install

Clone and create a virtualenv:

git clone https://github.com/sufehmi/AirWrapperLLM.git
cd AirWrapperLLM
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip wheel setuptools

Install PyTorch with the CUDA wheel matching your driver. The README’s recommendation is CUDA 12.8 for Blackwell GPUs (RTX 50-series, cc 10.0/12.0); check vast-capabilities | jq .hardware.gpu.cuda or your equivalent to confirm:

pip install torch --index-url https://download.pytorch.org/whl/cu128

Then everything else:

pip install -r requirements.txt

The pinned requirements include AirLLM 3.x, transformers 4.56.x (some remote-code models — Kimi K3 — refuse to load on 5.x), tiktoken, fla-core, bitsandbytes, compressed-tensors, fastapi, uvicorn, pydantic.

If your model uses flash attention, install it against the CUDA toolkit that matches your torch wheel:

CUDA_HOME=/usr/local/cuda-12.8 pip install flash-attn --no-build-isolation

There are no prebuilt flash-attn wheels for torch 2.11, so this takes a few minutes to compile. On Blackwell (cc 12.0) the compile is straightforward; on older arches you may need a CUDA toolkit match the driver, and on Hopper (cc 9.0) flash-attn 2.8.3 builds cleanly against CUDA 12.x.

Download a model

AirWrapperLLM is engine-agnostic over any AirLLM-compatible checkpoint. The current AirLLM readme highlights Kimi K3; smaller alternatives (Qwen3 32B, Llama 3.1 70B, DeepSeek-V3) work the same way.

python3 -c "
from huggingface_hub import snapshot_download
snapshot_download(
    'nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8',  # or any AirLLM-compatible model
    local_dir='./models/model-name',
    max_workers=16,
)
"

With max_workers=16 and a fast connection, expect roughly 10–15 minutes per 100 GB of weights. The Kimi K3 download alone is around 1.5 TB.

Start the server

python airwrapper.py \
  --model ./models/model-name \
  --host 0.0.0.0 \
  --port 20002 \
  --max-seq-len 32768 \
  --compression 4bit \
  --delete-original

Or use the bundled launch wrapper, which sources the right environment variables first:

./launch_docker.sh

On a successful start you’ll see:

================================================================
AirWrapperLLM — OpenAI-compatible server for AirLLM
================================================================
API key : air-XXXXXXXXXXXXXXXXXXXXXXXXXXXX
Model   : ./models/model-name
Listen  : 0.0.0.0:20002
Compress: 4bit
Delete original after split: True
Max seq : 32768
================================================================
...
[AirWrapperLLM] Model loaded in 1500s.
[AirWrapperLLM] Starting uvicorn on 0.0.0.0:20002
INFO:     Uvicorn running on http://0.0.0.0:20002

The first start is slow — AirLLM splits the checkpoint into per-layer shards on disk (and optionally 4bit/8bit-compresses them). Subsequent starts reuse the cached shards and are fast.

CLI flags and environment variables

Flag / env var Default Notes
--model / AIRWRAPPER_MODEL /workspace/kimi-k3 Path to a downloaded Hugging Face checkpoint.
--host / AIRWRAPPER_HOST 0.0.0.0 Bind address.
--port / AIRWRAPPER_PORT 20002 HTTP port.
--compression / AIRWRAPPER_COMPRESSION (empty) 4bit or 8bit. Halves disk reads at the cost of disabling layer prefetching.
--delete-original false After the first load, delete the original HF snapshot to save disk.
--dtype / AIRWRAPPER_DTYPE auto (bf16) Runtime dtype. bfloat16 is recommended for modern models.
--max-seq-len / AIRWRAPPER_MAX_SEQ_LEN 1048576 Maximum sequence length (context + generation). Memory use of attention layers scales with this; Mamba/SSM layers do not.
AIRWRAPPER_API_KEY_FILE /workspace/.airwrapper_api_key Where the API key is read/written.
AIRWRAPPER_DEVICE cuda:0 CUDA device for AirLLM to stream into.

Test it

Copy the API key from the server’s stdout, then:

API_KEY=air-XXXXXXXXXXXXXXXXXXXXXXXXXXXX

# Health check (no auth required)
curl http://localhost:20002/health
# {"status":"ready","version":"1.0.0","model":"...","uptime":...}

# Models
curl http://localhost:20002/v1/models \
  -H "Authorization: Bearer $API_KEY"

# Chat with thinking mode
curl http://localhost:20002/v1/chat/completions \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-model",
    "messages": [{"role":"user","content":"What is 17 * 23?"}],
    "max_tokens": 200,
    "temperature": 0.6,
    "reasoning": {"effort": "auto"}
  }'

Python with the official OpenAI SDK:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:20002/v1",
    api_key="air-XXXXXXXXXXXXXXXXXXXXXXXXXXXX",
)

resp = client.chat.completions.create(
    model="your-model",
    messages=[{"role": "user", "content": "Hello"}],
    max_tokens=100,
    extra_body={"reasoning": {"effort": "auto"}},
)
print(resp.choices[0].message.reasoning)   # thinking trace, if any
print(resp.choices[0].message.content)     # final answer

Tool calling

curl http://localhost:20002/v1/chat/completions \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-model",
    "messages": [{"role":"user","content":"What is the weather in Tokyo?"}],
    "tools": [{"type":"function","function":{
      "name":"get_weather",
      "description":"Get the current weather for a city",
      "parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}
    }}]
  }'

The response includes structured tool_calls:

{
  "choices": [{
    "finish_reason": "tool_calls",
    "message": {
      "role": "assistant",
      "tool_calls": [{
        "id": "call_abc123...",
        "type": "function",
        "function": {"name": "get_weather", "arguments": "{\"city\": \"Tokyo\"}"}
      }]
    }
  }]
}

Multi-turn

Just pass the full conversation history — the tokenizer’s chat template handles rendering:

resp = client.chat.completions.create(
    model="your-model",
    messages=[
        {"role": "user", "content": "My name is Alice. Remember it."},
        {"role": "assistant", "content": "Nice to meet you, Alice!"},
        {"role": "user", "content": "What is my name?"},
    ],
    max_tokens=50,
)

Run as a managed service

For production deployments you’ll want auto-restart on crash. The launch wrapper is supervisor-friendly:

[program:airwrapper]
environment=PROC_NAME="%(program_name)s"
command=/path/to/AirWrapperLLM/launch_docker.sh
autostart=true
autorestart=unexpected
startsecs=2
stopasgroup=true
killasgroup=true
stopsignal=TERM
stdout_logfile=/dev/stdout
redirect_stderr=true
stdout_logfile_maxbytes=0

Or under systemd. The server is single-process, single-GPU, so a simple supervisor/service is sufficient — no clustering, no shared state.

Operational notes

Rotating the API key: kill the server, regenerate the key file, restart.

echo "API_KEY=air-$(openssl rand -hex 24)" > /path/to/.airwrapper_api_key
chmod 600 /path/to/.airwrapper_api_key
supervisorctl restart airwrapper

Inspecting logs: with the supervisor config above, logs go to /dev/stdout and surface in the supervisor journal. The model’s quantization + loading phase prints tens of thousands of progress-bar lines — pipe through grep -v "Compressing\|Applying quant" if you want clean output.

When generation hangs: AirLLM is single-tenant. If a request times out on the client, the server still holds the generation lock. Wait for the in-flight run to finish (or use supervisorctl restart airwrapper to hard-stop — the server doesn’t expose a “cancel” signal yet).

Why no live token streaming? AirLLM returns the full sequence only after the last token. AirWrapperLLM emits the parsed result as SSE deltas when the run completes, not token-by-token as they generate. The OpenAI stream=true contract is honored, but in practice clients will see all tokens arrive at once. Improving this requires changes to AirLLM itself.

Limitations

  • Single request at a time. AirLLM has no batched-inference path. Two requests in flight queue; they don’t run in parallel.
  • No live token-by-token streaming. See above.
  • No LoRA / adapter support. AirLLM doesn’t expose this.
  • XTML parser is Kimi-K3-aware. Other models that use a different structural-token scheme may need a tweaked parser.
  • No multi-GPU. AirLLM runs on one device by design; AirWrapperLLM cannot override that without a different engine underneath.

Conclusion

AirWrapperLLM turns AirLLM’s “run any model from disk” engine into a normal OpenAI-compatible HTTP service: API key auth, multi-turn, thinking mode, tool calling, versioned releases. If you have a model that’s too big for your VRAM and you’re willing to wait for tokens, it’s the cleanest end-to-end stack available in 2026.

If you want real speed, use vLLM with a model that fits your VRAM. AirLLM and AirWrapperLLM exist for the case where that’s not an option.

AirWrapperLLM v1.0.0 — MIT licensed. Project: github.com/sufehmi/AirWrapperLLM. AirLLM is Apache 2.0 by Gavin Li.