Running Gemma4 31B at full 1 million token context with a multi-token-prediction (MTP) speculative drafter on a single H200 GPU. The result: 48 tokens/second for a 30B+ model with 1M context, served over OpenAI-compatible HTTPS, with thinking mode and tool calling.

With native 256K context, Gemma4-31B can actually run faster, up to 60 token/sec. But in this case, we need the 1M context for processing huge amount of data. Gemma4-31B with 1M context slowed down to 6 token/sec. Utilizing MTP drafter, the speed increased to 48 token/sec.

This post is the full build log: the hardware, the dead-ends, the math that mattered, and the final reproducible setup.

Hardware

• NVIDIA H200 (141 GB VRAM), CUDA 13.0
• 440 GB system RAM, 4 TB NVMe
• Debian 13 (arm64), llama.cpp build 529ac89 from Oct 14 2026
• Caddy 2.x as reverse proxy, systemd for supervision

The goal

One llama.cpp server that:

• Loads unsloth/gemma-4-31B-it-GGUF (UD-Q4_K_XL, 18.8 GB on disk, 30.7 B params)
• Exposes 1,048,576-token context (via --override-kv gemma4.context_length + YaRN extension factor 4 over the model’s native 256K)
• Uses Gemma 4 native thinking mode (--reasoning auto)
• Supports tool calling via the Jinja chat template (--jinja)
• Enforces a sha512-based API key on every request
• Listens on 127.0.0.1:10002
• Reaches the world at https://xxxxx.ai.id/gemma4/ through a Caddy subroute

The VRAM-budget iterations

This is where the real work was. My original “max 16 GB” cap turned out to be mathematically impossible for 1M context with a 30B model. We had to bump the cap twice.

Why 16 GB can’t work

llama.cpp’s flash-attention computes a per-graph scratch buffer that scales with the configured context size, not the actual prompt length. At 1M tokens, this buffer alone is ~10.6 GB of VRAM on the GPU, regardless of model size. The 30 B Q4_K_M model wants another 12-18 GB. You cannot fit 10.6 + 12 = 22.6 GB of work into a 16 GB cap.

Key numbers from real measurements (H200, CUDA 0, KV cache on CPU):

ngl	Context	Drafter	VRAM	Tok/s	Notes
30	1M	no	21.7 GB	6.2	Original. 31/61 layers on CPU = bottleneck.
40	1M	no	23.2 GB	~9
50	1M	no	25.5 GB	~12	Just over 24 GB cap.
60	1M	no	~29 GB	~15
99	1M	no	28.7 GB	~15	All on GPU, no drafter.
99	1M	yes	40.5 GB	48.4	Final config. 8× speedup, 4.5 GB over the 36 GB cap.

The MTP drafter discovery

Gemma 4 ships with a companion mtp-gemma-4-31B-it.gguf (491 MB on disk) — a 4-layer next-token-prediction assistant. When used as a speculative drafter with --spec-type draft-mtp, the drafter proposes N tokens that the target verifies in one forward pass. Acceptance rate on the workloads we measured: ~37 % (297 accepted of 801 drafted on a 500-token run).

That’s where the 8× speedup comes from.

The gotcha: the drafter inherits the target’s --ctx-size and therefore its own 1M context compute graph. The drafter is 0.5 GB of model + 2.3 GB of q8_0 KV, but the unavoidable 9 GB flash-attn scratch for the drafter’s 1M context is what pushes the total over the 36 GB cap. There is no --ctx-size-draft flag in llama.cpp as of this writing.

Final setup

1. Build llama.cpp

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j

2. Download the models

mkdir -p /home/myuser/gemma4
cd /home/myuser/gemma4
hf download unsloth/gemma-4-31B-it-GGUF \
  --include "gemma-4-31B-it-UD-Q4_K_XL.gguf"
hf download unsloth/gemma-4-31B-it-GGUF \
  --include "mtp-gemma-4-31B-it.gguf"

3. Generate an API key

head -c 64 /dev/urandom | sha512sum | awk '{print $1}' > /home/myuser/gemma4/.api-key
chmod 600 /home/myuser/gemma4/.api-key

4. start-gemma4.sh

#!/bin/bash
# Gemma 4 31B (UD-Q4_K_XL) - llama.cpp server
# 1M context + all-GPU + MTP drafter.
# VRAM ~40 GB on GPU 1 (H200).

set -e
export CUDA_VISIBLE_DEVICES=1
export LLAMA_CACHE=/home/myuser/llama.cpp/build/bin
export GGML_CUDA_DISABLE_GRAPHS=0

MODEL_DIR=/home/myuser/gemma4
MODEL_FILE="$MODEL_DIR/gemma-4-31B-it-UD-Q4_K_XL.gguf"
MTP_FILE="$MODEL_DIR/mtp-gemma-4-31B-it.gguf"
API_KEY_FILE="$MODEL_DIR/.api-key"

if [ ! -f "$MODEL_FILE" ]; then
    echo "ERROR: model not found" >&2; exit 1
fi
if [ ! -f "$MTP_FILE" ]; then
    MTP_ARG=""
else
    MTP_ARG="--model-draft $MTP_FILE \
             --spec-type draft-mtp \
             --spec-draft-n-max 4 \
             --n-gpu-layers-draft 999 \
             --cache-type-k-draft q8_0 \
             --cache-type-v-draft q8_0"
fi

API_KEY=$(cat "$API_KEY_FILE")

exec /home/myuser/llama.cpp/build/bin/llama-server \
    --model "$MODEL_FILE" \
    $MTP_ARG \
    --alias "gemma-4-31b" \
    --host 127.0.0.1 \
    --port 10002 \
    --api-key "$API_KEY" \
    --override-kv gemma4.context_length=int:1048576 \
    --ctx-size 1048576 \
    --n-gpu-layers 99 \
    --no-kv-offload \
    --cache-type-k q4_0 \
    --cache-type-v q4_0 \
    --batch-size 512 \
    --ubatch-size 128 \
    --parallel 1 \
    --threads 16 \
    --cont-batching \
    --flash-attn on \
    --load-mode none \
    --jinja \
    --reasoning auto \
    --reasoning-budget -1 \
    --no-warmup \
    --rope-scaling yarn \
    --yarn-orig-ctx 262144 \
    --yarn-ext-factor 4 \
    --yarn-attn-factor 1.0 \
    --temp 1.0 --top-p 0.95 --top-k 64 \
    --log-verbosity 4 \
    "$@"

5. Caddy subroute

# /etc/caddy/routes/gemma4.caddy
@gemma4 path /gemma4 /gemma4/*
handle @gemma4 {
    uri strip_prefix /gemma4
    reverse_proxy 127.0.0.1:10002 {
        flush_interval -1
    }
}

After dropping the file:

sudo systemctl reload caddy

6. systemd unit

# /etc/systemd/system/gemma4.service
[Unit]
Description=Gemma 4 31B (UD-Q4_K_XL) via llama.cpp — port 10002
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=sufehmi
Group=sufehmi
WorkingDirectory=/home/myuser/gemma4
Environment=CUDA_VISIBLE_DEVICES=1
Environment=GGML_CUDA_DISABLE_GRAPHS=0
Environment=LLAMA_CACHE=/home/myuser/llama.cpp/build/bin
ExecStart=/home/myuser/gemma4/start-gemma4.sh
Restart=on-failure
RestartSec=10
StandardOutput=append:/home/myuser/gemma4/logs/gemma4.log
StandardError=append:/home/myuser/gemma4/logs/gemma4.log
LimitNOFILE=65536
MemoryHigh=180G
MemoryMax=200G

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now gemma4

Verification

Auth + model info

API_KEY=$(cat /home/myuser/gemma4/.api-key)

# No key -> 401
curl -s -o /dev/null -w "HTTP %{http_code}\n" \
  https://xxxxx.ai.id/gemma4/v1/models

# Good key -> 200
curl -s -H "Authorization: Bearer $API_KEY" \
  https://xxxxx.ai.id/gemma4/v1/models \
  | python3 -c "import json,sys; d=json.load(sys.stdin); m=d['data'][0]['meta']; \
      print(f\"model={d['data'][0]['id']}  n_ctx={m['n_ctx']}  \
            n_params={m['n_params']/1e9:.1f}B  quant={m['ftype']}\")"

Expected output:

HTTP 200
model=gemma-4-31b  n_ctx=1048576  n_params=30.7B  quant=Q4_K - Medium

Thinking mode

curl -s -X POST https://xxxxx.ai.id/gemma4/v1/chat/completions \
  -H "Authorization: Bearer $API_KEY" -H "Content-Type: application/json" \
  -d '{
    "model":"gemma-4-31b",
    "messages":[{"role":"user","content":"What is 7*6?"}],
    "max_tokens":300, "temperature":0.3
  }' | python3 -c "import json,sys; d=json.load(sys.stdin); \
    m=d['choices'][0]['message']; \
    print('reasoning:', m.get('reasoning_content','')[:200]); \
    print('answer:   ', m.get('content','')); \
    t=d.get('timings',{}); \
    print(f\"speed:    {t['predicted_per_second']:.1f} tok/s\")"

The flags that actually matter

• --override-kv gemma4.context_length=int:1048576 — unlocks 1M beyond the model’s native 262 K. Without this, llama.cpp refuses to allocate a 1M context.
• --ctx-size 1048576 — actually allocates the 1M KV.
• --yarn-orig-ctx 262144 + --yarn-ext-factor 4 — extends the rotary embedding to cover 1M tokens.
• --n-gpu-layers 99 — puts all 60 transformer layers + output on GPU. 30 layers gave us 6 tok/s; 99 gave 48.
• --no-kv-offload — keeps the 22.5 GB 1 M-context KV cache on RAM. Removing this blows the 36 GB cap on its own.
• --cache-type-k q4_0 --cache-type-v q4_0 — KV cache at Q4 precision. Half the RAM, no measurable quality loss at 1 M.
• --flash-attn on — without this, 1M context takes 4× longer per token.
• --no-warmup — skips the warmup pass that would otherwise double the first-token latency.
• --load-mode none — no mmap. mmapping the 18.8 GB GGUF didn’t change VRAM but added 1 s of first-load latency.
• --jinja — uses the model’s built-in Jinja template (Gemma 4’s native format with <|think|> tokens). Without this, thinking mode is unavailable.
• --reasoning auto + --reasoning-budget -1 — enable Gemma 4’s native thinking, unlimited budget.

What we tried that didn’t help

• --batch-size 256 --ubatch-size 64 instead of 512/128 — compute buffer is determined by graph size, not batch; no VRAM savings.
• Drafter KV at q4_0 instead of q8_0 — only 0.6 GB savings; the 9 GB compute buffer dwarfs the KV.
• Drafter on CPU (--n-gpu-layers-draft 0) — saved only 0.5 GB because the drafter’s compute graph stays in GPU memory.
• Manual pkill before systemd started — the 1st systemd restart counter kept growing until we let it own the port (lesson: don’t fight systemd, let it win).

If you’re on tighter budgets

Drop the drafter and use --n-gpu-layers 99 alone: 28.7 GB VRAM, ~15 tok/s. The model is fully on GPU, just no speculative decoding. Lose 3× speed for 12 GB less.

Drop to 256 K context (--ctx-size 262144, drop --override-kv and YaRN): ~18 GB VRAM, ~50-80 tok/s. The model is trained for 256 K so quality is best here.

Files on disk

/home/myuser/llama.cpp/                # v0.6.0-dev, commit 529ac89
/home/myuser/gemma4/gemma-4-31B-it-UD-Q4_K_XL.gguf    # 18 GB
/home/myuser/gemma4/mtp-gemma-4-31B-it.gguf            # 491 MB
/home/myuser/gemma4/.api-key            # 128-char sha512, chmod 600
/home/myuser/gemma4/start-gemma4.sh     # runner
/home/myuser/gemma4/gemma4.service      # systemd unit draft
/home/myuser/gemma4/logs/gemma4.log
/etc/caddy/routes/gemma4.caddy           # installed
/etc/systemd/system/gemma4.service       # installed

OpenAI-compatible, with thinking, with tools, with auth, durable across reboots. The 8× speedup is the headline; the 1M context unlocked via --override-kv is the trick that makes it possible.