A field-tested walkthrough of running Mistral-Medium-3.5-128B in Q4_K_M GGUF on four Tesla V100-SXM2-32GB cards. Includes the reasoning, the dead ends, the patches, and the working OpenAI-compatible deployment — with 64K context, tool calling, reasoning, and API-key auth, served via llama-server.
llama.cpp + a properly-merged single-file GGUF + a one-line chat-template patch. End-to-end: ~80 seconds to load the model, ~9 tok/s generation, full OpenAI API compatibility, served via a Cloudflare Quick Tunnel. The article below walks through every wrong turn, what we learned from each, and the exact commands that finally worked.
The goal
Run Mistral Medium 3.5 (128B) on the four V100-SXM2-32GB cards in our compute box, expose it as an OpenAI-compatible API on port 20000, accessible from the public internet via a Cloudflare tunnel. Required features: multi-turn chat, tool calling, native reasoning mode, API-key authentication.
The hardware constraint shaped everything that followed. V100 is Volta — compute capability 7.0. It has no tensor cores, no FP8, no BF16, no hardware attention. Any “modern” inference stack that assumes Ampere or later will refuse to run, or run with a degraded backend.
Why not vLLM?
vLLM is the obvious choice. It speaks the OpenAI API natively, supports Mistral’s tool-calling format, has a --reasoning-parser mistral flag for the new Magistral-style reasoning tokens, and integrates with Hugging Face tokenizers. On paper it should “just work” with a GGUF — vLLM has a gguf.py loader since 0.7.
On V100, it does not. Three issues, in order of encounter:
1. compressed-tensors and awq quantization loaders are gated by SM
vLLM’s auto-select quant loader checks each layer’s required minimum compute capability. For the compressed-tensors backend: Min capability: 80 (Ampere). For the legacy awq backend: 75 (Turing). V100 is 70. We hit this immediately when trying to serve cyankiwi/Mistral-Medium-3.5-128B-AWQ-INT4:
RuntimeError: Quantization scheme is not supported for the current GPU.
Min capability: 80. Current capability: 70.That rules out every standard AWQ/GPTQ INT4 quant on this hardware. Which leaves either BF16/FP16 (won’t fit — 128B × 2 bytes = 256 GB, we have 128 GB total) or GGUF.
2. vLLM dropped Volta pre-built kernels at 0.9+
The last vLLM release with Volta-compatible pre-built CUDA binaries is 0.8.5. Newer versions may or may not work via PTX JIT fallback, but the community consensus is “don’t bother.” We pinned to vllm==0.8.5.
3. Triton 3.1/3.2 crashes on V100 with GGUF
This is the killer. vLLM successfully loads a GGUF model on V100, but the first inference call crashes:
LLVM ERROR: Failed to compute parent layout for slice layout.
ERROR [client.py:305] RuntimeError('Engine process (pid 24920) died.')This is a known Triton bug specific to SM 7.0: Triton’s linear-layout code generator was reworked for Hopper (SM 9.0+) and emits a layout the V100 backend can’t decode. Confirmed by issues #17152, #17449, and the vLLM 0.9 release notes.
We tried downgrading Triton to 2.3.0 (the old layout-free version). That produced a different error: ImportError: cannot import name 'triton_key'. Triton 3.0.0 sits between the two bugs but wasn’t on PyPI in the form we needed. We could have patched the Triton source, but “patching the JIT compiler of a major framework for one model deployment” is the kind of technical debt that wakes you up at 3 AM six months later. Pass.
llama.cpp. GGUF is llama.cpp’s native format. It supports Volta. It has tool-call parsers. It supports Mistral chat templates. It is what the GGUF was designed for.
Why llama.cpp
llama.cpp is the project that invented the GGUF format. Its llama-server binary speaks OpenAI-compatible HTTP, supports streaming, tool calling, multiple chat templates, and is the most thoroughly tested GGUF loader on consumer and datacenter hardware alike.
For our target hardware (4× V100-SXM2-32GB, NVLink), llama.cpp has the right answers baked in:
- Tesla V100 support: SM 7.0 is a first-class target.
- Multi-GPU:
--tensor-split a,b,c,ddistributes layers across GPUs. - Quantization support: native GGUF, including Q2_K through Q8_0 and the i-quants.
- OpenAI-compatible API:
/v1/chat/completions,/v1/models,/v1/embeddings. - Reasoning extraction:
--reasoning-format deepseekstrips[THINK]...blocks into a separatereasoning_contentfield. - Tool-call parsing: supports the Mistral tool format natively via the chat template.
Building it
git clone --depth=1 https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j$(nproc) --target llama-server~5 minutes on the V100 box. GGML_CUDA=ON is the only flag you need; CUDA toolkit 12.8 was already in the base image. We also built llama-gguf-split (the tool that knows how to merge split GGUF shards correctly) with the same command.
The model: which GGUF?
Mistral Medium 3.5 ships as a 128-billion-parameter dense transformer. The HF ecosystem offers several quantizations:
| Quant | Size | Quality | Notes |
|---|---|---|---|
| BF16 (original) | 248 GB | Reference | Won’t fit. Skip. |
| AWQ-INT4 (cyankiwi) | 153 GB | Good | vLLM rejects (SM 80 required). Skip on V100. |
| Q2_K (unsloth, single file) | 44 GB | Degraded | Too lossy for a 128B reasoning model. |
| Q4_K_M (unsloth, split) | 71 GB | Very good | Sweet spot. Fits 4× V100 with KV cache headroom. |
| Q5_K_M, Q6_K | 87 / 103 GB | Mixed | Tighter fit, less KV cache room. |
We picked unsloth/Mistral-Medium-3.5-128B-GGUF in Q4_K_M. The unsloth quantizer splits it into three shards (because GGUF metadata + tensors exceeds HF’s per-file recommendation). The breakdown:
- Shard 1 (8 MB): tokenizer metadata + the
output.weighttensor (kept at high precision Q6_K because Mistral intentionally preserves the output projection). - Shard 2 (47 GB): roughly the first half of the transformer blocks.
- Shard 3 (24 GB): the second half of the blocks.
The merge: the one hard part
llama-server needs a single GGUF file. The unsloth repo gives you three. Two options:
llama-gguf-split --merge(official llama.cpp tool). Knows how to merge split GGUFs while preserving all KV metadata, including thetokenizer.ggml.mergesARRAY (a list of 538,891 BPE merge rules that the tokenizer needs).- Roll your own merger in Python using the
ggufPyPI library. Do not do this.
We learned option 2 the hard way. Our first attempt used a custom Python script that read each shard with gguf.GGUFReader and wrote a merged file with gguf.GGUFWriter. The tensors copied fine, the scalar metadata (architecture, hyperparameters) copied fine. But the script silently dropped every ARRAY-typed KV (tokenizer merges, token list, token types). The resulting GGUF loaded into vLLM but crashed inside llama.cpp’s tokenizer init with the cheerful error: “cannot find tokenizer merges in model file.”
Switching to llama-gguf-split --merge fixed it in one command:
cd /workspace/models/mistral-medium-3.5-gguf/Q4_K_M
llama-gguf-split --merge Mistral-Medium-3.5-128B-Q4_K_M-00001-of-00003.gguf merged.gguf
# gguf_merge: merged.gguf merged from 3 split with 795 tensors.Verifying the merge
The gguf Python library is a useful sanity check:
from gguf import GGUFReader
r = GGUFReader("merged.gguf")
for k in ["tokenizer.ggml.merges", "tokenizer.ggml.tokens", "tokenizer.ggml.token_type"]:
f = r.fields[k]
print(f"{k}: {len(f.parts)} parts, type {f.types[0].name}")
# tokenizer.ggml.merges: 538891 parts, type ARRAY
# tokenizer.ggml.tokens: 262149 parts, type ARRAY
# tokenizer.ggml.token_type: 131077 parts, type ARRAY~538K BPE merge rules, ~262K vocab tokens, ~131K token types — all there. The official Mistral tokenizer (tekken v15, vocab size 131,072) is the one baked into this GGUF.
Server flags: what worked
The complete llama-server invocation:
llama-server \
--model /workspace/models/mistral-medium-3.5-gguf/Q4_K_M/Mistral-Medium-3.5-128B-Q4_K_M.gguf \
--host 0.0.0.0 \
--port 20000 \
--api-key "sk-..." \
--ctx-size 65536 \
--n-gpu-layers 99 \
--tensor-split "8,8,8,8" \
--parallel 1 \
--jinja \
--chat-template-file /workspace/models/mistral-medium-3.5-tokenizer/chat_template_patched.jinja \
--reasoning on \
--reasoning-format deepseek \
--temp 0.7 \
--top-p 0.95 \
--top-k 40 \
--alias mistral-medium-3.5Notes on each flag:
| Flag | Value | Why |
|---|---|---|
--model |
merged GGUF | Single file the server can mmap directly. |
--ctx-size |
65536 | 64K context per slot. Lower if you need more slots. |
--n-gpu-layers 99 |
99 | All layers to GPU; no CPU offload. |
--tensor-split |
8,8,8,8 | Distribute evenly: each V100 gets 1/4 of every linear layer’s weight matrix. |
--parallel 1 |
1 | One slot, full 64K context. Increase for concurrency at the cost of context per slot. |
--jinja |
flag | Use the Jinja chat template (vs. llama.cpp’s built-in C++ template). |
--chat-template-file |
patched.jinja | Custom template (see next section). |
--reasoning on |
Enable reasoning mode. | |
--reasoning-format deepseek |
Strip [THINK]... into a separate reasoning_content field. |
|
--alias |
mistral-medium-3.5 | The model name clients use. |
The chat template patch (the one thing nobody told us)
The Mistral-Medium-3.5 chat template that ships in the unsloth GGUF (and the one in the official mistralai repo) requires the client to pass reasoning_effort: "none" or "high". Any other value, or omitting the parameter entirely, throws:
Jinja Exception: reasoning_effort must be either "none" or "high"This breaks any client that defaults to "low" or "medium" (the Mistral API itself accepts those). We patched the template to:
- Default
reasoning_effortto"high"when the client omits it. - Normalize any non-standard value (
"low","medium", etc.) to"high"instead of raising.
{#- if reasoning_effort is not defined or reasoning_effort is none %}
{%- set reasoning_effort = 'high' %} {# was: 'none' #}
{%- endif %}
{%- if reasoning_effort not in ['none', 'high'] %}
{%- set reasoning_effort = 'high' %} {# was: raise_exception(...) #}
{%- endif %}The patch is at /workspace/models/mistral-medium-3.5-tokenizer/chat_template_patched.jinja and is loaded via --chat-template-file. After the patch, the API accepts any reasoning_effort value (or none) and the server always runs in reasoning mode. To disable reasoning, clients must explicitly send "reasoning_effort": "none".
Putting it under supervisor
Supervised by the existing supervisord in the base image. The wrapper script waits for the merged GGUF and the binary to be present, then execs llama-server with the right flags. startsecs=3600 (1 hour) gives the 128B model plenty of time to load without supervisor giving up on it.
The relevant files on the host:
/opt/supervisor-scripts/vllm-mistral-medium.sh # wrapper
/etc/supervisor/conf.d/vllm-mistral-medium.conf # supervisor service
/workspace/.vllm_credentials.env # API key + paths (mode 600)
/workspace/models/mistral-medium-3.5-gguf/Q4_K_M/Mistral-Medium-3.5-128B-Q4_K_M.gguf # 70 GB
/workspace/models/mistral-medium-3.5-tokenizer/chat_template_patched.jinja
/workspace/llama.cpp/build/bin/llama-server # binary
/var/log/portal/vllm-mistral-medium.log # logsPerformance numbers (V100-SXM2-32GB, Q4_K_M, 4× TP, 64K ctx)
| Metric | Value |
|---|---|
| Model load time | ~80 seconds |
| VRAM per card (idle) | 23-25 GB (out of 32 GB) |
| Prompt eval (cold) | ~127 tok/s |
| Prompt eval (KV cache hit) | ~9 MS total / ~370 tokens cached |
| Generation | ~9 tok/s |
| Time to first token (TTFT) | ~3 seconds for ~370-token prompt |
The 9 tok/s generation speed is the V100 tax. No tensor cores, no FlashAttention-2, no FP8. The model is doing honest FP16 GEMMs with XFormers attention. For comparison, an A100-40GB hits ~25 tok/s on the same model, an H100 hits ~50 tok/s. On V100, 9 tok/s is the ceiling for Q4_K_M Mistral-128B.
If you need faster throughput on V100, the trade-off is quantization. Q5_K_M (~87 GB) gets you a noticeable quality bump but uses more VRAM. Q3_K (~58 GB) loses some coherence on hard reasoning tasks but doubles decode speed. For our use case (reasoning-heavy agentic workflows), Q4_K_M was the right balance.
Talk to it
Once running, the API is a standard OpenAI-compatible endpoint. Example with curl:
curl -sS -H "Authorization: Bearer sk-..." \
-H "Content-Type: application/json" \
https://crossing-exhibit-mlb-compression.trycloudflare.com/v1/chat/completions \
-d '{
"model": "mistral-medium-3.5",
"messages": [{"role": "user", "content": "What is 17 * 23?"}],
"max_tokens": 2048
}'Returns:
{
"choices": [{
"finish_reason": "stop",
"message": {
"role": "assistant",
"content": "17 multiplied by 23 is **391**.",
"reasoning_content": "The user is asking for a simple multiplication: 17 * 23..."
}
}],
"usage": {"completion_tokens": 124, "prompt_tokens": 373, "total_tokens": 497}
}Tool calls work the same way as the OpenAI API: send tools: [...], the model emits tool_calls, you respond with role: "tool" + tool_call_id. Multi-turn with tool results composes naturally.
What I’d do differently next time
- Skip vLLM entirely on V100. llama.cpp is the right tool for this hardware. Save yourself the day of debugging Triton’s slice-layout bug.
- Use
llama-gguf-split --mergefrom the start. Don’t try to write your own GGUF merger in Python. The official tool knows about every KV metadata format, including the ARRAY-typed ones that look like they should be trivial but aren’t. - Patch the chat template before the first client hits the API. The
reasoning_effortvalidation is a footgun; better to relax it up front than to debug it during an incident. - Set
--parallel 1for the full 64K context. More slots means smaller per-slot context, which defeats the point of having a 256K-context model. One slot at full context is the right default for single-tenant deployments.
Conclusion
128-billion-parameter LLMs on commodity datacenter hardware (V100) are a solved problem if you pick the right stack. The stack is llama.cpp + a properly-merged single-file GGUF + a chat-template patch for client leniency. vLLM is the wrong default for Volta — it will fight you at every quantization step and lose to the Triton bug on inference.