Running a 120-billion-parameter reasoning model across two GPUs is no longer an exotic exercise — in 2026, with NVIDIA’s FP8 quantization and the Hopper architecture’s native FP8 tensor cores, it is an afternoon project that delivers genuinely production-grade speed. This article walks through the complete, end-to-end deployment of Nemotron-3-Super-120B-A12B on a server with 2× NVIDIA H200 (143 GB each), served by vLLM behind a standard OpenAI-compatible API, with a 1-million-token context window, thinking/reasoning mode, tool calling, multi-turn conversations, and API-key authentication.
There is already an excellent guide for running this model on a single Blackwell RTX PRO 6000 using the NVFP4 (4-bit floating point) checkpoint. This article is the companion piece for the Hopper / H200 case — and as you’ll see, the model variant and several flags must change, because Hopper and Blackwell have different native low-precision capabilities. They are a different hardware, so the installation process turned out to be quite different.
The Target Hardware
| Component | Specification |
|---|---|
| GPU | 2× NVIDIA H200 |
| VRAM (per GPU) | 143 GB (143,771 MiB) |
| Total VRAM | ~287 GB |
| Compute Capability | sm_90 (Hopper) |
| Driver | 580.105.08 / CUDA 13.0 |
| System RAM | 440 GB |
| Storage | 1.5 TB NVMe |
| OS | Ubuntu 24.04.3 LTS |
The Critical Decision: FP8, Not NVFP4
This is the single most important thing to get right, and it is where a blind copy of the Blackwell guide will bite you.
Nemotron-3-Super-120B-A12B ships in three quantization variants on Hugging Face:
- NVFP4 — 4-bit floating point weights (~80 GB). Requires Blackwell architecture (B200/B300, RTX PRO 6000 Blackwell) with native FP4 tensor cores. On Hopper, FP4 has no native tensor-core support — it would be emulated and slow.
- FP8 — 8-bit floating point weights (~128 GB). This is the native sweet spot for Hopper (H100/H200). The H200’s FP8 tensor cores run this at full speed.
- BF16 — 16-bit, ~240 GB. Overkill and a waste of VRAM unless you need the absolute maximum accuracy.
For 2× H200, the correct choice is nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8. At ~128 GB it fits comfortably across both GPUs with tensor parallelism, leaving roughly 160 GB of headroom for a large FP8 KV cache — which is exactly what lets you run the full 1M-token context.
Rule of thumb: Blackwell (sm_120+) ? NVFP4. Hopper (sm_90) ? FP8. Do not mix these up — the performance difference is dramatic.
The Model: Nemotron-3-Super-120B-A12B
A quick refresher on why this model is special and how it achieves its speed:
- 120 billion total parameters, only 12 billion active per token — a hybrid Latent Mixture-of-Experts design (22 experts routed out of 512 per layer). This is why it is fast despite its size.
- Mamba-2 hybrid layers interleaved with MoE and attention layers. The Mamba layers have O(1) recurrent state (no growing KV cache), which is precisely how the model supports a 1,048,576-token (1M) context.
- Configurable reasoning/thinking mode via the chat template’s
enable_thinkingflag — the model produces a reasoning trace, then a final answer. - Native tool calling (function calling) and multi-turn conversation support.
- Multi-Token Prediction (MTP) layers for faster speculative generation.
On Hugging Face it lives at nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8. It is public and ungated — no token or license acceptance required to download.
Step 1 — Connect to the Server
ssh root@YOUR-SERVER-IPVerify the GPUs are visible and idle:
nvidia-smiYou should see two H200s with ~143 GB each and 0 MiB used.
Step 2 — Install Docker and GPU Passthrough
Unlike the Blackwell guide (where Docker was pre-installed), our H200 server had no Docker at all — only the nvidia-container-toolkit package. Install Docker first:
apt-get update
apt-get install -y docker.io
systemctl enable docker
systemctl start dockerThen configure the NVIDIA container runtime so Docker can pass GPUs through to containers:
nvidia-ctk runtime configure --runtime=docker
systemctl restart dockerVerify both GPUs are visible inside a container:
docker run --rm --runtime=nvidia --gpus all ubuntu:22.04 \
nvidia-smi --query-gpu=name,memory.total --format=csv,noheaderYou should see:
NVIDIA H200, 143771 MiB
NVIDIA H200, 143771 MiBStep 3 — Create a Python Environment
We need a small venv on the host for downloading the model. On Ubuntu 24.04, the python3-venv package must be installed explicitly:
apt-get install -y python3.12-venv
python3 -m venv /opt/nemotron-venv
/opt/nemotron-venv/bin/pip install --upgrade pip wheel setuptools
/opt/nemotron-venv/bin/pip install "huggingface_hub[cli]" hf_transferStep 4 — Download the Model (128 GB)
mkdir -p /opt/models
HF_HUB_CACHE=/opt/models /opt/nemotron-venv/bin/python - <<'PY'
from huggingface_hub import snapshot_download
snapshot_download(
"nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8",
cache_dir="/opt/models",
max_workers=16,
)
PYThis pulls 26 safetensors shards (~128 GB) plus the chat template, custom modeling code (modeling_nemotron_h.py), and a reasoning parser (super_v3_reasoning_parser.py). With 16 parallel workers on a fast connection, expect roughly 10–15 minutes.
Note the snapshot path — you’ll need it later:
find /opt/models -type d -path "*Nemotron-3-Super*FP8*/snapshots/*"Step 5 — Generate an API Key
Create a random API key that clients must present as a Bearer token:
API_KEY="nm-$(openssl rand -hex 24)"
echo "$API_KEY"
mkdir -p /opt/nemotron-serve
echo "API_KEY=$API_KEY" > /opt/nemotron-serve/.api_key
chmod 600 /opt/nemotron-serve/.api_keyKeep this key secret — every request to the server must carry it.
Step 6 — Pull the vLLM Image
The model’s README pins vLLM 0.20.0. Use the official Docker image — it bundles a matched CUDA 13.x toolchain so all the FP8 kernels compile and load cleanly. Do not be tempted by pip install vllm; the pip path hits JIT compiler/header mismatches on Hopper that the Docker image avoids entirely.
docker pull vllm/vllm-openai:v0.20.0Step 7 — The Launch Script
This is the heart of the deployment. Create the launch script:
cat > /opt/nemotron-serve/launch_docker.sh <<'EOF'
#!/bin/bash
set -euo pipefail
API_KEY="$(grep -oP 'nm-[0-9a-f]+' /opt/nemotron-serve/.api_key)"
# Path as it appears INSIDE the container (the HF_HOME mount point)
MODEL_PATH="/root/.cache/huggingface/models--nvidia--NVIDIA-Nemotron-3-Super-120B-A12B-FP8/snapshots/SNAPSHOT_HASH"
echo "Using model path: $MODEL_PATH"
echo "Using API key: $API_KEY"
docker rm -f nemotron-serve 2>/dev/null || true
exec docker run --name nemotron-serve \
--runtime nvidia --gpus all \
--network host \
--shm-size 16g \
--restart unless-stopped \
-e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
-e HF_HUB_OFFLINE=1 \
-e HF_HOME=/root/.cache/huggingface \
-e HF_MODULES_CACHE=/root/.cache/hf_modules \
-v /opt/models:/root/.cache/huggingface:ro \
-v /opt/hf-cache/modules:/root/.cache/hf_modules:rw \
vllm/vllm-openai:v0.20.0 \
--model "$MODEL_PATH" \
--served-model-name nvidia/nemotron-3-super \
--host 0.0.0.0 \
--port 20000 \
--api-key "$API_KEY" \
--async-scheduling \
--dtype auto \
--kv-cache-dtype fp8 \
--tensor-parallel-size 2 \
--enable-expert-parallel \
--trust-remote-code \
--gpu-memory-utilization 0.90 \
--enable-chunked-prefill \
--max-model-len 1048576 \
--max-num-seqs 32 \
--mamba-ssm-cache-dtype float32 \
--reasoning-parser nemotron_v3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder
EOF
chmod +x /opt/nemotron-serve/launch_docker.sh
mkdir -p /opt/hf-cache/modulesReplace SNAPSHOT_HASH with the actual hash from Step 4.
What each key flag does
--tensor-parallel-size 2— splits the model across both H200s. Essential for a 128 GB checkpoint.--enable-expert-parallel— distributes the MoE experts across GPUs (NVIDIA’s recommended setting for multi-GPU Hopper).--max-model-len 1048576+VLLM_ALLOW_LONG_MAX_MODEL_LEN=1— unlocks the full 1M-token context. The model’smax_position_embeddingsis 262144, but the Mamba/attention hybrid legitimately extends to 1M; the env var lets vLLM exceed the conservative default.--kv-cache-dtype fp8— FP8 KV cache, halving attention memory and fitting longer contexts in VRAM.--mamba-ssm-cache-dtype float32— the Mamba-2 recurrent state in fp32 (required for the hybrid layers).--reasoning-parser nemotron_v3— loads the parser that splits reasoning traces into thereasoningfield of the response.--enable-auto-tool-choice --tool-call-parser qwen3_coder— enables function-calling in the OpenAI-compatible API.--max-num-seqs 32— allows up to 32 concurrent sequences for batching.--enable-chunked-prefill— processes long prompts in chunks for smoother scheduling.
Launch it
nohup /opt/nemotron-serve/launch_docker.sh > /opt/nemotron-serve/docker.log 2>&1 &Startup takes about 2 minutes: ~30 s to load 128 GB of weights across two GPUs, then ~90 s for torch.compile and CUDA graph capture. The server is ready when the log shows:
INFO ... Application startup complete.Follow along with:
tail -f /opt/nemotron-serve/docker.logThe Two Pitfalls (and Their Fixes)
I hit both of these during bring-up. The launch script above already avoids them, but they are worth understanding.
Pitfall #1 — --swap-space not recognized
Symptom: vllm: error: unrecognized arguments: --swap-space 0.
Cause: The model’s README (written for a newer vLLM) includes --swap-space 0, but vLLM 0.20.0’s argument parser does not accept this flag.
Fix: Simply remove --swap-space 0 from the launch script.
Pitfall #2 — Model path resolution inside the container
Symptom: HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/opt/models/...' or LocalEntryNotFoundError: Cannot find an appropriate cached snapshot folder.
Cause: There are two traps here. First, if you pass the host path (/opt/models/...), it does not exist inside the container (the weights are mounted at /root/.cache/huggingface/...). Second, if you pass the repo ID (nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8) with HF_HUB_OFFLINE=1, vLLM’s get_model_path() tries to look up a specific revision and fails because offline mode prevents the lookup.
Fix: Pass the path as it appears inside the container — i.e., under the HF_HOME mount point (/root/.cache/huggingface/models--nvidia--.../snapshots/HASH), not the host path. The launch script above does exactly this.
Step 8 — Verify Everything Works
Load the API key into a variable:
API_KEY=$(grep -oP 'nm-[0-9a-f]+' /opt/nemotron-serve/.api_key)List models — confirms the 1M context
curl -s http://localhost:20000/v1/models \
-H "Authorization: Bearer $API_KEY" | python3 -m json.toolThe response includes "max_model_len": 1048576 — your 1M context is active.
Thinking / reasoning mode
Pass "reasoning": {"effort": "auto"} in the request body to activate the reasoning parser. The model’s thinking trace appears in the reasoning field, and the final answer in content:
curl -s http://localhost:20000/v1/chat/completions \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "nvidia/nemotron-3-super",
"messages": [{"role":"user","content":"A bat and ball cost $1.10 total. The bat costs $1.00 more than the ball. How much is the ball?"}],
"max_tokens": 500,
"temperature": 0.6,
"reasoning": {"effort": "auto"}
}'You can also toggle thinking per-request via the chat template:
"chat_template_kwargs": {"enable_thinking": true}Tool calling
curl -s http://localhost:20000/v1/chat/completions \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "nvidia/nemotron-3-super",
"messages": [{"role":"user","content":"What is the weather in Tokyo?"}],
"max_tokens": 400,
"tools": [{"type":"function","function":{
"name":"get_weather",
"description":"Get current weather for a city",
"parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}
}}]
}'The model returns "finish_reason": "tool_calls" with a structured call to get_weather({"city": "Tokyo"}).
Multi-turn conversations
Simply pass the full message history — the model retains context across turns:
curl -s http://localhost:20000/v1/chat/completions \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "nvidia/nemotron-3-super",
"messages": [
{"role":"user","content":"My name is Alice. Remember it."},
{"role":"assistant","content":"Nice to meet you, Alice!"},
{"role":"user","content":"What is my name?"}
],
"max_tokens": 200
}'API key enforcement
# Wrong key ? HTTP 401
curl -o /dev/null -w "%{http_code}" http://localhost:20000/v1/models \
-H "Authorization: Bearer wrong-key"
# No key ? HTTP 401
curl -o /dev/null -w "%{http_code}" http://localhost:20000/v1/models
# Correct key ? HTTP 200
curl -o /dev/null -w "%{http_code}" http://localhost:20000/v1/models \
-H "Authorization: Bearer $API_KEY"Benchmarks
Measured against the live server with single requests (unless noted), using the OpenAI-compatible API:
| Test | Prompt | Output | Wall time | Throughput |
|---|---|---|---|---|
| Short generation | 20 tok | ~50 tok | 0.35 s | 144 tok/s |
| Medium generation | 26 tok | ~300 tok | 1.87 s | 160 tok/s |
| Long generation | 37 tok | ~800 tok | 4.91 s | 163 tok/s |
| Large prefill | 8,827 tok | 132 tok | 8.16 s | ~4,507 tok/s (prefill+decode) |
| Concurrent (8 parallel) | 200 tok | 1,600 tok | 5.69 s | 281 tok/s aggregate |
~160 tokens/second steady-state decode from a 120B-parameter model — and remarkably flat across output lengths (144 ? 163), confirming the FP8 KV cache and Mamba SSM state are healthy. This is roughly 1.7× faster than the ~94 tok/s achieved on a single Blackwell RTX PRO 6000, thanks to the H200’s higher memory bandwidth plus tensor parallelism across two GPUs.
Under 8-way concurrency, aggregate output throughput reaches 281 tok/s, demonstrating that the batching engine (--max-num-seqs 32) scales well. For context: a full 800-token coding response generates in under 5 seconds.
Using the Server
The server speaks the standard OpenAI API. Point any OpenAI-compatible client at it:
from openai import OpenAI
client = OpenAI(
base_url="http://YOUR-SERVER-IP:20000/v1",
api_key="nm-YOUR-API-KEY",
)
resp = client.chat.completions.create(
model="nvidia/nemotron-3-super",
messages=[{"role": "user", "content": "Design a REST API for a todo app."}],
max_tokens=2000,
temperature=0.6,
extra_body={"reasoning": {"effort": "auto"}},
)
print(resp.choices[0].message.reasoning) # thinking trace
print(resp.choices[0].message.content) # final answerTool calling works the same way — pass tools=[...] and the model emits structured function calls. Multi-turn conversations work as expected (pass the full message history).
Operations Cheatsheet
# check container status
docker ps --filter name=nemotron-serve
# tail logs
docker logs -f nemotron-serve
# restart (picks up the launch script)
docker restart nemotron-serve
# GPU usage
nvidia-smi
# rotate the API key
API_KEY="nm-$(openssl rand -hex 24)"
echo "API_KEY=$API_KEY" > /opt/nemotron-serve/.api_key
docker restart nemotron-serveGPU Memory Breakdown
With the 128 GB FP8 checkpoint split across two H200s at 90% memory utilization:
GPU 0: 129,757 MiB / 143,771 MiB used
GPU 1: 129,757 MiB / 143,771 MiB usedThat is ~65 GB of weights per GPU plus a large FP8 KV cache and Mamba SSM state — leaving healthy headroom for the 1M-token context window.
Conclusion
The era of running a 120-billion-parameter reasoning model on affordable datacenter GPUs is firmly here. With two H200s and vLLM, you get a 1M-context reasoning model at ~160 tok/s, with native tool calling and thinking mode, behind a standard OpenAI-compatible API secured with an API key.
The key insight — and the reason this article exists alongside the Blackwell guide — is matching the quantization variant to your architecture: NVFP4 for Blackwell, FP8 for Hopper. Get that one decision right, and the rest of the deployment is straightforward infrastructure work: Docker GPU passthrough, the correct model path inside the container, and the right cache/memory flags. Once those are solved, the server is stable, fast, and ready for production traffic.
Model: NVIDIA-Nemotron-3-Super-120B-A12B-FP8, governed by the NVIDIA Nemotron Open Model License. Serving: vLLM 0.20.0 on 2× NVIDIA H200. All benchmarks are real, measured live on the described hardware.