How to run Nemotron 3 Super 120B FP8 on vLLM on 2 x Nvidia H200 143GB VRAM

Running a 120-billion-parameter reasoning model across two GPUs is no longer an exotic exercise — in 2026, with NVIDIA’s FP8 quantization and the Hopper architecture’s native FP8 tensor cores, it is an afternoon project that delivers genuinely production-grade speed. This article walks through the complete, end-to-end deployment of Nemotron-3-Super-120B-A12B on a server with 2× NVIDIA H200 (143 GB each), served by vLLM behind a standard OpenAI-compatible API, with a 1-million-token context window, thinking/reasoning mode, tool calling, multi-turn conversations, and API-key authentication.

There is already an excellent guide for running this model on a single Blackwell RTX PRO 6000 using the NVFP4 (4-bit floating point) checkpoint. This article is the companion piece for the Hopper / H200 case — and as you’ll see, the model variant and several flags must change, because Hopper and Blackwell have different native low-precision capabilities. They are a different hardware, so the installation process turned out to be quite different.

The Target Hardware

Component	Specification
GPU	2× NVIDIA H200
VRAM (per GPU)	143 GB (143,771 MiB)
Total VRAM	~287 GB
Compute Capability	sm_90 (Hopper)
Driver	580.105.08 / CUDA 13.0
System RAM	440 GB
Storage	1.5 TB NVMe
OS	Ubuntu 24.04.3 LTS

The Critical Decision: FP8, Not NVFP4

This is the single most important thing to get right, and it is where a blind copy of the Blackwell guide will bite you.

Nemotron-3-Super-120B-A12B ships in three quantization variants on Hugging Face:

NVFP4 — 4-bit floating point weights (~80 GB). Requires Blackwell architecture (B200/B300, RTX PRO 6000 Blackwell) with native FP4 tensor cores. On Hopper, FP4 has no native tensor-core support — it would be emulated and slow.
FP8 — 8-bit floating point weights (~128 GB). This is the native sweet spot for Hopper (H100/H200). The H200’s FP8 tensor cores run this at full speed.
BF16 — 16-bit, ~240 GB. Overkill and a waste of VRAM unless you need the absolute maximum accuracy.

For 2× H200, the correct choice is nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8. At ~128 GB it fits comfortably across both GPUs with tensor parallelism, leaving roughly 160 GB of headroom for a large FP8 KV cache — which is exactly what lets you run the full 1M-token context.

Rule of thumb: Blackwell (sm_120+) ? NVFP4. Hopper (sm_90) ? FP8. Do not mix these up — the performance difference is dramatic.

The Model: Nemotron-3-Super-120B-A12B

A quick refresher on why this model is special and how it achieves its speed:

120 billion total parameters, only 12 billion active per token — a hybrid Latent Mixture-of-Experts design (22 experts routed out of 512 per layer). This is why it is fast despite its size.
Mamba-2 hybrid layers interleaved with MoE and attention layers. The Mamba layers have O(1) recurrent state (no growing KV cache), which is precisely how the model supports a 1,048,576-token (1M) context.
Configurable reasoning/thinking mode via the chat template’s enable_thinking flag — the model produces a reasoning trace, then a final answer.
Native tool calling (function calling) and multi-turn conversation support.
Multi-Token Prediction (MTP) layers for faster speculative generation.

On Hugging Face it lives at nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8. It is public and ungated — no token or license acceptance required to download.

Step 1 — Connect to the Server

ssh root@YOUR-SERVER-IP

Verify the GPUs are visible and idle:

nvidia-smi

You should see two H200s with ~143 GB each and 0 MiB used.

Step 2 — Install Docker and GPU Passthrough

Unlike the Blackwell guide (where Docker was pre-installed), our H200 server had no Docker at all — only the nvidia-container-toolkit package. Install Docker first:

apt-get update
apt-get install -y docker.io
systemctl enable docker
systemctl start docker

Then configure the NVIDIA container runtime so Docker can pass GPUs through to containers:

nvidia-ctk runtime configure --runtime=docker
systemctl restart docker

Verify both GPUs are visible inside a container:

docker run --rm --runtime=nvidia --gpus all ubuntu:22.04 \
  nvidia-smi --query-gpu=name,memory.total --format=csv,noheader

You should see:

NVIDIA H200, 143771 MiB
NVIDIA H200, 143771 MiB

Step 3 — Create a Python Environment

We need a small venv on the host for downloading the model. On Ubuntu 24.04, the python3-venv package must be installed explicitly:

apt-get install -y python3.12-venv

python3 -m venv /opt/nemotron-venv
/opt/nemotron-venv/bin/pip install --upgrade pip wheel setuptools
/opt/nemotron-venv/bin/pip install "huggingface_hub[cli]" hf_transfer

Step 4 — Download the Model (128 GB)

mkdir -p /opt/models

HF_HUB_CACHE=/opt/models /opt/nemotron-venv/bin/python - <<'PY'
from huggingface_hub import snapshot_download
snapshot_download(
    "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8",
    cache_dir="/opt/models",
    max_workers=16,
)
PY

This pulls 26 safetensors shards (~128 GB) plus the chat template, custom modeling code (modeling_nemotron_h.py), and a reasoning parser (super_v3_reasoning_parser.py). With 16 parallel workers on a fast connection, expect roughly 10–15 minutes.

Note the snapshot path — you’ll need it later:

find /opt/models -type d -path "*Nemotron-3-Super*FP8*/snapshots/*"

Step 5 — Generate an API Key

Create a random API key that clients must present as a Bearer token:

API_KEY="nm-$(openssl rand -hex 24)"
echo "$API_KEY"

mkdir -p /opt/nemotron-serve
echo "API_KEY=$API_KEY" > /opt/nemotron-serve/.api_key
chmod 600 /opt/nemotron-serve/.api_key

Keep this key secret — every request to the server must carry it.

Step 6 — Pull the vLLM Image

The model’s README pins vLLM 0.20.0. Use the official Docker image — it bundles a matched CUDA 13.x toolchain so all the FP8 kernels compile and load cleanly. Do not be tempted by pip install vllm; the pip path hits JIT compiler/header mismatches on Hopper that the Docker image avoids entirely.

docker pull vllm/vllm-openai:v0.20.0

Step 7 — The Launch Script

This is the heart of the deployment. Create the launch script:

cat > /opt/nemotron-serve/launch_docker.sh <<'EOF'
#!/bin/bash
set -euo pipefail

API_KEY="$(grep -oP 'nm-[0-9a-f]+' /opt/nemotron-serve/.api_key)"
# Path as it appears INSIDE the container (the HF_HOME mount point)
MODEL_PATH="/root/.cache/huggingface/models--nvidia--NVIDIA-Nemotron-3-Super-120B-A12B-FP8/snapshots/SNAPSHOT_HASH"

echo "Using model path: $MODEL_PATH"
echo "Using API key:    $API_KEY"

docker rm -f nemotron-serve 2>/dev/null || true

exec docker run --name nemotron-serve \
  --runtime nvidia --gpus all \
  --network host \
  --shm-size 16g \
  --restart unless-stopped \
  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
  -e HF_HUB_OFFLINE=1 \
  -e HF_HOME=/root/.cache/huggingface \
  -e HF_MODULES_CACHE=/root/.cache/hf_modules \
  -v /opt/models:/root/.cache/huggingface:ro \
  -v /opt/hf-cache/modules:/root/.cache/hf_modules:rw \
  vllm/vllm-openai:v0.20.0 \
    --model "$MODEL_PATH" \
    --served-model-name nvidia/nemotron-3-super \
    --host 0.0.0.0 \
    --port 20000 \
    --api-key "$API_KEY" \
    --async-scheduling \
    --dtype auto \
    --kv-cache-dtype fp8 \
    --tensor-parallel-size 2 \
    --enable-expert-parallel \
    --trust-remote-code \
    --gpu-memory-utilization 0.90 \
    --enable-chunked-prefill \
    --max-model-len 1048576 \
    --max-num-seqs 32 \
    --mamba-ssm-cache-dtype float32 \
    --reasoning-parser nemotron_v3 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder
EOF

chmod +x /opt/nemotron-serve/launch_docker.sh
mkdir -p /opt/hf-cache/modules

Replace SNAPSHOT_HASH with the actual hash from Step 4.

What each key flag does

--tensor-parallel-size 2 — splits the model across both H200s. Essential for a 128 GB checkpoint.
--enable-expert-parallel — distributes the MoE experts across GPUs (NVIDIA’s recommended setting for multi-GPU Hopper).
--max-model-len 1048576 + VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 — unlocks the full 1M-token context. The model’s max_position_embeddings is 262144, but the Mamba/attention hybrid legitimately extends to 1M; the env var lets vLLM exceed the conservative default.
--kv-cache-dtype fp8 — FP8 KV cache, halving attention memory and fitting longer contexts in VRAM.
--mamba-ssm-cache-dtype float32 — the Mamba-2 recurrent state in fp32 (required for the hybrid layers).
--reasoning-parser nemotron_v3 — loads the parser that splits reasoning traces into the reasoning field of the response.
--enable-auto-tool-choice --tool-call-parser qwen3_coder — enables function-calling in the OpenAI-compatible API.
--max-num-seqs 32 — allows up to 32 concurrent sequences for batching.
--enable-chunked-prefill — processes long prompts in chunks for smoother scheduling.

Launch it

nohup /opt/nemotron-serve/launch_docker.sh > /opt/nemotron-serve/docker.log 2>&1 &

Startup takes about 2 minutes: ~30 s to load 128 GB of weights across two GPUs, then ~90 s for torch.compile and CUDA graph capture. The server is ready when the log shows:

INFO ... Application startup complete.

Follow along with:

tail -f /opt/nemotron-serve/docker.log

The Two Pitfalls (and Their Fixes)

I hit both of these during bring-up. The launch script above already avoids them, but they are worth understanding.

Pitfall #1 — `--swap-space` not recognized

Symptom: vllm: error: unrecognized arguments: --swap-space 0.

Cause: The model’s README (written for a newer vLLM) includes --swap-space 0, but vLLM 0.20.0’s argument parser does not accept this flag.

Fix: Simply remove --swap-space 0 from the launch script.

Pitfall #2 — Model path resolution inside the container

Symptom: HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/opt/models/...' or LocalEntryNotFoundError: Cannot find an appropriate cached snapshot folder.

Cause: There are two traps here. First, if you pass the host path (/opt/models/...), it does not exist inside the container (the weights are mounted at /root/.cache/huggingface/...). Second, if you pass the repo ID (nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8) with HF_HUB_OFFLINE=1, vLLM’s get_model_path() tries to look up a specific revision and fails because offline mode prevents the lookup.

Fix: Pass the path as it appears inside the container — i.e., under the HF_HOME mount point (/root/.cache/huggingface/models--nvidia--.../snapshots/HASH), not the host path. The launch script above does exactly this.

Step 8 — Verify Everything Works

Load the API key into a variable:

API_KEY=$(grep -oP 'nm-[0-9a-f]+' /opt/nemotron-serve/.api_key)

List models — confirms the 1M context

curl -s http://localhost:20000/v1/models \
  -H "Authorization: Bearer $API_KEY" | python3 -m json.tool

The response includes "max_model_len": 1048576 — your 1M context is active.

Thinking / reasoning mode

Pass "reasoning": {"effort": "auto"} in the request body to activate the reasoning parser. The model’s thinking trace appears in the reasoning field, and the final answer in content:

curl -s http://localhost:20000/v1/chat/completions \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nvidia/nemotron-3-super",
    "messages": [{"role":"user","content":"A bat and ball cost $1.10 total. The bat costs $1.00 more than the ball. How much is the ball?"}],
    "max_tokens": 500,
    "temperature": 0.6,
    "reasoning": {"effort": "auto"}
  }'

You can also toggle thinking per-request via the chat template:

"chat_template_kwargs": {"enable_thinking": true}

Tool calling

curl -s http://localhost:20000/v1/chat/completions \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nvidia/nemotron-3-super",
    "messages": [{"role":"user","content":"What is the weather in Tokyo?"}],
    "max_tokens": 400,
    "tools": [{"type":"function","function":{
      "name":"get_weather",
      "description":"Get current weather for a city",
      "parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}
    }}]
  }'

The model returns "finish_reason": "tool_calls" with a structured call to get_weather({"city": "Tokyo"}).

Multi-turn conversations

Simply pass the full message history — the model retains context across turns:

curl -s http://localhost:20000/v1/chat/completions \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nvidia/nemotron-3-super",
    "messages": [
      {"role":"user","content":"My name is Alice. Remember it."},
      {"role":"assistant","content":"Nice to meet you, Alice!"},
      {"role":"user","content":"What is my name?"}
    ],
    "max_tokens": 200
  }'

API key enforcement

# Wrong key ? HTTP 401
curl -o /dev/null -w "%{http_code}" http://localhost:20000/v1/models \
  -H "Authorization: Bearer wrong-key"

# No key ? HTTP 401
curl -o /dev/null -w "%{http_code}" http://localhost:20000/v1/models

# Correct key ? HTTP 200
curl -o /dev/null -w "%{http_code}" http://localhost:20000/v1/models \
  -H "Authorization: Bearer $API_KEY"

Benchmarks

Measured against the live server with single requests (unless noted), using the OpenAI-compatible API:

Test	Prompt	Output	Wall time	Throughput
Short generation	20 tok	~50 tok	0.35 s	144 tok/s
Medium generation	26 tok	~300 tok	1.87 s	160 tok/s
Long generation	37 tok	~800 tok	4.91 s	163 tok/s
Large prefill	8,827 tok	132 tok	8.16 s	~4,507 tok/s (prefill+decode)
Concurrent (8 parallel)	200 tok	1,600 tok	5.69 s	281 tok/s aggregate

~160 tokens/second steady-state decode from a 120B-parameter model — and remarkably flat across output lengths (144 ? 163), confirming the FP8 KV cache and Mamba SSM state are healthy. This is roughly 1.7× faster than the ~94 tok/s achieved on a single Blackwell RTX PRO 6000, thanks to the H200’s higher memory bandwidth plus tensor parallelism across two GPUs.

Under 8-way concurrency, aggregate output throughput reaches 281 tok/s, demonstrating that the batching engine (--max-num-seqs 32) scales well. For context: a full 800-token coding response generates in under 5 seconds.

Using the Server

The server speaks the standard OpenAI API. Point any OpenAI-compatible client at it:

from openai import OpenAI

client = OpenAI(
    base_url="http://YOUR-SERVER-IP:20000/v1",
    api_key="nm-YOUR-API-KEY",
)

resp = client.chat.completions.create(
    model="nvidia/nemotron-3-super",
    messages=[{"role": "user", "content": "Design a REST API for a todo app."}],
    max_tokens=2000,
    temperature=0.6,
    extra_body={"reasoning": {"effort": "auto"}},
)
print(resp.choices[0].message.reasoning)  # thinking trace
print(resp.choices[0].message.content)     # final answer

Tool calling works the same way — pass tools=[...] and the model emits structured function calls. Multi-turn conversations work as expected (pass the full message history).

Operations Cheatsheet

# check container status
docker ps --filter name=nemotron-serve

# tail logs
docker logs -f nemotron-serve

# restart (picks up the launch script)
docker restart nemotron-serve

# GPU usage
nvidia-smi

# rotate the API key
API_KEY="nm-$(openssl rand -hex 24)"
echo "API_KEY=$API_KEY" > /opt/nemotron-serve/.api_key
docker restart nemotron-serve

GPU Memory Breakdown

With the 128 GB FP8 checkpoint split across two H200s at 90% memory utilization:

GPU 0: 129,757 MiB / 143,771 MiB used
GPU 1: 129,757 MiB / 143,771 MiB used

That is ~65 GB of weights per GPU plus a large FP8 KV cache and Mamba SSM state — leaving healthy headroom for the 1M-token context window.

Conclusion

The era of running a 120-billion-parameter reasoning model on affordable datacenter GPUs is firmly here. With two H200s and vLLM, you get a 1M-context reasoning model at ~160 tok/s, with native tool calling and thinking mode, behind a standard OpenAI-compatible API secured with an API key.

The key insight — and the reason this article exists alongside the Blackwell guide — is matching the quantization variant to your architecture: NVFP4 for Blackwell, FP8 for Hopper. Get that one decision right, and the rest of the deployment is straightforward infrastructure work: Docker GPU passthrough, the correct model path inside the container, and the right cache/memory flags. Once those are solved, the server is stable, fast, and ready for production traffic.

Model: NVIDIA-Nemotron-3-Super-120B-A12B-FP8, governed by the NVIDIA Nemotron Open Model License. Serving: vLLM 0.20.0 on 2× NVIDIA H200. All benchmarks are real, measured live on the described hardware.

harry.sufehmi.com

life is a struggle – information wants to be free

How to run Nemotron 3 Super 120B FP8 on vLLM on 2 x Nvidia H200 143GB VRAM

The Target Hardware

The Critical Decision: FP8, Not NVFP4

The Model: Nemotron-3-Super-120B-A12B

Step 1 — Connect to the Server

Step 2 — Install Docker and GPU Passthrough

Step 3 — Create a Python Environment

Step 4 — Download the Model (128 GB)

Step 5 — Generate an API Key

Step 6 — Pull the vLLM Image

Step 7 — The Launch Script

What each key flag does

Launch it

The Two Pitfalls (and Their Fixes)

Pitfall #1 — `--swap-space` not recognized

Pitfall #2 — Model path resolution inside the container

Step 8 — Verify Everything Works

List models — confirms the 1M context

Thinking / reasoning mode

Tool calling

Multi-turn conversations

API key enforcement

Benchmarks

Using the Server

Operations Cheatsheet

GPU Memory Breakdown

Conclusion

Leave a Reply

The Target Hardware

The Critical Decision: FP8, Not NVFP4

The Model: Nemotron-3-Super-120B-A12B

Step 1 — Connect to the Server

Step 2 — Install Docker and GPU Passthrough

Step 3 — Create a Python Environment

Step 4 — Download the Model (128 GB)

Step 5 — Generate an API Key

Step 6 — Pull the vLLM Image

Step 7 — The Launch Script

What each key flag does

Launch it

The Two Pitfalls (and Their Fixes)

Pitfall #1 — --swap-space not recognized

Pitfall #2 — Model path resolution inside the container

Step 8 — Verify Everything Works

List models — confirms the 1M context

Thinking / reasoning mode

Tool calling

Multi-turn conversations

API key enforcement

Benchmarks

Using the Server

Operations Cheatsheet

GPU Memory Breakdown

Conclusion

Leave a Reply

Pitfall #1 — `--swap-space` not recognized