How to setup Nemotron 3 Super 120B + vLLM as local LLM on Nvidia RTX6000 96GB VRAM

Etc

Introduction

Running a 120-billion-parameter reasoning model on a single workstation GPU used to be impossible. In 2026, thanks to NVIDIA’s NVFP4 quantization and the Blackwell architecture’s native FP4 tensor cores, it is now a practical afternoon project. This article walks through the complete, end-to-end deployment of nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 on a single RTX PRO 6000 Blackwell (96 GB) — every command, every pitfall, and the real benchmarks at the end.

We will cover: connecting to the server, choosing the right model artifact, installing the serving stack, fixing the four problems you will hit on Blackwell, and measuring throughput.

The Target Hardware

Component Specification
GPU NVIDIA RTX PRO 6000 Blackwell Workstation Edition
VRAM 96 GB (97,887 MiB)
Compute Capability sm_120 (Blackwell)
Driver 580.95.05 / CUDA 13.0
System RAM 147 GB
Storage 485 GB NVMe
OS Ubuntu 22.04

The Model: Nemotron-3-Super-120B-A12B

This is the star of the show. Nemotron-3-Super-120B-A12B is a hybrid Latent Mixture-of-Experts model:

  • 120 billion total parameters, but only 12 billion active per token (22 experts routed out of 512 per layer). This is why it runs fast despite its size.
  • Architecture: interleaved Mamba-2 state-space layers, MoE layers, and select attention layers. The Mamba layers have O(1) recurrent state — they do not grow a KV cache — which is exactly how the model supports a 1,048,576-token (1M) context on a single GPU.
  • NVFP4 quantization: experts are stored in 4-bit floating point (group size 16); attention and shared layers remain FP8. This shrinks the weights to ~80 GB, fitting comfortably in 96 GB VRAM with room for a large KV cache.
  • Capabilities: configurable reasoning/thinking mode (via the chat template’s enable_thinking flag), native tool calling, and a 1M-token context window.

On Hugging Face it lives at nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4. It is public and ungated — no token or license-acceptance gate is required to download it.

Important naming note: There is no model literally called “Nemotron 3 Super NVFP4.” The official repository id is nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4. An older, unrelated family member (nvidia/Llama-3.3-Nemotron-Super-49B-v1) is a DeciLM/NAS model — make sure you grab the right one.

Step 1 — Create a Python Environment

We do not need a full CUDA toolkit on the host; vLLM ships with everything inside its Docker image. We do need a small venv on the host for downloading the model and running benchmark scripts.

python3 -m venv /opt/nemotron-venv
/opt/nemotron-venv/bin/pip install --upgrade pip wheel setuptools
/opt/nemotron-venv/bin/pip install "huggingface_hub[cli]" hf_transfer

Step 2 — Download the Model (80 GB)

export HF_HUB_CACHE=/opt/models
mkdir -p /opt/models

/opt/nemotron-venv/bin/python - <<'PY'
from huggingface_hub import snapshot_download
snapshot_download(
    "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4",
    cache_dir="/opt/models",
    max_workers=16,
)
PY

This pulls 17 safetensors shards (~80 GB) plus the chat template and a custom reasoning parser (super_v3_reasoning_parser.py). Copy the parser somewhere stable — we will mount it into the container:

MODEL_PATH=$(find /opt/models -type d -path "*Nemotron-3-Super*/snapshots/*" | head -1)
cp "$MODEL_PATH/super_v3_reasoning_parser.py" /opt/super_v3_reasoning_parser.py

Step 3 — Generate an API Key

API_KEY="nm-$(openssl rand -hex 24)"
echo "$API_KEY"
mkdir -p /opt/nemotron-serve
echo "API_KEY=$API_KEY" > /opt/nemotron-serve/.api_key
chmod 600 /opt/nemotron-serve/.api_key

Keep this key secret — every request to the server must carry it as a Bearer token.

Step 4 — Fix Docker GPU Passthrough (Pitfall #1)

The host had Docker installed but no nvidia-container-toolkit, so --gpus all failed with could not select device driver "" with capabilities: [[gpu]]. The NVIDIA apt repository was already configured, so the fix was a one-liner:

apt-get install -y nvidia-container-toolkit
nvidia-ctk runtime configure --runtime=docker
systemctl restart docker

Verify the GPU is visible inside a container:

docker run --rm --runtime=nvidia --gpus all ubuntu:22.04 \
  nvidia-smi --query-gpu=name,memory.total --format=csv,noheader

# NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 97887 MiB

Step 5 — Pull the vLLM Image

The model README pins vLLM 0.20.0 with a purpose-built Blackwell image. Do not be tempted by a newer pip install vllm — the pip path hit two more problems (see the pitfalls below). Use the official image, which bundles a matched CUDA 13.x toolkit:

docker pull vllm/vllm-openai:v0.20.0

Step 6 — Launch the Server

Create the launch script. The flags come straight from the model’s official README, with two critical additions discovered during bring-up (explained in the pitfalls section).

cat > /opt/nemotron-serve/launch_docker.sh <<'EOF'
#!/bin/bash
set -euo pipefail

API_KEY="$(grep -oP 'nm-[0-9a-f]+' /opt/nemotron-serve/.api_key)"
MODEL_PATH="/root/.cache/huggingface/models--nvidia--NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4/snapshots/4f0cf9daaeb7a4d5e23f80a00e7ed15f0e03caf6"

docker rm -f nemotron-serve 2>/dev/null || true

exec docker run --name nemotron-serve \
  --runtime nvidia --gpus all \
  --network host \
  --shm-size 16g \
  --restart unless-stopped \
  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
  -e HF_HUB_OFFLINE=1 \
  -e HF_HOME=/root/.cache/huggingface \
  -e HF_MODULES_CACHE=/root/.cache/hf_modules \
  -v /opt/models:/root/.cache/huggingface:ro \
  -v /opt/hf-cache/modules:/root/.cache/hf_modules:rw \
  -v /opt/super_v3_reasoning_parser.py:/app/super_v3_reasoning_parser.py:ro \
  vllm/vllm-openai:v0.20.0 \
    --model "$MODEL_PATH" \
    --served-model-name nvidia/nemotron-3-super \
    --host 0.0.0.0 \
    --port 20000 \
    --api-key "$API_KEY" \
    --async-scheduling \
    --dtype auto \
    --kv-cache-dtype fp8 \
    --tensor-parallel-size 1 \
    --trust-remote-code \
    --gpu-memory-utilization 0.90 \
    --enable-chunked-prefill \
    --max-model-len 1048576 \
    --max-num-seqs 32 \
    --mamba-ssm-cache-dtype float16 \
    --reasoning-parser-plugin /app/super_v3_reasoning_parser.py \
    --reasoning-parser super_v3 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder
EOF
chmod +x /opt/nemotron-serve/launch_docker.sh

# writable HF modules dir for trust-remote-code
mkdir -p /opt/hf-cache/modules

# launch
nohup /opt/nemotron-serve/launch_docker.sh > /opt/nemotron-serve/docker.log 2>&1 &

What each key flag does

  • --max-model-len 1048576 + VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 — unlocks the full 1M-token context (the model’s max_position_embeddings is 262144, but the Mamba/attention hybrid legitimately extends to 1M; the env var lets vLLM exceed the conservative default).
  • --kv-cache-dtype fp8 — FP8 KV cache, halving attention memory and fitting longer contexts in VRAM.
  • --mamba-ssm-cache-dtype float16 — the Mamba-2 recurrent state in fp16 (required for the hybrid layers).
  • --reasoning-parser super_v3 + --reasoning-parser-plugin — loads the custom parser that splits reasoning traces into the reasoning_content field.
  • --enable-auto-tool-choice --tool-call-parser qwen3_coder — enables function-calling in the OpenAI-compatible API.
  • --max-num-seqs 32 — see Pitfall #4.

Startup takes about 90 seconds: ~25 s to load 80 GB of weights, then ~60 s for torch.compile and CUDA graph capture. The server is ready when the log shows:

INFO ... Application startup complete.

Step 7 — Verify It Works

API_KEY=$(grep -oP 'nm-[0-9a-f]+' /opt/nemotron-serve/.api_key)

# list models — confirms the 1M context
curl -s http://localhost:20000/v1/models \
  -H "Authorization: Bearer $API_KEY" | python3 -m json.tool

# chat
curl -s http://localhost:20000/v1/chat/completions \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nvidia/nemotron-3-super",
    "messages": [{"role":"user","content":"What is 17 * 23?"}],
    "max_tokens": 500,
    "temperature": 1.0,
    "top_p": 0.95
  }'

The Four Pitfalls (and Their Fixes)

I hit all four of these. If you follow the script above you will avoid them — but they are worth understanding.

Pitfall #1 — Docker GPU passthrough missing

Symptom: could not select device driver "" with capabilities: [[gpu]].

Cause: nvidia-container-toolkit not installed.

Fix: apt-get install -y nvidia-container-toolkit && nvidia-ctk runtime configure --runtime=docker && systemctl restart docker.

Pitfall #2 — Read-only filesystem for trust-remote-code

Symptom: OSError: [Errno 30] Read-only file system: '/root/.cache/huggingface/modules'.

Cause: The model uses trust-remote-code, which writes its custom nemotron_h Python module into a cache directory. I had mounted the HF cache as :ro.

Fix: Keep the weights mount read-only, but add a separate writable volume for HF_MODULES_CACHE: -e HF_MODULES_CACHE=/root/.cache/hf_modules -v /opt/hf-cache/modules:/root/.cache/hf_modules:rw.

Pitfall #3 — max_num_seqs exceeds available Mamba blocks

Symptom: ValueError: max_num_seqs (1024) exceeds available Mamba cache blocks (685). Each decode sequence requires one Mamba cache block, so CUDA graph capture cannot proceed.

Cause: The hybrid Mamba-2 layers allocate one fixed-size state block per concurrent sequence. vLLM’s default max_num_seqs of 1024 was too high for the VRAM left after the 69 GB of weights.

Fix: Add --max-num-seqs 32 (the model README suggests 4 for DGX Spark, but 32 fits comfortably in the 685 available blocks and gives better batch throughput).

Pitfall #4 — Why not just pip install vllm?

I tried a native pip install vllm (which pulled v0.26.0 with torch 2.11 / CUDA 13). It hit two Blackwell-specific JIT problems that the Docker image avoids entirely:

  1. Missing nvcc at runtime: vLLM’s CUTLASS NVFP4 kernels need to JIT-compile. The host had no CUDA toolkit, only the driver. Fixable by pointing CUDA_HOME at the pip-installed nvidia/cu13 package — but that led to problem #2.
  2. CUDA compiler / header mismatch: FlashInfer’s sm_120 FP4/NVFP4 GEMM JIT (gen_gemm_sm120) failed with #error "CUDA compiler and CUDA toolkit headers are incompatible". The pip nvidia-cuda-nvcc (13.3) disagreed with the bundled CCCL/libcudacxx headers.

The purpose-built vllm/vllm-openai:v0.20.0 image contains a coherently-matched CUDA 13.x toolchain, so all the NVFP4 kernels compile and load cleanly on the first try. Use the Docker image.

Benchmarks

Measured against the live server with a single concurrent request:

Test Prompt Output Wall time Throughput
Short generation 25 tok 60 tok 0.66 s 90.3 tok/s
Medium generation 28 tok 300 tok 3.20 s 93.7 tok/s
Long generation 38 tok 800 tok 8.51 s 94.0 tok/s
Large prefill 2,427 tok 120 tok 1.43 s ~1,780 tok/s (prefill+decode)

~94 tokens/second steady-state decode from a 120B-parameter model on a single workstation GPU. That is remarkable, and it comes from three things working together: NVFP4 weights (4× compression with minimal accuracy loss), Blackwell’s native FP4 tensor cores, and the 12B-active Mamba/MoE hybrid architecture that keeps the per-token FLOPs low. Throughput is flat across output lengths, confirming the KV and Mamba caches are healthy.

Using the Server

The server speaks the standard OpenAI API. Point any OpenAI-compatible client at it:

from openai import OpenAI

client = OpenAI(
    base_url="http://175.28.230.22:20000/v1",
    api_key="nm-<your-key>",
)

resp = client.chat.completions.create(
    model="nvidia/nemotron-3-super",
    messages=[{"role": "user", "content": "Design a REST API for a todo app."}],
    max_tokens=2000,
    temperature=1.0,
    top_p=0.95,
)
print(resp.choices[0].message.content)

Tool calling works the same way — pass tools=[...] and the model emits structured function calls. Multi-turn conversations work as expected (pass the full message history).

Thinking mode

The model’s chat template defaults to enable_thinking=True. When thinking is on, the model first produces a reasoning trace inside <think>...</think> tags, then the final answer. The super_v3 reasoning parser splits these into the response’s reasoning_content and content fields. To toggle it per-request:

resp = client.chat.completions.create(
    model="nvidia/nemotron-3-super",
    messages=[{"role": "user", "content": "Prove the Pythagorean theorem."}],
    extra_body={"chat_template_kwargs": {"enable_thinking": True}},
    ...
)

Operations Cheatsheet

# check container status
docker ps --filter name=nemotron-serve

# tail logs
docker logs -f nemotron-serve

# restart (picks up the launch script)
docker restart nemotron-serve

# GPU usage
nvidia-smi

# rotate the API key
API_KEY="nm-$(openssl rand -hex 24)"
echo "API_KEY=$API_KEY" > /opt/nemotron-serve/.api_key
docker restart nemotron-serve

Conclusion

The era of “a 120-billion-parameter model on one GPU” has arrived. The Nemotron-3-Super-120B-A12B design — LatentMoE with Mamba-2 hybrid layers, trained natively in NVFP4 — is purpose-built for exactly this hardware moment. With an RTX PRO 6000 Blackwell you get a 1M-context reasoning model at 94 tok/s, behind a standard OpenAI-compatible API, for the cost of the electricity to run one card.

The tricky parts are all infrastructure: getting Docker GPU passthrough working, matching the CUDA toolchain for the NVFP4 JIT kernels, and sizing the Mamba/sequence caches correctly. Once those are solved (and the launch script above solves them), the deployment is stable and fast.

Model: NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4, governed by the NVIDIA Nemotron Open Model License. Serving: vLLM 0.20.0. All benchmarks are real, measured live on the described hardware.

Leave a Reply

Your email address will not be published. Required fields are marked *