This article documents exactly what it took to get Nemotron-3-Nano-30B-A3B running on vLLM with 1M context, thinking mode, tool calling, and API-key auth on a 2x RTX 5090 (Blackwell, sm_120) box running Ubuntu 22.04. The model itself is a hybrid Mamba-2/MoE/Attention architecture with NVFP4 weights – one of the first production models that exercises every cutting- edge path in vLLM at once.
The article is split into:
1. The plan and feasibility math
2. The install
3. The eight things that go wrong on Blackwell + Nemotron-H + NVFP4
4. The launch script
5. Benchmarking (with real numbers)
6. Gotchas and lessons learned
1. THE PLAN
Hardware (confirmed via nvidia-smi):
2x NVIDIA GeForce RTX 5090 (32 GB each, sm_120 / Blackwell)
Driver 580.95.05, CUDA 13.0 available, 131 GB RAM, 478 GB disk free
Ubuntu 22.04.5 LTS, Python 3.10.12, no nvcc initially, port 20000 free
Model (nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4):
Architecture : NemotronHForCausalLM (hybrid Mamba-2 + MoE + 6 attn layers)
Quantization : NVFP4 (modelopt, weights ~16 GB, kv_cache_quant=fp8)
Native context : 262 144 (256k)
Hidden size : 2688
Layers : 52 (23 Mamba + 23 MoE + 6 Attention)
Attention heads : 32, KV heads = 2 (heavy GQA), head_dim = 128
MoE : 128 routed experts, 1 shared expert, top-6
Active params : ~3B (out of ~30B total)
Chat template : includes
Tool calling : Hermes format (chat_template.jinja ships in repo)
Gated : NO – public download, no HF token needed
Why 1M context actually fits on 32 GB GPUs:
Only 6 of 52 layers have a KV cache. The Mamba-2 SSM is
recurrent with O(1) state per sequence (it does NOT
grow with context length), and the MoE layers have
no attention at all.
KV @ 1M ctx (bf16, TP=2 replicated):
= 2 (K+V) * 2 (KV heads) * 128 (head dim) * 2 bytes * 6 layers
= 6.1 KB / token
= 5.95 GB total per GPU at 1M tokens
Add fp8 KV cache (NVIDIA pre-configured this in
hf_quant_config.json with kv_cache_quant_algo: FP8)
and we're at ~2.9 GB per GPU.
So 1M context is comfortable; the trick is YaRN to extend
attention positions from the native 256k out to 1M.
2. THE INSTALL
Step 1 – Set up a Python 3.10 venv at /opt/vllm-venv:
apt-get install -y python3.10-venv
python3 -m venv /opt/vllm-venv
/opt/vllm-venv/bin/pip install --upgrade pip wheel setuptools
Step 2 – Install PyTorch + CUDA 13.0 system-wide (needed for nvcc + JIT header paths that vLLM workers look up via $CUDA_HOME):
DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
cuda-nvcc-13-0 \
cuda-cudart-dev-13-0 \
cuda-cccl-13-0 \
cuda-driver-dev-13-0 \
cuda-nvrtc-dev-13-0 \
libcusparse-dev-13-0 \
libcusparselt0-dev-cuda-13 \
libcublas-dev-13-0 \
libcurand-dev-13-0 \
libcudnn9-dev-cuda-13
Reasoning: vLLM nightly is a single abi3 wheel that bundles its own CUDA-13 Python runtime, but the worker subprocesses still spawn nvcc to JIT-compile flashinfer CUTLASS kernels for the local GPU arch.
That nvcc needs the matching dev headers; otherwise you get “fatal error: cublasLt.h: No such file or directory” deep inside a ninja build.
IMPORTANT: after this install, /usr/local/cuda is re-pointed via update-alternatives to cuda-13.0 (so compute_120 is in nvcc’s –list-gpu-arch output). If you ever install a different cuda-* package later, re-check that symlink – it’s the silent foot-gun.
Step 3 – Install vLLM nightly (stable has NO Blackwell cu128 wheel):
/opt/vllm-venv/bin/pip install --extra-index-url \
https://wheels.vllm.ai/nightly vllm
This pulls vllm-0.26.0rc1 + torch 2.11.0+cu130 + flashinfer 0.6.14 + a full CUDA-13 Python stack. The venv ends up around 11 GB.
/opt/vllm-venv/bin/pip install --quiet ninja huggingface_hub
Step 4 – Download the model:
/opt/vllm-venv/bin/hf download nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 \
--local-dir /root/models/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 \
--max-workers 8
~19 GB across 18 files (5 safetensors shards + chat_template.jinja + the nano_v3_reasoning_parser.py and modeling_nemotron_h.py that vLLM auto-imports via trust_remote_code-style auto_map).
Step 5 – Patch config.json to add YaRN rope scaling.
vLLM 0.26 dropped the –rope-scaling CLI flag (you set it in the model’s config.json now, or via JSON overrides baked in at serve time).
Edit /root/models/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4/config.json and add:
"rope_parameters": {
"rope_type": "yarn",
"factor": 4.0,
"original_max_position_embeddings": 262144
}
(New transformers >= 4.50 reads “rope_parameters”; “rope_scaling” still works as legacy fallback but you get a warning. Use rope_parameters.)
Also keep a backup so you can revert if you decide to stay at native 256k context:
cp config.json config.json.bak
3. THE EIGHT THINGS THAT GO WRONG (and exactly how to fix each)
If you skip straight to “vllm serve” you will hit, in order:
(1) “unrecognized arguments: –rope-scaling / –rope-theta / –swap-space” -> vLLM 0.26 removed these CLI flags. Move them into config.json (rope_parameters) and drop –swap-space entirely.
(2) “User-specified max_model_len (1056768) is greater than the derived max_model_len (max_position_embeddings=1048576.0)” -> Set max_model_len = 1048576 exactly (= 262144 * 4), not higher. If you must exceed it, export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 (it warns about NaN risk on out-of-range rope positions).
(3) “Could not find nvcc and default cuda_home=’/usr/local/cuda’ doesn’t exist” -> Install cuda-nvcc-12-8 OR cuda-nvcc-13-0 via apt, then set PATH=/usr/local/cuda/bin:$PATH in the launch env. For Blackwell sm_120 you need nvcc 13.0 – 12.8 errors with “SM 12.x requires CUDA >= 12.9” when probing GPU archs.
(4) “FlashInfer requires GPUs with sm75 or higher” -> Deceptive error: flashinfer isn’t failing to detect the GPU, it’s failing because its TARGET_CUDA_ARCHS list is empty. Caused by nvcc –list-gpu-arch returning empty. Fixed by installing nvcc 13.0 so compute_120 shows up.
(5) “‘ninja’ not found” -> pip install ninja into the venv, and put /opt/vllm-venv/bin FIRST in PATH so the worker subprocesses inherit it.
(6) “fatal error: curand_kernel.h: No such file or directory” -> apt install libcurand-dev-13-0. flashinfer CUTLASS paths include device-side curand headers for random sampling in MoE routing.
(7) “fatal error: cublasLt.h / nvrtc.h / cusparse.h not found” -> apt install libcublas-dev-13-0, cuda-nvrtc-dev-13-0, libcusparse-dev-13-0, cuda-cccl-13-0. These are the dev-13.0 packages; -dev-13-1/-13-2/-13-3 exist too but 13-0 matches what vLLM’s bundled nvcc 13.0.88 was compiled against.
(8) “Free memory on device cuda:1 (19.67/31.36 GiB) on startup is less than desired GPU memory utilization (0.9, 28.22 GiB)” -> A previous failed vLLM run left zombie VLLM::Worker_TP0/TP1 processes holding 11 GB each. pkill -f “vllm serve” doesn’t kill them because their process name is different. Use nvidia-smi –query-compute-apps=pid,process_name then `kill -9
There’s a 9th gotcha that’s not an error but will save you 10 minutes:
(9) The model directory must contain nano_v3_reasoning_parser.py at the root. Pass –reasoning-parser-plugin
4. THE LAUNCH SCRIPT
# /root/start-nemotron.sh - launches Nemotron vLLM fully detached
set -u
VENV=/opt/vllm-venv
MODEL_DIR=/root/models/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4
API_KEY=$(cat /root/vllm-api.key)
LOG=/var/log/vllm/nemotron.log
PIDFILE=/var/run/nemotron-vllm.pid
mkdir -p /var/log/vllm /var/run
pkill -9 -f "vllm serve" 2>/dev/null || true
sleep 2
export PATH=/opt/vllm-venv/bin:/usr/local/cuda/bin:$PATH
export CUDA_HOME=/usr/local/cuda
export HF_HUB_OFFLINE=1
export TRANSFORMERS_OFFLINE=1
export NCCL_P2P_DISABLE=0
export TOKENIZERS_PARALLELISM=false
export VLLM_LOGGING_LEVEL=INFO
export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
# Generate the API key once:
# python -c "import secrets; print(secrets.token_urlsafe(32))" > /root/vllm-api.key
# chmod 600 /root/vllm-api.key
setsid bash -c "
exec $VENV/bin/vllm serve $MODEL_DIR \
--host 0.0.0.0 \
--port 20000 \
--served-model-name nemotron-nano \
--api-key $API_KEY \
--tensor-parallel-size 2 \
--max-model-len 1048576 \
--max-num-seqs 16 \
--gpu-memory-utilization 0.90 \
--enable-auto-tool-choice \
--tool-call-parser hermes \
--reasoning-parser nano_v3 \
--reasoning-parser-plugin $MODEL_DIR/nano_v3_reasoning_parser.py \
--enable-prefix-caching \
--enable-chunked-prefill \
--dtype bfloat16 \
>> $LOG 2>&1
" /dev/null 2>&1 &
echo $! > "$PIDFILE"
disown 2>/dev/null || true
echo "Launched. PID=$PID, log=$LOG"
Run it with `bash /root/start-nemotron.sh`.
First boot takes 5-10 minutes because the worker JIT-compiles flashinfer CUTLASS sm_120 kernels (a few hundred cu files, ~150 nvcc invocations).
Subsequent restarts skip the kernel cache at /root/.cache/flashinfer/0.6.14/120f/ and start in ~60s.
Watch the log for the magic line: INFO: Application startup complete.
5. BENCHMARKS
Using the OpenAI-compatible /v1/chat/completions endpoint with stream=true and stream_options.include_usage=true:
SINGLE-STREAM DECODE (512 generation tokens, varying prompt size):
prompt gen TTFT total decode_tok/s
----- ---- ---- ----- ------------
243 145 0.16s 0.77s 189.4
927 162 0.14s 0.81s 200.7
3658 200 0.41s 1.23s 162.5
14580 275 0.33s 1.43s 192.5
58270 364 0.64s 2.11s 172.7
116524 450 8.31s 10.13s 44.4 <- prompt processing dominates
LONG GENERATION (1024 output, 24-token prompt): prompt=24 gen=1024 total=4.36s decode_tok/s=235.0
CONCURRENT (8 parallel streams, 512 gen each): total=4096 tokens wall=3.24s aggregate=1263 tok/s (about 158 tok/s per stream under 8-way concurrency)
THINKING MODE (reasoning_effort=medium, "What is 17 * 24?"): gen=529 total=2.19s decode_tok/s=241.5 -> response.choices[0].message.reasoning has the chain-of-thought -> response.choices[0].message.content has the final answer
TOOL CALL (force tool_choice=function for "weather in Tokyo?"): tool_calls[0].function.name = "get_weather" tool_calls[0].function.arguments = '{"location":"Tokyo"}' -> Hermes parser emits clean OpenAI-format tool_calls
TOOL CALL WITHOUT tool_choice: Often returns the answer in plain text. The model has the ability but chooses prose. Force it with `tool_choice` or a clear system prompt like "You MUST call the get_weather function".
These numbers are sane for a 30B-A3B NVFP4 hybrid on 2x Blackwell. For comparison, the BF16 sibling would be ~2-2.5x slower on decode (weight memory bandwidth bound) and would NOT fit on 32 GB GPUs at 1M context.
6. LESSONS LEARNED
- NemotronH is a hybrid model. 23 of 52 layers are Mamba-2 (recurrent, fixed-state), 23 are MoE (no attention), only 6 are attention. So prompt-context cost is mostly MoE weight bandwidth and a small KV cache, NOT quadratic attention.
- vLLM nightly is the only path to Blackwell + NemotronH + NVFP4 right now. PyPI stable has no cu128/cu130 wheel for Blackwell, and the NemotronHForCausalLM modelopt loader landed in late-2025.
- The launch environment matters enormously. /usr/local/cuda must point to a CUDA version whose nvcc can list compute_120. PATH must include /opt/vllm-venv/bin (for ninja) and /usr/local/cuda/bin (for nvcc). CUDA_HOME must be set. All four are needed.
- The first cold start compiles a few hundred flashinfer CUTLASS cu files. Don't panic if the log is just ninja [N/56] lines for 5-10 minutes. Look for the actual fatal error (one of the eight above) and fix it; subsequent restarts are fast.
- After a failed launch, ALWAYS nvidia-smi --query-compute-apps before restarting. The VLLM::Worker_TP0/TP1 children survive pkill -f "vllm serve" because their argv[0] is different, and they hold ~11 GB of GPU memory each. Kill them by PID.
- Reasoning output goes in a separate field. response.choices[0].message has .reasoning (chain-of-thought) AND .content (final answer). If you only see .content, your reasoning parser isn't loaded correctly.
- For 1M context attention quality, expect some degradation past the native 256k. The model was trained with rope up to 262144; YaRN with factor=4 extends it but is not free. For workloads that mostly live under 256k, you can drop YaRN entirely (just remove the rope_parameters block) and you'll get full training-quality attention.
- The serve command does NOT need --trust-remote-code. vLLM 0.26 resolves NemotronHForCausalLM natively via the auto_map entry in config.json. Modeling code lives in the model dir.
- Bind to 0.0.0.0 if you want reachability from inside the same host or via SSH tunnel. If you want public reachability, open the cloud firewall (security group / cloud firewall rule / iptables) to allow inbound TCP on 20000 from your client IPs.
APPENDIX A - Quick health check
# Is it up?
curl -sS http://localhost:20000/v1/models \
-H "Authorization: Bearer $(cat /root/vllm-api.key)"
# Quick chat test
curl -sS http://localhost:20000/v1/chat/completions \
-H "Authorization: Bearer $(cat /root/vllm-api.key)" \
-H "Content-Type: application/json" \
-d '{
"model": "nemotron-nano",
"messages": [{"role":"user","content":"In one sentence, what is the capital of France?"}],
"max_tokens": 60,
"temperature": 0
}'
# GPU usage
nvidia-smi --query-gpu=index,memory.used,memory.free,utilization.gpu --format=csv
# vLLM logs
tail -f /var/log/vllm/nemotron.log
# Restart cleanly
pkill -9 -f "vllm serve"; nvidia-smi --query-compute-apps=pid \
| awk -F, 'NR>1 {print $1}' | xargs -r kill -9
bash /root/start-nemotron.sh
APPENDIX B - File layout on disk
/opt/vllm-venv/ Python 3.10 venv, 11 GB
/root/models/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4/
config.json patched with rope_parameters
config.json.bak original (no rope scaling)
chat_template.jinja reasoning + tool-call template
nano_v3_reasoning_parser.py reasoning parser plugin
modeling_nemotron_h.py remote modeling code
configuration_nemotron_h.py remote config class
model-00001-of-00005.safetensors ~4 GB each
...
/root/vllm-api.key API key, mode 0600
/root/start-nemotron.sh launch script
/var/log/vllm/nemotron.log server log
/var/run/nemotron-vllm.pid current PID
/root/.cache/flashinfer/0.6.14/120f/ JIT-compiled CUTLASS .so's
/root/.cache/vllm/torch_compile_cache/ AOT-compiled torch graphs