This article documents exactly what it took to get Nemotron-3-Nano-30B-A3B running on vLLM with 1M context, thinking mode, tool calling, and API-key auth on a 2x RTX 5090 (Blackwell, sm_120) box running Ubuntu 22.04. The model itself is a hybrid Mamba-2/MoE/Attention architecture with NVFP4 weights – one of the first production models that exercises every cutting- edge path in vLLM at once.
The article is split into:
1. The plan and feasibility math
2. The install
3. The eight things that go wrong on Blackwell + Nemotron-H + NVFP4
4. The launch script
5. Benchmarking (with real numbers)
6. Gotchas and lessons learned
1. THE PLAN
Hardware (confirmed via nvidia-smi):
2x NVIDIA GeForce RTX 5090 (32 GB each, sm_120 / Blackwell)
Driver 580.95.05, CUDA 13.0 available, 131 GB RAM, 478 GB disk free
Ubuntu 22.04.5 LTS, Python 3.10.12, no nvcc initially, port 20000 free
Model (nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4):
Architecture : NemotronHForCausalLM (hybrid Mamba-2 + MoE + 6 attn layers)
Quantization : NVFP4 (modelopt, weights ~16 GB, kv_cache_quant=fp8)
Native context : 262 144 (256k)
Hidden size : 2688
Layers : 52 (23 Mamba + 23 MoE + 6 Attention)
Attention heads : 32, KV heads = 2 (heavy GQA), head_dim = 128
MoE : 128 routed experts, 1 shared expert, top-6
Active params : ~3B (out of ~30B total)
Chat template : includes
Tool calling : Hermes format (chat_template.jinja ships in repo)
Gated : NO – public download, no HF token needed
Why 1M context actually fits on 32 GB GPUs:
Only 6 of 52 layers have a KV cache. The Mamba-2 SSM is
recurrent with O(1) state per sequence (it does NOT
grow with context length), and the MoE layers have
no attention at all.
KV @ 1M ctx (bf16, TP=2 replicated):
= 2 (K+V) * 2 (KV heads) * 128 (head dim) * 2 bytes * 6 layers
= 6.1 KB / token
= 5.95 GB total per GPU at 1M tokens
Add fp8 KV cache (NVIDIA pre-configured this in
hf_quant_config.json with kv_cache_quant_algo: FP8)
and we're at ~2.9 GB per GPU.
So 1M context is comfortable; the trick is YaRN to extend
attention positions from the native 256k out to 1M.
2. THE INSTALL
Step 1 – Set up a Python 3.10 venv at /opt/vllm-venv:
apt-get install -y python3.10-venv
python3 -m venv /opt/vllm-venv
/opt/vllm-venv/bin/pip install --upgrade pip wheel setuptools
Step 2 – Install PyTorch + CUDA 13.0 system-wide (needed for nvcc + JIT header paths that vLLM workers look up via $CUDA_HOME):
DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
cuda-nvcc-13-0 \
cuda-cudart-dev-13-0 \
cuda-cccl-13-0 \
cuda-driver-dev-13-0 \
cuda-nvrtc-dev-13-0 \
libcusparse-dev-13-0 \
libcusparselt0-dev-cuda-13 \
libcublas-dev-13-0 \
libcurand-dev-13-0 \
libcudnn9-dev-cuda-13
Reasoning: vLLM nightly is a single abi3 wheel that bundles its own CUDA-13 Python runtime, but the worker subprocesses still spawn nvcc to JIT-compile flashinfer CUTLASS kernels for the local GPU arch.
That nvcc needs the matching dev headers; otherwise you get “fatal error: cublasLt.h: No such file or directory” deep inside a ninja build.
IMPORTANT: after this install, /usr/local/cuda is re-pointed via update-alternatives to cuda-13.0 (so compute_120 is in nvcc’s –list-gpu-arch output). If you ever install a different cuda-* package later, re-check that symlink – it’s the silent foot-gun.
Step 3 – Install vLLM nightly (stable has NO Blackwell cu128 wheel):
/opt/vllm-venv/bin/pip install --extra-index-url \
https://wheels.vllm.ai/nightly vllm
This pulls vllm-0.26.0rc1 + torch 2.11.0+cu130 + flashinfer 0.6.14 + a full CUDA-13 Python stack. The venv ends up around 11 GB.
/opt/vllm-venv/bin/pip install --quiet ninja huggingface_hub
Step 4 – Download the model:
/opt/vllm-venv/bin/hf download nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 \
--local-dir /root/models/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 \
--max-workers 8
~19 GB across 18 files (5 safetensors shards + chat_template.jinja + the nano_v3_reasoning_parser.py and modeling_nemotron_h.py that vLLM auto-imports via trust_remote_code-style auto_map).
Step 5 – Patch config.json to add YaRN rope scaling.
vLLM 0.26 dropped the –rope-scaling CLI flag (you set it in the model’s config.json now, or via JSON overrides baked in at serve time).
Edit /root/models/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4/config.json and add:
"rope_parameters": {
"rope_type": "yarn",
"factor": 4.0,
"original_max_position_embeddings": 262144
}
(New transformers >= 4.50 reads “rope_parameters”; “rope_scaling” still works as legacy fallback but you get a warning. Use rope_parameters.)
Also keep a backup so you can revert if you decide to stay at native 256k context:
cp config.json config.json.bak
3. THE EIGHT THINGS THAT GO WRONG (and exactly how to fix each)
If you skip straight to “vllm serve” you will hit, in order:
(1) “unrecognized arguments: –rope-scaling / –rope-theta / –swap-space” -> vLLM 0.26 removed these CLI flags. Move them into config.json (rope_parameters) and drop –swap-space entirely.
(2) “User-specified max_model_len (1056768) is greater than the derived max_model_len (max_position_embeddings=1048576.0)” -> Set max_model_len = 1048576 exactly (= 262144 * 4), not higher. If you must exceed it, export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 (it warns about NaN risk on out-of-range rope positions).
(3) “Could not find nvcc and default cuda_home=’/usr/local/cuda’ doesn’t exist” -> Install cuda-nvcc-12-8 OR cuda-nvcc-13-0 via apt, then set PATH=/usr/local/cuda/bin:$PATH in the launch env. For Blackwell sm_120 you need nvcc 13.0 – 12.8 errors with “SM 12.x requires CUDA >= 12.9” when probing GPU archs.
(4) “FlashInfer requires GPUs with sm75 or higher” -> Deceptive error: flashinfer isn’t failing to detect the GPU, it’s failing because its TARGET_CUDA_ARCHS list is empty. Caused by nvcc –list-gpu-arch returning empty. Fixed by installing nvcc 13.0 so compute_120 shows up.
(5) “‘ninja’ not found” -> pip install ninja into the venv, and put /opt/vllm-venv/bin FIRST in PATH so the worker subprocesses inherit it.
(6) “fatal error: curand_kernel.h: No such file or directory” -> apt install libcurand-dev-13-0. flashinfer CUTLASS paths include device-side curand headers for random sampling in MoE routing.
(7) “fatal error: cublasLt.h / nvrtc.h / cusparse.h not found” -> apt install libcublas-dev-13-0, cuda-nvrtc-dev-13-0, libcusparse-dev-13-0, cuda-cccl-13-0. These are the dev-13.0 packages; -dev-13-1/-13-2/-13-3 exist too but 13-0 matches what vLLM’s bundled nvcc 13.0.88 was compiled against.
(8) “Free memory on device cuda:1 (19.67/31.36 GiB) on startup is less than desired GPU memory utilization (0.9, 28.22 GiB)” -> A previous failed vLLM run left zombie VLLM::Worker_TP0/TP1 processes holding 11 GB each. pkill -f “vllm serve” doesn’t kill them because their process name is different. Use nvidia-smi –query-compute-apps=pid,process_name then `kill -9
There’s a 9th gotcha that’s not an error but will save you 10 minutes:
(9) The model directory must contain nano_v3_reasoning_parser.py at the root. Pass –reasoning-parser-plugin
4. THE LAUNCH SCRIPT
# /root/start-nemotron.sh - launches Nemotron vLLM fully detached
set -u
VENV=/opt/vllm-venv
MODEL_DIR=/root/models/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4
API_KEY=$(cat /root/vllm-api.key)
LOG=/var/log/vllm/nemotron.log
PIDFILE=/var/run/nemotron-vllm.pid
mkdir -p /var/log/vllm /var/run
pkill -9 -f "vllm serve" 2>/dev/null || true
sleep 2
export PATH=/opt/vllm-venv/bin:/usr/local/cuda/bin:$PATH
export CUDA_HOME=/usr/local/cuda
export HF_HUB_OFFLINE=1
export TRANSFORMERS_OFFLINE=1
export NCCL_P2P_DISABLE=0
export TOKENIZERS_PARALLELISM=false
export VLLM_LOGGING_LEVEL=INFO
export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
# Generate the API key once:
# python -c "import secrets; print(secrets.token_urlsafe(32))" > /root/vllm-api.key
# chmod 600 /root/vllm-api.key
setsid bash -c "
exec $VENV/bin/vllm serve $MODEL_DIR \
--host 0.0.0.0 \
--port 20000 \
--served-model-name nemotron-nano \
--api-key $API_KEY \
--tensor-parallel-size 2 \
--max-model-len 1048576 \
--max-num-seqs 16 \
--gpu-memory-utilization 0.90 \
--enable-auto-tool-choice \
--tool-call-parser hermes \
--reasoning-parser nano_v3 \
--reasoning-parser-plugin $MODEL_DIR/nano_v3_reasoning_parser.py \
--enable-prefix-caching \
--enable-chunked-prefill \
--dtype bfloat16 \
>> $LOG 2>&1
" /dev/null 2>&1 &
echo $! > "$PIDFILE"
disown 2>/dev/null || true
echo "Launched. PID=$PID, log=$LOG"
Run it with `bash /root/start-nemotron.sh`.
First boot takes 5-10 minutes because the worker JIT-compiles flashinfer CUTLASS sm_120 kernels (a few hundred cu files, ~150 nvcc invocations).
Subsequent restarts skip the kernel cache at /root/.cache/flashinfer/0.6.14/120f/ and start in ~60s.
Watch the log for the magic line: INFO: Application startup complete.
5. BENCHMARKS
Using the OpenAI-compatible /v1/chat/completions endpoint with stream=true and stream_options.include_usage=true:
SINGLE-STREAM DECODE (512 generation tokens, varying prompt size):
prompt gen TTFT total decode_tok/s
----- ---- ---- ----- ------------
243 145 0.16s 0.77s 189.4
927 162 0.14s 0.81s 200.7
3658 200 0.41s 1.23s 162.5
14580 275 0.33s 1.43s 192.5
58270 364 0.64s 2.11s 172.7
116524 450 8.31s 10.13s 44.4 <- prompt processing dominates
LONG GENERATION (1024 output, 24-token prompt): prompt=24 gen=1024 total=4.36s decode_tok/s=235.0
CONCURRENT (8 parallel streams, 512 gen each): total=4096 tokens wall=3.24s aggregate=1263 tok/s (about 158 tok/s per stream under 8-way concurrency)
THINKING MODE (reasoning_effort=medium, "What is 17 * 24?"): gen=529 total=2.19s decode_tok/s=241.5 -> response.choices[0].message.reasoning has the chain-of-thought -> response.choices[0].message.content has the final answer
TOOL CALL (force tool_choice=function for "weather in Tokyo?"): tool_calls[0].function.name = "get_weather" tool_calls[0].function.arguments = '{"location":"Tokyo"}' -> Hermes parser emits clean OpenAI-format tool_calls
TOOL CALL WITHOUT tool_choice: Often returns the answer in plain text. The model has the ability but chooses prose. Force it with `tool_choice` or a clear system prompt like "You MUST call the get_weather function".
These numbers are sane for a 30B-A3B NVFP4 hybrid on 2x Blackwell. For comparison, the BF16 sibling would be ~2-2.5x slower on decode (weight memory bandwidth bound) and would NOT fit on 32 GB GPUs at 1M context.
6. LESSONS LEARNED
- NemotronH is a hybrid model. 23 of 52 layers are Mamba-2 (recurrent, fixed-state), 23 are MoE (no attention), only 6 are attention. So prompt-context cost is mostly MoE weight bandwidth and a small KV cache, NOT quadratic attention.
- vLLM nightly is the only path to Blackwell + NemotronH + NVFP4 right now. PyPI stable has no cu128/cu130 wheel for Blackwell, and the NemotronHForCausalLM modelopt loader landed in late-2025.
- The launch environment matters enormously. /usr/local/cuda must point to a CUDA version whose nvcc can list compute_120. PATH must include /opt/vllm-venv/bin (for ninja) and /usr/local/cuda/bin (for nvcc). CUDA_HOME must be set. All four are needed.
- The first cold start compiles a few hundred flashinfer CUTLASS cu files. Don't panic if the log is just ninja [N/56] lines for 5-10 minutes. Look for the actual fatal error (one of the eight above) and fix it; subsequent restarts are fast.
- After a failed launch, ALWAYS nvidia-smi --query-compute-apps before restarting. The VLLM::Worker_TP0/TP1 children survive pkill -f "vllm serve" because their argv[0] is different, and they hold ~11 GB of GPU memory each. Kill them by PID.
- Reasoning output goes in a separate field. response.choices[0].message has .reasoning (chain-of-thought) AND .content (final answer). If you only see .content, your reasoning parser isn't loaded correctly.
- For 1M context attention quality, expect some degradation past the native 256k. The model was trained with rope up to 262144; YaRN with factor=4 extends it but is not free. For workloads that mostly live under 256k, you can drop YaRN entirely (just remove the rope_parameters block) and you'll get full training-quality attention.
- The serve command does NOT need --trust-remote-code. vLLM 0.26 resolves NemotronHForCausalLM natively via the auto_map entry in config.json. Modeling code lives in the model dir.
- Bind to 0.0.0.0 if you want reachability from inside the same host or via SSH tunnel. If you want public reachability, open the cloud firewall (security group / cloud firewall rule / iptables) to allow inbound TCP on 20000 from your client IPs.
APPENDIX A - Quick health check
# Is it up?
curl -sS http://localhost:20000/v1/models \
-H "Authorization: Bearer $(cat /root/vllm-api.key)"
# Quick chat test
curl -sS http://localhost:20000/v1/chat/completions \
-H "Authorization: Bearer $(cat /root/vllm-api.key)" \
-H "Content-Type: application/json" \
-d '{
"model": "nemotron-nano",
"messages": [{"role":"user","content":"In one sentence, what is the capital of France?"}],
"max_tokens": 60,
"temperature": 0
}'
# GPU usage
nvidia-smi --query-gpu=index,memory.used,memory.free,utilization.gpu --format=csv
# vLLM logs
tail -f /var/log/vllm/nemotron.log
# Restart cleanly
pkill -9 -f "vllm serve"; nvidia-smi --query-compute-apps=pid \
| awk -F, 'NR>1 {print $1}' | xargs -r kill -9
bash /root/start-nemotron.sh
APPENDIX B - File layout on disk
/opt/vllm-venv/ Python 3.10 venv, 11 GB
/root/models/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4/
config.json patched with rope_parameters
config.json.bak original (no rope scaling)
chat_template.jinja reasoning + tool-call template
nano_v3_reasoning_parser.py reasoning parser plugin
modeling_nemotron_h.py remote modeling code
configuration_nemotron_h.py remote config class
model-00001-of-00005.safetensors ~4 GB each
...
/root/vllm-api.key API key, mode 0600
/root/start-nemotron.sh launch script
/var/log/vllm/nemotron.log server log
/var/run/nemotron-vllm.pid current PID
/root/.cache/flashinfer/0.6.14/120f/ JIT-compiled CUTLASS .so's
/root/.cache/vllm/torch_compile_cache/ AOT-compiled torch graphs
Selama beberapa minggu ini software 
Ternyata instalasinya cukup simpel. Berjalan dengan cepat, dan softwarenya langsung berjalan. Hanya menanyakan username & password saya satu kali, lalu tidak ada lagi mengusik kita. Saya perhatikan, softwarenya juga cukup stabil. Lalu saya cek sekilas disitusnya, datanya juga sudah mulai masuk di account saya. Oke, mari kita biarkan dulu dia berjalan selama beberapa hari.
