This article documents exactly what it took to get Nemotron-3-Nano-30B-A3B running on vLLM with 1M context, thinking mode, tool calling, and API-key auth on a 2x RTX 5090 (Blackwell, sm_120) box running Ubuntu 22.04. The model itself is a hybrid Mamba-2/MoE/Attention architecture with NVFP4 weights – one of the first production models that exercises every cutting- edge path in vLLM at once.

The article is split into:

1. The plan and feasibility math
2. The install
3. The eight things that go wrong on Blackwell + Nemotron-H + NVFP4
4. The launch script
5. Benchmarking (with real numbers)
6. Gotchas and lessons learned

1. THE PLAN

Hardware (confirmed via nvidia-smi):

2x NVIDIA GeForce RTX 5090 (32 GB each, sm_120 / Blackwell)
Driver 580.95.05, CUDA 13.0 available, 131 GB RAM, 478 GB disk free
Ubuntu 22.04.5 LTS, Python 3.10.12, no nvcc initially, port 20000 free

Model (nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4):

Architecture : NemotronHForCausalLM (hybrid Mamba-2 + MoE + 6 attn layers)
Quantization : NVFP4 (modelopt, weights ~16 GB, kv_cache_quant=fp8)
Native context : 262 144 (256k)
Hidden size : 2688
Layers : 52 (23 Mamba + 23 MoE + 6 Attention)
Attention heads : 32, KV heads = 2 (heavy GQA), head_dim = 128
MoE : 128 routed experts, 1 shared expert, top-6
Active params : ~3B (out of ~30B total)
Chat template : includes … reasoning tags
Tool calling : Hermes format (chat_template.jinja ships in repo)
Gated : NO – public download, no HF token needed

Why 1M context actually fits on 32 GB GPUs:

    Only 6 of 52 layers have a KV cache. The Mamba-2 SSM is 
    recurrent with O(1) state per sequence (it does NOT 
    grow with context length), and the MoE layers have 
    no attention at all.

    KV @ 1M ctx (bf16, TP=2 replicated):
      = 2 (K+V) * 2 (KV heads) * 128 (head dim) * 2 bytes * 6 layers
      = 6.1 KB / token
      = 5.95 GB total per GPU at 1M tokens

    Add fp8 KV cache (NVIDIA pre-configured this in 
    hf_quant_config.json with kv_cache_quant_algo: FP8) 
    and we're at ~2.9 GB per GPU.

    So 1M context is comfortable; the trick is YaRN to extend 
    attention positions from the native 256k out to 1M.

2. THE INSTALL

Step 1 – Set up a Python 3.10 venv at /opt/vllm-venv:


apt-get install -y python3.10-venv
python3 -m venv /opt/vllm-venv
/opt/vllm-venv/bin/pip install --upgrade pip wheel setuptools


Step 2 – Install PyTorch + CUDA 13.0 system-wide (needed for nvcc + JIT header paths that vLLM workers look up via $CUDA_HOME):


DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
cuda-nvcc-13-0 \
cuda-cudart-dev-13-0 \
cuda-cccl-13-0 \
cuda-driver-dev-13-0 \
cuda-nvrtc-dev-13-0 \
libcusparse-dev-13-0 \
libcusparselt0-dev-cuda-13 \
libcublas-dev-13-0 \
libcurand-dev-13-0 \
libcudnn9-dev-cuda-13


Reasoning: vLLM nightly is a single abi3 wheel that bundles its own CUDA-13 Python runtime, but the worker subprocesses still spawn nvcc to JIT-compile flashinfer CUTLASS kernels for the local GPU arch.

That nvcc needs the matching dev headers; otherwise you get “fatal error: cublasLt.h: No such file or directory” deep inside a ninja build.

IMPORTANT: after this install, /usr/local/cuda is re-pointed via update-alternatives to cuda-13.0 (so compute_120 is in nvcc’s –list-gpu-arch output). If you ever install a different cuda-* package later, re-check that symlink – it’s the silent foot-gun.

Step 3 – Install vLLM nightly (stable has NO Blackwell cu128 wheel):


/opt/vllm-venv/bin/pip install --extra-index-url \
https://wheels.vllm.ai/nightly vllm


This pulls vllm-0.26.0rc1 + torch 2.11.0+cu130 + flashinfer 0.6.14 + a full CUDA-13 Python stack. The venv ends up around 11 GB.


/opt/vllm-venv/bin/pip install --quiet ninja huggingface_hub


Step 4 – Download the model:


/opt/vllm-venv/bin/hf download nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 \
--local-dir /root/models/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 \
--max-workers 8


~19 GB across 18 files (5 safetensors shards + chat_template.jinja + the nano_v3_reasoning_parser.py and modeling_nemotron_h.py that vLLM auto-imports via trust_remote_code-style auto_map).

Step 5 – Patch config.json to add YaRN rope scaling.

vLLM 0.26 dropped the –rope-scaling CLI flag (you set it in the model’s config.json now, or via JSON overrides baked in at serve time).

Edit /root/models/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4/config.json and add:


"rope_parameters": {
"rope_type": "yarn",
"factor": 4.0,
"original_max_position_embeddings": 262144
}


(New transformers >= 4.50 reads “rope_parameters”; “rope_scaling” still works as legacy fallback but you get a warning. Use rope_parameters.)

Also keep a backup so you can revert if you decide to stay at native 256k context:

cp config.json config.json.bak


3. THE EIGHT THINGS THAT GO WRONG (and exactly how to fix each)

If you skip straight to “vllm serve” you will hit, in order:

(1) “unrecognized arguments: –rope-scaling / –rope-theta / –swap-space” -> vLLM 0.26 removed these CLI flags. Move them into config.json (rope_parameters) and drop –swap-space entirely.

(2) “User-specified max_model_len (1056768) is greater than the derived max_model_len (max_position_embeddings=1048576.0)” -> Set max_model_len = 1048576 exactly (= 262144 * 4), not higher. If you must exceed it, export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 (it warns about NaN risk on out-of-range rope positions).

(3) “Could not find nvcc and default cuda_home=’/usr/local/cuda’ doesn’t exist” -> Install cuda-nvcc-12-8 OR cuda-nvcc-13-0 via apt, then set PATH=/usr/local/cuda/bin:$PATH in the launch env. For Blackwell sm_120 you need nvcc 13.0 – 12.8 errors with “SM 12.x requires CUDA >= 12.9” when probing GPU archs.

(4) “FlashInfer requires GPUs with sm75 or higher” -> Deceptive error: flashinfer isn’t failing to detect the GPU, it’s failing because its TARGET_CUDA_ARCHS list is empty. Caused by nvcc –list-gpu-arch returning empty. Fixed by installing nvcc 13.0 so compute_120 shows up.

(5) “‘ninja’ not found” -> pip install ninja into the venv, and put /opt/vllm-venv/bin FIRST in PATH so the worker subprocesses inherit it.

(6) “fatal error: curand_kernel.h: No such file or directory” -> apt install libcurand-dev-13-0. flashinfer CUTLASS paths include device-side curand headers for random sampling in MoE routing.

(7) “fatal error: cublasLt.h / nvrtc.h / cusparse.h not found” -> apt install libcublas-dev-13-0, cuda-nvrtc-dev-13-0, libcusparse-dev-13-0, cuda-cccl-13-0. These are the dev-13.0 packages; -dev-13-1/-13-2/-13-3 exist too but 13-0 matches what vLLM’s bundled nvcc 13.0.88 was compiled against.

(8) “Free memory on device cuda:1 (19.67/31.36 GiB) on startup is less than desired GPU memory utilization (0.9, 28.22 GiB)” -> A previous failed vLLM run left zombie VLLM::Worker_TP0/TP1 processes holding 11 GB each. pkill -f “vllm serve” doesn’t kill them because their process name is different. Use nvidia-smi –query-compute-apps=pid,process_name then `kill -9 ` directly. You can also drop gpu-memory-utilization to 0.85 to give yourself a 5% buffer.

There’s a 9th gotcha that’s not an error but will save you 10 minutes:

(9) The model directory must contain nano_v3_reasoning_parser.py at the root. Pass –reasoning-parser-plugin /nano_v3_reasoning_parser.py AND –reasoning-parser nano_v3 on the CLI. If you use a different parser name (like deepseek_r1) the reasoning extraction silently breaks – test it by checking that response.choices[0].message has a “reasoning” field, not just “content”.

4. THE LAUNCH SCRIPT


# /root/start-nemotron.sh - launches Nemotron vLLM fully detached
set -u
VENV=/opt/vllm-venv
MODEL_DIR=/root/models/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4
API_KEY=$(cat /root/vllm-api.key)
LOG=/var/log/vllm/nemotron.log
PIDFILE=/var/run/nemotron-vllm.pid

setsid bash -c "
exec $VENV/bin/vllm serve $MODEL_DIR \
--host 0.0.0.0 \
--port 20000 \
--served-model-name nemotron-nano \
--api-key $API_KEY \
--tensor-parallel-size 2 \
--max-model-len 1048576 \
--max-num-seqs 16 \
--gpu-memory-utilization 0.90 \
--enable-auto-tool-choice \
--tool-call-parser hermes \
--reasoning-parser nano_v3 \
--reasoning-parser-plugin $MODEL_DIR/nano_v3_reasoning_parser.py \
--enable-prefix-caching \
--enable-chunked-prefill \
--dtype bfloat16 \
>> $LOG 2>&1
" /dev/null 2>&1 &
echo $! > "$PIDFILE"
disown 2>/dev/null || true
echo "Launched. PID=$PID, log=$LOG"


Run it with `bash /root/start-nemotron.sh`.

First boot takes 5-10 minutes because the worker JIT-compiles flashinfer CUTLASS sm_120 kernels (a few hundred cu files, ~150 nvcc invocations).

Subsequent restarts skip the kernel cache at /root/.cache/flashinfer/0.6.14/120f/ and start in ~60s.

Watch the log for the magic line: INFO: Application startup complete.

5. BENCHMARKS

Using the OpenAI-compatible /v1/chat/completions endpoint with stream=true and stream_options.include_usage=true:

SINGLE-STREAM DECODE (512 generation tokens, varying prompt size):

    prompt  gen      TTFT    total     decode_tok/s
    -----  ----     ----    -----     ------------
      243   145    0.16s    0.77s          189.4
      927   162    0.14s    0.81s          200.7
     3658   200    0.41s    1.23s          162.5
    14580   275    0.33s    1.43s          192.5
    58270   364    0.64s    2.11s          172.7
   116524   450    8.31s   10.13s           44.4   <- prompt processing dominates

LONG GENERATION (1024 output, 24-token prompt): prompt=24 gen=1024 total=4.36s decode_tok/s=235.0

CONCURRENT (8 parallel streams, 512 gen each): total=4096 tokens wall=3.24s aggregate=1263 tok/s (about 158 tok/s per stream under 8-way concurrency)

THINKING MODE (reasoning_effort=medium, "What is 17 * 24?"): gen=529 total=2.19s decode_tok/s=241.5 -> response.choices[0].message.reasoning has the chain-of-thought -> response.choices[0].message.content has the final answer

TOOL CALL (force tool_choice=function for "weather in Tokyo?"): tool_calls[0].function.name = "get_weather" tool_calls[0].function.arguments = '{"location":"Tokyo"}' -> Hermes parser emits clean OpenAI-format tool_calls

TOOL CALL WITHOUT tool_choice: Often returns the answer in plain text. The model has the ability but chooses prose. Force it with `tool_choice` or a clear system prompt like "You MUST call the get_weather function".

These numbers are sane for a 30B-A3B NVFP4 hybrid on 2x Blackwell. For comparison, the BF16 sibling would be ~2-2.5x slower on decode (weight memory bandwidth bound) and would NOT fit on 32 GB GPUs at 1M context.

6. LESSONS LEARNED

- NemotronH is a hybrid model. 23 of 52 layers are Mamba-2 (recurrent, fixed-state), 23 are MoE (no attention), only 6 are attention. So prompt-context cost is mostly MoE weight bandwidth and a small KV cache, NOT quadratic attention.

- vLLM nightly is the only path to Blackwell + NemotronH + NVFP4 right now. PyPI stable has no cu128/cu130 wheel for Blackwell, and the NemotronHForCausalLM modelopt loader landed in late-2025.

- The launch environment matters enormously. /usr/local/cuda must point to a CUDA version whose nvcc can list compute_120. PATH must include /opt/vllm-venv/bin (for ninja) and /usr/local/cuda/bin (for nvcc). CUDA_HOME must be set. All four are needed.

- The first cold start compiles a few hundred flashinfer CUTLASS cu files. Don't panic if the log is just ninja [N/56] lines for 5-10 minutes. Look for the actual fatal error (one of the eight above) and fix it; subsequent restarts are fast.

- After a failed launch, ALWAYS nvidia-smi --query-compute-apps before restarting. The VLLM::Worker_TP0/TP1 children survive pkill -f "vllm serve" because their argv[0] is different, and they hold ~11 GB of GPU memory each. Kill them by PID.

- Reasoning output goes in a separate field. response.choices[0].message has .reasoning (chain-of-thought) AND .content (final answer). If you only see .content, your reasoning parser isn't loaded correctly.

- For 1M context attention quality, expect some degradation past the native 256k. The model was trained with rope up to 262144; YaRN with factor=4 extends it but is not free. For workloads that mostly live under 256k, you can drop YaRN entirely (just remove the rope_parameters block) and you'll get full training-quality attention.

- The serve command does NOT need --trust-remote-code. vLLM 0.26 resolves NemotronHForCausalLM natively via the auto_map entry in config.json. Modeling code lives in the model dir.

- Bind to 0.0.0.0 if you want reachability from inside the same host or via SSH tunnel. If you want public reachability, open the cloud firewall (security group / cloud firewall rule / iptables) to allow inbound TCP on 20000 from your client IPs.

APPENDIX A - Quick health check

# Restart cleanly
pkill -9 -f "vllm serve"; nvidia-smi --query-compute-apps=pid \
| awk -F, 'NR>1 {print $1}' | xargs -r kill -9
bash /root/start-nemotron.sh


APPENDIX B - File layout on disk


/opt/vllm-venv/ Python 3.10 venv, 11 GB
/root/models/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4/
config.json patched with rope_parameters
config.json.bak original (no rope scaling)
chat_template.jinja reasoning + tool-call template
nano_v3_reasoning_parser.py reasoning parser plugin
modeling_nemotron_h.py remote modeling code
configuration_nemotron_h.py remote config class
model-00001-of-00005.safetensors ~4 GB each
...
/root/vllm-api.key API key, mode 0600
/root/start-nemotron.sh launch script
/var/log/vllm/nemotron.log server log
/var/run/nemotron-vllm.pid current PID
/root/.cache/flashinfer/0.6.14/120f/ JIT-compiled CUTLASS .so's
/root/.cache/vllm/torch_compile_cache/ AOT-compiled torch graphs


End of article. Tested August 2026
on 2x RTX 5090 + vLLM nightly 0.26.0

I thought I’d never see the day – BSA (Business Software Alliance) is currently being sued by an Indonesian company, Multisari, due to the illegal raids carried out to find illegal / pirated software.

I know, I know – what an irony indeed, right ? 🙂 breaking the law to find the breakers of the law, gosh.
What a bunch of genius. 😀

Context : Here in Indonesia, some(often?)times there are rogue officials, from BSA or Police, who conducted the checks (raids) illegally.

Imagine having your computers / servers seized suddenly, due to suspicion of having illegal software in it.
What a nasty way to halt a company’s daily operation eh? Yet that’s what happened with a lot of companies here.

No search warrant. No warning whatsoever.
Just some officers suddenly showing up in your office, do some quick check – and there goes your computers.

However, most of the victims chose to stay silent. Or, do a backhand deal with said officers, involving some cash of course, to secure the return of their computer.

Some of the victims of these illegal BSA raids also chose to migrate to Linux 🙂 which then are welcomed warmly by the local F/OSS community.
Thanks BSA for sending them our way ! Bill Gates & Steve Ballmer would be SO happy that you did such a great job 😀

Anyway, Multisari chose to stood its ground. It chose to strike back at these BSA bullies. And I’m so happy to be able to witness such a day. Good for them !

Quoted :

“In case number 517/Pdt.G/2011/PN.Jkt.Pst, Multisari Langgeng Inc. sued BSA Singapore, BSA Indonesia, and BSA Washington DC”

“Multisari sued BSA because of the raids conducted by BSA Singapore (**) and BSA Indonesia to its office on September 22dn, 2011”

(**) What the hell is BSA Singapore doing here, raiding Indonesian companies, on Indonesian soil ?
This is an outrage.

Source : http://www.bisnis.com/articles/hukum-bisnis-perkara-multisari-vs-bsa-masuk-mediasi
(hint: Google Translate is your friend)