This guide documents a complete, working deployment of orcarouter/Qwen3.8-Flash-Next-Uncensored-GGUF (IQ3_XXS quant) on Strata (Niko1221/Strata, the llama.cpp-based local LLM launcher) on a fresh Ubuntu 24.04 server using only 24GB of VRAM on the GPU, and offloding everything else to 120GB of RAM. By the end of the run you have:
- A Strata instance on
127.0.0.1:10001pinned to GPU 1, 24 GB VRAM cap, 1 M-token context, multi-client and tool calling - An HTTPS front door at
https://xxxxx.ai.id/strata2/via Caddy, with an API key gate - Three MCP servers wired in (
fake,fetch, and a customhttp_inspectfor full HTTP header inspection)
The uncensored fine-tune is the same Qwen3.8-Flash-Next architecture (125 B total, 6 B active per token MoE), so the Strata engine works after a one-time iq_pack.py --compat-bf16 step. Total wall time including a fresh 85 GB download, pack, and verification is roughly 90 minutes on a 50 MB/s link.
Why Strata + OrcaRouter
Strata is a purpose-built inference engine for exactly one model: Qwen3.8-Flash-Next 125 B. The engine exploits four architecture-specific features (Qwen Sparse Attention, Gated DeltaNet, a 51 B n-gram embedding, the model’s own MTP draft head) that no generic llama.cpp build can use. The trade-off is limited model support, only Qwen3.8-Flash-Next is currently supported.
OrcaRouter ships community GGUFs of the same model with a community safety profile (uncensored), split into standard llama.cpp quants (IQ2_M, IQ2_XXS, IQ3_XXS, IQ4_XS, Q2_K, Q3_K_*, Q4_K_*, Q5_K_*, Q6_K, Q8_0) plus an MTP-draft GGUF. Strata’s installer menu only knows its own quants, so for Orca you bypass setup.sh entirely and pack the GGUF manually with tools/iq_pack.py --compat-bf16.
Prerequisites
- Hardware: enough VRAM for the cap you choose plus enough RAM for the resident expert arena. Orca IQ3_XXS needs ~50 GiB RAM for the experts; the rest can be on disk or RAM – but this guide will put everything else on RAM, never disk, for maximum performance.
- Driver: NVIDIA 580 or newer. Check with
nvidia-smi. - Disk: ~90 GB for the two GGUF shards plus a few GiB for the pack.
- Python 3.10+ with
venvandpipon PATH. - Hugging Face account that has accepted the gating on the orcarouter repo.
Hugging Face: gated access
The OrcaRouter repo is gated. A bare curl to resolve/main returns 401 even with a valid HF token if you have not accepted the license. You need to do these simple two steps in the browser:
- Open the model page and click Agree and access. The form for an uncensored model is longer than usual; expect a content disclaimer.
- Create a read-only access token at huggingface.co/settings/tokens.
Put the token on the server. Never paste it in chat:
echo 'export HF_TOKEN=hf_xxx...xxxx' > ~/.hf_token_env
chmod 600 ~/.hf_token_env
# verify without exposing the value
source ~/.hf_token_env && echo "PREFIX=${HF_TOKEN:0:3} LEN=${#HF_TOKEN}"A correct token echoes something like PREFIX=hf_ LEN=37.
Strata engine and compile
Strata compiles its own CUDA engine the first time you run setup.sh, so the Orca instance reuses the same engine binary. Clone and run a no-op setup to populate .venv:
cd ~
git clone --depth 1 https://github.com/Niko1221/Strata.git
cd Strata
./setup.sh --yes --family unsloth --model UD-Q4_K_XL --no-start \
--host 0.0.0.0 --api-key "$(head -c 48 /dev/urandom | sha512sum | awk '{print $1}')"--no-start is the key flag. It runs the CUDA compile, builds the venv, then exits without downloading a model. Expect five minutes of compilation.
Add the Python tools the pack step needs:
source .venv/bin/activate
pip install gguf numpy regex
pip install mcp mcp-server-fetchDownload the OrcaRouter GGUF
Two shards, ~85 GB total for IQ3_XXS. Keep them in a data directory separate from the first instance:
mkdir -p ~/Strata-data-orca/orca
cd ~/Strata-data-orca/orca
. ~/.hf_token_env
curl -L --retry 5 -H "Authorization: Bearer $HF_TOKEN" \
-o Qwen3.8-Flash-Next-Uncensored-IQ3_XXS-00001-of-00002.gguf \
"https://huggingface.co/orcarouter/Qwen3.8-Flash-Next-Uncensored-GGUF/resolve/main/Qwen3.8-Flash-Next-Uncensored-IQ3_XXS-00001-of-00002.gguf"
curl -L --retry 5 -H "Authorization: Bearer $HF_TOKEN" \
-o Qwen3.8-Flash-Next-Uncensored-IQ3_XXS-00002-of-00002.gguf \
"https://huggingface.co/orcarouter/Qwen3.8-Flash-Next-Uncensored-GGUF/resolve/main/Qwen3.8-Flash-Next-Uncensored-IQ3_XXS-00002-of-00002.gguf"A 50 MB/s link finishes the pair in about 20 minutes. The script writes the bytes to two separate *.exit files; poll du -sh ~/Strata-data-orca/orca to watch.
Pack the model
OrcaRouter’s GGUFs use the regular llama.cpp quant layout. Strata expects its own format, which the engine’s tools/iq_pack.py produces from a single shard. --compat-bf16 dequantizes the small non-native projections to BF16 (a 1.22 GiB rounding pass; experts and the 28.8 GB PLE table stay unchanged):
cd ~/Strata
.venv/bin/python tools/iq_pack.py \
--gguf /home/myuser/Strata-data-orca/orca/Qwen3.8-Flash-Next-Uncensored-IQ3_XXS-00001-of-00002.gguf \
--out /home/myuser/Strata/packs/orca-iq3_xxs \
--compat-bf16The pack takes 10–20 minutes. Look for compat-bf16: 363 tensors, 1.22 GiB; expert and PLE table bytes unchanged in the log. The output lands at ~/Strata/packs/orca-iq3_xxs/ with dense.bin, index.txt, and a tokenizer directory.
Engine config
Strata normally writes the config during setup.sh. The manual pack path needs a hand-written JSON. Save as strata-orca-iq3_xxs.json in the Strata repo root. The "gpu": 1 line pins the engine to the second GPU; the VRAM reserve computes as 143771 - 24576 = 119195 MiB, which gives the engine exactly 24 GiB on the card.
{
"exe": "/home/myuser/Strata/engine/strata",
"args": [
"--pack", "/home/myuser/Strata/packs/orca-iq3_xxs",
"--native", "/home/myuser/Strata-data-orca/orca/Qwen3.8-Flash-Next-Uncensored-IQ3_XXS-00001-of-00002.gguf",
"--ple-io", "ram",
"--expert-profile", "/home/myuser/Strata/data/expert-profile.bin",
"--expert-cache", "auto",
"--prefill", "512",
"--spec", "4",
"--spec-min-p", "0.5",
"--mtp", "/home/myuser/Strata-data/mtp/rt",
"--max-context", "1000000",
"--rope-scaling", "yarn",
"--rope-scale", "3.8147",
"--kv", "int8",
"--kv-resident", "32768",
"--resident-budget-gib", "90",
"--vram-reserve-mib", "119195",
"--conversation-cache-mib", "8192"
],
"cwd": "/home/myuser/Strata",
"tokenizer": "/home/myuser/Strata/packs/orca-iq3_xxs/tokenizer",
"model_name": "orcarouter-qwen3.8-flash-next-uncensored-iq3_xxs",
"log": "/home/myuser/Strata/strata-orca-iq3_xxs.log",
"host": "127.0.0.1",
"port": 10001,
"gpu": 1,
"api_key": "9d9d705a593290841c26c707e70ee30ea8376e1b3728c3a0fccb9b8c7302deeaec47b9c2d6ef",
"parallel": 4,
"mcp_servers": {
"fake": {"command": "python3", "args": ["/home/myuser/Strata/serve/mcp_fake_server.py"]},
"fetch": {"command": "/home/myuser/Strata/.venv/bin/mcp-server-fetch"},
"http_inspect": {"command": "python3", "args": ["/home/myuser/Strata/mcp_servers/http_inspect.py", "--body-chars", "20000"]}
},
"mcp": {"timeout_s": 60, "max_result_chars": 20000, "max_rounds": 8}
}Two non-obvious choices:
--prefillmust be a chunk number (512 here) for native IQ packs, notauto. The engine refuses to start otherwise.--ple-io ramforces the 28.8 GB n-gram table into RAM instead of letting it sit on disk. The 440 GB RAM budget makes this safe and removes per-token SSD reads.
Start the Strata instance
cd /home/myuser/Strata
setsid .venv/bin/python -m serve.server \
--engine strata \
--config /home/myuser/Strata/strata-orca-iq3_xxs.json \
--port 10001 \
</dev/null >/home/myuser/strata-orca.out 2>&1 &
sleep 5
curl -s -H "Authorization: Bearer $KEY" http://127.0.0.1:10001/healthFirst start is two to three minutes. The model loads the resident RAM arena (~50 GiB), warms the MTP draft head, and answers the first probe. Subsequent starts are faster (~30 s) because the OS file cache holds the GGUF pages.
HTTPS via Caddy
Add a route for the Orca port. Drop this at /etc/caddy/routes/strata2.caddy:
# Strata Orca (127.0.0.1:10001) — path /strata2/
# Auth ditegakkan backend Strata sendiri (wajib Authorization: Bearer, no-key = 401)
@strata2 path /strata2 /strata2/*
handle @strata2 {
uri strip_prefix /strata2
reverse_proxy 127.0.0.1:10001 {
flush_interval -1
}
}sudo cp strata2.caddy /etc/caddy/routes/
sudo caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile
sudo systemctl restart caddyUse restart, not reload. On Caddyfile import /etc/caddy/routes/*.caddy, a partial reload can leave a transient “test” process holding the new ports and a stale main process. The full restart collapses both and re-adapts cleanly.
MCP and the http_inspect tool
Strata exposes MCP tools to the chat model via mcp_servers in the config (see JSON above). The shipped fetch server returns body only. For headers, save the following as ~/Strata/mcp_servers/http_inspect.py and reference it in the config:
"""Small stdio MCP server exposing HTTP inspection tools.
Tools:
http_get - GET a URL; returns status, headers (all), body (truncated).
http_head - HEAD a URL; returns status and headers only.
http_options - OPTIONS; returns Allow + CORS headers.
curl_like - custom method/url/headers/body; status, headers, body.
"""
import json, sys, urllib.request, urllib.error, argparse
TOOLS = [
{"name": "http_get", "description": "GET a URL. Returns JSON with status, headers, body text.",
"inputSchema": {"type": "object",
"properties": {"url": {"type": "string"},
"max_body_chars": {"type": "integer"}},
"required": ["url"]}},
{"name": "http_head", "description": "HEAD a URL. Returns JSON with status and headers (no body).",
"inputSchema": {"type": "object",
"properties": {"url": {"type": "string"}},
"required": ["url"]}},
{"name": "http_options", "description": "OPTIONS a URL. Returns Allow + CORS headers.",
"inputSchema": {"type": "object",
"properties": {"url": {"type": "string"}},
"required": ["url"]}},
{"name": "curl_like", "description": "Custom method/headers/body. Returns status, headers, body.",
"inputSchema": {"type": "object",
"properties": {"url": {"type": "string"},
"method": {"type": "string", "default": "GET"},
"headers": {"type": "object"},
"body": {"type": "string"}},
"required": ["url"]}},
]
def text(s):
return {"content": [{"type": "text", "text": s}]}
def do_request(url, method="GET", headers=None, body=None, timeout=15):
req = urllib.request.Request(url, method=method, headers=headers or {})
data = body.encode("utf-8") if isinstance(body, str) else body
try:
with urllib.request.urlopen(req, data=data, timeout=timeout) as resp:
return {"ok": True, "status": resp.status,
"headers": dict(resp.headers),
"body": resp.read().decode("utf-8", "replace")}
except urllib.error.HTTPError as e:
return {"ok": True, "status": e.code,
"headers": dict(e.headers.items()) if e.headers else {},
"body": (e.read() if hasattr(e, "read") else b"").decode("utf-8", "replace")}
except Exception as e:
return {"ok": False, "error": f"{type(e).__name__}: {e}"}
def call(name, args, body_chars):
args = args or {}
if name == "http_head":
r = do_request(args["url"], "HEAD")
r.pop("body", None)
return text(json.dumps(r, indent=2))
if name == "http_get":
r = do_request(args["url"], "GET")
if r.get("ok") and "body" in r:
cap = int(args.get("max_body_chars") or body_chars)
r["body"] = r["body"][:cap]
r["truncated"] = len(r["body"]) >= cap
return text(json.dumps(r, indent=2))
if name == "http_options":
r = do_request(args["url"], "OPTIONS")
r.pop("body", None)
return text(json.dumps(r, indent=2))
if name == "curl_like":
r = do_request(args["url"], (args.get("method") or "GET").upper(),
args.get("headers"), args.get("body"))
if r.get("ok") and "body" in r:
r["body"] = r["body"][:body_chars]
return text(json.dumps(r, indent=2))
return text(json.dumps({"ok": False, "error": f"unknown tool: {name}"}))
def handle(msg, body_chars):
rid = msg.get("id"); method = msg.get("method"); params = msg.get("params") or {}
if "id" not in msg: return None
if method == "initialize":
return {"jsonrpc": "2.0", "id": rid, "result": {
"protocolVersion": params.get("protocolVersion", "2025-06-18"),
"capabilities": {"tools": {}},
"serverInfo": {"name": "http_inspect", "version": "0.1"}}}
if method == "tools/list":
return {"jsonrpc": "2.0", "id": rid, "result": {"tools": TOOLS}}
if method == "tools/call":
return {"jsonrpc": "2.0", "id": rid,
"result": call(params.get("name"), params.get("arguments"), body_chars)}
return {"jsonrpc": "2.0", "id": rid, "error": {"code": -32601, "message": method}}
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--body-chars", type=int, default=20000)
args = ap.parse_args()
for line in sys.stdin:
line = line.strip()
if not line: continue
try: msg = json.loads(line)
except: continue
resp = handle(msg, args.body_chars)
if resp is not None:
sys.stdout.write(json.dumps(resp) + "\n"); sys.stdout.flush()
if __name__ == "__main__":
main()No external dependencies. Drop it in ~/Strata/mcp_servers/ and reference it from mcp_servers.http_inspect. Discovery via GET /strata2/mcp lists all four tools with their full schemas.
Verification
KEY="9d9d705a593290841c26c707e70ee30ea8376e1b3728c3a0fccb9b8c7302deeaec47b9c2d6ef"
# 1. Health and auth gate
curl -s -H "Authorization: Bearer $KEY" https://xxxxx.ai.id/strata2/health
curl -s -o /dev/null -w '%{http_code}\n' https://xxxxx.ai.id/strata2/v1/models # 401 expected
# 2. VRAM actually respects the 24 GiB cap
nvidia-smi --query-gpu=index,memory.used --format=csv,noheader # GPU 1 should be near 26 GiB
# 3. MCP tools are live
curl -s -H "Authorization: Bearer $KEY" https://xxxxx.ai.id/strata2/mcp | python3 -m json.tool
# 4. http_head returns real headers via the model
curl -s -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
-d '{"model":"strata","strata_mcp":true,
"messages":[{"role":"user","content":
"Use the http_head tool to inspect https://huggingface.co/. Tell me the Server and Cache-Control headers."}],
"max_tokens":400,"reasoning_effort":"low"}' \
https://xxxxx.ai.id/strata2/v1/chat/completions | python3 -m json.toolA passing run reports the model orcarouter-qwen3.8-flash-next-uncensored-iq3_xxs, 1 M context, api_key: true, the second GPU at ~26 GiB (a little over the 24 GiB cap is normal because the engine reports the in-flight expert cache), and an answer that quotes real response headers rather than admitting defeat.
Troubleshooting
- Engine refuses to start with
native (IQ) pack: it needs --native SHARD1, --spec T (T >= 2) and --prefill CHUNK. The--prefillvalue must be a number (e.g.512), notauto. The--spec 4 --spec-min-p 0.5pair is also required for native IQ packs. - Token set in
~/.hf_token_envbutHF_TOKENis empty in the SSH session. The script that does the download mustsourcethe env file explicitly. Add. ~/.hf_token_envat the top of the download script, or setHF_TOKENin~/.bashrcand usebash -lc. - Sometimes
killreportscommand not foundfor a PID. The PID may belongs to a process that has already exited. Usess -tlnp | grep :10001to find the actual listener and kill that one.
Results and numbers
Tested on a server with 2 GPUs running an instance of Strata each:
| Metric | Instance 1 (UD-Q4_K_XL) | Instance 2 (Orca IQ3_XXS) |
|---|---|---|
| Public path | /strata/ | /strata2/ |
| GPU / VRAM cap | 0 / 32 GiB | 1 / 24 GiB |
| VRAM in use | ~38 GiB | ~26 GiB |
| Context | 1 M (YaRN) | 1 M (YaRN) |
| Decode | ~21 tok/s | ~30 tok/s |
| Draft acceptance | (MTP Qwen) | ~80% |
| Parallel slots | 4 | 4 |
| MCP servers | none | fake, fetch, http_inspect |
| RAM used | 325 GiB / 440 GiB | |
The Orca instance is ~30% faster on the same prompts than the cautious Q4_K_XL instance at the same context, which is consistent with the IQ3_XXS cache being smaller and KV streaming being able to stay hotter. The uncensored profile is the differentiator; the architecture and the engine path are identical.
Closing
Strata is a sharp instrument: one model, one engine, one storage layout, four GPU/RAM knobs. That constraint is exactly what makes the 125 B model usable on a single card with sub-second first-token latency. OrcaRouter is the simplest way to get the same model with relaxed safety without changing the engine. The combination is reproducible, auditable, and one iq_pack.py invocation away on any Linux host with the right driver.