NeMo Switchyard: Route Agent Workloads, Cut Costs 74% - Aoyii

Long-running AI agents burn most of their token budget on routine execution — tool calls, validation, retrieval formatting, summarization, classification. Send all of it to a frontier model and you overspend. Send it to a small model and quality collapses on the hard steps. NVIDIA dropped two open-source pieces on August 11 that make the middle path practical: Nemotron 3.5 Lightning, a 30B MoE with only 3B active parameters built for that execution layer, and NeMo Switchyard, an open-source model routing library that decides which model handles which step.

What you're getting

Nemotron 3.5 Lightning — fully open weights, free to download, use, and modify commercially. Up to 4× output speed vs. same-class models and ~30% faster agent task completion on NVIDIA's PinchBench. Hybrid Mamba-transformer backbone with multi-token prediction and speculative decoding (D-Flash / D-Spark). Ships in BF16 and NVFP4; the NVFP4 checkpoint runs on consumer RTX GPUs and DGX Spark (~71 tok/s single-stream on a DGX Spark).
NeMo Switchyard — open source routing library (github.com/NVIDIA-NeMo/Switchyard). Provider-agnostic SDK, ships a server that accepts OpenAI / Anthropic / Responses API requests, and logs which model was chosen plus why — so you can audit a live route.
Post-training data + recipes — including the Nemotron-RL-Agentic-Terminal-Pivot dataset used to train coding-agent capabilities. Fine-tuning is unusually transparent for this model class.

Why the numbers matter

Routing a request once per task instead of always hitting the biggest model produces results that are hard to ignore:

LangChain — 145 multi-turn agent tasks, only 7% of calls to a frontier model: cost −74% at a ~6-point accuracy tradeoff.
Ramp — matched frontier performance on RampSWE-Bench with cost −58%, runtime −33%.
Cognition — NeMo Switchyard staged routing inside Devin Desktop: mean cost −28% vs. a single frontier model.
Boomi — 100% domain-routing accuracy, 59% of traffic to a 5× faster fine-tuned model, later-turn latency −21%.
NVIDIA internal — frontier-level accuracy at roughly one-third the cost of Opus 4.8 alone.

How the router decides

Routing signals come from three areas: model capability (which model can solve this correctly), cost profile (latency + tokens per model), and infrastructure (handoff/state). Switchyard gives you four strategies out of the box:

LLM classifier — an LLM judge picks a model target and keeps session affinity across turns, so it doesn't re-classify work that hasn't changed.
Stage router — watches tool activity. Errors, repeated unproductive work, or prolonged exploration push the turn toward a capable model; steady writes and passing tests favor the efficient one.
Escalation router — starts every session cheap, and a judge promotes the session only when it detects sustained difficulty. This is the config behind the LangChain 74% result.
Prefill router (tunable) — learns from the model's residual stream to predict each candidate's success probability, then blends accuracy with cost/latency constraints.

Minimal setup

Quickest way to feel Lightning's latency without any hardware: hit the free OpenRouter endpoint.

curl https://openrouter.ai/api/v1/chat/completions \
  -H "Authorization: Bearer $OPENROUTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nvidia/nemotron-3.5-lightning:free",
    "messages": [{"role": "user", "content": "Classify this security alert: multiple failed SSH logins from 10.0.0.44 in 3 minutes"}]
  }'

For a local agent setup, clone Switchyard and define a model pool + policy. The exact schema lives in the repo README; the shape is roughly:

# switchyard.yaml — model pool + routing policy
targets:
  - name: lightning-local          # semantic name, provider-agnostic
    provider: openai-compatible   # local NIM / vLLM endpoint
    model: nvidia/Nemotron-3.5-Lightning-30B-A3B-NVFP4
    base_url: http://localhost:8000/v1
  - name: frontier
    provider: anthropic
    model: claude-opus-4.8

router:
  type: escalation        # start cheap, escalate on sustained difficulty
  judge: frontier
  default: lightning-local
  fallback: frontier

from switchyard import SwitchyardClient

client = SwitchyardClient("switchyard.yaml")
resp = client.chat(messages=[...], session_id="sess-42")
print(resp.model_used, resp.rationale)  # which target served this turn, and why

Practice advice

Route once per task or session, not per turn. Re-routing every message adds a decision layer and breaks prompt caching — the single most common mistake with routers.
Don't put Lightning at the front of a pipeline fed untrusted input. Its known weak spot is prompt-injection resistance. Put an orchestrator or stronger model above it; let Lightning own the repetitive execution layer underneath.
Fine-tuning is cheap enough to try. Reported cases: ~1 epoch in under 3 hours for roughly $100 (Unsloth has scripts for consumer GPUs); CodeRabbit trained a router agent in ~2 hours / $85 using the NeMoAuto recipe.
Pick the right checkpoint. NVFP4 for local inference on RTX/DGX Spark; BF16 if you're fine-tuning and want full fidelity.
You don't have to build the gateway. LiteLLM is adding Switchyard as a plugin, Kong ships it natively in its AI Gateway, and LangChain / Nous Research (Hermes) already integrate it.

Resources

NeMo Switchyard (GitHub): github.com/NVIDIA-NeMo/Switchyard
Nemotron 3.5 Lightning weights (Hugging Face): NVFP4 checkpoint
RL dataset: Nemotron-RL-Agentic-Terminal-Pivot
NIM microservice: build.nvidia.com · OpenRouter: nemotron-3.5-lightning:free
NVIDIA dev blog: Route AI Agent Workloads Across Models