Route LLM Traffic with NVIDIA NeMo Switchyard - Aoyii

One model for everything no longer makes sense. Coding agents like Claude Code and Codex spend most of their tokens on repetitive, mechanical steps — reading files, running tests, fixing lints — while only a fraction of calls actually need frontier-level reasoning. That mismatch is exactly what NVIDIA attacked on August 11 with a two-part release: Nemotron 3.5 Lightning, a 30B open-weight MoE model (3B active per token) tuned for high-volume agent work, and NeMo Switchyard, an Apache-2.0 Rust proxy and library that routes LLM traffic across models and providers.

Switchyard is the piece to try first. It sits between your agent and your model backends and does three things:

Protocol translation — converts between OpenAI Chat, OpenAI Responses, and Anthropic Messages formats, so Claude Code keeps speaking its native API while the request is served by vLLM, NVIDIA NIM, Ollama, or any OpenAI-compatible endpoint.
Typed routing algorithms — LLM-as-classifier, signal-driven stage routing, escalation, random A/B splits, or a custom algorithm you write yourself.
Operational metrics — Prometheus metrics covering requests, errors, latency, tokens, and routing overhead.

The fastest start: launcher path

If you want Claude Code, Codex, or OpenClaw running through a smart route in two minutes:

curl -LsSf https://astral.sh/uv/install.sh | sh
source "$HOME/.local/bin/env"
uv tool install --python 3.12 "nemo-switchyard[cli]"

export OPENROUTER_API_KEY="your-key"
switchyard launch claude --model switchyard
# or: switchyard launch codex --model switchyard
# or: switchyard launch openclaw --model switchyard

The packaged deployment exposes a route ID named switchyard. The launcher starts the native Rust server and points the agent at it — no config file needed for this path.

Custom routing: standalone server

When you want your own model tiers, run the server directly. Install it, then write a routes.toml:

cargo install --locked switchyard-server

schema_version = 1

[llm_clients.openrouter]
format = "openai_chat"
base_url = "https://openrouter.ai/api/v1"
api_key_env = "OPENROUTER_API_KEY"

[targets.weak]
id = "openai/gpt-4o-mini"
llm_client = "openrouter"

[targets.strong]
id = "openai/gpt-4o"
llm_client = "openrouter"

[routes.smart]
id = "switchyard"
type = "llm_classifier"
mode = "capability"
classifier_target = "weak"
strong_target = "strong"
weak_target = "weak"
base_threshold = 0.5

This declares two targets and one route: a classifier model decides, per request, whether the weak or strong model should answer. Secrets live in environment variables via api_key_env — never in the TOML. Validate and start:

export OPENROUTER_API_KEY="your-key"
switchyard-server --config routes.toml --dry-run
switchyard-server --config routes.toml --host 127.0.0.1 --port 4000

Any client that speaks OpenAI Chat, Anthropic Messages, or OpenAI Responses can connect. The route id is the model name clients use:

curl http://localhost:4000/health
curl http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"switchyard","messages":[{"role":"user","content":"hello"}]}'

Which routing algorithm?

Strategy	Use it when
`random`	You need a fixed weighted split for A/B tests or cost baselines.
`llm_classifier`	Request content should decide between a weak and strong tier.
`llm_classifier` + `mode = "escalation"`	Every turn runs on the weak tier first; a judge re-runs the same request on the strong tier only when the answer is weak.
`stage_router`	Signals already in the conversation (tool results, errors) should route most turns without an extra model call.

For agent loops, stage_router is the interesting one: most steps are "did the tool succeed, keep going" — cheap decisions that don't need a frontier model.

It's not theory — the numbers

LangChain: across 145 multi-turn DeepAgents tasks, only 7% of calls went to frontier models → 74% cost reduction at a 6% accuracy cost.
Ramp: parity with frontier models on RampSWE-Bench, with 58% lower cost and 33% less runtime.
Cognition: DevinDesktop integration reports ~28% lower average cost than running a single frontier model.
Boomi: 100% domain routing accuracy, sending 59% of traffic to a 5× faster fine-tuned model, with 21% lower latency on later turns.

Kong has also integrated Switchyard into its AI Gateway, so routing can live in your gateway layer instead of inside the agent.

Practical tips

Run --dry-run before starting: it validates the schema, env lookups, and route construction without binding a port.
Telemetry is a single X-Switchyard-Version header on outbound calls (no content). Opt out with export SWITCHYARD_TELEMETRY_OPT_OUT=1.
Switchyard is pre-alpha — the API will change before v1.0. Pin versions for anything serious.
Embed routing in your own Rust service with switchyard-libsy: algorithms only make decisions and hand model calls back to you, so it drops into an existing gateway without owning an HTTP stack.
Want a self-hosted cheap tier? Pair it with Nemotron 3.5 Lightning — 30B total / 3B active, 1M context, runs on a single DGX Spark (GB10) or H100, with NVFP4 and BF16 checkpoints under the permissive OpenMDW-1.1 license.

Resources

GitHub: github.com/NVIDIA-NeMo/Switchyard
Getting Started: docs/getting_started.md
NVIDIA blog: Route AI Agent Workloads Across Models with NVIDIA NeMo Switchyard
Nemotron 3.5 Lightning: huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4