Run 35B MoE Models on an RTX 4060 with FreeToken

AI Guide

Researchers from UC Berkeley and MIT just open-sourced FreeToken, an inference engine that runs frontier Mixture-of-Experts (MoE) models on consumer hardware. On an 8GB RTX 4060 laptop it serves Qwen3.6-35B at roughly 39 tokens/second; on an RTX 5090 it handles DeepSeek-V4-Flash (284B). The idea: treat your PC as an elastically schedulable heterogeneous compute pool instead of a resource-starved data-center node. If you want to cut cloud-API spend and self-host large models, this is worth a weekend.

The core idea: never stall the GPU

Sparse MoE only computes a fraction of parameters per token, but decoding still has to route through hundreds of billions of inactive weights. In the data center, NVLink hides the weight-transfer overhead. On consumer hardware, PCIe bandwidth (typically 16–64 GB/s) and host-memory latency become the bottleneck.

Legacy edge runtimes (Ollama, llama.cpp layer offloading) rely on static expert offloading: inactive weights sit in system RAM and get transferred synchronously when activated — on a cache miss, execution stalls completely. FreeToken replaces this with a dynamic co-scheduling scheme called the q* policy. On a cache miss the GPU keeps computing; token work is split between CPU cores and GPU tensor cores based on live interconnect throughput.

Three things that make it work

1. Bandwidth-adaptive execution (q* scheduling)

Instead of fixed offload rules (as in KTransformers), FreeToken computes a closed-form optimal allocation per layer in real time, splitting work across CPU and GPU dynamically.

2. FTW fast weight format + whole-layer double buffering

Weight transfers over PCIe overlap completely with execution of the active compute layers, hiding the transfer cost.

3. Semantic-anchor checkpoints (built for agents)

Coding assistants and autonomous agents constantly mutate prompts, append tool results, and generate reasoning blocks — so the prefix changes often. Standard engines drop the linear KV cache and pay for a full recompute. FreeToken caches intermediate attention states and recurrent activations at logical task boundaries, so when an agent edits a tool argument or injects an external result, previously computed subsequences are reused instead of invalidating the whole prompt cache.

Quick start

# Clone FreeToken (Linux / Windows, NVIDIA RTX 30/40/50-series)
git clone https://github.com/FlashML-org/FreeToken
cd FreeToken

# CLI: point it at a model and go — scheduling is handled automatically
# Example: run a 35B MoE model on 8GB VRAM
freetoken run --model Qwen3.6-35B

# Or use the desktop client from FlashML.ai to watch it work.
# Resident-expert and KV-cache allocations can be resized at runtime
# without reloading the model.

How it compares

  • Ollama / llama.cpp: tuned for GGUF quantization and per-layer offloading, but can't dynamically split sparse expert load between host and device. On the same MoE model, FreeToken decodes 3–4x faster and pre-fills 6–30x faster.
  • vLLM / SGLang: optimized for data-center throughput via PagedAttention and continuous batching; they assume high-bandwidth interconnects, not a heterogeneous memory hierarchy.
  • KTransformers: static CPU/GPU offload rules; FreeToken computes a per-layer optimal split at runtime.

Practice tips

The community is pairing used RTX 3090/4080 cards with standard DDR4/DDR5 RAM to self-host agentic inference, cutting recurring API costs to near zero and keeping code private. Caveat: the closed-form q* solution is sensitive to real-world CPU scheduling latency and memory contention, so benchmark it on your own model and agent workload before replacing a hand-tuned llama.cpp setup.

Resources

Scroll to top