Kimi K3 Deployment Guide: Cluster Configs and Inference Tuning for the 2.8T Open Model

On July 27, Moonshot AI pushed Kimi K3’s complete weights onto Hugging Face as promised: 2.8 trillion parameters, MoE architecture, 896 experts, native million-token context — the largest open-weight model in the world, full stop.

Three numbers matter most for developers: native MXFP4 quantization, a 1.4TB minimum memory footprint, and Day-0 vLLM/SGLang support. K3 is not a look-at-the-paper model — it is an inference system you can put into production. This guide skips the benchmarks and goes straight to running it.

K3 Architecture in Brief: Why Deployment Logic Differs

Three facts about K3 drive your cluster configuration:

1. Total parameters vs active parameters

Total is 2.8T, but each token activates only 16 of the 896 experts — roughly 104B active parameters. The catch: all 896 experts’ weights must sit in memory, because you never know which ones the router will pick.

2. MXFP4 is native, not post-hoc

K3 went through quantization-aware training from the SFT stage onward. MXFP4 is not a post-quantization artifact — it is the factory setting. No separate calibration pass is needed.

Precision	Bytes/param	Weight memory
BF16/FP16	2	~5.6 TB
MXFP8	1	~2.8 TB
MXFP4 (recommended)	0.5	~1.4 TB

3. Attention: KDA + AttnRes

KDA (Kimi Delta Attention) is a hybrid linear attention; combined with Attention Residuals for cross-layer retrieval, the cost of million-token contexts stops scaling quadratically with sequence length. vLLM and SGLang completed their adaptations before July 27.

Put the three facts together and K3’s deployment profile reads unlike a dense model of similar capability. A dense 104B model would be trivial to host; a dense 2.8T model would be impossible. K3 sits in the awkward middle — its serving cost tracks the 104B active parameters while its memory requirement tracks the full 2.8T, which is why every configuration question below is really a memory-capacity and interconnect question rather than a compute one. Plan for the weights first, the traffic second.

Three Cluster Configurations

All three plans assume MXFP4 (the recommended precision).

Plan A: 8× B300 (minimum viable)

# Single node, 8x B300 (192GB VRAM per card)
# Theoretical capacity: 8 x 192 = 1536 GB > 1400 GB needed
# Leave headroom for KV cache and activation memory

# vLLM launch example
python -m vllm.entrypoints.openai.api_server \
  --model moonshotai/Kimi-K3 \
  --tensor-parallel-size 8 \
  --expert-parallel-size 8 \
  --max-model-len 131072 \
  --dtype auto \
  --gpu-memory-utilization 0.95 \
  --trust-remote-code

Plan B: 16× B200 (production recommended)

# Two nodes, 8x B200 each (144GB VRAM per card)
# Theoretical capacity: 16 x 144 = 2304 GB — comfortable headroom
# Supports larger batch sizes and longer context windows

# SGLang launch example
python -m sglang.launch_server \
  --model-path moonshotai/Kimi-K3 \
  --tp 16 \
  --ep 16 \
  --context-length 524288 \
  --quantization mxfp4 \
  --trust-remote-code \
  --host 0.0.0.0 \
  --port 8000

Plan C: 1× GB200 NVL72 (full power)

# One rack, 72 cards; BF16-equivalent compute ~2.5x per H100
# Can run BF16 directly, no quantization downgrade
# Handles 1M context plus heavy production batch traffic

# Three-tier parallelism: expert + pipeline + tensor
python -m vllm.entrypoints.openai.api_server \
  --model moonshotai/Kimi-K3 \
  --tensor-parallel-size 8 \
  --pipeline-parallel-size 3 \
  --expert-parallel-size 72 \
  --max-model-len 1048576 \
  --dtype bfloat16 \
  --enable-chunked-prefill \
  --gpu-memory-utilization 0.9

Which plan fits depends on what you are honest about regarding traffic. Plan A proves the pipeline works and serves light internal use. Plan B is where most production deployments will land — two nodes is a materially simpler procurement than a rack, and 512K contexts cover the overwhelming majority of real workloads. Plan C is for the specific case where million-token contexts or very high concurrency are the product, not an option. The jump between tiers is a jump in cost discipline, not just capability.

Three Open-Source Infrastructure Components

K3 shipped with three open-sourced training/inference infrastructure pieces:

MoonEP

A high-performance communication library for hyperscale fine-grained MoE. Under expert parallelism it maintains peak communication efficiency even with imbalanced loads — and it ports directly into your own MoE deployments.

git clone https://github.com/MoonshotAI/MoonEP
# Integrate as a custom expert-parallel backend in vLLM

FlashKDA

High-performance CUDA kernels for KDA: on H20, prefill runs 1.72–2.22× faster than the flash-linear-attention baseline. Drops in as a replacement backend.

AgentEnv

A sandbox system built with KVCache.ai providing high-fidelity isolation for large-scale agent training — snapshots, restore, and fork for parallel agent workflows.

# AgentEnv sandbox example
# Isolated Python runtimes with fast snapshotting
pip install agent-env
agent-env run --snapshot --max-concurrency 128

Calling the API

Once deployed, K3 exposes a standard OpenAI-compatible API:

# Text inference
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Kimi-K3",
    "messages": [{"role": "user", "content": "Write a Python function implementing quicksort"}],
    "max_tokens": 1024,
    "temperature": 0.7
  }'

# Multimodal (image understanding)
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Kimi-K3",
    "messages": [
      {"role": "user", "content": [
        {"type": "text", "text": "What is in this image?"},
        {"type": "image_url", "image_url": {"url": "https://example.com/photo.jpg"}}
      ]}
    ],
    "max_tokens": 512
  }'

Practical Advice

Don’t run BF16. A 5.6TB memory requirement means at least 30 H100s — terrible value. MXFP4 is the official recommendation with QAT baked in, so precision loss is minimal.
H20 users: Huawei Ascend’s CANN stack shipped Day-0 native MXFP4 support on July 27 — see the Ascend 950PR/DT deployment docs.
Trim context length to fit. If you don’t need 1M context, set max-model-len to 131072 and KV cache memory drops to an eighth.
MoonEP + FlashKDA deserve separate attention: neither is bound to K3 — both port into any MoE model’s deployment pipeline.