TokenSpeed: A Speed-of-Light LLM Inference Engine for Agentic Workloads

TokenSpeed is a speed-of-light LLM inference engine designed from first principles for agentic workloads. Agentic workloads typically consist of many successive model calls within a single task, placing heavy demands on scheduling, parallel execution, and the management of intermediate state such as the KV cache. TokenSpeed is built around these demands: it incorporates a compiler-backed modeling mechanism for parallelism, a high-performance scheduler, a safe KV resource reuse restriction, a pluggable layered kernel system that supports heterogeneous accelerators, and SMG integration.