The scaling playbook that remade language models has quietly hit a wall in video — and a Tsinghua-rooted team just published its way past it. Sand.ai has open-sourced MAGI-2-preview, the world’s first video generation model built on a hundred-billion-parameter Mixture-of-Experts architecture: 114 billion total parameters, roughly 6 billion active per forward pass, and a 10-second 1080P clip that costs about seven US cents to generate on eight H100s.
Both the code and the weights are public from day one. That alone makes it a landmark release; what makes it interesting is everything the team had to reinvent to get there.
The Impossible Triangle of Video Generation
Text models scale relatively cheaply because language is a sequence of discrete tokens: stack parameters, add compute, and the model keeps getting better. Video models confront an entire dynamic world. Every frame shatters into a swarm of spatial patches, consecutive frames must encode character motion, object relationships, and camera movement, and stacking text and audio on top multiplies sequence lengths all over again.
That produces the field’s core contradiction: the model needs to be bigger, the videos need to be longer, and the cost needs to stay survivable — a genuine impossible triangle. Keep scaling a dense architecture and training-plus-inference costs go vertical. Abandon scaling and you accept a hard quality ceiling.
The standard escape hatch is MoE, the Mixture-of-Experts design that keeps total model capacity enormous while activating only a sliver of it per token. Nothing about the idea is new. Making it work for video turned out to be far harder than anyone expected.
Why Video MoE Is Much Harder Than LLM MoE
MoE has been thoroughly proven in language models — DeepSeek and others run it at scale every day. Moving it to video means clearing three hurdles that language never posed.
First, communication. Conventional expert parallelism routes each token to its top-K experts, wherever those experts live across the GPU fleet. The more experts you activate, the more data crosses device boundaries. Video models already juggle super-long sequences, so traffic quickly swamps the actual expert computation it was meant to serve.
Second, training stability. Routing decisions that shift step to step, expert loads that swing wildly, and the inherently heterogeneous mix of text, video, and audio data — each is a known scaling headache on its own. Combined in a video MoE, they multiply.
Third, the architecture gap. A language model thinks in sequences of thousands of tokens. A video model thinks in sequences of millions. A naive transplant of an LLM’s MoE design collapses on cross-device communication and multimodal alignment at that length.
That is why almost nobody took this road. Sand.ai is the first team to carry a hundred-billion-scale video MoE from theory into a shipped, open model.
How MAGI-2-preview Cracks It
The answer is an end-to-end rebuild — architecture, systems, communication, and data, redesigned as one piece.
One stream for sight and sound
MAGI-2-preview extends the single-stream architecture Sand.ai validated in daVinci-MagiHuman: text, video, and audio all enter the same Transformer and exchange information directly inside every self-attention layer. The conventional design runs video and audio down separate channels and stitches them together with cross-attention at the end. Starting joint modeling from layer one yields far finer audio-visual correspondence, and a unified backbone means lower latency and maintenance cost than chained pipelines — which also makes it a natural fit for MoE scaling.
Expert routing at extreme granularity
The technical centerpiece is Multi-Head LatentMoE. The model splits its 3,072-dimensional hidden representation into twelve 256-dimensional heads, and each head routes independently. The same token gets examined in a dozen low-dimensional subspaces — one head may track a character’s appearance, another their motion and timing, each specialist doing its own job.
Across the 36-layer body, every layer holds 3,072 independent expert units. Each token selects 6 experts within each head — 72 small experts activated in total. For comparison, mainstream MoE giants like DeepSeek-V4-Pro still run on the order of hundreds of experts per layer. MAGI-2-preview pushes to the thousands: more experts, each owning a smaller, more sharply defined subspace, so the model can assemble a far richer set of capability combinations.
Infrastructure built to move 3,000 experts
Off-the-shelf MoE frameworks cannot schedule experts this fine, so Sand.ai built its own stack:
- MagiMoE kernel library — fuses routing, sorting, and expert computation into one pipeline, cutting intermediate VRAM transfers, with forward and backward passes tuned specifically for multi-head MoE.
- Head Parallel — distributes computation by head before routing ever happens, so devices exchange fixed-shape head representations instead of a fluctuating flood of routed tokens. Traffic stays constant regardless of how many experts activate, which is what finally takes long video sequences off the critical path.
- A hybrid optimizer — Muon for matrix parameters, AdamW for expert parameters, each class of weights updating on its own rhythm.
Data: organized, not filtered
Most teams process data by progressive filtering, ending with a clean but narrow corpus. Generative models need the opposite: diversity — the wider the world a model has seen, the more convincingly it can invent new ones. MAGI-2-preview’s pipeline first expands effective data volume and preserves breadth, then organizes it through precise annotation. Pretraining aims for full coverage; post-training handles preference alignment, controllability, and safety.
What an Open Hundred-Billion MoE Actually Changes
The economics first. With 114 billion total parameters and about 6 billion active per pass, renting eight H100s gets you a 10-second 1080P clip for roughly seven US cents — about a tenth of what mainstream models charge. On quality, MAGI-2-preview sits sixth on the AA video generation leaderboard, brushing the top tier on 6 billion active parameters alone.
The real shockwave is in the open-source ecosystem, in three directions.
For researchers, this is the first time anyone can dissect a hundred-billion-scale video MoE rather than squint at a paper: how do experts divide their labor, does routing stay stable, how is communication organized, do the scaling laws reproduce? These questions finally have a public specimen.
For enterprises, openness means an option for private deployment and domain fine-tuning. In finance, healthcare, and industrial settings, keeping data local and continuing to train on proprietary material matters as much as raw generation quality — sometimes more.
For the industry’s structure, the open-versus-closed comparison is being redefined. It used to be a pure quality contest; now whether a model keeps training, and whether its interface answers to a single platform, are part of the score. If open models keep closing the quality gap, video generation will likely retrace the language model path: closed models compete on product experience, open models win on deployment control.
Redrawing the Triangle
MAGI-2-preview’s answer to the impossible triangle is architectural: change the triangle’s boundaries instead of picking two sides. MoE is not a new idea — but hundred-billion scale, video, and full openness had never been combined, and the team that combined them has form. Founder Cao Yue co-first-authored the Swin Transformer, and Kai-Fu Lee has called Sand.ai “the DeepSeek of AI video generation.” From autoregressive video to joint audio-video generation to an open MoE, this group keeps taking the road with no footprints on it.
Honest caveats apply. This is a preview, and the team itself notes that 6 billion active parameters does not price like a 6-billion dense model — expert communication, routing, memory, and deployment all add overhead. The preview still has to give way to a formal release.
But one claim is already settled: hundred-billion-scale MoE no longer belongs to language models alone. The weights are on GitHub — the dissection can begin.
