While most labs are still arguing about whether to release model weights, Stanford's Marin project did something stranger: it started training a 535-billion-parameter model in front of everyone — code, data recipe, loss curves and GitHub issues included — with no guarantee it won't fail halfway through.
Percy Liang, the Stanford CS professor and Together AI co-founder behind the project, announced the launch of Marin 535B-A23B on X last week. The post passed 800,000 views and drew a rare public endorsement from Andrew Ng, who called the project "a precious demonstration" of AI openness and reminded the field that publicly releasing AI research "used to be the industry norm." For the launch itself, see our earlier coverage; this piece goes deeper into the mechanism — the MoE math, hand-written expert parallelism, and the scaling ladder that turn a public run into a controlled experiment.
The thing being opened is not just the model. It's the recipe — and the process itself.
What an "open lab" actually looks like
Marin runs on an open lab mechanism. Every experiment starts as a GitHub issue declaring its hypothesis and goals before training begins. Configurations are submitted as code and pull requests, open for external researchers to review. Once a run starts, W&B training metrics go public. Successes, failures, and mid-course corrections are all logged as they happen. The data, code, recipe, and final weights stay open afterward.
The numbers for this run: 18.75 trillion tokens (~80% pretraining, 20% mid-training), 11 NVIDIA GB200 NVL72 systems, roughly three months, and about 2.7×10²⁴ FLOPs in total — after which the project moves to post-training.
The MoE math: why 535B is not 535B
Marin 535B-A23B is a mixture-of-experts model. The "535B" is total parameters; the "A23B" means only ~23 billion parameters are active for any given token. A routing module decides which experts handle each token — which is exactly why MoE became the mainstream route again: total capacity can keep growing without per-token compute growing in lockstep.
The design details matter. Each layer keeps 2 shared experts plus 8 routed experts, all half-width, with routed experts additionally 2× compressed. Roughly a third of expert compute comes from always-on shared experts — a deliberate choice to lower the risk of token dropping, the silent killer of MoE training quality.
Token dropping is the load-balancing tax: when a batch's tokens land unevenly across experts, capacity limits force some tokens to be dropped rather than fully processed. Marin's own runs saw dropping climb from ~7% to ~40% when context length went from 4K to 65K — because with a fixed token batch, longer contexts mean fewer independent sequences per batch and worse expert load balance. That's why Marin 535B starts at 4K context, and why their experimental pooled/wave expert parallelism cut dropping to ~3% at 4K. They openly admit it may spike again at 65K.
The infrastructure gamble: hand-written EP on GB200
MoE training bottlenecks are rarely raw FLOPs. They're all-to-all communication, expert load imbalance, and memory access. When a token is routed to an expert sitting on another GPU or rack, the system must ship the token there and ship the result back.
Marin's 8B and 32B runs lived on Google TPUs. The 535B run moves to NVIDIA GB200 NVL72 GPU clusters — so the team hand-wrote an expert parallelism implementation for JAX/XLA/Levanter, having found no production-ready option in that stack. GB200 NVL72 packs 72 Blackwell GPUs and 36 Grace CPUs into a rack-scale NVLink domain, which is exactly the shape of hardware a communication-heavy MoE wants. On the other side of the hardware race, OpenAI's first chip, Jalapeño, shows how the inference tier is being reinvented too.
The scaling ladder: scaling laws as a fault-detection system
Marin didn't pour all its compute into 535B directly. It first trained a four-rung scaling ladder of smaller MoE models — from 1.6B total params (61M active) up to 27.7B (~1.2B active) — for roughly 1% of the final compute.
The ladder does three jobs: it predicts what loss the 535B run should hit at each stage (so a deviation flags a problem early, instead of after trillions of tokens); it exposes stability issues at low cost (earlier ladder runs found gradient norms climbing past 4 on long schedules, leading to logit z-loss to prevent divergence); and it separates normal fluctuation from pre-divergence warning signs.
There's also a stated contingency plan: if problems hit within the first 25% of the token budget, the team may shorten the final token count and re-tune learning-rate decay and data mix — rather than mechanically pushing through.
From open weights to open process
This is not the first public training run. BigScience's BLOOM (2022) streamed TensorBoard logs; Pythia, LLM360, and Ai2's OLMo opened data, code, logs, and checkpoints. Marin's announcement explicitly credits these predecessors — EleutherAI, AI2, Hugging Face, BigScience, BigCode, LLM360 — and, alongside heavy open-weight bets such as Nvidia's $6B open-model push, says it wants to go one step further: making openness the lab's default working mode, from hypothesis to failure, rather than a cleaned-up post-mortem after training finishes.
The software analogy is precise. On GitHub, developers can read issues, submit code, review PRs, and reproduce bugs. Foundation-model experiments today run in closed clusters; outsiders see finished weights and a technical report — not the decisions, not the failed runs. Marin is attempting to import the GitHub workflow into model training itself.
What it means if it works (or fails)
Open weights answered "who can use the model." Marin is asking "who gets to know how the model was made." If the run succeeds, the open community gets a rare large-scale MoE training specimen. If it fails mid-run, the public failure path is itself research value.
Either way, the transferable lessons — expert routing behavior, token-dropping dynamics, gradient anomalies, long-context scaling, JAX expert parallelism on GB200 — are reusable at any model size. For teams that can't afford 2.7×10²⁴ FLOPs, this is a free course in how frontier-scale training actually behaves.
Keep the caveats on the table: openness is not the same as frontier performance. A 535B MoE is not equivalent to a 535B dense model; pretraining loss only measures fit to data; code, math, tool use, and agent capabilities depend heavily on mid-training and post-training. The project is early — even the token count is still being aligned (the GitHub title says 18T; the announcement says 18.75T).
What to do with this
- If you train or fine-tune MoE models: watch the token-dropping numbers and the pooled/wave EP approach. The "start at 4K, extend later" strategy is a cheap, immediately usable lesson.
- If you're a researcher without big compute: the scaling-ladder methodology replicates at small scale — gradient-norm monitoring and z-loss cost almost nothing and catch divergence early.
- If you run an open-source AI organization: the open-lab workflow (issues → PRs → public metrics) is a concrete template for transparency beyond weights.
- Everyone else: bookmark the GitHub repo and the W&B dashboards. Watching a ~100-day run unfold in public is the closest thing to a live case study of frontier training economics.
FAQ
Q: Why is watching the Marin 535B run worthwhile?
A: It is the first frontier training run treated as an open experiment — 18.75 trillion tokens, 11 GB200 NVL72 systems, roughly three months, about 2.7×10²⁴ FLOPs. Success yields a large-scale MoE specimen; a mid-run failure leaves a public failure path. Either way, it is research value.
Q: How is a 535B MoE different from a 535B dense model?
A: 535B is total parameters; A23B means only about 23 billion activate per token, so per-token compute stays far below a same-size dense model — exactly why MoE became mainstream again.
Q: What can I use in my own training right away?
A: Three things: start at 4K context (dropping falls from about 40% to about 3%), run a scaling ladder first (about 1% of final compute), and monitor gradient norms plus logit z-loss to prevent divergence (near-zero cost).
