Bottom line: Stanford professor Percy Liang has kicked off a fully public training run for a 535-billion-parameter model. Live loss curves, training-data mixes, configs and engineering logs are open to anyone. This is the first serious attempt to make frontier-scale training an observable science — and the beginning of the end for the training black box.
Percy Liang, Stanford professor and founder of Simile AI, has launched the hero run for Marin 535B-A23B, a Mixture-of-Experts model with 535B total parameters and 23B activated. The entire process is being streamed in public: real-time loss curves on Weights & Biases, data composition, training configuration, and engineering logs published as they happen. For anyone who has ever wondered what actually goes on inside a GPT-4-class training run, this is the first time the answer is displayed in the open instead of sealed inside a lab.
The Plan: A 535B MoE, Trained in the Open
Marin 535B-A23B is not a small science project. The lab prepared 18.75 trillion tokens of training data and deployed 11 GB200 NVL72 systems — roughly 792 GB200 GPUs. The run is expected to last about three months and consume around 2.7e24 FLOPs, followed by a post-training phase. In other words: for the next few months, anyone can watch a frontier model grow from zero — data in, loss curves out.
The Scaling Ladder: Four Small Runs Before the Hero Run
Before the hero run, the team trained a four-stage Scaling Ladder, from a 1.6B-A61M model on 48B tokens up to a 27.7B-A1.2B model on 926B tokens. Percy Liang gives two reasons: first, catch and debug problems at small scale before spending real money; second, use small-run behavior to calibrate predictions for the main run. He is also honest about the limits of prediction — this is the largest training run his team has done, and he expects unexpected things to happen. The ladder is the closest thing yet to a public, reproducible methodology for de-risking a frontier run.
What Open Training Actually Reveals
For GPT-4, Claude and Gemini, outsiders see only final benchmarks and the occasional paper. Data mixing ratios, failed runs, effective hyperparameters, loss trajectories, whether the scaling laws held — all unknown. Marin publishes them: the data mix and how each domain enters the model, training configs, code, experiment design, live loss, model state, and stage-by-stage predictions. Even the small artifacts become teaching material. Community member James Thewlis noticed a norm spike around step 500 and traced it to router_bias, the MoE token-balancing heuristic — a parameter not trained by gradients. That question-and-answer loop is the whole point: training becomes discussable and collaborative, like open-source software or a published paper.
Why This Is a Structural Shift, Not a Stunt
Three reasons this matters beyond the spectacle. First, observability turns scaling claims into checkable facts — if an extrapolation is wrong, everyone sees it in real time. Second, transparency is a precondition for meaningful AI governance: you cannot audit what you cannot see. Third, live training is apprenticeship at unprecedented scale; thousands of engineers can learn how an 800-GPU run actually behaves. Most fundamentally, it resets what open means — not just released weights but an open process. When a roughly three-month, multi-million-dollar compute run is operated in public on purpose, openness stops being a PR posture and becomes the strategy itself. It is the natural next step after the industry-wide shift to open-weight models, such as Nvidia’s $6 billion move into the open-weight race. Our mechanism deep-dive breaks down the MoE math, hand-written expert parallelism, and the scaling ladder.
What You Can Do Now
- Watch the run: live curves on W&B at wandb.ai/marin-community/marin_moe, the data-composition overview, and the GitHub thread at github.com/marin-community/marin/issues/8435.
- Study the artifacts: the router_bias norm spike around step 500 is a ready-made case study in MoE dynamics.
- Copy the playbook: before a large run of your own, run a small scaling ladder first — the four-stage approach transfers to any compute budget.
- If you run a lab: treat process transparency as governance and recruiting infrastructure, not a cost.
Q: Can I really watch a 535B model train live?
A: Yes. The Marin lab publishes real-time training curves on Weights & Biases (wandb.ai/marin-community/marin_moe), plus a data-composition overview and a GitHub thread with training details. The run lasts roughly three months.
Q: What exactly is Marin 535B-A23B?
A: A Mixture-of-Experts LLM with 535B total parameters and 23B active parameters, trained on 18.75 trillion tokens using 11 GB200 NVL72 systems (about 792 GB200 GPUs) — roughly 2.7e24 FLOPs of compute.
Q: Why is fully open training a big deal?
A: Frontier labs keep data mixes, failed runs and hyperparameters secret; you only see final benchmarks. Marin publishes the process itself — data, configs, code and live loss — turning training into an observable, auditable science.
