Ornith released three open-weight coding models this month — a 397B MoE flagship, a 35B-A3B, and a 9B dense — trained with a loop no public release has shown before: the model generates its own training tasks, builds its own agent scaffolding, and runs its own solution trajectories, all optimized together by reinforcement learning. On Terminal-Bench 2.1, the flagship's self-measured score climbed from 77.5 to 86.1 in two months, above the 85.0 Ornith recorded for Claude Opus 4.8 on its own setup. Treat the headline number with context, but the mechanism behind it — automating the training-data bottleneck — is the real story, and it points to where the next open-vs-closed contest will be fought: the question bank.
The bottleneck isn't compute. It's training questions
Training a model that can actually do coding and agent work costs more than GPUs — it costs training tasks. A usable question needs a runnable environment, a verifier that judges right from wrong, protections against the model digging up old submissions, and difficulty set exactly one notch above what the model already knows. Too easy, and the model learns nothing; too hard, and not a single successful trajectory survives to reinforce. Then you need enough volume to actually move the needle.
Humans have been doing this job, and it simply cannot keep up: by the time a batch of tasks is curated, the model has already finished eating through it. That supply lag is one reason high-quality agent-training data is the scarcest input in the industry right now — arguably scarcer than compute, at a moment when the price of inference is going up, not down.
Ornith's answer: let the model set its own homework.
A closed loop: generate the task, build the scaffold, run the trajectory
The June release, Ornith-1.0, already did something unusual: it turned agent scaffolding — the workbench a model uses while solving a task — into a learnable object. Each reinforcement-learning step had two stages: first mutate the scaffolding based on the task and the previous scaffold version, then run solution trajectories on top of it. Reward flowed back to both stages. The model learned to build its own workbench, but humans still supplied the questions.
Ornith-1.5 takes over that last human job. Given a codebase, a high-level description of the task type, and the model's own past solution records, the system generates a batch of new questions pitched deliberately harder than what the model has already solved — aimed at its own capability gaps. It then generates or improves task-specific scaffolding: instructions, tools, decomposition strategy, orchestration logic. Finally, it runs solution trajectories under both the task and the scaffold, and the reward propagates back to all three stages, updated together with GRPO. Task generation, workbench construction, and problem-solving are now one closed loop — and the difficulty curve climbs itself, without a human tuning knob.
This is worth contrasting with the ongoing fight over whose agent scaffolding wins in production: the same scaffolding question that vendors are fighting over in the field, Ornith has moved inside the training loop itself.
Three reward signals — why the model can't grade its own homework
The obvious cheat: generate easy questions, inflate the score, learn nothing. Ornith blocks this by multiplying three signals — validity × frontier difficulty × novelty. Multiplication is the key design choice: if any term approaches zero, the entire reward for the question collapses to zero.
Validity is a hard gate. Does the scaffold actually run? Does a high-confidence correct solution pass? Does an obviously wrong solution fail? Does the judging logic match the task description? Anything that fails the gate scores zero — this filters out "looks hard but can't actually be judged" junk.
Frontier difficulty is the clever part. The system samples several trajectories per question and computes the empirical success rate; reward peaks near a 20% target. A question the model solves eight times out of ten is already learned — it has no training value. A question it never solves yields no successful trajectory, so reinforcement learning just spins in place. The target is a moving one: as a class of questions gets mastered, the reward for that class decays automatically, and the generator must keep raising the bar. The curriculum calibrates itself against the model's current ability.
Novelty carries the lowest weight and exists only to de-duplicate near-copies of existing questions — the team explicitly notes it is not a reward for weird questions.
Read the 86.1 with a grain of salt
The headline number is self-measured: a five-run average on the team's own environment and configuration, including a four-hour timeout, 32 cores, 48GB of RAM — and modifications to configuration fields the official leaderboard locks down. The official Terminal-Bench 2.1 leaderboard (17 entries as of August 19) does not include Ornith at all. On that board, Claude Opus 4.8 with Claude Code sits at 78.9 (5th place), and the top entry is Claude Code + Fable 5 at 83.8. The same Opus 4.8 scores 85.0 in Ornith's table and 78.9 on the official one — a 6.1-point gap. Agent benchmarks measure the combination of model, scaffold, timeout, context, and resources; change the rig, and you change the meaning of the number.
License terms deserve the same precision. The MIT label covers the weights. The public GitHub repository currently holds a README, the license, and showcase materials — no full training implementation, no task-generation dataset, no reproduction evaluation scripts. Open weights are not the same as open everything.
The quiet stars of the release are the middle sizes. The 35B-A3B activates only about 3B parameters per token yet scores 67.8 on Terminal-Bench 2.1 (Gemma 4-31B: 42.1, Muse Glimmer-30B: 51.7) and 79.0 on SWE-bench Verified (vs 52.0). The 9B dense — 5.63GB compressed, with an MLX build for Apple Silicon — leads its size class on coding and reasoning but loses on long tool chains: MCP-Atlas 54.2 vs 55.0, Toolathlon-Verified 41.2 vs 52.8. Training method can compensate for parameter count on pure coding; it cannot yet compensate on multi-step tool calling. This is the same compute-efficiency pattern reshaping inference economics elsewhere, from custom chips designed for inference to cheaper per-token serving.
The next contest is the question bank
For two years, open models chased closed flagships along two paths: scaling parameters and copying recipes. Both are, in effect, copying someone else's answer sheet. Ornith has opened a third path — write the exam. With data production moved inside the reinforcement-learning loop, the model that keeps generating better questions keeps its own fuel supply flowing. The upper bound on a model is no longer just compute; it is the quality of the curriculum it trains on.
Three practical takeaways:
- For anyone reading benchmarks: treat self-reported agent scores as directional, not comparable, until they appear on an independently verified leaderboard with locked configuration.
- For labs: task quality is becoming a durable asset in the way pretraining data was — the curriculum is becoming the moat.
- For users: the most cost-effective pick today is the 35B-A3B tier — near-flagship coding ability at ~3B activated parameters, cheap to self-host.
What remains for humans in the loop is the syllabus: defining which task types are worth practicing, judging validity, and deciding what the model should learn next. The loop is closed. The curriculum is not.
FAQ
Q: Is Ornith 1.5 actually better than Claude Opus 4.8 at coding?
A: Not proven. The 86.1 is self-measured with modified configuration; the official Terminal-Bench 2.1 leaderboard (17 entries as of August 19) has no Ornith, and Opus 4.8 + Claude Code sits at 78.9 there. Treat the claim as directional until an independent run appears.
Q: What does "open source" mean for Ornith 1.5?
A: The weights are MIT-licensed, but the training implementation, the task-generation dataset, and reproduction scripts are not public yet. Open weights ≠ full open source.
Q: Why does Ornith target a 20% success rate for generated tasks?
A: A question solved 8/10 times teaches nothing new; one solved 0/10 produces no successful trajectory to learn from. Around 2/10 sits at the informative edge — and the target rises automatically as the model improves.
