A fashionable narrative is quietly taking root: AI writes code ever better, and the programmer’s value is being eroded. The narrative reactivates with every GPT release — GPT-5.6 just shipped, and the same voices returned. But look closely at one concrete case, and you find the opposite story.
James, an engineer at PostHog, rewrote the company’s SQL parser with Claude. The result: 16K lines of parser code, 5K lines of tooling, a 70× performance improvement (454× measured in production), zero divergence on all real queries.
But the notable thing is not the numbers. It is that he wrote almost none of the code. He did not accomplish the rewrite by “writing code” — he did it by “building a verification loop.” That is precisely the structural shift underway in what programmers do: from hand-writing code to building verification systems.
Why PostHog Needed a Rewrite — and Why It Couldn’t Before
PostHog is an open-source product-analytics platform where users query data by writing SQL directly. Underneath sits ClickHouse, and PostHog must transpile in between — converting user SQL into SQL ClickHouse understands.
That requires a SQL parser. Every query the parser processes is untrusted input, and the downstream access control and performance optimization all depend on the AST (abstract syntax tree) it produces. If the parser is wrong, the foundation of the whole system is wrong. PostHog previously used an ANTLR-generated parser. ANTLR is an open-source parser generator — you define grammar rules and it generates the parser code. Powerful and flexible, at the cost of performance: ANTLR compiles grammar rules into an ATN (augmented transition network) and walks it at runtime with a generic interpreter. Every parse goes through abstraction and indirection; there is no hand-written parseExpression() — everything is generic graph traversal.
Before AI, hand-writing a parser was simply unrealistic — months of work, brutal maintenance, far from worth it for a performance win. ANTLR traded runtime speed for development efficiency: the classic trade between generic tooling and specialized implementation. AI tilted the balance.
The Verification Loop: the Programmer’s New Working Method
James’s rewrite was, at its core, a continuously iterating verification loop:
Generate test cases → find divergences → fix the parser → run regression → repeat
The loop contains three layers. Layer one: oracle comparison. James had a “ground truth” — the existing C++ ANTLR parser. He made it the oracle, and the new parser’s goal was exact agreement on every real query. Simple to state, nontrivial to execute: divergence points do not surface themselves; you must go find them. Layer two: automated divergence discovery. James used four methods: a regression test suite for fast early validation; property-based testing (PBT) with the Hypothesis library to auto-generate SQL statements hunting for disagreements with the oracle; anonymized queries sampled from production logs; and “think hard about edge cases” — Claude agents proactively generating nasty test cases in the background. PBT was the core weapon: he even wrote a tool that generates SQL generators from ANTLR grammar files — “to write a SQL parser, first write a parser for .g4 grammar files,” he noted with a laugh.
The key insight: he let PBT run continuously in the background, appending new failing cases to a file. Whenever Claude idled, it pulled those cases and fixed the parser. His CPU ran PBT at full load while Claude’s inference ran at full load fixing the parser — two production lines running simultaneously. Layer three: coverage-guided generation. Later, James added coverage-based test generation — the generator identified which grammar structures remained uncovered and targeted them. Not strictly necessary (production accuracy was near 100% already), but it surfaced several extremely subtle edge cases.
The final loop: generate failing cases (PBT + real corpora + regression + edge-case thinking) → minimize and add to regression suite → think through the best fix, preferring general solutions → implement, producing a summary for human review → run regression → loop automatically.
The Verification-Loop Framework: the Programmer’s New Capability Model
From the PostHog case, a general frame abstracts out — the Verification Loop — in three layers from basic to advanced. Layer one: oracle definition. You must have a standard of “correct”: an existing system, human annotation, or a theoretical model. No oracle, no verification; no verification, no control over AI output quality. Layer two: automated divergence discovery. Never count on eyeballs to find every difference — search systematically with property-based testing, fuzzing, and coverage guidance. The ROI beats manually reviewing AI output by a wide margin. Layer three: loop autonomy. Chain divergence discovery and fixing into an automatic cycle. The human shifts from “reviewing every line of code” to “reviewing the design of the loop itself,” making the key calls: are the test cases sufficient? Is the fix general? Is the loop converging?
James’s own summary is precise: “While I didn’t write any of the code by hand, I would never call this vibes-based coding. My PBT configuration generates inputs from grammar files and uses coverage-guided generation — this is near state-of-the-art for parser fuzzing.”
The Framework’s Reach Beyond Parsers
The verification loop is not exclusive to parser rewrites — it applies to nearly every scenario where AI generates code. Code refactoring: to have AI rewrite an old Python system in Go, generating directly yields code that “looks right but may not be.” Run the loop instead — the old system’s output and behavior serve as the oracle; build automated diff tooling comparing old against new; let AI fix until every test passes. AI-assisted unit testing: test generation is the verification loop’s native application — AI writes cases, you run the suite, find uncovered branches, and have AI fill them. Multi-agent collaboration: when agents work together, the verification loop becomes “consensus checking between agents” — each agent’s output serves as another’s oracle, and divergence points are the conflicts. This is exactly what the CoAgent framework tackles.
One frame, three layers, many scenarios. The verification loop is not PostHog’s private invention — it is becoming the foundational paradigm of software engineering in the AI era.
What Programmers Should Do
Front-line developers: stop spending energy reviewing every line of AI code — that is the inefficient path. Start building verification tooling: test frameworks, diff tools, property-based tests. Your core output is no longer code; it is the confidence that this code is correct. Engineering managers: do not measure team efficiency by AI-generated line counts. Build a “verification-loop maturity” metric — does the team systematically construct oracle, divergence discovery, and closed loops on every project? Invest in test infrastructure; its return beats buying more AI credits. Architects: make verifiability the first design priority. If a module’s output cannot be automatically verified, it should not be AI-generated. Consider shadow mode — run the AI-generated implementation in parallel with the old system and compare output differences.
This time, James did in days what once took months. He wrote no code — he built a system where AI generates code under correct constraints, automated tests guarantee quality, and the verification loop drives convergence. AI has not made programmers obsolete. It has promoted them from writing code to building loops — a more senior job. Programmers who master verification loops have not lost value; leverage has multiplied it.
Source: PostHog Engineering Blog — I wrote a 70x faster SQL parser while barely looking at the code
