Every CLAUDE.md starts small. Then someone adds a rule after one failed run. A month later, another rule. Six months in, the file has tripled, and nobody dares delete a single line — because deleting means proving the rule no longer matters, and that proof is expensive. This isn't a discipline problem. It's now a measured phenomenon with a name: catastrophic remembering.
Independent researcher Kushal Chakrabarti's paper Why Does CLAUDE.md Keep Growing? Catastrophic Remembering in Agentic Coding (arXiv:2608.11095) tracked 247,694 instruction lifetimes across 1,867 GitHub repositories — and quantified what every developer has felt: agentic instruction files like CLAUDE.md, AGENTS.md and copilot-instructions.md grow by 226% on average over their lifetime, adding a net 4.9 instructions per commit. And the older an instruction gets, the less likely it is to ever be deleted.
The Asymmetric Cost Behind the Bloat
Why does a CLAUDE.md only ever grow? The paper's answer is brutally simple: adding is cheap, deleting is expensive.
Appending a new instruction costs almost nothing — a one-line note after a failure, instantly verifiable. But months later, deleting that rule requires answering a chain of questions: Was the original failure a fluke or a pattern? Do newer rules already cover it? Are there invisible interactions with other instructions? The paper formalizes this: auditing one instruction requires reconstructing its "latent reasoning" — the context of why it exists — and without that context, verification costs are exponential (O(2^|D|) subset probes in a prompt of |D| instructions). With the rationale recorded, the cost drops to O(1).
The data confirms the mechanism: the longer an instruction survives, the more likely its author has left and its context has faded. Deletion hazard actually declines with age (log-hazard −0.032 per commit) — the opposite of what natural staleness would predict. Old rules aren't removed because their justification has evaporated. The rule stays; the reason is gone.
Catastrophic Remembering, the Mirror Image of Forgetting
Machine learning has long studied catastrophic forgetting — a model overwrites what it should have kept. Chakrabarti names the agentic-coding mirror: a maintainer keeps what they should have overwritten. Both fail the same way: the information that would license an update is missing.
This also explains why the common escape hatch — deleting the whole file and rewriting from scratch — doesn't work. Maintainers who can't prune a single rule will happily nuke the entire file. The study found rewrite-wholesale doesn't stop the cycle: the same growth pattern returns, because the underlying incentive structure never changed. And the problem compounds inside agent workflows: agents that rewrite their own instructions, as explored in DeepSeek's self-evolving agent research, inherit whatever fragility the file already contains.
The Fix: A Comment That Says Why
The paper's solution is almost anticlimactic: annotate each long-lived instruction with a comment that records why it exists. Not "useful." Not "I want to add." A real comment: what failure triggered the rule, what hypothesis the author held, and whether the rule actually fixed it.
The results are dramatic. In controlled experiments, structured comments encoding this latent reasoning compressed prompt excess from 211.3% down to 1.4% — eliminating roughly 99.3% of redundant instructions. And shorter files didn't hurt performance: with realistic, noisy instructions, adding effective comments improved instruction-following accuracy by up to 23.1%. Rules aren't a "more is better" game; irrelevant, duplicated or conflicting instructions actively distract the model. Comments give maintainers the evidence to cut the noise.
The experiment also found what doesn't work: placeholder comments ("I think so." / "works.") perform identically to no comment at all, and half-recorded attempts can actively mislead — a note that says "suspect token truncation, tried max_tokens 2048→4096" without recording the outcome turns an unverified guess into a false convention for the next maintainer.
The Deeper Problem Is Tooling, Not User Discipline
Read the paper carefully and the sharpest point isn't about comment hygiene. It's that the entire agent-tooling stack lacks maintainability primitives. Code has comments, version history, tests, and architecture decision records. A long-lived agent instruction has none of that — no author, no trigger event, no validation result, no expiry condition.
AI vendors keep telling users to keep prompts short while giving them no basis to decide what to delete. "Keep it short" is a demand without evidence. The toolchain should surface stale rules proactively: "This rule hasn't fired in six months, its dependency has been replaced, and tests pass without it — review for deletion?" Until then, every maintainer rationally chooses "keep it, don't risk it," and the aggregate result is a file that only grows. This is the same class of gap as Anthropic's multi-agent experiments — we're discovering agent behavior and maintenance rules empirically, after the systems shipped.
What You Can Do Today
- Comment every long-lived rule. Three fields: why it was added, what problem it solved, and whether it's still validated. That's the entire fix.
- Audit by age. The oldest instructions are the least trustworthy — their rationale has had the most time to rot. Run a "why does this exist?" pass on anything older than a quarter.
- Treat instruction files like code. Review changes, keep a changelog, and add a test or check that exercises the rule. A rule you can verify is a rule you can delete.
Catastrophic remembering is the first named, measured disease of agent memory. The cure is boring: write down why. The deeper lesson is that agent tools won't reach production maturity until remembering — and forgetting — are first-class features.
