Causal inference is the last place most teams trust an LLM. One confounder you didn't control for and your "effect" is noise — and unlike a code typo, a bad estimand doesn't crash. It quietly misleads the business. That's exactly why this week's release from Netflix is worth your time: oci-agent (Netflix-Skunkworks/oci-agent) is a reference implementation of an agentic workflow for observational causal inference (OCI). An actor agent fills templated Jupyter notebooks from a human-written analysis plan and executes them; a critic agent audits the results and demands revisions; the loop repeats until the report passes. It's not production-grade, but it's the clearest public blueprint yet for automating the toil of causal analysis without surrendering human judgment.
Why this pattern matters
Netflix frames OCI as target trial emulation — finding the optimal A/B test for your question using observational data. Their case study makes the point: estimating the retention impact of a new entertainment type, the workflow produced an effect estimate just 25% of a naive Claude linear-regression baseline, because the critic flagged early-adopter bias and a failed placebo test before anyone shipped the number. The scaffolding is the product.
Core design: the actor-critic loop
The pipeline is a single loop with inspectable artifacts at every step:
plan → actor.draft → spec → nb_runner → results.json → critic.evaluate → oci_report.md
↑ │
└────── actor.revise ◀────────── critique.json ◀──────┘Three moving parts, all under oci_agent/:
- Actor (
actor.py) — translates a markdown analysis plan into a spec YAML; on later iterations applies the critic's suggested changes. - Runner (
nb_runner.py) — injects spec parameters into a notebook's configuration cell, executes the notebook, then appends a results-serialization cell. - Critic (
critic.py) — reviewsresults.jsonagainst thewriting-reportsskill and emits a three-tier verdict (fully_satisfactory/satisfactory_with_caveats/not_satisfactory) plus concrete spec changes.
Two design choices make it work. First, templated notebooks: the agent operates inside a fixed, reviewed analysis structure, so best practices are the default rather than a hope. Second, skills as markdown: writing-specs, changing-notebooks, running-notebooks, writing-reports, suggesting-remedies are loaded at runtime — swap the text, change the agent's behavior, no code changes.
Run it yourself
Python 3.10+ required. Note the pinned numpy<2: econml's transitive shap dependency references the removed np.bool8 in numpy ≥ 2.0.
python3 -m venv .venv && source .venv/bin/activate
pip install -e .
export ANTHROPIC_API_KEY=sk-ant-... # actor/critic call the Claude Messages API
# Generate a synthetic ACIC-shaped dataset for a quick smoke test
python evals/generate_synthetic_acic.py
# Drive the loop one step at a time
oci-agent draft --plan plans/tryitout.md --specs-dir specs
oci-agent run specs/tryitout/iter_01.yaml --output-dir output/tryitout
oci-agent evaluate output/tryitout --plan plans/tryitout.md
oci-agent revise output/tryitout --specs-dir specs/tryitout
# Or run the full loop in one command
oci-agent loop specs/tryitout/iter_01.yaml --output-dir output/tryitout \
--specs-dir specs/tryitout --plan plans/tryitout.md --iterations 1The CLI auto-increments output directories (iter_02/, iter_03/, …) and spec filenames, so you never track iteration counters yourself. Need a proxy instead of the public Anthropic API? Set ANTHROPIC_BASE_URL.
Does the loop actually help? The numbers
On the ACIC 2016 benchmark battery (77 data-generating processes), the workflow lands solid coverage with low bias:
Estimand |Bias| RMSE Cov95
ATE 0.015 0.173 84.8%
ATT 0.017 0.083 96.1%
ATO 0.014 0.066 97.0%ATT ranks 5th of 16 by bias against the official ACIC black-box benchmark. The head-to-head ablation is the headline: across 10 ACIC datasets, the full actor-critic loop gives ATT mean |error| = 0.054 and covers truth in 9/10 runs — while the same model (Sonnet 4.6) given only the plan and a 5-row data head, with no tools or skills, has mean |error| = 2.572 (~48× worse) and covers truth in only 3/10. The critic's deterministic verdicts also agree with an independent LLM judge on 666/693 records (96%).
Practical takeaways
- Keep the human at the top. You frame the question and sign off results; the agent handles sensitivity analysis, iteration tracking, and report writing — the exact work that burns analyst hours.
- Audit artifacts, not vibes. Every iteration leaves
spec.yaml,results.json, the executed notebook,critique.json, andoci_report.md. Wire those into your review flow and you get process transparency, not just output. - Don't ship it as-is. This is a Skunkworks reference release: no PRs accepted, no maintenance commitment. Model your own workflow on it — the actor-critic skeleton transfers cleanly to other analysis domains since the skills are just markdown.
Resources
- GitHub: github.com/Netflix-Skunkworks/oci-agent
- Netflix Tech Blog: A Human-Augmenting Agentic Workflow for Causal Inference
- InfoQ coverage: Netflix Open-Sources Agentic Workflow for Causal Inference
