SciForge Architecture: Turning Research Agents from One-Shot Chats into Auditable Pipelines

Agents like Codex and Claude Code are formidable: they retrieve papers, write code, and call tools without breaking a sweat. But real research doesn’t fit in one conversation. Experiments iterate over days, data and code keep changing, and conclusions must trace back to a specific version of the parameters. Stretch a task long enough and the questions pile up: what problem is the agent actually solving? Which version of the data? What supports the conclusion? And when the next person takes over, must they re-read the entire chat log?

SciForge is built directly against those pain points: let the agent carry the execution, while the research process stays continuable, inspectable, and hand-off-able. The project is open source (GitHub: AGI4Sci/SciForge), with a full paper and eight end-to-end real-world cases — required reading for anyone doing agent engineering.

Core Idea: Decouple the Research from the Conversation

Most agent tasks are prompt-centric — goals, constraints, and evaluation criteria all live inside the chat. Stretch the session, change the environment, or hand off to a colleague, and the context collapses.

SciForge stores research goals separately from any specific conversation. A project explicitly records the research question, scope, evaluation metrics, stopping conditions, and publication requirements. Every agent run leaves behind its inputs, artifacts, and judgments. The next researcher opens the project view and immediately sees: what the goal is, which results are confirmed, and which remain candidates.

Design 1: Scientific Files Are Not “Uploaded Attachments”

Research files carry scientific semantics. “Discuss the PDB file” might mean one specific residue of one chain; a plot only makes sense alongside the data and script versions that produced it. If those relationships live only in ephemeral chat, models easily mix versions and objects.

SciForge passes file identity between the workspace, domain-specific viewers, and conversations through structured scientific object references. A reference includes:

{
  "file_path": "structures/9VMR.pdb",   // file path
  "content_hash": "sha256:ab12...",      // content hash, prevents version confusion
  "version": "v3",                       // current version
  "selection": {                         // selection in the viewer
    "model": 0,
    "chain": "A",
    "residue": 86
  }
}

Protein sequences, structures, and small molecules route through a Scientific Model Router that invokes domain models to produce structured observations before the main agent interprets them against the research question; single-cell transcriptomics goes through a Cell2Sentence (C2S) worker. “Describing an image” becomes “scientific Q&A with an object, a location, and a provenance.”

Design 2: Two-Tier Evidence Chains — Session DAG and Project DAG

A complete log is not enough; researchers need a traceable path from every conclusion back to its evidence. SciForge organizes this with two DAGs:

• Session-level Evidence DAG: records the sources, reasoning, runs, artifacts, and research claims within one agent session — plus the relationships between them: supports, conflicts, derives-from.
• Project-level Project DAG: collects evidence snapshots submitted across sessions, placing long-term goals, cross-session conclusions, disagreements, and decisions in one view — never rewriting the original evidence, only preserving the traceability path.

// Concept: relationship types between Evidence DAG nodes
source --supports--> claim
run   --produces--> artifact
claim --conflicts--> claim
claim --derives_from--> run

// Measured scale from the paper's cases
// Single session: 32 nodes / 21 edges, located one inflated and
//   fabricated PDB citation
// Project level: 4 sessions / 25 evidence items / 16 claims
//   -> 45 nodes / 48 relationships

Evidence auditing runs in the background without interrupting operations — it only flags unsupported claims, contradictory conclusions, and weak sources. The final call stays with the human.

Design 3: Human-Agent Division of Labor — Pause at Checkpoints, Don’t Approve Everything

SciForge supports per-task autonomy levels: routine exploration proceeds automatically; risky tasks pause at designated checkpoints; high-impact operations wait for explicit confirmation. Researchers intervene at four points:

• Confirming the problem scope, evaluation metrics, and stopping conditions
• Judging whether key scientific objects and evidence were understood correctly
• Handling conflicts, anomalies, and evidence gaps
• Deciding whether results proceed to the next experiment round or go external

The system separately records “candidate results” and “confirmed published status,” retaining the agent’s suggestion, the evidence behind it, and the human decision together — that is where auditability comes from.

Eight Real Cases — the Three Most Instructive

• ESMC-6B ContactProbe hyperparameter search: the agent iterated 24 times under a fixed 7-minute training budget, allowed to modify only train.py — no touching feature extraction or evaluation logic. Every run recorded its Git version, metrics, and a KEEP/DISCARD/CRASH status. This is the standard template for constraint-bounded auto-tuning.
• MCFST spatial-transcriptomics paper reproduction: the agent extracted the model, hyperparameters, and evaluation protocol from the paper, then reproduced ARI 0.7007 on the Visium dataset (paper reported 0.693) — and faithfully flagged the discrepancy between 25 runs and the script log’s 5. No varnish; that is what credible reproduction looks like.
• AI-guided protein design: RFdiffusion generated 3 scaffolds → ProteinMPNN produced 15 sequences → Boltz-2 + ESMFold structural evaluation → 2 lead candidates selected, with the report’s source-mismatch records preserved. The full chain is auditable.

Practical Takeaways

• To try it directly: download a release (github.com/AGI4Sci/SciForge/releases) and run the paper’s reproduction case first — watch how the Evidence DAG gets generated.
• To study the architecture: read the paper (gui branch, paper/sciforge-report.pdf) and work through Figures 1 and 2 against the two-tier DAG data model.
• To apply in your own agent engineering: even outside research, porting the three moves — decouple goals from conversations, structured object references, traceable conclusions — will markedly improve the maintainability of long-running task systems.
• With a real research need: submit it at agi4sci.github.io/SciForge/submit/; requests are tracked as public GitHub issues.

Resources

• GitHub repository: github.com/AGI4Sci/SciForge
• Releases: github.com/AGI4Sci/SciForge/releases
• Paper PDF: sciforge-report.pdf
• Submit a research need: agi4sci.github.io/SciForge/submit/
• Original coverage (BAAI Hub): hub.baai.ac.cn/view/56913