Before a model is trained, its corpus must be parsed, cleaned, deduplicated, scored, tokenized, and assembled into samples. At petabyte scale, data engineers aren't just fighting compute bills (compute keeps concentrating in cloud giants) — they're drowning in hundreds of tables, ever-growing feature sets, and a handful of bad rows that can force an entire batch to rerun.
Ant Group's OmniTable — winner of the best industrial-track paper at VLDB 2026 — attacks the most overlooked stage in the LLM pipeline: data preparation. In production it manages more than 35 PB and 305 billion rows of training data across web, code, PDF, and SFT domains (training data is becoming a key self-play iteration asset). On one real SFT preparation task, the end-to-end cycle dropped from about 14 days to 2.5, and manual steps fell from 45 to 12.
The result didn't come from a faster machine. OmniTable changes how engineers organize data and features: one data domain appears as a single logical wide table on top, while underneath it stays split by scale, access pattern, and compute engine. Features stop being ad-hoc calculations inside scripts and become system assets with definitions, versions, dependencies, and lineage.
Why one feature can drag in 106 tables
Traditional LLM data processing is built around physical tables. Parse results land in one table, cleaning in another; quality scores, domain labels, dedup signatures, and safety markers each spawn new tables or intermediates. Web, code, PDF, and SFT each run their own pipeline. Add features, and the maintenance surface explodes: to add one quality feature, engineers had to touch 106 tables on a task canvas.
Worse, tables store only results, not how those results were computed. UDFs live in scattered codebases, with weak links between input columns, operator versions, run batches, and downstream training tasks. Tracing one anomalous sample means jumping across tables and scripts; a feature version change forces teams to decide which historical batches need recomputation.
OmniTable frames these as three engineering costs: data hard to locate, features hard to backfill, and results hard to trace. Its design move is direct — promote data batches and feature columns to first-class objects, and demote physical tables to a storage-layer implementation detail.
Logical unification, physical separation
The core principle is "logically unified, physically separate." At the logical layer, each row is a traceable data entity and each column holds a processing-stage state or a derived feature. RawData, ProcessedData, and TrainableData map to raw, intermediate, and training-ready forms, with quality, domain, safety, and dedup columns added as needed. One domain, one logical view.
This is the same move database systems made decades ago: separate the schema you reason about from the storage that serves it. LLM data engineering is finally getting the equivalent — a schema layer over a chaotic physical mess.
Why this matters beyond one company
Model capability is increasingly gated not by architecture but by data discipline. Every frontier lab now runs the same training loop; the differentiator is how clean, traceable, and reproducible the data behind it is — the same data-quality-driven evolution that reshaped protein AI. OmniTable is a signal that the data-engineering layer of AI is maturing into a first-class engineering discipline, with real tooling, real abstractions, and real performance numbers.
If you run any serious training or fine-tuning pipeline, the takeaway is transferable: treat features as versioned, lineage-tracked assets rather than scripted afterthoughts; give every row an identity; and keep a clean logical view on top of whatever physical mess lives underneath.