Ant Group's OmniTable Wins VLDB 2026 Best Industrial Paper, Speeds Up LLM Data Prep by 5.6x

OmniTable is already managing more than 35 petabytes and over 305 billion records of LLM training data in production, covering Web, code, PDF, and SFT data domains. In a real SFT data preparation task, the system reduced the end-to-end cycle from about 14 days to 2.5 days, with manual operation steps falling from 45 to 12. The paper says the 5.6x speedup does not come from a faster machine, but from reorganizing how data engineers model data and features, turning fragmented physical tables and ad-hoc scripts into a unified logical view with system-managed metadata.

Large-model data processing traditionally revolves around physical tables. Each stage — parsing, cleaning, quality scoring, domain labeling, deduplication, and safety marking — often produces a new table or intermediate result, with separate pipelines for Web, code, PDF, and SFT. As data sources and features grow, the number of objects to maintain expands rapidly. The paper records a real case where adding one feature required an engineer to handle 106 tables on a task canvas. Tables also store only results, rarely documenting how those results were computed; UDFs are scattered across codebases, making it hard to trace an anomalous sample or decide which historical batches must be recomputed after a feature version changes.

OmniTable's core principle is "logically unified, physically separated." In the logical layer, each row represents a traceable data entity, and each column stores a processing state or derived feature. RawData, ProcessedData, and TrainableData correspond to raw, intermediate, and trainable forms. Two system fields, _ai_unique_id_ and _ai_append_name_, provide a global primary key and an access batch identifier, giving stable anchors for backfills, point lookups, and lineage tracking. The "one table" is a logical contract: production is divided into four domain-wide logical tables for Web, code, PDF, and post-SFT, with the largest Web table holding about 25 PB and over 300 billion records across more than 800 logical columns and 200 registered features. A catalog maps logical columns to physical storage, allowing splits, merges, or materialized views underneath without changing the logical schema.

Features are registered as first-class assets with input columns, output columns, UDF or model inference logic, version, and CPU/GPU execution preferences. When an engineer submits a backfill job, they specify only the target batch and feature. OmniTable queries the current computation state, follows the column-level dependency DAG to build a minimal closure, and generates a physical execution plan. Completed ancestors are reused; missing ones are added. After success, the catalog atomically records the batch-feature status, version, physical location, and column-level lineage.

Fault isolation also shifts from task-level to record-level under OmniTable. Common UDF failures such as OOM, timeout, or uncaught exceptions are caught per sample; the system logs the sample ID and error summary, marks the record as NULL, and continues processing the rest. Errors are stored in an error table for later repair. In a controlled comparison on a 500 GB task with about 600 million records containing 31,247 abnormal records (0.005%), record-level failover allowed about 99.995% of records to complete in roughly 6.2 hours without manual intervention. Without the feature, the task failed, and the old flow required three rounds of debugging and deletion, totaling about 52 hours, including 18 hours of human work. The paper notes the record-level wrapping adds 3-5% execution overhead and cannot handle all failures, but it addresses the most common time sink: a few bad records dragging down an entire batch.

OmniTable also reduces repeated scans through operator fusion. In one experiment, eight CPU/Spark features all read parsed_text from about 2.5 PB of data with over 300 billion records. After fusion, scanning dropped from eight passes to one, CPU hours fell from 42,000 to 18,500, and end-to-end time dropped from 38 to 14 hours, a 2.7x improvement. Adaptive tuning, which selects execution backends and resource parameters based on user declarations, operator profiles, engine capabilities, and cluster load, achieved first-submission success rates of 100% across 50 GB, 500 GB, and 2 TB batch sizes, with cost within 5% of expert-tuned configurations in the reported BERT feature task.

A background governance service continuously monitors data scale and access patterns, automatically performing small-file merges, row/column splits, and materialized view construction using a Prepare-Execute-Commit cycle. This lets logical schemas exceed the physical column limits of an underlying engine; in tests on about 2 PB of data, logical columns increased from 200 to 2,500 while P95 latency for point queries stayed manageable. Global ID indexes support single-sample lookup, with P50 of 8.3 seconds and P99 of 14.7 seconds for querying a complete logical row on the 25 PB Web dataset.

In the end-to-end SFT comparison, the old process required about 2 days to locate and ingest data, 9.5 days to backfill features, and 2.5 days to write multi-table joins and export results, involving about 45 manual steps, 24 independent pipelines or scripts, and 35 physical tables. OmniTable cut ingestion to about 0.5 days, feature backfilling to 1.7 days, and filtered export to 0.3 days, totaling 2.5 days, with 12 manual steps, 10 independent commands, and a single logical wide table for the SFT domain. The speedup mainly comes from less per-table orchestration, fewer repeated scans, fewer full-batch reruns caused by occasional bad data, and proactive physical layout adjustments instead of reactive manual tuning after performance degrades.