Chinese AI Developers Rework Pre-Training as Dirty Data Burns Time and Compute

One company spent a year on a trillion-parameter model whose deployed performance was beaten by models one-hundredth its size; the team traced the gap to dirty data. Others were slowed by vague labeling rules, contaminated evaluation sets and redundant junk corpora, and some concluded their data was beyond repair, rebuilt their data teams and started over.

Zhao Xiang, a data specialist at a leading foundation model company, told Leiphone that the industry's early consensus was that more data was enough and that slightly worse data did not matter because models would absorb it. Companies scraped web junk and unverified cleaning packages to reach hundreds of billions or even trillions of tokens, producing first-generation models that stalled at scores of 60 or 70 out of 100 while overseas models reached 90.

Wang Bin, a former executive at a model company who now runs an AI startup, said contaminated underlying data wastes cards, money and manpower and costs teams their time window. He described a company in central China that fed scraped technical blogs into its pre-training corpus and, midway through training, found that many were machine-generated aggregator pages repeating the same paragraph tens of thousands of times. The model began repeating itself on evaluation sets, tens of thousands of GPU-hours had already been consumed, and the version was scrapped.

Wang Yunhe, founder of Jiyuan Ludong, said in a media interview that data is the foundation and that teams should not blame compute. He added that data teams need some authority over pre-training teams, that pre-training teams should in turn make demands of data teams, and that assessing the two separately causes confusion. Zhao said judging a data team only on volume is absurd, because large models depend on the information density and cleanliness of data rather than its raw size; several hundred terabytes of dirty data can make a model worse or cause training to collapse.

Media reports have described Tencent's Hunyuan team as having had unclear labeling rules paired with acceptance thresholds set too high, so staff produced large volumes of unusable data to meet quotas, while leaderboard data and redundant data added contamination, forcing the team to restart and clean the data from scratch.

Data built up during the internet era has not transferred to this fight. Xu Dong, an algorithm lead at a foundation model company, told Leiphone that local services, social, gaming, e-commerce and search data once treated as moats matter less for general-purpose capability. Google holds the most scenario data but has not performed best, he said, while Anthropic and OpenAI, which had no data accumulation, moved faster, and domestic companies show the same pattern.

Zhao said there are few secret techniques in data work, which is closer to hard labor: teams must keep spending people and patience to move data from 60 points to 90, weighing optimization cost against training cycles and improving in stages rather than verifying every item before training begins. Post-training expert data and long-horizon agent trajectory data are also costly and hard to obtain.

Zhou Jun, a business lead at an AI data vendor, cited a rough industry breakdown in which every 100 yuan spent on AI training goes 40 to compute, 30 to talent, 20 to advertising and 10 to data. Expert long-horizon task data is in heavy demand and several large companies budget hundreds of millions of yuan, he said, but pay reluctantly: a single item for US-based HLE research starts at 10,000 to 20,000 dollars, while domestic rates run about 1,000 to 2,000 yuan. Capacity, he added, is a tighter constraint than money, and some model makers quietly run token-forwarding relay services that generate revenue while capturing user requests and results, yielding task data closer to effective training samples than public scraped corpora. Routing startups are also eyeing that asset, though large companies' own routing platforms prioritize in-house models and attract limited users.

The data flywheel, in which stronger models draw harder tasks and higher-quality data flows back into training, remains largely unspun. Zhao and others point to three blockages: organizational restructuring that has yet to dismantle data walls between departments; a shortage of data cleaning, pipeline, engineering and research staff, roles long treated as support work and only recently in demand, according to AI recruiters; and thin endogenous data. Long-horizon trajectory data comes mainly from AI coding and AI office work, but overseas models Claude and Codex hold a large share of the coding market and office products are still at an early stage with limited scenarios.

Several people told Leiphone they expect more tear-down-and-restart cases in the next year or two, describing the rework as an industry-wide lesson rather than any single company's failure. Compute can be bought and talent hired, they said, but high-quality data has no shortcut.