Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data

> The latest studies demonstrate model collapse is not a given

Which studies? [edit: I'll assume you mean these two given by @dorolow: https://arxiv.org/abs/2404.01413 https://arxiv.org/abs/2406.07515]

> It can't be used as the -only- data source of course, but that's not how it is being used

Right, so human data creation would also have to scale up exponentially, and that's not gonna happen.

> because there's no difference between synthetic data and the data people can create

I mean, that's obviously false, otherwise model collapse wouldn't exist. The difference is statistical, but it's there.

> It is accelerating how quickly researchers and engineers can do their jobs. > https://news.mit.edu/2026/ai-helps-design-new-materials-that...

That's pretty clearly a hype article, the headline even says "The CrysVCD tool developed at MIT COULD cut the huge amounts of time and money spent". I'm asking for empirical measurements of timelines, not hypotheticals.

> This is only the beginning, too... Look ahead a year or two.

Lol that excuse is getting really old

We use large amounts of synthetic data for training at work and have not observed any sort of model collapse when done properly.

Edit: https://arxiv.org/abs/2404.01413 https://arxiv.org/abs/2406.07515

That paper has had a pretty turbulent reception and looks pretty conclusively wrong at this point.

It used an incorrect theoretical framing that assumed that data was being replaced rather than accumulated as a result of more training (see https://arxiv.org/abs/2404.01413 which explores this). This is incorrect because this simply isn't how real-world datasets are created via synthetic data generation (which generally accumulate more data over time rather than replace their data). As a result most of the theoretical results were invalid.

Empirical evidence has also cast a considerable amount of doubt on the paper. For example, Microsoft Phi-4 was an empirical test in specifically what happens if the majority of your training data is synthetic rather than human and it turns out that Phi-4 did significantly better than previous models which relied primarily on human data.

There's some nuance to all of this in how exactly you do this, but the original claims of the paper are looking really shaky at this point.

"Model collapse" is often overstated, as this paper demonstrates: https://arxiv.org/pdf/2404.01413

The original model collapse paper assumes you train networks on 100% synthetic data produced by the previous generation. But if you maintain some portion of real data then the problem is mitigated.