A new study from the University of Oxford and University of Cambridge has revealed a major threat to large language models like ChatGPT, calling it “model collapse.
Researchers at Oxford and Cambridge have identified a fundamental flaw threatening the future of large language models like ChatGPT, and “Big Short” investor Michael Burry says the finding backs up a warning he’s been making about the technology’s core limitations for a while. The research shows that when generative AI systems are repeatedly trained on synthetic data produced by earlier models, they begin to lose the “tails” of the original data distribution. In practice, this means models forget the creative, fringe, and unique nuances of human writing, collapsing into repetitive echo chambers.
Feeding AI-generated content back into training indiscriminately causes what researchers describe as irreversible defects in the resulting model, according to the research. Specifically, the model begins losing the “tails” of the original data distribution, meaning it gradually forgets the creative, unusual, and distinctly human nuances present in real human writing. Over successive generations, this causes the model to collapse into an increasingly narrow, repetitive echo chamber of its own outputs. Crucially, the researchers found this isn’t unique to ChatGPT or language models specifically. Their theoretical framework showed the collapse phenomenon is ubiquitous across learned generative models more broadly, appearing in large language models as well as in variational autoencoders and Gaussian mixture models.
The concern centers on a shift already underway across the internet: as generative AI produces more and more of the text online, future AI models risk being trained on data that was itself generated by earlier AI systems, rather than by humans.
Burry’s skepticism lands in direct contrast to comments made earlier this month by Nvidia CEO Jensen Huang, who declared that AGI has already arrived, citing OpenAI’s GPT-6 Astra model as proof. Because human knowledge itself is too limited relative to the scale of what AI companies are trying to build, and because human wants, needs, and questions are inherently repetitive and redundant, writing on X, Burry tied the research directly to an argument he’s made before: that compression of available information is essentially inevitable,. Huang’s claim drew immediate pushback from parts of the AI research community.