Will AI end up eating itself? What happens when AI models run out of human generated inputs and are trained on content created by AI? Is there a risk that AI models trained on AI slop become ‘inbred’ and get into a self-reinforcing death spiral of reducing quality? So many questions.
AI model collapse refers to the progressive degradation of generative AI models when they are trained predominantly on data generated by other AI systems. This may seem like a fanciful idea at this relatively early stage of AI model development but it potentially presents a significant risk that may become very real sooner than we think.
A study published in Nature highlighted that indiscriminate use of AI-generated content in training can cause irreversible defects in models, leading to a loss of diversity and the disappearance of rare but important data patterns. Recursive training of this kind leads to a feedback loop where errors and biases are amplified over successive model generations, resulting in a decline in performance and outputs that are less accurate, less diverse, and increasingly detached from real-world data.
Model collapse matters for a few key reasons:
- Erosion of data quality: As AI-generated content proliferates online, an inability to distinguish between human and machine-generated data could well lead to a decline in input diversity and overall AI performance.
- Innovation stagnation: Models trained on homogenized data may fail to capture the richness and variability of human language and creativity, resulting in outputs that are repetitive and lack novelty.
- Reputational, societal and operational risks: Increased misinformation or biased outputs from collapsing models could lead to consumer mistrust, bias and reputational harm.
In this opinion piece Louis Rosenberg points out that even if we solve the inbreeding problem, a widespread reliance on AI could be stifling to human culture because GenAI systems ‘are explicitly trained to emulate the style and content of the past, introducing a strong backward-looking bias’. If systems are not designed to avoid model collapse we may well see increasingly repetitive or generic outputs, rising error rates, and a proliferation of misinformation and bias. Sturgeon’s Law, coined by science fiction writer Theodore Sturgeon, states that ‘90% of everything is crap’. It reminds us that mediocrity is common across all fields and the art to making any kind of dent on the universe is to elevate ourselves out of the mediocre and into the exceptional 10%. Perhaps the biggest risk from widespread model collapse is that this 90% becomes closer to 100%.
Of course there are strategies that businesses and AI developers can take to mitigate generative inbreeding: a ‘human-in-the-loop’ approach to ensure diverse and authentic training data and deliberate use of human-generated inputs in hybrid models that can benefit from the efficiency of AI while retaining the creativity and contextual understanding of human contributors; systems to track the origin of data used in training; and involving human reviewers to assess and correct AI outputs.
One of the more ‘out there’ ideas to mitigate collapse is to consider integrating embodied AI systems that learn from real-world interactions (such as robotics) to gather ‘authentic’ data and experiences. Scholars at Google Deepmind are already talking about how AI training data is too static, and is restricting AI’s ability to reach new heights. In a paper posted last week (PDF) they argue for more focus on reinforcement learning – a type of machine learning where an agent learns to make decisions by interacting with an environment, receiving rewards or penalties for actions, and optimising its behaviour to maximise cumulative reward over time.
It’s been almost a decade since the AI model AlphaGo defeated world champion Lee Sedol in a best-of-five Go match (Go being the complex Asian game). AlphaGo and its successor AlphaZero both used reinforcement learning through self-play to become better than the best human at something specific. But this form of learning was restricted to limited applications like games where all the rules are known. Gen AI models are different in that they can deal with spontaneous human input without explicit rules to determine outputs.
The authors of the paper say that discarding reinforcement learning has meant that ‘something was lost in this transition: an agent’s ability to self-discover its own knowledge’. LLMs, they say, rely on human pre-judgement at the prompt stage which is limiting. They argue that AI should be allowed to have ‘experiences’ and interact with the world to create goals and understanding based on the knowledge it gains from interacting with dynamic environments. They call this the ‘era of experience’.
This ‘streams’ approach, as they call it, would flip AI model inputs from learning from static datasets to continuous experiential learning and, they argue, better equip AI to deal with complex, fast-changing scenarios and environments. By grounding AI learning in real-life experiences, this approach aims to preserve data diversity and authenticity, mitigating the risks associated with recursive training on AI-generated content.
It’s an interesting idea that would seem to take current capability to a whole different level. But so many questions.
A version of this post appeared on my weekly Substack of AI and digital trends, and transformation insights. To join our community of over ten thousand subscribers you can sign up to that here.
To get posts like this delivered straight to your inbox, drop your email into the box below.
Photo by Brian Kelly on Unsplash
Leave a Reply