The Invisible Ingredient That Determines Everything
In the spring of 2023, researchers at Stanford University’s Center for Research on Foundation Models published a detailed audit of the datasets underpinning some of the world’s most powerful AI systems. What they found was disquieting: the training corpora for these models—vast digital libraries scraped from the internet, digitized books, and licensed databases—were riddled with duplicated content, toxic language, and demographic skews that their creators had never fully mapped. The models trained on this data were already deployed in millions of applications. The damage, if any existed, had long since been baked in.
This is the peculiar and profound predicament at the heart of modern artificial intelligence: by the time anyone realizes that the raw material was flawed, the product has already been shipped to the world.
Understanding how training data shapes AI models is not merely an academic exercise. It is an inquiry into how power gets encoded in systems, how errors propagate at scale, and who bears the consequences when invisible technical decisions go wrong. As AI models become embedded in hiring software, medical diagnostics, judicial risk assessments, and creative tools used by hundreds of millions of people, the question of what goes into those models—and who decides—has become one of the defining governance questions of the century.
What Training Data Actually Is and How It Works
To understand why data matters so profoundly, it helps to understand the mechanics of how modern AI models learn.
The dominant paradigm in contemporary AI—particularly for large language models (LLMs) like GPT-4, Claude, and Google’s Gemini—is called supervised or self-supervised learning on massive datasets. In simplified terms, a neural network is exposed to enormous quantities of text, images, or other data, and it adjusts billions of internal numerical parameters (called weights) to become better at predicting patterns within that data. The model doesn’t memorize content in the way a hard drive stores files; it distills statistical relationships—which words tend to follow which other words, which visual features correspond to which labels, which concepts cluster together.
This process means the model is, in a very real sense, a compressed representation of its training data. Its knowledge of history comes from whatever historical texts were included. Its understanding of human emotion derives from whatever emotional expression appeared in its corpus. Its concept of “normal” is calibrated to whatever distribution of content it saw most frequently. OpenAI’s GPT-3, released in 2020, was trained on roughly 570 gigabytes of filtered text from the internet (a dataset called WebText2), 67 gigabytes of books, and portions of Wikipedia and Common Crawl, a web scrape representing billions of pages. GPT-4’s training data has not been fully disclosed, but estimates suggest it encompasses trillions of tokens—individual chunks of text that collectively dwarf any single human life of reading.
Scale, however, is not the same as quality, representativeness, or safety. “The field spent a decade chasing bigger and bigger datasets as though size were a virtue in itself,” says Dr. Timnit Gebru, founder of the Distributed AI Research Institute and one of the authors of a landmark 2021 paper on the risks of large language models. “But if you scrape more of the internet, you also scrape more of the internet’s worst tendencies.”
When Bad Data Produces Bad Behavior
The clearest illustrations of how training data shapes model behavior come from the field’s most embarrassing failures.
In 2016, Microsoft released a Twitter-based chatbot called Tay that was designed to learn conversational patterns from user interactions in real time. Within 24 hours, coordinated users had taught it to generate racist, sexist, and conspiratorial content. Microsoft shut it down. The lesson—that models reflect the data environments they inhabit—was stark, but the industry did not fully absorb it.
More structurally significant are the biases baked into models through their static training sets. Amazon scrapped an internal AI recruiting tool in 2018 after discovering it systematically downgraded résumés from women. The system had been trained on ten years of hiring data from a company that had historically hired mostly men; it learned that maleness was correlated with being hired and generalized accordingly. The data was an accurate reflection of Amazon’s past. That was precisely the problem.
Face recognition systems have documented similar dynamics. Research by Joy Buolamwini and Timnit Gebru, published in 2018 as the “Gender Shades” study, found that commercial facial analysis tools from Microsoft, IBM, and Face++ showed error rates for darker-skinned women that were up to 34 percentage points higher than for lighter-skinned men. The disparity traced directly to training datasets that were overwhelmingly composed of lighter-skinned faces—a reflection of who had historically been photographed, digitized, and made available in machine-readable form.
These are not edge cases. A 2023 study published in the journal Nature Medicine examined radiology AI tools and found that many performed significantly worse on patients from demographic groups that were underrepresented in training data. In healthcare, that statistical gap can mean the difference between a correct and a missed diagnosis.
Language models carry their own flavor of embedded distortion. Research has consistently shown that LLMs trained on general internet text associate Black-sounding names with more negative sentiment, associate women with domestic roles, and replicate cultural assumptions about national and religious groups at scale. A 2021 paper from researchers at the Allen Institute for AI found that even after extensive filtering, the Common Crawl dataset—a backbone of many major AI training sets—contained substantial quantities of content from unreliable news sites, conspiracy forums, and extremist platforms.
The Provenance Problem and the Consent Debate
If the composition of training data is so determinative, an obvious question follows: where does the data come from, and who authorized its use?
The dominant model in the industry, particularly during the foundational years of large AI development, was essentially extraction without compensation or consent. Web scraping allowed AI companies to harvest text from news articles, academic papers, personal blogs, Reddit threads, creative writing platforms, literary fiction, and scientific journals—often without the knowledge of authors, and almost always without payment.
This approach is now facing serious legal and ethical challenges. In 2023 and 2024, a cascade of lawsuits arrived from content creators. The New York Times sued OpenAI and Microsoft, alleging that millions of its articles had been used to train GPT models without authorization. A class-action suit from authors including John Grisham and Jodi Picoult made similar claims against Meta. Getty Images sued Stability AI over the use of its photographs to train image generation models.
The legal outcomes remain unsettled, but the debate has crystallized something that technologists had long treated as a solved or ignorable problem: training data has provenance, and provenance has moral and legal weight. “There’s a tendency in the AI industry to treat data as if it just exists in the world, like sunlight or air,” says Ryan Calo, a professor of law at the University of Washington who specializes in technology regulation. “But data is produced by human activity, and the people who produced it have legitimate interests in how it’s used.”
The counterargument—advanced by AI companies and some intellectual property scholars—is that training a model on text is analogous to a human author reading widely and developing a style or knowledge base from that reading. Just as readers don’t owe royalties to every author they’ve encountered, the argument goes, AI systems should not require blanket licensing for pattern recognition. The courts have not yet definitively resolved where this analogy holds and where it breaks.
Meanwhile, questions of what should be excluded from training data are equally contested. Many major AI developers now apply content filters and human review processes (sometimes called Reinforcement Learning from Human Feedback, or RLHF) to reduce harmful outputs after initial training. But post-hoc filtering is widely acknowledged to be imperfect. A model trained on vast quantities of violent or manipulative content develops internal representations of that content even if it’s later coached not to express them—a phenomenon researchers call “latent capability,” and one that jailbreak techniques routinely exploit.
Data Governance: The Policy Frontier Nobody Fully Prepared For
The accumulation of these problems has produced a growing consensus—though not yet corresponding action—that data governance for AI must become a serious regulatory discipline.
The European Union’s AI Act, which came into force in 2024, includes provisions requiring that high-risk AI systems maintain documentation of their training data, including its sources, scope, and any known limitations or biases. This is a meaningful step, though critics note that it places the burden of compliance largely on the companies that built the systems, with limited independent verification mechanisms.
In the United States, the Biden administration’s October 2023 executive order on AI called for the development of standards around training data transparency, but binding rules remained elusive as of mid-2024. The Federal Trade Commission has signaled interest in investigating whether AI companies’ data practices constitute unfair or deceptive trade, but regulatory action has proceeded slowly against a backdrop of rapid commercial deployment.
Beyond legality, there is a growing academic and practitioner movement around what’s being called “data-centric AI”—the idea that improving the quality, diversity, and documentation of training datasets may yield greater improvements in model behavior than simply scaling up model size. Andrew Ng, one of the most prominent figures in machine learning, has become a prominent advocate for this approach, arguing that the industry’s obsession with model architecture has consistently undervalued the importance of the data those models consume.
Several organizations are working to build curated, ethically sourced training datasets as alternatives to indiscriminate web scraping. The Common Voice project from Mozilla, for example, has collected voice data from voluntary contributors in dozens of languages, with explicit consent and open licensing. EleutherAI’s Pile and Hugging Face’s datasets initiative represent attempts to document and version training data in ways that allow for auditing and reproducibility. These are encouraging signals, but they operate at a scale that remains modest compared to the closed, proprietary datasets of the largest commercial AI labs.
The Feedback Loops Nobody Fully Understands
There is a final, disquieting dimension to the training data problem that is only beginning to receive serious attention: the question of what happens as AI-generated content becomes an increasingly large share of the text available on the internet.
Models trained today are trained primarily on human-generated content. But models deployed today are generating enormous quantities of new text—articles, social media posts, marketing copy, code documentation, academic papers—much of which flows back into the internet and will be scraped by future models. Researchers call this “model collapse” or “data poisoning by dilution,” and early experimental evidence suggests it is not theoretical. A 2023 paper from researchers at Oxford and Cambridge demonstrated that when language models were iteratively retrained on their own outputs, the resulting models showed degraded performance, reduced diversity of expression, and amplified biases from the original generation.
If AI-generated content comes to dominate the training sets of future AI models, those models may increasingly inherit not human knowledge and expression but an AI’s compressed, statistically regularized version of it. The cultural and epistemic implications are difficult to overstate.
“We’re potentially looking at a situation where the informational ecosystem that future AI learns from is increasingly a reflection of what previous AI thought the world looked like,” says Yejin Choi, a professor of computer science at the University of Washington and a MacArthur Fellow. “That’s a very strange loop to be standing inside of.”
What Comes Next
The trajectory of AI development over the next decade will be substantially shaped by choices made right now about data—choices that are technical, legal, ethical, and political all at once. The decisions include: whether training data must be disclosed and to whom; whether creators whose work was used to train commercial models are entitled to compensation; whether governments have the authority and the will to mandate audits of AI training corpora; and whether the industry will invest seriously in the unglamorous work of dataset curation, documentation, and diversification.
What is already clear is that treating training data as a mere technical input—an industrial raw material no different from electricity or server capacity—is no longer intellectually or ethically defensible. The data is the model, in the most meaningful sense. Its composition encodes assumptions about what knowledge matters, whose voices count, and what the world looks like. Those assumptions then get reproduced at scale, billions of times, in contexts their original architects never imagined.
The chatbot that gives a patient medical advice, the algorithm that evaluates a loan application, the writing assistant used by a student in Lagos or São Paulo or rural Idaho—all of them are downstream of choices made, often quietly and without public input, about which data to include, which to exclude, and what to do when the available data was never equitable to begin with.
The machines are only as honest as the mirrors we hold up to them. Right now, those mirrors deserve a great deal more scrutiny than they are getting.