A twelve year old has heard roughly 100 million words. Meta's Llama 3.1 trained on 15 trillion tokens.
Researchers describe the difference as a stack of paper 20 metres high against one that reaches past the International Space Station.
Both end up fluent. Only one of them can explain how.
The obvious answers have been tried
MIT Technology Review covered the state of this problem, and the interesting content is the list of things that did not work.
The BabyLM challenge, organised by researchers at Stanford, Georgetown, UC San Diego and Boston University, trains models on child scale datasets of 100 million or 10 million words. It is an annual competition specifically designed to close this gap.
Curriculum learning was the obvious candidate. Order the training data the way a child actually encounters it, simple before complex, concrete before abstract. It underperformed.
Multimodal training was the second obvious candidate. The SAYCam project at Princeton used head mounted video recorded from children, giving models the visual and auditory stream a child receives alongside language. Those attempts have struggled too.
Michael C. Frank puts the scale of the failure plainly. Train GPT-2 on 30 million words and you get a nonsense generator, not a child.
So more data of a different kind did not solve it, and better ordering of the same data did not solve it. The remaining candidates involve active exploration and social interaction, which are considerably harder to put in a training loop.
Both failed approaches share a shape worth noticing. Each assumed the missing ingredient sat in the input: better ordering, richer modality, more senses.
What neither could supply is the thing a child does constantly and a training run never does. A child acts on the world, gets a response, and updates. The correction arrives unprompted, from outside, and it is specific to what that child just got wrong.
That loop is expensive to simulate and nobody has run it at scale for language.
What this means if you are buying, not building
The industry operates on an assumption that capability scales with data and compute. For four years that assumption has been correct and spectacularly profitable.
But it is a statement about a method, not about intelligence. The one system we know reaches fluency does it on five orders of magnitude less input, and we cannot reproduce the trick.
Which means the current path is not the efficient path. It is the path we found first, and it is being scaled because it works, not because it is optimal.
Two commercial consequences follow, and they point in opposite directions.
If capability keeps requiring more data, then data access becomes the moat rather than model architecture. That has been visible for a while in how aggressively labs pursue corpora, including shredding physical books for clean training data. Architecture gets published. Corpora get bought, and the buying is getting more expensive.
And if the cost curve depends on ever larger training runs, the assumption baked into most three year AI budgets, that unit costs keep falling on the same slope, is not guaranteed. It has held so far through efficiency gains at inference. Training economics are a different curve.
The upside case
There is a genuine opportunity buried in this research, and it is not a technical one.
Closing the data efficiency gap would let minority language communities build capable models on the corpora they actually have. That changes the economics of every market that does not operate in English.
For anyone running marketing or product in a smaller language market, and I am writing this from Bucharest, that is the difference between waiting for a lab to decide your language is worth a training run and building something yourself.
Right now the answer is wait. If the efficiency problem cracks, the answer changes.
The version that applies on Monday
Strip out the cognitive science and one practical rule remains.
The model does not learn your business from three prompts. It learns your business from the corpus you assemble for it, and that corpus is an asset somebody has to fund, maintain and own.
Most companies have not started. They have prompts, scattered across individuals, rediscovered weekly. What they do not have is a structured, versioned body of their own material that any model can be pointed at.
Start with the material that already exists and is badly organised. Past proposals, call transcripts, the reasoning behind pricing decisions, the accounts you lost and why. That corpus is small by training standards and enormous by competitive ones, because nobody else has it.
That work is unglamorous and it compounds, which is the profile of every asset worth building. It is also the practical answer to why the model is not your moat.
The frontier labs figured this out first, which is why they are buying books rather than tuning architectures.
Every child in the world solves this problem before they can tie their shoes. Fifteen trillion tokens later, we still cannot say how.