Large language models are commonly trained on very large token corpora — but the correct scale is typically hundreds of billions to trillions of tokens, not a few billion. For clarity, here are typical orders of magnitude and a few public approximations:
Typical orders of magnitude
- Small LLMs (≤ 10B parameters): approximately 100–500 billion tokens
- Mid-size LLMs (10B–100B parameters): approximately 500 billion to 2 trillion tokens
- Very large LLMs (≥ 100B parameters): roughly 1 trillion up to 10 trillion tokens or more
Public examples (approximate figures)
- OpenAI GPT-3: about 300 billion tokens
- Google PaLM: about 780 billion tokens
- Meta LLaMA 2: about 2 trillion tokens
These numbers show that modern LLM training sets are often many orders of magnitude larger than a few billion tokens. Quantity matters for statistical pattern learning, but raw token counts still don’t provide grounding, interaction, or the sensory richness needed for human-like understanding.
Why raw internet text falls short
- Lack of grounding: Internet text is largely symbolic — words referring to things, not direct sensorimotor experience. Symbols without grounding lose causality and affordances.
- Sparse interaction traces: Most posts are one-shot utterances, not long-lived interactive episodes with goals, planning, consequences and rewards.
- Missing multimodality at scale: Real-world intelligence depends on audio, video, continuous proprioception, and correlated sensor streams. Pure text lacks these modalities.
- Socio-cultural biases and gaps: Public internet content over-represents some languages, cultures and contexts while under-representing others and rare-but-critical experiences.
- No reliable long-term memory: The internet is fragmented; a single user’s long-term, continuous history of goals and corrective feedback is rarely available.
Because of these gaps, training on even trillions of tokens of internet text allows spectacular pattern prediction and linguistic competence, but does not guarantee the causal, embodied, and goal-directed competencies we associate with human-level general intelligence.
Estimated scale for richer learning
We can give rough orders-of-magnitude to illustrate what’s missing, not to claim exact thresholds:
- Internet text only: typically 10^11–10^13 tokens for large public models (see examples above)
- Multimodal + interactive corpora likely needed: at least 10^12–10^15 token-equivalents when you count high-bandwidth video/audio/robot streams (i.e., orders of magnitude more raw sensory datapoints than text-only corpora)
Put another way, when you convert continuous multimodal recordings into token-equivalents (frames, audio samples, proprioceptive vectors), realistic embodied datasets quickly produce orders-of-magnitude more data than scraped text. That extra data carries grounding, temporality, and causal signals that pure text lacks.
Which data specifically matters if we want to attempt AGI?
- Multimodal aligned streams: high-resolution video + audio + captions + object and event timestamps (synchronous streams are essential).
- Sensorimotor logs / robot traces: continuous control signals, proprioception, force/torque, and environment state while agents act and receive feedback.
- Longitudinal interaction records: dialog and behavior traces spanning months/years for the same agent/person, showing goal pursuit, learning, correction and memory consolidation.
- Rewarded task logs: large numbers of episodes where actions are tied to clear objective signals or human-provided reward/ratings, enabling goal-directed learning.
- Instrumented experiments: controlled datasets with causal interventions and ground truth outcomes (scientific experiments, simulation-with-ground-truth).
- Diverse human-grounded datasets: carefully curated dialogues with humans in-the-loop, role-play, demonstrations, and paired naturalistic annotation of intent and belief.
- Cross-cultural, multilingual, and private-context corpora: to reduce blind spots and better model variation in values and norms.
Simple diagram (data types → role)
graph LR
T["Internet text<br/>(10^11–10^13 tokens)"]
M["Multimodal streams<br/>(video, audio, images)"]
R["Robot & sensorimotor logs"]
I["Interactive traces<br/>(dialogue, long-term logs)"]
S["Scientific / causal datasets"]
AGI["AGI-ready training signal"]
T -->|language patterns| AGI
M -->|grounding + perception| AGI
R -->|embodiment + causality| AGI
I -->|memory + goals| AGI
S -->|causal knowledge| AGI
Practical implications
- If the goal is practical progress toward AGI, researchers should invest in collecting and curating high-quality multimodal and interactive datasets, instrumentation for robot and agent behavior, and longitudinal human-in-the-loop data with careful consent and ethical controls.
- More data alone is not a silver bullet — the right data matters: causal, grounded, interactive, and diverse. Quantity helps, but so does structure, feedback signals, and embodiment.
Conclusion
Even trillions of internet tokens are a powerful resource for language modeling, but they are not sufficient for AGI. Achieving human-like general intelligence will require orders-of-magnitude more diverse, multimodal, and grounded data together with architectures and learning objectives that exploit those signals.