Paul Buchheit on AGI, open source models, and the future of AI freedom

Executive overview

Google was always an AI company at its core — its mission was to gather the world's training data and feed it into a large AI system. Yet Google failed to lead the AI revolution it helped create, paralysed by regulatory fear and the need to protect its search monopoly.

The real risk now is not AI itself but how power concentrates as AI scales. Open source models are the best defence against centralisation — they represent freedom of thought, not just freedom of speech.

The central choice is between centralisation — where AI amplifies the power of states and corporations — and freedom, where every individual gains the tools to be a better version of themselves.

Google's AI origins and failure to lead

• PageRank was a foundational ML algorithm; Google's mission was always to gather training data and run it through a large AI system
• The "Did You Mean" spell corrector was the first widely used AI product — trained on web data and query logs, no dictionary, could correct proper nouns
• Noam Shazeer built that system in his first two weeks at Google; he later co-authored the Transformer paper and founded Character.AI
• Google had all the ingredients — data, compute, talent — but became focused on protecting the search monopoly
• AI threatened to break the search ad model: give users the right answer and they stop clicking on ads
• Internal AI projects were heavily restricted; researchers couldn't generate images of humans; a chatbot originally named "Human" had to be renamed Lambda
• Sergey and Larry leaving created a leadership vacuum; no one with sufficient credibility could bet the company on AI

OpenAI's founding and why it worked

• By the mid-2010s, deep learning results were impressive enough that AI shifted from "indefinite future" to "definite future"
• Sam Altman was alarmed by Elon Musk's calls for AI regulation; Buchheit argued it was better to build the AI and steer it than try to outlaw it
• OpenAI began as YC Research — meant to keep AI development open to the startup ecosystem rather than locked inside Google
• The pitch to researchers: your work will be published, not buried; that attracted talent who wanted to ship
• Elon's emails pushed to remove the YC branding; the YC roots were largely erased
• GPT-2 was the inflection point — next-word prediction is deceptively simple but requires building a model of reality to work
• OpenAI succeeded partly because Google was slow; startups often win by competing against incumbents with the wrong internal incentives

The path to AGI

• AI crossed a critical threshold when investment started producing returns, which attracted more investment — the same cycle that turned ARPAnet into the internet
• Current LLMs run on stream-of-consciousness reasoning; giving models time to think (system-two reasoning) is the obvious next step
• Founder workflows — chain-of-thought, multi-agent pipelines — are hard-coding system-two thinking as a stopgap; over time the model takes on more of that itself
• Jan LeCun's scepticism is noted but not shared; Buchheit believes incremental progress keeps compounding toward AGI
• By 2033, Zoom-based knowledge workers could be transparently replaced: AI can watch someone do their job, learn the patterns, and deep-fake their presence in calls
• All the required data — camera feed, audio, keyboard and mouse input — is already digital and capturable

Open source as a freedom issue

• The long-term question is not capability but power distribution: does AI concentrate in governments and big tech, or does it empower individuals?
• Open source is a litmus test for freedom; locked models with restricted outputs are equivalent to restrictions on freedom of thought
• Meta releasing Llama is net positive — driven by competitive self-interest but the effect is deflationary for closed-API gross margins and good for the ecosystem
• Relying solely on Meta is dangerous; a broad coalition in favour of open source is needed
• Compute costs are centralising by nature; legislative protections and hardware/algorithm efficiency gains are needed to counteract that
• The human brain runs on ~15 watts; current AI training is far less efficient, leaving large room for algorithmic improvement

Regulation, control, and geopolitics

• Legislation like SB 1047 — holding model builders criminally liable for model outputs — would make AI development untouchable, handing total control to regulators
• COVID-era restrictions on discussing the virus's origins are a recent example of how information control causes catastrophic harm
• China's authoritarian approach forces models to lie (e.g. about Tiananmen Square), which is a structural disadvantage in building reliable AI
• The "doomer" position has a long history — Population Bomb, Limits to Growth — and consistently advocates for central control and degrowth
• The worst-case scenario is AI enabling a permanent totalitarian system where even thoughts are censored; that requires resisting the centralisation impulse now
• YC's structural role — funding 19-year-olds to build large companies — is itself a counterweight to power concentration

What an optimistic AI future looks like

• Everyone with a 200 IQ equivalent is the target: AI as a tool that makes every individual smarter and more capable
• A child producing Pixar-quality animation on their own is a concrete example of what democratised AI enables
• Central planning cannot predict the right outcome; the correct move is to move in the right direction and trust distributed creativity
• ChatGPT's most important contribution may already be done: making AI a public phenomenon rather than something locked inside a lab
• Developing AI in the open, across many perspectives and organisations, is the best path to a good outcome
• Building it in a secret government laboratory is how you get Skynet