#86 – David Silver: AlphaGo, AlphaZero, and Deep Reinforcement Learning — Summary & Takeaways

At a Glance

David Silver is the head of reinforcement learning research at DeepMind, lead researcher on AlphaGo and AlphaZero, and co-lead on AlphaStar and MuZero. This issue focuses on how deep reinforcement learning masters games (Go, Chess, Shogi, Atari) from scratch and progressively moves toward artificial general intelligence. Silver's core judgment is that the definition of the reinforcement learning problem itself is sufficient to cover most of the essence of intelligence, and the AlphaZero-style self-play plus deep neural network is currently the closest path to a "general intelligence algorithm"—the same algorithm, without any modification, can achieve superhuman performance across multiple completely different domains.

Theme 1: Why Go Is the "Ultimate Touchstone" for AI — The Dual Challenge of Intuition and Search

David Silver argues that Go is considered an insurmountable challenge for AI because it simultaneously demands "intuitive positional assessment" and an "astronomical search space," both of which are precisely the weaknesses of traditional symbolic AI.

Minimal rules, maximal complexity: Go is played on a 19×19 board, with rules so simple that "if aliens exist, we might not be able to communicate, but we could play Go together." Yet the state space reaches 10¹⁷⁰, far exceeding the number of atoms in the universe (approximately 10⁸⁰).
Positional assessment cannot rely on "scoring": Unlike chess, Go has no piece values. The number of stones played by both sides is nearly equal, and victory hinges on "which territory ultimately belongs to whom" — requiring the prediction of outcomes 300 moves ahead from a sparse set of 20–30 stones. Silver notes: "This ability for intuitive judgment is precisely what enables humans to play Go, and it was something computers could not simulate at all before."
Historical comparison: In the 1990s, heuristic search conquered chess (Deep Blue), checkers, Othello, and others. But by 2000, the strongest Go program could still be defeated by a 9-year-old child with a 9-stone handicap, while the computer Go expert himself could beat the same program with a 29-stone handicap. Silver recalls: "At the time, people firmly believed that conquering Go meant conquering something akin to human intuition — and that was the holy grail of AI."

Theme 2: From MoGo to AlphaGo — The Breakthrough and Ceiling of Monte Carlo Tree Search

Silver points out that Monte Carlo Tree Search (MCTS) was the first revolution in Go, but it quickly hit a ceiling — and AlphaGo broke through that ceiling using deep learning.

Core idea of MCTS: Instead of evaluating positions based on human knowledge, the system repeatedly "plays out randomly" from the current position and uses the average win rate as the evaluation. Silver comments: "The fact that random play can yield useful information is itself remarkable — it hints at the fundamental role of randomness in understanding the structure of the search tree."
MoGo's achievements and limitations: MoGo, based on MCTS, was the first to reach the level of a human master on a 9×9 board. However, on the full 19×19 board, no matter how much computing power was invested, it could never surpass the amateur dan level, remaining far from professional players.
AlphaGo's breakthrough: In 2014–2015, Silver's team used deep neural networks to directly learn "predicting the best move from a position" and "predicting the outcome from a position." Pure deep learning (without search) already matched the level of the strongest MCTS systems at the time. Silver recalls: "We made an internal bet — whether pure deep learning could beat an amateur dan player. The system won. At that moment, I realized that reaching world champion level was inevitable."

Theme 3: The Magic of Self-Play — How AlphaZero Surpasses Millennia of Human Knowledge from Scratch

Silver believes that AlphaZero's most important contribution is not its performance, but the proof that the process of "starting from random and purely correcting errors through self-play" can infinitely approach optimality, without relying on any human data or domain knowledge.

Motivation from AlphaGo to AlphaZero: AlphaGo still relied on pre-training with human game records, which led to "delusion" — the system would persistently misjudge certain board positions. Silver's logic is: "The only way to completely eliminate errors is to let the system correct its own mistakes. And if this logic holds even starting from a random point, then no human data is needed."
Mechanism of self-play: The system begins with completely random moves and gradually improves through a cycle of "trying → seeing results → correcting." Silver describes: "It first discovers that randomness is foolish, so it becomes slightly better than random; then it finds that this slightly better system also has problems, and corrects again... This process can continue indefinitely."
Falsifiable prediction: Silver offers a clear scientific prediction: "If someone runs the AlphaZero algorithm with more computing power than we used, it will beat the previous version 100:0; repeating this process a few years later yields the same result — at least within a human lifetime, this process will not stop." He adds that the state space of Go (10¹⁷⁰) ensures the theoretical ceiling is far out of reach.
Resonance with human knowledge: During training, AlphaZero independently discovered all the "joseki" (standard sequences) that humans accumulated over millennia, and then began to discard some of them, creating its own new joseki — these new sequences are now incorporated into the regular training of professional players.

Theme 4: MuZero — When Rules Are No Longer Given

Silver views MuZero as a key step from games to the real world: the system is no longer told the rules of the game, but instead learns an "implicit model" on its own through interaction with the environment, and then uses this model for planning.

Core innovation: AlphaZero requires knowledge of the game rules (e.g., "how the board changes after a move"), whereas MuZero has no knowledge of the rules at all. It only receives raw pixel inputs (e.g., Atari game screens) and learns through trial and error: "If I take this action, what will happen next?"
Performance validation: The same MuZero algorithm, without any modifications, achieves or surpasses AlphaZero's level across Atari games, Go, chess, and shogi. Silver emphasizes: "This proves that even without rules, the system can learn a sufficiently good world model through pure trial and error and use it for planning."
Real-world significance: The real world has no "rulebook." MuZero's approach — learning environmental dynamics autonomously through interaction — is a necessary condition for reinforcement learning to move toward real-world scenarios such as robotics and autonomous driving.

Theme 5: The Multi-Layered Structure of Intelligence and the Problem of "Meaning"

When confronted with the philosophical question, "What is the meaning of human life?", Silver offers a scientific framework based on the hierarchical structure of reinforcement learning: each layer serves the goals of the layer above it, and intelligence itself is a natural product of this hierarchical structure.

From the Universe to the Hierarchy of Intelligence: Borrowing from physicist Max Tegmark's perspective, Silver proposes a chain of layers: the physical laws of the universe (entropy increase) → evolution (a mechanism for efficiently dissipating energy) → organisms (maximizing reproductive success) → the brain (flexible learning and decision-making) → artificial systems (tools that surpass human capabilities).
Adherence to a "Single Reward Function": Silver emphasizes that while a system may contain countless sub-goals and intrinsic motivations, there must be a measurable ultimate goal; otherwise, the problem itself is undefined. "If you don't have a clearly defined goal, you won't get a clearly defined answer."
AI as a New Layer: Silver notes that his original desire to "build a Go program that plays better than me" was essentially about creating a new layer—one that could achieve the goal of "playing Go well" more effectively than a human. In the future, AI systems themselves may generate sub-goals, forming even deeper layers.

Mentioned Positions

Position	Guest Attitude	Key Data
AlphaGo	Milestone achievement	Defeated Lee Sedol 4:1 in 2016; subsequent version defeated other top players 60:0
AlphaGo Zero	Superior version	Defeated original AlphaGo 100:0; discovered all human joseki and created new ones autonomously in 40 days of training
AlphaZero	Breakthrough in general algorithm	Same algorithm defeated world's strongest chess program Stockfish and shogi program Elmo; Magnus Carlsen credited studying AlphaZero game records for his personal rating record high
MuZero	Key technological advancement	Same algorithm achieved superhuman performance on Atari, Go, chess, and shogi; no need to know the rules
MoGo	Early pioneer	First Go program to reach human master level on 9×9 board (2006-2007)
Deep Blue	Historical comparison	Defeated Kasparov in 1997, but relied on handcrafted evaluation functions and could not transfer to other domains

Judgments Worth Remembering

1. "Go's intuitive judgment is precisely what makes it possible for humans to play Go, and it was something computers had been completely unable to simulate before." (Silver) — Position evaluation in Go cannot be "scored" like in chess; it requires predicting territorial ownership 300 moves ahead from a sparse board state, forcing AI to learn a human-like intuition.

2. "Pure deep learning (without search) already matched the level of the strongest MCTS system at the time — at that moment, I realized reaching world champion level was inevitable." (Silver) — Early AlphaGo experiments in 2014-2015 demonstrated that deep neural networks could directly predict moves and outcomes from board states, without needing Monte Carlo tree search.

3. "AlphaZero's self-play process: starting from random, realizing random is stupid → becoming slightly better than random → discovering that this slightly better system also has flaws → correcting again… This process can continue indefinitely." (Silver) — This is the core mechanism of AlphaZero and the basis for its falsifiable prediction: more compute inevitably yields a stronger version.

4. "If someone runs AlphaZero with more compute, it will beat the previous version 100:0; repeat a few years later, the same result — at least for as long as humans are alive, it won't stop." (Silver) — A clear, testable scientific prediction, grounded in the fact that Go's state space of 10¹⁷⁰ remains far out of reach.

5. "AlphaZero autonomously discovered all human joseki, then began discarding some of them and creating its own new joseki — these new joseki are now part of professional players' regular training." (Silver) — This is the most intuitive example of "machine creativity": the system not only learned human knowledge but surpassed it.

6. "MuZero proved that even without rules, a system can learn a sufficiently good world model through pure trial and error, and use it for planning." (Silver) — The same algorithm achieved superhuman performance on Atari, Go, chess, and shogi, marking a key step toward applying reinforcement learning to the real world.

7. "If you don't have a clearly defined goal, you won't get a clearly defined answer." (Silver) — Silver insists that reinforcement learning must have a measurable ultimate objective, even if the system can internally have countless sub-goals and intrinsic motivations.

8. "The multi-layered structure of intelligence: the physical laws of the universe → evolution → organisms → brains → artificial systems — each layer serves the goals of the layer above." (Silver) — Borrowing Max Tegmark's perspective, Silver reframes "the meaning of life" as a hierarchical optimization problem, with AI as the latest layer in this chain.