At a Glance
Noam Brown, a research scientist at Meta AI (FAIR), co-founded the team that created the first AI to achieve superhuman/human-level performance in both heads-up and six-player no-limit Texas Hold'em and the board game Diplomacy. The main thread of this episode traces the evolution of AI from poker to Diplomacy, with the core shift moving from purely adversarial zero-sum games to complex social games requiring natural language cooperation and trust-building. The most weighty judgment in the entire episode: Noam Brown argues that in games requiring cooperation, pure self-play is no longer sufficient to defeat humans — AI must learn and adapt to human behavioral patterns, including irrationality and emotional responses; otherwise, even strategically "correct" AI will be jointly ostracized by human players.
Theme 1: Breakthroughs in Poker AI – Search Matters More Than Neural Networks
Noam Brown argues that in imperfect-information games like poker, the improvement from real-time search far exceeds that from scaling up precomputed strategies.
- Historical Context: In 2015, Brown’s poker AI suffered a crushing defeat against human professional players because it relied solely on precomputed strategy tables, while humans spent minutes reasoning at critical moments. In 2017, Libratus introduced real-time search—starting mid-game (post-flop) to search all possible action paths to the end—and defeated four top heads-up players by approximately $2,000,000 (virtual chips) over 120,000 hands at $50/$100 blinds.
- Mechanism Breakdown: The core of search is not simply evaluating the current hand, but simultaneously searching about 1,000 possible hand combinations (52 choose 2) that the AI might hold, to maintain unpredictability. The system learns through Counterfactual Regret Minimization (CFR) in self-play: it simulates the game, calculates the “regret value” of each action relative to alternatives, and adjusts future action probabilities proportionally to regret. The integration of search with precomputed strategies works as follows: the precomputed value function evaluates states at the search endpoint (e.g., from flop to turn), avoiding the need to search all the way to the end.
- Data Chain: The heads-up version of No-Limit Texas Hold’em has approximately 10¹⁶¹ decision points (more than the square of the number of atoms in the universe). Libratus’s training cost was about $100,000 (cloud computing resources), while the subsequent Pluribus (six-player poker) reduced costs to <$150 through depth-limited search—searching only a few steps ahead and then substituting with value estimates, rather than searching to the end.
- Key Analogy: Noam Brown notes that even AlphaGo/AlphaZero, if stripped of real-time search (Monte Carlo Tree Search), would see its ELO rating drop from about 5,200 to roughly 3,000—below the human top level (about 3,600). “No one has built a Go AI that achieves superhuman performance using only raw neural networks.” Search is an indispensable component in these breakthroughs.
Theme 2: The Unexpected Discovery of Poker AI — “Overbet” Changed Human Play
During the competition, Libratus spontaneously adopted the “overbet,” a move rarely used by humans, which proved to be an extremely effective strategy and has since been widely adopted by professional human players.
- Mechanism Breakdown: Brown’s team added the option for Libratus to “bet any amount” (including 10 times the pot) at the last moment, initially assuming the AI would not use it. However, during the competition, the AI frequently bet $20,000 into a $1,000 pot — forcing human players holding the “second-best hand” into an extreme dilemma: either suspect the AI was bluffing and call, or fear the AI had the nuts and fold. Humans sometimes took 5–10 minutes to make a decision.
- Data and Reasoning: Brown observed that humans struggled intensely in such situations and realized that the AI was actually “doing the right thing” — it was simply maximizing expected value, and “making opponents uncomfortable” was a byproduct. “This was the first lesson humans took away from the competition: they now also started using overbets.” Today, the overbet has become a common weapon in high-level poker.
- Falsification Condition: If human players could adapt to this betting pattern and adjust their call/fold frequencies, the effectiveness of overbets would diminish. However, Brown believes that under the Nash equilibrium framework, the equilibrium point of this strategy still favors the AI.
Theme 3: The Challenge of Diplomacy — Cooperation and Trust Cannot Be Solved by Pure Self-Play
Noam Brown argues that Diplomacy represents a fundamental leap for AI from purely adversarial games to understanding human behavioral patterns — an AI trained purely through self-play will be "ganged up on and eliminated" by human players in a seven-player game.
- Mechanism Breakdown: Diplomacy is a seven-player war game centered on natural language negotiation: players privately negotiate alliances and coordinate actions, but can break promises at any time. The game has approximately 50 territories, and each turn players simultaneously submit movement orders. The support mechanism allows players to aid allies or betray them. "The core of the game is people, not pieces."
- Why Pure Self-Play Fails: Brown's team trained a pure self-play Diplomacy AI (without human data), which was "destroyed" when playing against humans in a seven-player game. Two reasons: 1) The AI developed a "robot language" only it could understand, making communication with humans impossible; 2) The AI made mathematically optimal but socially unacceptable moves — for example, when jointly targeting a leader, the AI would use spare forces to seize allies' territory. "Humans get angry, then deliberately lose the game to retaliate against you." This irrational yet real human behavior cannot be learned from zero-sum self-play.
- Data Chain: The Cicero system used approximately 50,000 human Diplomacy games (over 10 million natural language messages) as training data. The language model was first pre-trained on internet data, then fine-tuned on Diplomacy data. The key innovation is intent conditioning: the system uses reinforcement learning + planning to compute "what I want to do" and "what I want the other to do," then feeds these intents into the language model to generate messages. A filtering network also evaluates the expected value of each message — if a message would lead the other party to take an unfavorable action (e.g., telling them "I will attack you"), it is blocked.
- Inference: Cicero ranked second in 40 online games (out of approximately 80 players, 19 of whom played more than 5 games). Brown emphasizes that the benchmark is not "rank first among all experts," but "outperform strong human players in a real-world environment with mixed skill levels" — just as testing autonomous vehicles should not be on a stunt driver track, but on real city roads.
Theme 4: Humanized AI — Balancing "Correctness" with "Collaborability"
Noam Brown proposes that by adjusting the weight of the "anchor policy" (a human-anchored strategy) during self-play, it is possible to create AI that is both powerful and thinks like a human — with profound implications for training, cheat detection, and game design.
- Mechanism Breakdown: The core approach involves introducing regularization into self-play: the AI is encouraged to maximize expected rewards, but is simultaneously penalized for deviating from human behavioral patterns (the "anchor policy" derived from supervised learning on human data). A parameter controls the relative weight between the two. "You can say, 'Play as human-like as possible, but if a certain move has exceptionally high expected value, you are allowed to deviate.'"
- Application Cases: Brown's team has applied this technique to chess and Go. The result: the AI maintains near-superhuman strength while exhibiting a human-like style. "You can tune a 2,800 ELO human-style player — rather than the inhuman style of Stockfish." Going further, in theory, it is possible to model specific players (e.g., Magnus Carlsen) for targeted training or weakness analysis.
- Ethics and Risks: This technique also introduces challenges for cheat detection — if AI can perfectly mimic human style, it becomes harder to distinguish humans from bots in online games. Brown acknowledges: "This raises profound questions about the role of AI systems in society." Additionally, the ability to deceive in Diplomacy raises ethical concerns — "Developing language models that can lie to humans is a thorny issue."
- Falsification Conditions: If human data is insufficient or of poor quality, the anchor policy will be inaccurate, causing the AI to be either too weak (over-imitating flawed humans) or too non-human-like (over-optimizing). Brown believes data scale is the primary bottleneck.
Mentioned Positions
| Position |
Guest Attitude |
Key Data |
| Libratus (Poker AI) |
Successful (superhuman) |
Defeated 4 top heads-up players in 2017, earned ~$2,000,000 (virtual) over 120,000 hands, training cost ~$100,000 |
| Pluribus (Six-player Poker AI) |
Successful (superhuman) |
Training cost <$150, achieved through deep limited search |
| Cicero (Diplomacy AI) |
Human-level (ranked second) |
40 online games, ranked second among 19 active players; used data from 50,000 human games |
| AlphaGo/AlphaZero |
Cited as a comparative case |
ELO dropped from 5,200 to 3,000 without search (below top human level of 3,600) |
| Human Poker Player (Daniel Negreanu) |
Positive about his ability to adapt to AI |
One of the few veteran top players keeping pace with AI development |
| Human Poker Player (Phil Hellmuth) |
Neutral (style mentioned) |
Known for a "suboptimal but effective" playing style |
Judgments Worth Remembering
1. Noam Brown: In games requiring cooperation, pure self-play is no longer sufficient—AI must learn human behavioral patterns, including irrationality and emotional responses. Support: In Diplomacy, a pure self-play AI was "ganged up on and eliminated" by human players because it made mathematically optimal but socially unacceptable moves (e.g., seizing allied territory), triggering human retaliation.
2. Noam Brown: Search is the key to AI breakthroughs, not the neural network itself—even AlphaGo would fall below human level without search. Support: AlphaZero without search saw its ELO drop from 5,200 to 3,000; Libratus turned a poker AI from a 2015 failure into a 2017 crushing victory through real-time search.
3. Noam Brown: Overbetting was an unexpected discovery by Libratus and has since changed human poker strategies. Support: The AI bet $20,000 into a $1,000 pot, forcing humans into extreme dilemmas; human players later admitted, "We need to start doing this."
4. Noam Brown: In Diplomacy, trust matters more than deception—long-term lying reduces AI scores. Support: Once an AI is caught lying, human players permanently lose trust and refuse to cooperate; Cicero was designed to minimize lies because "in the long run, lying leads to lower scores."
5. Noam Brown: By regularizing with "human-anchored strategies," AI can be made both powerful and human-like—this enables targeted training but also poses challenges for cheat detection. Support: This method has been applied to chess and Go, producing human-style players at specific ELO levels; however, it also makes it harder to distinguish humans from bots in online games.
6. Noam Brown: The "anti-AI bias" in Diplomacy is human nature—once players suspect an AI, they will team up to eliminate it first. Support: In non-language version experiments, human players spent entire games trying to identify the AI and then jointly attacked it; this mirrors the social psychology of "when aliens appear, humans unite."
7. Noam Brown: Data efficiency is the biggest bottleneck for AI today—humans can learn Go in thousands of games, while AI requires millions. Support: He believes the solution may lie in enabling AI to leverage broad background knowledge (e.g., language model pretraining) like humans do, rather than learning each task from scratch.
8. Noam Brown: Diplomacy is a major step for AI toward the real world—because it uses natural language, involves trust and deception, and requires understanding human irrationality. Support: Compared to StarCraft or Dota, Diplomacy's "action space" consists of natural language sentences, closer to real-world negotiation scenarios; however, deploying it in the real world still faces challenges in defining reward functions and action spaces.