#106 – Matt Botvinick: Neuroscience, Psychology, and AI at DeepMind

This Issue at a Glance

Guest Matt Botvinick is the Director of Neuroscience Research at DeepMind, spanning cognitive psychology, computational neuroscience, and artificial intelligence. The main thread of this episode revolves around the core question of how the brain generates behavior, exploring how neuroscience and AI research can inspire each other. The most significant judgment in the entire episode is: Botvinick believes that the ultimate goals of neuroscience, psychology, and AI research should be unified—understanding the brain is understanding behavior, and AI models (especially meta-reinforcement learning and distributional coding) are constantly validating this view.

The "Middle Layer" Mystery of Cognition: A Vast Gap Between High-Level Functional Descriptions and Low-Level Physical Mechanisms

Matt Botvinick argues that current understanding of the brain is in a "strange" intermediate state: we know quite a lot about its high-level functions (e.g., "what to do") and low-level mechanisms (e.g., individual neuron firing), but the gap in between remains unfilled.

• Historical Context: Botvinick uses the history of genetics as an analogy. Mendelian genetics predated Watson and Crick's discovery of the DNA double helix. The concept of "gene" used in the former, though metaphorical, provided direction for finding the physical mechanism. He believes that concepts like "attention" and "memory" in current cognitive psychology are similar, valuable metaphors.
• Mechanism Deconstruction: He argues that the ultimate goal of neuroscience is to study the brain's "purpose" (i.e., generating adaptive behavior), and psychology is the science that studies this "functional structure." Therefore, these two disciplines should not be separated. He likens the brain to a kidney: first, its function (filtering blood) must be clarified, then its specific mechanisms can be studied.
• Inference and Signals: Botvinick predicts that neuroscience will see a revival of "behavioral research" in the future, using new technologies (e.g., virtual reality environments for mice) to introduce richer behavioral paradigms and combine them with AI computational mechanisms to bridge the middle gap. Verification Signal: Whether more research combining precise psychological experiments with high-throughput neural recording techniques will emerge.

Meta-Reinforcement Learning: An Automatic "Learning to Learn" Mechanism Revealed by AI

Botvinick elaborates on the core concept of "meta-reinforcement learning," pointing out that this is a key insight from AI research feeding into neuroscience—a spontaneously emergent, rather than artificially designed, "learning to learn" mechanism.

• Mechanism Deconstruction: Meta-learning refers to the process where one learning algorithm gives rise to another learning algorithm. Botvinick's team found that when training a recurrent neural network (RNN) with a reinforcement learning algorithm to perform a series of related tasks, a nearly "magical" phenomenon occurs: the slow weight adjustment process shapes the network's activation dynamics, and this dynamic itself becomes a brand new, fast learning algorithm. Even with frozen weights, the RNN can still rationally explore and exploit in new tasks through its internal dynamics.
• Data Chain: The core of this research is that as long as the system has memory (e.g., the recurrent structure of an RNN or a memory-augmented neural network), its function is shaped by reinforcement learning, and the training environment includes a series of tasks sharing an abstract structure, meta-learning happens automatically and inevitably. (Cited from the paper Meta-Learning in Memory Augmented Neural Networks)
• Inference and Signals: Botvinick believes that the prefrontal cortex (with its highly recurrent structure and working memory) in the human brain may support reinforcement learning through this exact mechanism. The dopamine system is responsible for slow synaptic learning, while the prefrontal cortex uses its recurrent dynamics to achieve fast, activation-based learning. Falsification Signal: If direct neural evidence consistent with the meta-reinforcement learning framework cannot be found in animal experiments for the relationship between prefrontal cortex dynamics and behavioral strategy updates.

Dopamine: A Distributed "Surprise Signal" Encoder

Botvinick introduces new findings from his team regarding dopamine, suggesting that this key neurotransmitter may not convey a single "prediction error" but instead use a refined "distributional code" to represent future uncertainty.

• Background and Mechanism: In traditional reinforcement learning, value signals and prediction errors are typically represented as a single number (e.g., "expected reward of 5 yuan"). "Distributional reinforcement learning" generalizes this representation to a probability distribution (e.g., "50% chance of getting 10 yuan, 50% chance of getting 0 yuan").
• Data Chain: AI research has shown that using distributional codes can significantly accelerate reinforcement learning and lead to better policies. The reason is that it preserves the richness of internal representations, avoiding the "squeezing" of different situations (with the same expected value) into one, thereby retaining more effective information for subsequent learning.
• Inference and Signals: Botvinick's team, in collaboration with Naoshige Uchida's lab at Harvard University, made a specific prediction: if the brain also uses distributional codes, then in specific behavioral tasks, the activity patterns of dopamine neurons should not be a single signal but rather reflect the probability distribution of different outcomes. Experimental data confirmed this prediction (paper A distributional code for value in dopamine-based reinforcement learning). Verification Signal: Whether other labs can replicate this finding in independent experiments.

The "Warmth Dimension" of Human-AI Interaction: The Ultimate Turing Test for AGI

Botvinick believes that AI research is overly focused on the "competence" dimension while neglecting the "warmth" dimension. Creating an AI system that people feel is equally warm and trustworthy is the real grand challenge.

• Mechanism Deconstruction: He cites research by social psychologist Susan Fiske, pointing out that humans evaluate others primarily along two dimensions: competence and warmth. Current AI efforts are almost entirely focused on enhancing competence (e.g., superhuman Go, protein folding), but have barely touched the "warmth" dimension.
• Inference and Signals: Botvinick argues that the "warmth" of AI is not simply achieved by putting a cute face on a machine; it requires a complex set of behaviors that make humans feel that its "care" is genuine, not simulated. He suggests this could be the ultimate Turing test: an AI system must not only pass a conversational test but also make humans genuinely feel "this is a good guy." Falsification Condition: If no algorithmic framework can be designed that enables humans to develop genuine emotional trust (rather than simple manipulation) in long-term interaction, then achieving this goal will be far off.

Judgments Worth Remembering

1. "Meta-reinforcement learning happens automatically, not by design." (Matt Botvinick) — As long as a system with memory (e.g., a recurrent neural network) is trained with a reinforcement learning algorithm and experiences enough variation across a task distribution, meta-learning emerges automatically, forming the internal dynamics of "learning to learn." This is a common underlying mechanism that AI and the brain may share.

2. "Dopamine may not be a single surprise signal but a distributional code." (Matt Botvinick) — The traditional view holds that dopamine neurons encode a single "reward prediction error" (a number). But new research (including AI validation) suggests that dopamine signals may encode various possibilities of future rewards in a distributional form, rather than a simple weighted average, preserving information and accelerating learning.

3. "The ultimate Turing test for AI is not competence but 'warmth'." (Matt Botvinick) — When humans evaluate others, competence and warmth are two independent dimensions. Creating an AI that is exceptionally capable but also makes humans feel sincere, trustworthy, and willing to form an emotional connection is the real challenge.

4. "Cognition is the result of the interaction between environment and system, not solely determined by brain structure." (Matt Botvinick) — To understand human cognition, equal importance must be given to environmental structure and the cognitive system itself. Self-play AI (like AlphaGo) is the best example, where the opponent becomes part of the environment, and competition (environment) drives learning.

5. "The prefrontal cortex is 'anti-habit'; it allows you to override automatic behavior when needed." (Matt Botvinick) — The brain has a habit system (automatic behavior) and a goal-directed system dominated by the prefrontal cortex. The prefrontal cortex enables you to remember "now touch elbows," thereby overriding the automatic habit of extending a hand for a handshake.

6. "Psychology and neuroscience should be the same science, both aiming to understand behavior." (Matt Botvinick) — He argues that the goal of neuroscience is to understand the brain's "purpose", i.e., generating behavior. Therefore, psychology (studying the functional structure of behavior) and neuroscience (studying its physical mechanisms) are two sides of the same coin and should not be separated.