#258 – Yann LeCun: Dark Matter of Intelligence and Self-Supervised Learning — Summary & Takeaways

This Issue Overview

Guest Background: The guest for this issue is Yann LeCun (Meta Chief AI Scientist, NYU Professor, Turing Award winner).

Main Theme: LeCun systematically expounds his core theory — self-supervised learning is the "dark matter" driving intelligence development, and points out the efficiency bottleneck of current AI systems (relying on supervised learning and reinforcement learning), and how to break through this bottleneck by building a Joint Embedding Predictive Architecture to move towards general artificial intelligence.

The Most Weighty Judgment in the Entire Episode: LeCun believes that the efficiency of current AI systems (especially through reinforcement learning) is extremely low, and the root cause is the lack of the ability to learn predictive models by observing the structure of the world; the core of solving this problem lies in "self-supervised learning", that is, building a world model by predicting missing information. This is the essence of intelligence, not the current mainstream supervised or reinforcement learning paradigm.

Topic 1: Why is "Self-Supervised Learning" the "Dark Matter" of Intelligence?

Yann LeCun argues that humans and animals learn far more efficiently than current AI, primarily because they use a "self-supervised" approach to build rich background knowledge (world models) simply by observing the world, without external labels.

The Efficiency Gap: LeCun points out that the two dominant AI paradigms today—supervised learning and reinforcement learning—are extremely inefficient. "Supervised learning requires a huge number of samples to learn anything, while reinforcement learning requires a ridiculously large number of trials for the system to learn anything." He uses the example of human learning to drive: a teenager can learn to drive in about 20 hours of practice, while an autonomous driving system, even after millions of hours of simulated practice, still cannot reliably learn to drive. Behind this vast gap lies the "background knowledge" or "common sense" that humans possess, accumulated by observing the world.
The "Dark Matter" Analogy: LeCun calls this form of learning, which is largely absent from current AI systems, the "dark matter" of intelligence. He believes that in the first few months after birth, infants learn how the world works almost entirely through pure observation, thereby building a world model. This learning is not task-specific and is not driven by external rewards; instead, it fills information gaps through prediction—for example, predicting what happens next after seeing a video clip, or predicting a masked word in a sentence. "So, in a sense, the essence of intelligence is the ability to predict."
Data Volume Comparison: LeCun uses a "cake" analogy to quantify the amount of information per feedback received by different learning paradigms. In reinforcement learning, each feedback is just a scalar (good or bad); in supervised learning, such as ImageNet classification, each feedback is about 10 bits (1000 categories). In self-supervised learning, however, such as predicting the next frame of a video, the feedback is a whole video segment, carrying massive information. "In a self-supervised setting, the signal is much richer than in supervised or reinforcement learning settings."

Topic 2: Joint Embedding Predictive Architecture—How to Handle Prediction Uncertainty?

Yann LeCun points out that the key challenge in implementing self-supervised learning is how to handle the uncertainty of the world, and he proposes the Joint Embedding Predictive Architecture (JEPA) as a solution.

Core Challenge: The world is inherently uncertain (for example, predicting the future from a video clip has countless possible outcomes). Traditional generative models (such as predicting pixels) need to handle distributions in high-dimensional continuous spaces, which is extremely difficult. Language succeeds because its prediction space is discrete (a large vocabulary) and can be easily represented as a probability distribution.
Paradigm Shift of JEPA: LeCun suggests that instead of directly predicting pixels, the system should learn an abstract, informative representation. Specifically, two parallel neural networks process the input (e.g., the beginning of a video) and the target (e.g., the continuation of the video). The training objective is for these two networks to output highly mutually predictable representations, but filter out unpredictable details (such as background texture). "You want the system to learn a function whose output does not change under those specific transformations of the input."
Non-Contrastive Methods: LeCun emphasizes that his current most exciting direction is "non-contrastive methods," such as Barlow Twins and VICREG. These methods do not require "negative samples" (i.e., dissimilar image pairs); instead, they prevent "information collapse" (the system outputting the same vector for all inputs) by maximizing the mutual information between the outputs of the two networks. "VICREG... is the most exciting thing in machine learning for me in the last 15 years." This approach can effectively learn representations useful for tasks like image classification, but it loses localization information, which precisely confirms the view that different tasks require different levels of representation.

Topic 3: The Future of AI Learning Paradigms—From "Engineered" to "End-to-End Learning"

Yann LeCun believes that the development trajectory of AI, like all other pattern recognition fields, will gradually evolve from "heavy hand-engineering" to "fully end-to-end learning," and self-supervised learning is the key to achieving this goal.

Historical Inevitability: LeCun illustrates this evolution using examples from OCR, speech recognition, and computer vision: initially, people relied on hand-crafted feature extractors (e.g., Fourier transforms, SIFT features) and then applied a simple machine learning model on top. As data and computing power increased, people began to train the entire system, from raw input to final output, using a single large neural network. "I see a continuous evolution from less and less hand-crafting to more and more learning." He believes this is analogous to the evolutionary process in biology.
Views on Multimodality and Active Learning: LeCun believes that multimodal learning (e.g., combining images and text) is an "interesting short-term problem," but not the fundamental solution to intelligence. He acknowledges that active learning (exploring world uncertainty through interaction) and learning causal models (observing outcomes through actions) are crucial for improving learning efficiency, but they are not the core issue. "I worry about active learning, once that question is answered... If active learning or interaction improves efficiency by orders of magnitude, then it might become completely different." He emphasizes that the fundamental problem of "how to learn background models through observation" must be solved first.
Generality of AI and Emotions: LeCun firmly believes that strong AI will eventually be achieved, "Machines will eventually become smarter than humans in all areas of human intelligence—that is an unreserved 'yes' from me." However, he thinks this will take a long time and is much harder than many people (like Musk) imagine. He stresses that emotions are an indispensable component of autonomous intelligent systems, because a system with intrinsic motivation and long-term prediction capabilities will inevitably generate emotions such as fear and joy. "The idea in Star Trek that Commander Data can turn off his emotion chip—I think that is absurd."

Mentioned Tickers

(This section is an interview and does not involve specific investable ticker position moves or in-depth business analysis, so the table format is not applicable.)

Memorable Judgments

1. Yann LeCun: "The essence of intelligence may simply be a specific form of statistics." Support: He argues that learning causal models, world models, and other deep mechanisms is inherently achieved through statistical learning. The key is that such statistics must be "smart" enough to infer structural causal relationships from observations, rather than simple pattern matching.

2. Yann LeCun: "The blind spot of current AI is that we have not yet been able to replicate a cat's brain." Support: Using the 800M neuron cat brain as a benchmark, he points out that cats possess rich intuitive physics and causal models, while the most advanced AI systems today fall far short of a domestic cat in terms of common sense. Solving this "low-level" intelligence problem is the necessary path to artificial general intelligence.

3. Yann LeCun: "Data augmentation is a 'necessary evil' but will eventually be replaced by purer prediction masking methods." Support: Current non-contrastive learning methods rely heavily on pre-designed data augmentations (e.g., rotations, color changes), which themselves introduce prior knowledge. LeCun is more optimistic about methods similar to the "masked denoising autoencoder" in natural language processing, such as training by randomly masking out some pixels in an image (e.g., Masked Autoencoder), which does not require complex data augmentation.

4. Yann LeCun: "Consciousness is not the result of our brain's power, but rather a manifestation of its limitations." Support: The human brain has only one "world model engine" and can only handle one task at a time. Consciousness acts as an "executive controller" that configures this world model for the current scenario. If there were multiple models that could run simultaneously, this "consciousness" would not be needed. This explains why highly skilled actions (e.g., driving) become "subconscious."

5. Yann LeCun: "Complexity is determined by the 'observer's' glasses." Support: Applying a random but fixed pixel permutation to an MNIST digit makes it complex for humans but simple for an "alien" who knows the permutation rule. Therefore, any measure of complexity contains an "arbitrarily large additive constant," which hinders theoretical understanding of complex systems (e.g., self-organization, the origin of life).

6. Yann LeCun: "The biggest challenge in AI over the next 10 years is to enable machines to learn to handle uncertainty and build predictive models that cope with the complexity of the real world." Support: Classical model predictive control (MPC) relies on hand-crafted deterministic models. What AI needs is to automatically learn, like humans, a world model through observation that can handle complex, nonlinear, and uncertain phenomena, such as rubber, water flow, crowd interactions, and other phenomena that are difficult to describe from first principles.