This interview features AI pioneer Ian Goodfellow, inventor of GANs. He argues deep learning isn't about learning abstract layers but like a program that refines results step by step. He's optimistic about GANs, which let AI generate realistic images and learn with few labels. Three key items: GANs (his invention that creates fake but convincing images); Deep Boltzmann Machines (an older technique that couldn't handle color photos); DCGAN (a 2015 upgrade that made GANs generate faces and became a template for later models).
Ian Goodfellow discussed core perspectives on deep learning and Generative Adversarial Networks (GANs) on the Lex Fridman podcast. He noted that the primary limitation of deep learning is its need for large amounts of data, especially labeled data. For example, humans can learn to play Pong without
Ian Goodfellow (author of the deep learning textbook, proposer of GANs, and former Google Brain researcher) discusses with Lex Fridman the essence of deep learning, the mechanisms and evolution of generative adversarial networks (GANs), as well as the prospects for AI safety and general intelligence. The most weighty judgment in the entire video: Goodfellow argues that "deep learning is essentially learning a multi-step sequential program, rather than layer-by-layer abstract representations"—this perspective redefines neural networks from a "feature hierarchy pyramid" to a "reasoning process that progressively refines states."
Ian Goodfellow argues that the core value of deep learning lies not in learning hierarchical abstract representations, but in enabling multi-step sequential processing.
Goodfellow likens neural networks to computer programs: depth represents the number of sequential steps executed, while width represents the number of parallel steps executed. He recalls that when Hinton demonstrated deep belief networks in 2006, the academic community widely believed that "the first layer learns edges, the second layer learns corners, and the final layer produces 'grandmother cells' that recognize specific objects." But today, "I think most people prefer to view it as a computer program—each additional layer means one more update before you output the final number. I don't think the 150th layer of ResNet is a grandmother cell, nor is the 100th layer something like contours."
He further explains that this processing is more like "refinement" than "replacement": "It is not a process of replacing representations, but a process of refinement. In a sense, this is somewhat like reasoning—not in the form of deductive reasoning, but taking an idea and refining it repeatedly until it is good enough."
Regarding the current limitations of deep learning, Goodfellow points out two aspects: first, it requires large amounts of data (especially labeled data), "humans do not need to play Pong 2 million times to learn, but reinforcement learning algorithms do"; second, deep learning is typically just a submodule of a larger system, "AlphaGo uses deep learning models to estimate value functions, but it also has other components. No one has truly proposed using deep learning alone as the sole component of intelligence."
Goodfellow recounts the origin story of GANs and explains why they succeeded where Deep Boltzmann Machines failed.
In 2014, Goodfellow conceived the idea of GANs while arguing with friends at a bar. His friends' skepticism was rooted in the belief that "training one neural network is already difficult enough; it's impossible to train a second network inside the outer network's inner loop"—a problem akin to the difficulty of synchronizing the positive and negative phases in Deep Boltzmann Machines (DBMs). Goodfellow admits: "Many people tried in the lab to get DBMs beyond MNIST and generate color photos, but they just couldn't keep the two processes in sync."
Why did GANs succeed? Goodfellow confesses there is still no single explanation: "There are many theoretical papers showing that, under certain theoretical assumptions, the GAN algorithm does converge. But those assumptions are restrictive enough that they may not explain all the results we see in practice."
The core mechanism of GANs: A two-player game—the Generator produces fake data, and the Discriminator judges whether the data is real or fake. The Nash equilibrium occurs when the generator perfectly captures the true data distribution, leaving the discriminator to guess randomly. Goodfellow highlights a counterintuitive fact: "If generative models actually did what we ask them to do, they would only do one thing—memorize the training data. The optimal solution for maximum likelihood estimation is to assign all probability mass to the training samples. For GANs, the game is also played on the training set—to become invincible, you have to memorize the training samples."
Why doesn't the generator simply memorize the training data? Goodfellow cites the work of his former intern Vaishnav Nagarajan: "From a statistical learning theory perspective, it's actually very hard for the generator to memorize the training data—it requires many learning steps and latent variable observations." However, he acknowledges: "We still don't have a good explanation for why the generated new samples are convincing images rather than garbage that differs from the training set. Given how vast the space of possible images is and how few the generative model sees during training, this seems unreasonable."
Goodfellow believes part of the answer lies in the "Deep Image Prior": "You can take a convolutional network, without even learning the parameters, and use only the model architecture to perform image inpainting. This suggests that the convolutional network architecture itself captures important information about image structure, without needing to acquire all of it through learning."
Goodfellow outlines key milestones for GANs from 2014 to the present, demonstrating applications beyond image generation.
Historical Timeline:
Application Expansion:
1. Semi-Supervised Learning: Goodfellow's OpenAI colleague Tim Salomons (now at Brain) demonstrated in 2016 that using the GAN discriminator as a classifier achieved an error rate below 1% on MNIST with only 100 labeled samples—compared to the approximately 60,000 labeled samples required by traditional methods, a 600-fold reduction.
2. Domain Adversarial Learning: The feature extractor competes with the domain classifier, making features statistically consistent across different domains. "The author had this idea before the GAN paper, but the paper was published later, though it politely cited the GAN paper."
3. Differentially Private Data Generation: Work from Casey Green's lab showed that GANs can be trained with differential privacy, and the generated synthetic data (e.g., medical records) carries the same privacy guarantees, making it available for other researchers.
4. Fairness: Using a domain-adversarial approach, the model is trained so that it cannot infer sensitive variables (e.g., gender) from features. Goodfellow also envisions using CycleGAN to transform data from one group to another for fairness audits.
Latest Advances: The Brain Zurich team, building on DeepMind's BigGAN, matched BigGAN's performance on ImageNet using only 10% of the labeled data (ImageNet has approximately 1.2 million images, all labeled). The approach uses a clustering algorithm to enable the discriminator to learn object grouping.
Goodfellow’s thinking on adversarial examples has undergone a major shift—from “revealing fundamental flaws in machine learning” to “primarily a security liability.”
Evolution of Perspective: “When I first started researching adversarial examples, I mainly saw them as revealing a big problem in machine learning, and we hoped to close the gap between how models and humans respond to adversarial examples. After more research, I still believe adversarial examples are important, but now I view them more as a security liability, rather than necessarily indicating that machine learning has a unique problem compared to humans.”
On the Trade-off Between Adversarial Training and Accuracy: Goodfellow notes that the first adversarial training in 2014 improved accuracy on both adversarial and clean examples on MNIST, but “this did not hold true on other datasets, especially when facing strong adversaries. When you face a truly powerful adversary, it seems you have to make a sacrifice.”
Adversarial Attacks in Speech Recognition: This has evolved from “hidden voice commands” in 2016 (inaudible to humans but recognized as commands by phones) to “sounds that appear normal but are recognized as different sentences.” “The perceptibility of adversarial perturbations remains relatively high—they sound like rustling in the background, but these rustling sounds are precisely the adversarial perturbations that cause the phone to hear entirely different content.”
Future Defense Directions: Goodfellow is particularly optimistic about dynamic models—“models that change with each prediction.” “Currently, we train a model and then freeze it, always outputting the same answer for the same input, which is a sitting duck from a security perspective. If the model continuously updates its predictions, it becomes harder for adversaries to find and repeatedly exploit the same error.”
Goodfellow is optimistic about the path to AGI but emphasizes the need for fundamental changes in environment and data.
Prerequisites:
1. Better training environment: "We want an agent to have extremely diverse experiences. Today we have multiple models, each doing one thing, typically trained on a single dataset or in a single RL environment. We don't have an agent that can seamlessly switch from one experience to another, truly integrating all the different experiences across its lifetime."
2. Massive compute: "It's hard to imagine exactly how much is needed."
3. Multimodal integrated data: "The human brain receives all different senses, with experiences like riding a bike, driving a car, talking to people, reading, etc. When we obtain this integrated dataset, combined with machine learning models that can interact in a closed loop, we may find algorithms not too different from today's that, when scaled and trained on massive multimodal data, learn something truly interesting."
Goodfellow's "less glue" intelligence test: "Imagine not having to go to the CIFAR website to download CIFAR-10, write a Python script to parse it—you just point the CIFAR-10 problem to an agent, and it downloads, extracts the data, trains the model, and starts making predictions on its own. No need for humans to assemble the pipeline at every step—if it knows how to preprocess the data to successfully complete the task, it's hard to argue that it doesn't truly understand the task in some fundamental sense."
On interpretability: Goodfellow believes this is one of the most "mature" areas for breakthroughs today. "We don't even have the right definition yet. It's like differential privacy—Cynthia Dwork and others gave a technical definition; before that, everything was vague, and once the definition existed, we could design randomized algorithms that guarantee privacy. If we can define a measurable concept related to interpretability, that would be a huge leap forward, even without new algorithms that increase that quantity."
| Position | Analyst View | Key Data |
|---|---|---|
| GANs (Generative Adversarial Networks) | Bullish (core contribution) | Proposed in 2014; <1% error rate on MNIST with 100 labeled samples (600x reduction); Brain Zurich matched BigGAN performance with 10% labeled data |
| Deep Boltzmann Machine (DBM) | Risk Warning (historical limitations) | Unable to scale beyond MNIST to color photo generation |
| DCGAN | Bullish (milestone) | Proposed by Alec Radford et al. in 2015; became the "backbone" of subsequent GANs |
| BigGAN (DeepMind) | Neutral (benchmark) | All 1.2 million images on ImageNet fully labeled |
| LapGAN | Neutral (historical contribution) | Emily Denton et al. achieved the first high-resolution GAN |
| ResNet | Neutral (as example) | Layer 150 is not a "grandmother cell"; layer 100 is not a "contour" |
| PixelCNN (Autoregressive Model) | Neutral (alternative approach) | Generates images pixel by pixel; slow but density computable |
| TruePick (Startup) | Bullish (certification direction) | Provides image authenticity verification mechanism |
1. Goodfellow: Deep learning is essentially learning multi-step sequential programs, not layer-by-layer abstract representations
— Depth = number of sequential steps, width = number of parallel steps; each layer does not learn different levels of abstraction but progressively refines the state.
2. Goodfellow: Generative models should theoretically memorize training data, yet in practice they generalize to produce reasonable new samples, which still lacks a satisfactory explanation
— "Given how large the space of possible images is and how much the generative model sees during training, this seems unreasonable." Part of the answer comes from the "deep image prior"—the convolutional network architecture itself captures image structure.
3. Goodfellow: Adversarial training improves both clean and adversarial accuracy on MNIST, but must make trade-offs against strong adversaries
— The positive results from the first experiment in 2014 were not replicated on other datasets, indicating a fundamental trade-off.
4. Goodfellow: There is still no single explanation for the true reason behind the success of GANs
— "There are many theoretical papers showing convergence under certain settings, but these settings are insufficient to explain all practical results." This contrasts with the failure of deep Boltzmann machines, which also have a dual process but cannot synchronize.
5. Goodfellow: Adversarial examples are more of a security liability than a fundamental difference between machine learning and humans
— The view has evolved from "revealing fundamental flaws" to "security liability"; the defense direction favors dynamic models (changing with each prediction).
6. Goodfellow: What the field of interpretability most lacks is a measurable definition, similar to the breakthrough of differential privacy
— "Now everyone talks about interpretability, but it's just personal opinion. If we could define a measurable concept, algorithms might catch up quickly."
7. Goodfellow: AGI requires a "less glue" test—an intelligent agent must complete the entire pipeline from data acquisition to prediction on its own
— "Give it a URL or a text description, and it downloads, extracts, trains, and predicts on its own—no need for humans to assemble each step."
8. Goodfellow: The simple version of using GANs for data augmentation (generating more data of the same class to train a classifier) has not been successful
— A more feasible direction is: train multiple different generative models and leverage their different generalization methods to form an ensemble; or generate data with different properties (such as differential privacy).