Jeremy Howard: fast.ai Deep Learning Courses and Research

Understood. Following your instructions, here is the analysis and interpretation based on the podcast transcript.

At a Glance

Guest Jeremy Howard is the founder of fast.ai, a Distinguished Research Scientist at the University of San Francisco, and a former President of Kaggle. The main thread of this episode explores the accessibility of deep learning education, practical methodologies, and the constraints programming languages place on research efficiency. The most significant judgment of the entire episode is: Howard believes that all major breakthroughs in AI over the next 20 years will be achieved on a single GPU, and multi-GPU training is a waste of time for the vast majority of researchers.

Deep Learning Education: Lowering the Barrier, Empowering Domain Experts

Jeremy Howard believes the greatest social value of deep learning lies in empowering domain experts, rather than cultivating more algorithm specialists.

• Motivation & Mission: Howard's founding of fast.ai stemmed from his frustration in a medical AI startup. He found that rather than becoming an expert in every field himself, it was more effective to teach those who already possessed data and domain knowledge (e.g., doctors, biologists) how to use deep learning tools. He observes that in Kaggle and the medical field, deep learning has reached a tipping point and is about to become the mainstream method in all areas, but the biggest bottleneck is the lack of practitioners who can apply it to real-world problems.
• Course Philosophy: The core of the fast.ai course is to "cut the BS," focusing on what is practical and cutting-edge. Howard emphasizes that the course is designed so that a "very proficient programmer" can complete it in 70 hours and be able to train world-class models. He believes the key to success is not intelligence, but "tenacity" — those who don't give up will eventually succeed.
• Teaching Practice: Howard's teaching method emphasizes "training a lot of models." He believes real learning happens through hands-on practice: from creating your own dataset (e.g., scraping data from Google Image Search with a script), to fine-tuning a pre-trained model, to analyzing the model's misclassifications. He specifically notes that in the second lesson of the course, students can create a web application that distinguishes between different types of bears, a process that takes only a few minutes.

Programming Language Bottleneck: Python's "Unhackability" and the Hope of Swift

Jeremy Howard believes the biggest bottleneck in current deep learning research is the Python language itself, which severely hinders algorithmic innovation.

• Python's Flaws: Howard points out that Python is slow, making it "extremely unhackable." Once a researcher needs to optimize underlying algorithms (e.g., loops in recurrent neural networks, sparse convolution kernels), they must delve into CUDA C, which is beyond the capability of most researchers. This leads to many promising research directions (e.g., sparse convolutional neural networks, active learning) being neglected because they are too difficult to implement.
• Historical Perspective: Howard reviews the history of programming languages, arguing that the mainstream world has overlooked the branch of "array-oriented languages" (e.g., APL, J, K). He highly praises the "expressiveness, composability, and beautiful design" of the J language, and mentions its commercial offshoot, the K language. Due to its extreme speed and compactness (the entire runtime can fit into a CPU's L3 cache), it is used by the world's top hedge funds, but costs $100,000 per CPU.
• Future Hope: Howard pins his hopes on Swift and the MLIR (Multi-Level Intermediate Representation) project. He believes Swift aims to be "infinitely hackable," while compiler technologies like MLIR (e.g., Halide, Tensor Comprehensions) allow researchers to describe computations using domain-specific languages (DSLs), which the compiler then automatically optimizes into efficient GPU code. He estimates it will take another three years for Swift to become practical for deep learning.

Critique of the Research Ecosystem: Neglected Practical Innovations

Jeremy Howard severely criticizes the current research culture in deep learning academia, arguing it rewards "safety" over "practicality," causing many valuable discoveries to be buried.

• "Useless" Research: Howard states bluntly that most research in deep learning is a "complete waste of time." Because scholars need to publish papers, they tend to work on "safe" topics that are familiar to their peers and yield marginal improvements, rather than directions that have significant practical impact but are "unpopular."
• Neglected Gems: He uses the phenomenon of Super-Convergence as an example. Researcher Leslie Smith discovered that with specific learning rate schedules, certain networks could train 10 times faster with better generalization. However, this paper was rejected by the academic community because it could not explain the underlying principle. Howard argues that the deep learning field "has no interest in publishing experimental results that cannot be explained," which is starkly different from other scientific fields like physics. He drew key inspiration from such unpublished papers and applied them to fast.ai's courses and the DawnBench competition.
• Misleading Datasets: Howard believes that large datasets like ImageNet (1.3 million images) encourage "uncreative thinking." He released two alternative datasets: ImageNette (an easy-to-classify subset of ImageNet) and ImageWoof (a hard-to-classify subset containing only dog breeds). These two datasets can be trained on a single GPU in 10 minutes, and the results are almost directly transferable to the full ImageNet, allowing researchers to iterate on ideas much faster.

Position Moves

Position	Guest's Stance	Key Data
NVIDIA	Risk Warning	"Massively overcharging" for high-end enterprise GPUs due to a lack of competition.
Google (TPU)	Risk Warning	TPUs are almost "completely unprogrammable" because Google restricts direct access to protect its IP.
Google (TensorFlow)	Risk Warning	The TensorFlow codebase is "extremely complex and messy"; TF-Eager mode is 10 times slower than PyTorch.
IBM (Watson)	Risk Warning	Like Google, promoting the wrong idea that "more data and more computation" is needed.
Amazon (AWS)	Neutral	Setting up GPU instances is "still too difficult," less convenient than platforms like GCP and PaperSpace.
Microsoft (Access)	Bullish	Believes its programming environment (VBA + relational database) is "the best ever created" and remains unsurpassed.

Judgments Worth Remembering

1. "All major breakthroughs in AI over the next 20 years will be achieved on a single GPU." (Jeremy Howard) — Rationale: All key breakthroughs of the past 20 years (BatchNorm, ReLU, Dropout, ConvNets, GANs) did not require multiple GPUs. Multi-GPU training is a "waste of time" for the vast majority of researchers and slows down iteration speed.

2. "Most research in deep learning is a waste of time." (Jeremy Howard) — Rationale: Academia rewards "safe" marginal improvements over innovations with significant practical impact. Discoveries like Super-Convergence, which can train models 10 times faster, are rejected for publication because the principle cannot be explained.

3. "Python is the biggest bottleneck for deep learning innovation." (Jeremy Howard) — Rationale: Python is slow, making it "extremely unhackable." Researchers cannot easily innovate on core algorithms for RNNs or sparse CNNs because they must delve into CUDA C. He pins his hopes on Swift and MLIR to solve this.

4. "The key to success is not intelligence, but tenacity." (Jeremy Howard) — Rationale: In his teaching at fast.ai, he observes that students from diverse backgrounds, as long as they don't give up, eventually succeed. Giving up is the only reason for failure.

5. "Don't try to become a deep learning expert; become an expert at using deep learning to solve problems in a field you love." (Jeremy Howard) — Rationale: Society doesn't need more scholars studying "slightly evolved" topics; it needs domain experts who can use AI to diagnose malaria, analyze media bias, or study fisheries ecology. True innovation comes from solving real problems.

6. "Array-oriented languages (like J, K) are far more powerful than mainstream languages but have been ignored for 60 years." (Jeremy Howard) — Rationale: The J language is "the most expressive, composable, and beautifully designed language"; its commercial offshoot, the K language, has a runtime that fits into a CPU's L3 cache, wins all benchmarks for data processing speed, but costs $100,000 per CPU.

7. "VC-backed startups are more terrifying than bootstrapped ones." (Jeremy Howard) — Rationale: VCs force you to pursue "thousand-fold returns" on growth, whereas bootstrapping allows you to grow at your own pace and achieve a "lifestyle exit" (e.g., selling for $10 million), which is enough for a founder to achieve financial freedom.

8. "To learn anything, first learn 'how to learn'." (Jeremy Howard) — Rationale: He uses Anki (spaced repetition software) to learn Chinese and creates "funny, disgusting, sexy, or related to acquaintances" stories for each word to enhance memory. For 25 years, he has spent at least half of each day learning or practicing something new.