Jeremiah Lowin: Explaining the New AI Paradigm - [Invest Like the Best, EP.307] — Summary & Takeaways · Colossus (Invest Like the Best / Business Breakdowns)

Jeremiah Lowin: Explaining the New AI Paradigm - [Invest Like the Best, EP.307]

At a Glance

Jeremiah Lowin is the founder and CEO of Prefect, a data orchestration company, and a long-time technical advisor to Patrick O'Shaughnessy. This episode centers on the new AI paradigm triggered by OpenAI's release of ChatGPT, focusing on explaining how pre-trained Transformer models work, the technological evolution of generative AI, and the implications of these technologies for investing and entrepreneurship. Lowin argues that the most underestimated power of current AI models lies in the "latent space"—it allows humans to map abstract concepts into mathematical representations, thereby eliminating friction between "idea and output" in unprecedented ways.

1. Pre-trained Transformer: Not a Dictionary Translation, but an "Attention"-Driven Sequence Transformation

Jeremiah Lowin argues that the core capability of the Transformer model does not lie in word-for-word correspondence, but in processing contextual relationships within a sequence through the "attention mechanism."

Mechanism Breakdown: The Transformer transforms an input sequence (e.g., an English sentence) into an output sequence (e.g., a French sentence), but it does not simply look up a dictionary. Lowin points out that differences in word order, gender, and idioms across languages make literal translation infeasible. The Transformer applies context through the "attention" mechanism—where the model learns to "focus on" more important parts of the sequence—allowing subsequent decisions to be based on previously seen information.
Analogy: Lowin uses translation as an example—"Adjective positions differ, grammar differs, idioms cannot be translated literally—the complexity explodes instantly." It is through this attention mechanism that the Transformer handles such explosive complexity.
Key Data: These models are called "foundational models" because they can process multiple forms of input—text, images, time series, and more.

> "These Transformers are a way of processing sequences, applying different techniques, such as focusing on what's more important—literally 'paying attention'—or applying context in a certain way so that you can make subsequent decisions based on what you've seen before." —Jeremiah Lowin

2. Latent Space: The Mathematical Bridge from "Spider-Man Smashing a Watermelon" to Pixels

Lowin believes the core magic of generative AI lies in the "latent space"—an intermediate layer that maps abstract concepts into independently adjustable mathematical variables.

Mechanism Breakdown: The mapping from input (e.g., the text prompt "Spider-Man smashing a watermelon") to output (pixel image) is extremely complex, making direct conversion nearly impossible. The model first maps the input into the latent space—a high-dimensional mathematical space where each concept (e.g., "red," "hand," "watermelon," "action") is encoded as an independent numerical dimension. It then generates the output from the latent space.
Historical Context: Lowin uses the previous-generation GAN (Generative Adversarial Network) model to illustrate—in the GAN's UI, users can see sliders for "smile level," "hair length," etc., each corresponding to a dimension in the latent space. Modern diffusion models have massive latent space dimensions, but the principle is the same.
Key Insight: The core difference between Stable Diffusion and DALL·E 2 is that DALL·E 2 performs diffusion (iterative denoising) in pixel space, while Stable Diffusion performs diffusion in latent space. Lowin points out that this "small insight" leads to a huge difference in quality and expressiveness, because the latent space is not subject to the spatial constraints of pixel space (e.g., having to generate five fingers).

> "The latent space is the source of power for these models. We map the input to a space where each concept is as independently identifiable as possible, which makes the work of output transformation much easier." —Jeremiah Lowin

3. Model Iteration: Not Just "More Data" — Architectural Innovation Is the Key to Leaps

Lowin believes that progress between model versions comes from three levels: more training/data, new training methods, and entirely new architectures — the latter carries the highest risk but also the highest reward.

Data chain: Lowin provides a specific example — the progression from Stable Diffusion 1.4 to 1.5 mainly involved weight updates (more training) rather than architectural changes. In contrast, the shift from DALL·E 1 to DALL·E 2 involved a completely different architecture (from a non-diffusion model to a diffusion model), representing "a huge risk."
Mechanism breakdown: The advancement of ChatGPT (based on GPT-3.5) came from a new form of reinforcement learning — more efficient, faster convergence, and requiring less data. OpenAI also uses other AI models to evaluate outputs and assign reinforcement scores, creating a cycle of "AI training AI."
Unique insight: Midjourney's "secret sauce" lies in the consistently high aesthetic quality of its images. Lowin speculates that Midjourney may have found some architecture or guidance mechanism that consistently steers outputs toward high-quality regions — "it's hard to get Midjourney to produce a bad image."

The essence of the training process: A model has an architecture (processing pipeline) and weights (tens of billions of numbers). Training involves changing these weights through repeated iterations — running inputs, judging the quality of outputs, and mathematically calculating how to adjust each weight to move closer to the target. This process is extremely compute-intensive and time-consuming.

> "What we call 'state of the art' will be obsolete 12 months from now. I'm not talking about more training, but new architectures, new applications, entirely new things." — Jeremiah Lowin

4. Competitive Landscape: Capital Is Not a Barrier; Proprietary Data and Model "Secret Sauce" Are the Moat

Lowin argues that the capital threshold for training frontier models is overestimated, and the true competitive moat lies in proprietary datasets and model-level "secret sauce."

Data Pipeline: Training the original Stable Diffusion model used approximately 4,000 A100 GPUs, with an annual cost of around $40 million—"a very reasonable, even small cost for a well-funded company." This made it the 10th most powerful supercomputer in the world at the time.
Competitive Projection: Lowin believes that the future may not see 17 different diffusion models competing against each other. Instead, there will likely be one open-source base model (e.g., Stable Diffusion), with millions of fine-tuned versions built on top—tailored to specific industries or styles (e.g., "only output Van Gogh style").
Sources of the Moat:

1. Proprietary Data: Datasets not publicly available (e.g., Prefect's user Q&A records, internal design documents)

2. Model Secret Sauce: For example, Midjourney's ability to guide high-quality outputs

3. User Interface: Lowering the barrier to "prompt engineering"—currently, users must input hacks like "comma, trending on ArtStation, comma, 8K, comma, 4K" to achieve high-quality outputs. Lowin believes this is not the future.

Reader's Note: Lowin is the founder of Prefect, and his firm Positive Sum has invested in Prefect. His discussion of "proprietary data as a moat" reflects a position-holder's perspective.

5. Copilot and Entrepreneurial Opportunities: AI Excels at "Patterned Variation," Not Handling Extreme Edge Cases

Lowin believes that AI currently performs best not in handling all edge cases, but in scenarios that are "well-defined yet stochastic"—and this is precisely where GitHub Copilot delivers value.

Copilot Experience: Lowin describes the feeling of writing code with Copilot—it does not merely autocomplete words but writes entire lines and blocks of code. He finds himself "waiting for Copilot to complete" and feels frustrated when it does not appear—"This is already my programming partner."
Analogy: Lowin draws an analogy with autonomous driving—his father worries whether AI can handle edge cases like "a school bus stopping but the stop sign not fully extended." However, Lowin points out that AI's value lies in handling high-frequency scenarios such as "driving straight on the New Jersey Turnpike" or "stuck in traffic on the way to school." The same applies to code—writing CRUD APIs, tests, and documentation, repetitive tasks that "vary slightly each time."
Entrepreneurial Methodology: Lowin advises entrepreneurs not to directly build "cooler versions"—such as AI-generated avatars ("indefensible, not a sustainable business"). Instead, they should look for areas where "generating something is hard," such as customer support—where users have unique questions, need quick customized responses, and companies need to scale without linearly adding headcount. Prefect has already deployed a GPT-powered bot on Slack to handle the first 80% of common questions.

> "What Copilot really does is free up my time, allowing me to focus on the unique, idiosyncratic parts of the code that deliver core value. I define what this thing does, and then hand it over to fill in the blanks." —Jeremiah Lowin

6. Future Focus: Video Generation, Neural Rendering, and "AI Learning to Use Tools"

Lowin is closely monitoring three directions: video generation, more interactive text generation, and neural rendering—the latter could fundamentally change how we capture and present 3D scenes.

Video generation: Early examples are just beginning to emerge
Interactive text: ChatGPT gives the same answer every time the same question is asked—"This is good for information retrieval, but bad from a conversational perspective"
Neural rendering: The direction Lowin is most excited about. Taking Luma Labs (commercializing NERF technology) as an example, AI can capture 3D scenes with high fidelity and render them in a browser. Potential applications include video calls (without fixed camera positions), film special effects, VR, etc.
Extrapolation: Lowin believes AI will eventually learn to use existing tools (such as Unreal Engine 5), but the question is "whether AI exists as a standalone interface or is baked into existing tools." He is confident that in the future, users will be able to say "project a castle and a dragon on my desktop and let my kids play," but the current technical bottleneck lies in the fidelity of generated scenes—"We are still working on making sure all hands have five fingers"

> "Latent space is like—if I ask you 'what are you thinking,' don't tell me, but 'show me your thoughts.' That's latent space. You can choose a thousand different ways to turn that thought into language, or draw it out. What if we could use machines to accelerate that process?" —Jeremiah Lowin

Mentioned Positions

Position	Guest Attitude	Key Data
OpenAI (ChatGPT/GPT-3.5/DALL·E)	Bullish on its technological leadership, but investment stance not explicitly stated	ChatGPT uses a new form of reinforcement learning; DALL·E 2 is a diffusion model, with a completely different architecture from DALL·E 1
Stability AI (Stable Diffusion)	Neutral, as an important case of open-source foundation models	Training used approximately 4,000 A100 GPUs, with an annual cost of about $40 million; at the time, the world's 10th largest supercomputer
Midjourney	Bullish on its aesthetic quality, believes it has a "secret sauce"	Iterations from V1 to V4 consistently produce high-quality images
GitHub Copilot	Positive experience, viewed as a programming partner	Can write entire lines and blocks of code; Lowin has formed a habit of "waiting for it to autocomplete"
DeepMind	Mentioned as an important player, not elaborated	—
Google	Mentioned as an important player, not elaborated	—
Luma Labs	Bullish on its neural rendering technology	Commercializes NERF (Neural Radiance Fields) technology
NVIDIA	Mentioned its AI eye contact correction demo	Uses AI to make eyes naturally look at the camera during video calls
Prefect	Lowin's own company, used as a case study	Has deployed a GPT-powered customer service bot on Slack to handle the first 80% of inquiries

Judgments Worth Remembering

1. "The latent space is the source of AI's power" (Lowin) — It maps abstract concepts into independently adjustable mathematical variables, allowing the model to "increase the degree of a smile" or "change a watermelon from green to orange" without retraining.

2. "Today's 'state-of-the-art' will be obsolete in 12 months" (Lowin) — Progress comes from new architectures and new applications, not just more training. Stable Diffusion performs diffusion in the latent space rather than the pixel space, and this "small insight" alone brought about a revolutionary change.

3. "Capital is not a barrier; intuition and patience are" (Lowin) — Training a frontier model costs approximately $40 million per year, a reasonable expense for well-funded companies. The real constraint is the iterative process of "a group of people making bets and waiting months to see the results."

4. "Don't look for 'cooler solutions'; look for 'problems where generating things is hard'" (Lowin) — AI avatar generation is indefensible, but customer support (where users have unique questions, need fast customized answers, and companies need to scale) is a perfect application scenario.

5. "AI excels at 'regular variations' but struggles with extreme edge cases" (Lowin) — The value of Copilot lies in handling repetitive tasks that vary slightly each time (writing CRUD APIs, tests, documentation), not in dealing with anomalies never seen before.

6. "Hacks in prompt engineering (comma, 8K, comma, 4K) are not the future" (Lowin) — Current users need to input a large number of "negative prompts" to prevent the model from generating garbage; lowering this barrier will be an important competitive dimension.

7. "The moat comes from proprietary data + model recipes" (Lowin) — Proprietary datasets (such as Prefect's user Q&A records) and model-level guidance mechanisms (such as Midjourney's high-quality output guidance) are sustainable advantages.

8. "Neural rendering will change how we interact with virtual worlds" (Lowin) — AI can capture 3D scenes with high fidelity and render them in a browser, with potential applications ranging from video calls to movie special effects. The current bottleneck lies in generation fidelity ("all hands have five fingers").

Jeremiah Lowin: Explaining the New AI Paradigm - [Invest Like the Best, EP.307]

In plain words