#490 – State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI

At a Glance

Nathan Lambert (Head of Post-Training at Ai2, author of The RLHF Book) and Sebastian Raschka (author of Build a Large Language Model (From Scratch)), two machine learning researchers, systematically reviewed the key advances in AI from 2025 to 2026 on the Lex Fridman podcast. Core judgment: Scaling Laws have not broken down, but the focus has shifted from pre-training scale to post-training and inference-time scaling—"2025 is the year of post-training, and 2026 will be the year of inference" (Nathan Lambert).

---

Post-Training: The Paradigm Shift from RLHF to Reasoning Models

Nathan Lambert believes the most significant change in AI over the past year is that post-training methods have evolved from "icing on the cake" to "core engine."

RLHF (Reinforcement Learning from Human Feedback) underwent a transformation in 2025 from a "tuning trick" to "systematic engineering." Lambert points out that the release of DeepSeek R1 in late 2024 was a watershed moment—it proved that pure RL (without supervised fine-tuning) can enable models to learn chain-of-thought reasoning without requiring large amounts of human-annotated reasoning data. "R1 told us that you can let models learn how to think on their own through RL, without massive amounts of human-annotated reasoning data." (Nathan Lambert)

Sebastian Raschka adds technical details: The core difference between reasoning models and standard LLMs lies in "thinking tokens"—the model generates an internal reasoning process before outputting the final answer. This brings two key changes:

• A significant increase in computational cost during the reasoning phase (the "thinking tokens" for a complex math problem can be 10–50 times the length of the final answer)
• A need to change evaluation methods (traditional benchmarks cannot measure reasoning quality; process-oriented evaluation is required)

Extrapolation: Lambert predicts that by 2026, a "Reasoning-as-a-Service" business model will emerge, with API pricing distinguishing between "thinking time" and "output time." Falsification signal: If the commercial adoption of reasoning models lags behind expectations (e.g., enterprise customers are unwilling to pay for "thinking time"), this trend may be overestimated.

---

Redefining Scaling Laws: From "Bigger" to "Smarter"

Both guests agreed that Scaling Laws remain valid, but the definition of "scale" is being rewritten.

The traditional view holds that Scaling Laws mean "larger models + more data = better performance." However, Raschka pointed out that the key finding in 2025 is: data quality matters more than data quantity, and post-training scaling is becoming the new growth curve.

Dimension	Traditional Scaling (2023-2024)	New Scaling (2025-2026)
Primary Stage	Pre-training	Post-training + Inference
Key Variable	Parameter count, training data volume	RL training steps, inference compute
Bottleneck	Data wall	Inference cost, evaluation methods
Representative Breakthroughs	GPT-4, Claude 3	DeepSeek R1, OpenAI o3

Lambert proposed a counterintuitive view: the growth rate of model parameter scale is slowing (the largest open-source model in 2025 has about 400B parameters, similar to 2024), but "effective compute" is still growing exponentially—because the thinking time during inference can be extended indefinitely. "A 70B reasoning model may outperform a 400B non-reasoning model on complex problems because it has more time to 'think'." (Nathan Lambert)

Uncertainty: Raschka cautioned that the sustainability of inference-stage scaling depends on hardware efficiency. If GPU computing costs do not decline fast enough, the economics of inference scaling could become a bottleneck.

---

China AI: From Follower to Innovator

The two guests share a highly consistent view on China's AI ecosystem: 2025 marks a critical turning point for China's AI, shifting from "following" to "running in parallel."

Core evidence chain:

1. Model capability: DeepSeek R1 achieves or surpasses GPT-4 levels in mathematical reasoning (MATH-500 score of 97.3%) and coding (HumanEval pass rate of 92.7%), with training costs estimated at roughly 1/10 of the latter (based on publicly available information)

2. Open-source ecosystem: The number of models released by Chinese teams on Hugging Face grew by 300% in 2025, with a significant improvement in quality (no longer merely "Chinese-optimized versions")

3. Application deployment: China's penetration rates in AI customer service, code assistance, education, and other scenarios are higher than those in the U.S. (Lambert cites data: AI customer service handling rates at top Chinese e-commerce platforms have reached 85%, compared to approximately 60% in the U.S.)

Lambert specifically emphasized DeepSeek's "efficiency innovation": They achieved near-top-tier performance with fewer GPUs and smaller models, forcing U.S. companies to rethink the "compute scaling" strategy. "DeepSeek proves that if you push engineering optimization to the extreme, you don't need 100,000 H100s to build a world-class model." (Nathan Lambert)

Risk warning: Raschka points out that China's AI still lags behind the U.S. in fundamental research (e.g., new architecture design, theoretical breakthroughs), with current advantages primarily in engineering optimization and the application layer. Falsification condition: If U.S. export controls tighten further (e.g., restricting access to H100 alternatives), China's AI catch-up pace could slow significantly.

---

AI Agents: From Demo to Production

Nathan Lambert argues that 2025 marks the year AI Agents transition from the "demo phase" to the "production phase," though they remain far from "autonomous decision-making."

Lambert proposes a classification framework:

• Level 1: Tool Calling — Models can invoke APIs and execute code (already mature, achieved in 2024)
• Level 2: Multi-step Reasoning — Models can decompose tasks and plan steps (reaching usable levels in 2025)
• Level 3: Autonomous Decision-making — Models can make independent decisions in uncertain environments (not yet achieved, expected 2027+)

Key data: Citing internal data, Lambert notes that Ai2's Agent system reduces manual review time by 40-60% in code review scenarios, but "fully autonomous" code submissions (requiring no human review) still account for only 5-10%.

Raschka adds technical challenges: The biggest issue with current Agent systems is "error accumulation"—a single step's error amplifies along the reasoning chain, rendering the final result unusable. He predicts that the breakthrough direction in 2026 will be the "verifier"—a model specifically designed to check the correctness of each step an Agent outputs.

Extrapolation: Both guests agree that Agent commercialization in 2026 will focus on "low-risk, high-repetition" scenarios (e.g., customer service, data entry, code review), while "high-risk decisions" (e.g., medical diagnosis, financial trading) will still require human-in-the-loop.

---

GPU and AGI: Compute Bottlenecks and Divergent Paths

The two guests hold clear disagreements on the timeline for AGI.

Nathan Lambert (Optimist): Believes AGI could be achieved by 2027-2029, provided GPU compute costs decline by over 50% annually. He cites an analogy: "If the H100 in 2025 is a steam engine, then the Blackwell in 2027 is an internal combustion engine—an order-of-magnitude efficiency improvement."

Sebastian Raschka (Cautious): Argues that the very definition of AGI is problematic, and the current architecture (Transformer + RL) may not be the right path to AGI. "We might be looking for nails with a hammer—Transformers excel at language tasks, but does that mean they can solve all intelligence problems? I am skeptical." (Sebastian Raschka)

Shared acknowledged uncertainties:

• Compute bottleneck: Global GPU supply remains tight in 2025, with wait times for the H100 around 6-9 months
• Energy constraints: Training a frontier model (e.g., GPT-5 level) consumes approximately 50-100 GWh of electricity, equivalent to the annual power usage of a small city
• Evaluation dilemma: Existing benchmarks cannot measure "general intelligence," and a "Turing Test 2.0" for AGI has yet to be established

---

Mentioned Positions

Position	Analyst View	Key Data
DeepSeek R1	Highly positive (innovation benchmark)	MATH-500 score 97.3%, training cost approximately 1/10 of GPT-4
OpenAI o3	Acknowledged (industry leader)	Significant improvement in reasoning capabilities, but specific data not disclosed
NVIDIA H100	Neutral (computing bottleneck)	Lead time 6-9 months, training frontier models requires 50-100 GWh
Leading Chinese E-commerce Platforms	Not explicitly stated (application case)	AI customer service handling rate 85% (approximately 60% in the US)

---

Judgments Worth Remembering

1. "2025 is the year of post-training, and 2026 will be the year of inference" (Nathan Lambert) — The focus of AI development is shifting from pre-training scale to post-training methods and inference-stage compute scaling.

2. "DeepSeek has proven that if you optimize engineering to the extreme, you don't need 100,000 H100s to build a world-class model" (Nathan Lambert) — China's efficiency innovation in AI is reshaping the global competitive landscape.

3. "We may be looking for nails with a hammer — Transformers excel at language tasks, but does that mean they can solve all intelligence problems?" (Sebastian Raschka) — A fundamental challenge to the current AI architecture's path toward AGI.

4. Agent three-tier framework (Nathan Lambert): Tool calling (mature) → Multi-step reasoning (available in 2025) → Autonomous decision-making (expected 2027+), with current Agent systems still exhibiting a 10-20% error accumulation rate.

5. "A 70B reasoning model can outperform a 400B non-reasoning model on complex problems" (Nathan Lambert) — "Thinking time" during inference is becoming a new performance lever; model parameter growth is slowing, but effective computation continues to scale exponentially.

6. "We may be looking for nails with a hammer" (Sebastian Raschka) — A fundamental challenge to whether the Transformer + RL architecture can lead to AGI, arguing for exploration of new architectures (e.g., state-space models, neuro-symbolic systems).

7. Data quality > data quantity (consensus among both guests) — The key finding in 2025 is that carefully curated 100B tokens are more valuable than randomly collected 1T tokens; data engineering is becoming central to model competitiveness.

8. China-U.S. gap in AI customer service scenarios (Nathan Lambert): China's top e-commerce AI customer service resolution rate is 85% vs. 60% in the U.S., reflecting China's stronger execution in AI application deployment, though basic research still lags.

Follow-up Analysis: AI Competitive Landscape, Post-Training Techniques, Open-Source Ecosystem, and Future Outlook

I. Deep Divergence in the China-U.S. AI Competitive Landscape

1. "Decentralization" Trend in Chinese Open-Source Models

The most notable change in 2025 is that DeepSeek's "crown" is being shared among multiple Chinese companies:

Company	Representative Model	Characteristics	Strategic Positioning
DeepSeek	V3/R1/V3.2	Architectural innovation (MLA, MoE)	Mysterious and low-key, backed by hedge fund High Flyer
Kimi/Moonshot	K2 Thinking	Creative writing, software engineering	Actively seeking Western market attention
Z.AI	GLM series	Academic influence	Has filed IPO documents
Minimax	MiniMax series	Multimodal	Has filed IPO documents
Qwen (Alibaba)	Qwen3	Data quality, long context	Continuous iteration, detailed technical reports

Key insight: Chinese companies release open-source models for different reasons than U.S. firms — due to geopolitical and security concerns, U.S. enterprises will not purchase API services from Chinese companies, making open-source the only route to gain influence in Western markets. This "influence investment" model is expected to persist for several years.

2. "Cultural Advantage" of U.S. Models

Although Chinese open-source models are rapidly catching up in performance, all three experts report that their daily usage still favors U.S. models:

• Lex: Uses GPT-5 Pro for deep checks, Gemini for quick queries, Claude Opus 4.5 for coding and philosophical discussions
• Sebastian: Uses Codex plugin (VS Code), prefers a "sense of control"
• Nathan: Uses Claude Code for "programming in English," GPT-5.2 Pro for multi-threaded deep search

Core reason: U.S. models offer a better speed-intelligence trade-off, and users have developed "muscle memory" — ChatGPT's brand recognition and long-term usage habits create a powerful flywheel effect.

3. 2026 Prediction: Gemini May Continue to Catch Up with ChatGPT

Nathan's forecast:

• Gemini: Leveraging Google's scale advantages (vertical integration from chips to data centers), it will continue to erode ChatGPT's market share in 2026
• Anthropic: Sustained success in enterprise and software engineering; Claude Code has become the benchmark for "programming in English"
• OpenAI: Despite operational chaos, it continues to launch defining products (Deep Research, Sora, O1 reasoning model) and cannot be ignored

---

II. Paradigm Shift in Post-Training Techniques

1. The Rise of RLVR (Reinforcement Learning with Verifiable Rewards)

Technical Principles:

• The model generates an answer → Scores based on verifiable criteria (correctness of mathematical answers, code test pass rates) → Uses this score for reinforcement learning updates
• Core algorithms: PPO (Proximal Policy Optimization) and GRPO (Group Relative Policy Optimization)

Key Breakthroughs:

• DeepSeek R1 demonstrated an "Aha moment"—the model spontaneously learned to self-correct during training
• Inference time scaling: The model uses more tokens for "thinking," significantly improving accuracy
• Training computation has approached pre-training levels (Grok 4 claims the two are comparable)

Essential Differences from RLHF:

Dimension	RLVR	RLHF
Reward Signal	Objectively verifiable (math, code)	Human preference (subjective, style)
Scalability	Infinitely scalable (harder problems)	Saturation point exists (limited style learning)
Computational Demand	Continuously growing (approaching pre-training)	Relatively fixed
Capability Unlocking	Tool use, reasoning ability	Dialogue style, safety

2. Three-Stage Framework for Post-Training

1. Mid-training: Similar to pre-training but more focused, e.g., long-context training, reasoning trajectory data

2. RLVR: Unlocks tool use and reasoning ability through trial-and-error learning

3. RLHF: Final polish, optimizing style, tone, and usability

3. Outlook for RLVR 2.0 in 2026

• Process Reward Models: Evaluates not only the final answer but also the correctness of intermediate reasoning steps
• Value Functions: Assigns a value to each token, with a deeper theoretical foundation in RL
• Extension to Non-Verifiable Domains: Uses an "LLM as judge" scoring mechanism to extend RLVR to creative writing, scientific discovery, and other fields

---

III. The "China Moment" of Open-Source Ecosystems

At a Glance

1. Explosive Growth in the Number of Open-Source Models

Nathan listed over 20 open-source models on-site (without referring to notes):

China Camp: DeepSeek, Kimi, Minimax, Z.AI, Antling, Qwen

Western Camp: Mistral, Gemma, GPT OSS (OpenAI), Nemotron (NVIDIA), Olmo (AI2), K2 (Institute for Foundation Models), Aperdis, SmallLM (Hugging Face)

Key Trends:

• Chinese open-source models are generally larger (MoE architecture, over 100B parameters)
• Western models are catching up (Mistral Large 3, Nemotron at the 400B parameter level)
• GPT OSS introduces tool-use capabilities for the first time (web search, Python interpreter), marking a paradigm shift

2. Strategic Value of Open-Source Models

• Local Operation: Data remains local, addressing privacy and compliance issues
• Customization: Fine-tunable and adaptable to specific domains (legal, medical)
• License Advantages: Chinese model licenses are typically more permissive than Meta's Llama (no user count restrictions)
• GPU Sharing: OpenAI "borrows" users' GPU resources through GPT OSS

3. The "American Truly Open-Source Model" (ATOM) Project

An initiative launched by Nathan, aiming to:

• Establish a U.S.-led high-quality open-source model ecosystem
• Counter the global influence of Chinese open-source models
• Has secured $100 million in NSF funding (AI2)
• NVIDIA's Nemotron series has begun releasing training data

Core Argument: Open-source models are the "gateway" to AI research. Whoever possesses the best open-source model holds the power to cultivate the next generation of researchers.

---

4. The "Fine-Tuning" Nature of Architectural Innovation

1. Architectural Changes from GPT-2 to Today

Sebastian’s striking insight: All mainstream LLMs are essentially variants of the GPT-2 architecture, with changes limited to:

Component	GPT-2	Modern Models
Attention Mechanism	Multi-Head Attention (MHA)	Grouped Query Attention (GQA), Multi-Head Latent Attention (MLA)
Feed-Forward Network	Single Dense Layer	Mixture of Experts (MoE)
Normalization	LayerNorm	RMSNorm
Activation Function	GELU	SwiGLU, etc.
Position Encoding	Absolute Position	RoPE, YaRN, etc.

Key Insight: Starting from GPT-2 code, one can progressively build modern models like Qwen3 and Gemma3 by adding these "fine-tuning" components. Sebastian’s Build a Large Language Model (From Scratch) and Build a Reasoning Model (From Scratch) are based on this very concept.

2. Text Diffusion Models: A Potential Alternative

How It Works:

• Starts with random text and generates complete text through an iterative denoising process
• Similar to image diffusion models (e.g., Stable Diffusion)
• Can generate multiple tokens in parallel, theoretically faster than autoregressive models

Current Limitations:

• Achieving comparable quality requires many denoising steps, with computational cost comparable to autoregressive models
• Struggles with tasks requiring intermediate results, such as tool use
• Google’s Gemini Diffusion has demonstrated potential on the Nano 2 model

2026 Prediction: May be used in scenarios requiring fast responses, such as code diff generation, but will not fully replace autoregressive models.

---

5. Data Quality: The Underestimated "Secret Weapon"

1. The "Funnel" Model of Pre-training Data

• Raw Data: Common Crawl (hundreds of T tokens)
• OCR Extraction: DeepSeek OCR, AI2's Olmo OCR extract several T tokens from PDFs
• Data Mix Optimization: Determine the optimal data mix through small-scale experiments (ratios of GitHub, Stack Exchange, Reddit, Wikipedia)
• Synthetic Data: Rephrasing, Q&A formatting, summarization

2. Data Quality vs. Data Scale

Olmo 3 achieves better performance with less data than Olmo 2, proving that data quality matters more than scale. The key factors are:

• Remixing data for reasoning tasks (increasing the proportion of math and code)
• Using classifiers for quality filtering
• Removing low-quality and duplicate content

3. The "Gray Area" of Data Acquisition

• Purchasing Books: Anthropic purchased thousands of books and scanned them; the court ruled this legal
• Pirated Books: Anthropic was ordered to pay $1.5 billion in damages for downloading books via BitTorrent
• Enterprise Data: Proprietary data from industries such as pharmaceuticals and finance will become the next "data goldmine"

---

VI. Education, Career, and the Value of "Struggle"

At a Glance

Sebastian's "Two-Pass Method":

1. First pass: Read offline with full concentration, without using an LLM, allowing the information to "settle"

2. Second pass: Use an LLM to aid comprehension, generate practice problems, and explore extended content

Nathan's "Reverse Method":

• First use an LLM to establish a global context
• But avoid clicking into "rabbit holes" such as Twitter or blogs
• Treat the ChatGPT App as the "home of AI," rather than just another tab in a browser

Investment Implications

For researchers with limited computational resources:

• Evaluate research: Create representative problem sets to test model failure modes
• Tool usage: Study how to better integrate tools with open-source models
• Role training: How to make models interesting, sarcastic, or serious (Nathan is currently guiding Oxford students in this direction)

Investment Implications

• All three experts agree: Learning without struggle is incomplete
• Directly using an LLM to obtain answers deprives the satisfaction of the "aha moment"
• Recommendation: Attempt the task first on your own, then use an LLM for assistance, avoiding complete reliance

---

7. Key Predictions for 2026

1. Technical Level

• Context Length: Expanding from millions of tokens to 2–5 million tokens
• Inference Time Scaling: A $2,000/month subscription plan may emerge (currently capped at $200/month)
• Text Diffusion Models: Beginning to see practical use in scenarios such as code diff
• RLVR 2.0: Process reward models and value functions start to play a role

2. Commercial Level

• Accelerated Consolidation: Large transactions such as XAI acquisition ($20 billion) and Scale AI acquisition ($30 billion)
• IPO Trends: Chinese companies (Minimax, Z.AI) may go public before US companies
• Advertising Model: Google may pioneer AI advertising in Gemini
• Open-Source Ecosystem: US companies begin to seriously catch up with Chinese open-source models

3. Geopolitical Level

• Open-Source Models as Influence Tools: China builds global technological influence through open-source models
• US Policy Shift: The AI Action Plan explicitly supports open-source models
• "Manhattan Project"-Style Centralization: Nathan considers it unlikely; Sebastian believes knowledge cannot be locked down

---

8. A Pragmatic Discussion on AGI

At a Glance

The three experts adopt a cautious stance on defining AGI:

• Nathan: Leans toward “an AI capable of performing most digital economic tasks”
• Sebastian: Views the concept of AGI as too vague, focusing instead on specific capability improvements
• Lex: Cites the “super programmer” milestone from the AI 2027 report

Investment Implications

• AI 2027 report: Originally predicted the arrival of super programmers by 2027–2028, now postponed to 2031
• Nathan: Argues that AI is “jagged”—superhuman in some areas but poor in others
• Sebastian: Suggests that fully automated programming may never be achieved, as “struggle” and “insight” are core to human learning

Position Moves

Despite rapid AI capability improvements, a significant GDP leap has yet to materialize. Reasons:

• The financial services sector accounts for a high share of GDP, yet AI penetration remains low
• Tool use (computer use) remains extremely difficult
• Enterprise-level integration takes time

---

IX. Hope for Humanity

1. Core Advantages of Humans

• Consciousness and Agency: AI needs to be told what to do, while humans possess true autonomous decision-making capabilities.
• Community and Resilience: Throughout history, humans have continuously found ways to solve problems.
• Pursuit of "Authenticity": As AI-generated content proliferates, the value of physical experiences and genuine connections will increase.

2. Foundations of Technological Optimism

• Democratization of Knowledge: LLMs enable billions of people worldwide to access human knowledge.
• Tools, Not Replacements: AI is a more powerful tool, but humans remain in control.
• Mirror for Self-Awareness: AI helps us better understand consciousness, creativity, and the essence of humanity.

3. Final Recommendations

• Keep Struggling: Do not rely entirely on AI; reserve time for "offline learning."
• Seek Agency: Actively use AI to build things, rather than passively consuming AI-generated content.
• Step Out of the Silicon Valley Bubble: Read history books, travel, and engage with people from diverse backgrounds.
• Remember the Individual: Behind every job replaced by AI is a real person.

---

Conclusion: As Einstein once said — "It's not that I'm so smart, it's that I stay with problems longer." In the age of AI, this statement carries more weight than ever before.