This interview is about how to actually use machine learning in investing. The author says ML is just a fancy version of linear regression that finds complex patterns, but financial data changes over time and is easy to misuse. He warns against using complex models for simple problems—like using a chainsaw to cut butter. His advice: start with a simple problem you understand deeply, and use ML to help human judgment, not replace it. Key points: data quality is the real edge—public datasets everyone can buy are unlikely to add value; watch out for 'data leakage' where the model accidentally sees future info; and avoid the '42 problem'—people often search for answers without knowing what question they're asking.
In this episode of Invest Like the Best, former quantitative risk control expert and founder of Prefect, Jeremiah Lowin, discusses the application of machine learning in investing. The core argument is that while machine learning is powerful, it is often overhyped, particularly for predicting non-st
Former quantitative risk control expert and Prefect founder Jeremiah Lowin discusses with host Patrick O'Shaughnessy the application of machine learning in investing. The core judgment: Machine learning is merely an "enhanced linear regression," adept at uncovering complex correlations, but the time-series nature of financial markets makes it highly prone to misuse. Lowin emphasizes that data quality and feature engineering are the true sources of excess returns, while model complexity often proves counterproductive.
Jeremiah Lowin argues that machine learning tools are often overused, with many people "using a chainsaw to cut butter." He bluntly states that AI is inherently dumb — "They're like puppies. They can do a small number of things, and sometimes they can do them well enough to survive." The only thing AI excels at is discovering complex correlations in data, which is equivalent to human "experience."
Key distinctions:
Lowin points out that building an AI model itself is easy — "it's just stacking a bunch of regression models together with a bit of fine-tuning" — but training is where the real difficulty lies and requires lifelong learning.
Lowin emphasizes that the three-way split of training set, validation set, and test set is a critical defense against overfitting:
| Dataset | Purpose | Key Note |
|---|---|---|
| Training Set | Enables the model to learn the correlation structure between inputs and labels | Can be used multiple times |
| Validation Set | Evaluates model performance during parameter tuning | Once used for tuning, it is no longer truly "out-of-sample" |
| Test Set | Final evaluation of the model's true performance | The model has never seen it and uses it only once |
Classic case of overfitting: Training an AI to recognize tanks, with all tank photos taken at night. The model actually learned to "determine whether it is nighttime" rather than recognize tanks. This "latent correlation" is the most insidious source of error in machine learning.
Lowin argues that the core difference lies in purpose, not technology:
"A parameter may turn out to only even be useful when you see an email that mentions Nigeria... It may sit dormant the rest of the time. This is hard to capture in a world that expresses statistical confidence through means and standard errors."
Stock prices follow a non-stationary distribution — as Amazon rises from $100 to $1,000, the model must adapt accordingly. The solution is to convert prices into returns, which approximate a stationary distribution (though with heavier tails).
Lowin argues that in finance, "the output is not obvious" — what constitutes a "good trade"? How long should the holding period be? When to exit? What about portfolio impact? This is a "cross-temporal credit assignment problem."
His core philosophy: The market is not always predictable, only occasionally so. Therefore, a successful ML strategy requires "not only predicting the market, but also predicting when the market can be predicted." The smartest approach is to "not play this game" and instead seek other, more predictable aspects of the investment process where ML can be deployed.
Lowin categorizes models into four quadrants:
| Supervised Learning | Unsupervised Learning | |
|---|---|---|
| Classification | Logistic regression, decision trees (e.g., spam classification) | K-means clustering (e.g., grouping data into clusters) |
| Regression | Linear regression, neural networks (e.g., predicting beta) | Autoencoders, etc. (e.g., dimensionality reduction) |
On decision trees/random forests: Lowin describes them as "set it and forget it" models—few hyperparameters, stable performance, and hard to find a reason not to use them. However, they are unsuitable for scenarios like image recognition, where the complex relationships between pixels require convolutional neural networks to capture.
Lowin asserts that data acquisition is the most critical link in ML investing: "If you're buying the same data set that everyone else can buy for 250 bucks... I'd be shocked if there was $250 of value in there."
A common trait among successful practitioners: the ability to feed their own actions, trades, research, and even mistakes back into the current model, creating a feedback loop.
Lowin summarizes three common pitfalls:
1. Over-engineering: Using complex models to solve simple problems (using a chainsaw to cut butter)
2. The "42 Problem": Not knowing what question you are asking — referencing The Hitchhiker's Guide to the Galaxy, where people waited millions of years for the answer "42," only to realize they had no idea what the question was
3. Data Leakage: The model inadvertently "sees the future" — "I can't count how many times I've heard 'Eureka,' only to find that the computer had cached future outputs in some completely innocent way"
Recommendation: Start with simple problems you understand most deeply, using ML to enhance rather than replace human judgment.
Lowin argues that interpretability is overrated, but not entirely inexplicable:
Gradient descent is the core mechanism for training neural networks — analogous to a marble rolling down a mountain, where each step follows the steepest direction (gradient) to minimize error. However, there is a local minimum problem (getting stuck in a valley and unable to descend further), with solutions including the introduction of momentum, allowing the model to maintain its forward direction.
Lowin's summary: "It's all well and good to figure out how you're going to get down that mountain as fast as possible. But if you look around and realize you were on the wrong mountain in the first place, you've got a problem."
No specific investable positions are discussed in this section.
1. AI does only one thing: discover complex correlations (Lowin). It does not think or reason; it merely "like a puppy" redeploys experience. Understanding this is the prerequisite for correctly using ML.
2. "Using a chainsaw to cut butter" is the most common mistake in ML investing (Lowin). Tools like TensorFlow make it too easy to build complex models, but financial data is noisy with weak signals, so simple models are often more effective.
3. Markets are not always predictable, only occasionally so (Lowin). A successful ML strategy requires "not only predicting the market, but also predicting when the market can be predicted"—the smartest move is to "not play this game" and instead look for other, more predictable links in the investment process.
4. Label construction is the most underestimated aspect of financial ML (Lowin). What constitutes a "good trade"? This is a "cross-temporal credit assignment problem"—holding period, exit timing, and portfolio impact all make the answer ambiguous.
5. The "42 problem": not knowing what you are asking (Lowin). Citing The Hitchhiker's Guide to the Galaxy—people get the answer "42" only to realize they never knew the question. Many ML projects fail because the problem to be solved was never properly defined.
6. Data quality is the true competitive advantage (Lowin). A public dataset bought for $250 is almost impossible to generate excess returns from; successful players can feed their own behavior, trades, and mistakes into the model.
7. Interpretability is overrated (Lowin). Explaining a neural network with billions of parameters one by one is meaningless; Google once "fired" human interveners because they kept overriding the search algorithm—letting the model run on its own actually improved accuracy.
8. The model's ability to "shrug" is crucial (Lowin). In occasionally predictable markets, the model must be able to say "I don't know" rather than being forced to output an answer. Probabilistic models are more suitable for financial scenarios than deterministic ones.