I'm working on a quantitative research system where I'm trying to build a data-driven pattern-discovery layer rather than starting with specific trading ideas and asking a backtester to test them and I’ve hit a result I’m having trouble interpreting.
The basic idea is to reverse the usual workflow:
Instead of:
“I think X causes Y, so let's write a strategy and backtest it.”
I'm trying to do:
“Here is a defined historical event dataset. Let the detector identify potentially meaningful relationships first, then independently test whether those relationships survive on unseen data.”
I'm deliberately trying to avoid hand-feeding the system a specific trading hypothesis, while still using proper train/validation/sealed testing so that “it found something” doesn't automatically mean “we found an edge.”
That's what led me into a surprisingly frustrating comparison between five detector families...
I’m not looking for the generic “ML overfits financial data” answer. I’m trying to understand the mechanism behind what happened, because the difference between the model families was much more dramatic than I expected.
The setup
The goal is not “predict the stock market.”
I’m testing whether different detectors can recover deliberately planted relationships inside historical ES intraday event data.
There are roughly 23 features, and the benchmark contains several synthetic relationship types: additive effects, nonlinear/zonal effects, feature confluence/interactions, regime-dependent effects, and a no-added-effect background.
We use chronological discovery/validation splits, a purged final sealed period, and multiple seeds. The purpose is to see whether a detector can discover a relationship and then still recognize it on data it did not train on.
The five detector families were:
- ElasticNet
- GAM
- Decision Tree
- Random Forest
- Gradient Boosting
For the tree models, the original configurations were approximately:
DecisionTreeRegressor(
max_depth=2,
min_samples_leaf=20
)
RandomForestRegressor(
n_estimators=150,
max_depth=4,
min_samples_leaf=15,
max_features=0.7
)
GradientBoostingRegressor(
n_estimators=100,
learning_rate=0.05,
max_depth=2,
min_samples_leaf=15
)
One complication was that RF/GB were not directly converted into trading rules. Their predictions were passed through a smaller Decision Tree surrogate, roughly:
DecisionTreeRegressor(
max_depth=2,
min_samples_leaf=20
)
The surrogate's leaf paths became the candidate rules.
So we suspected two possible problems:
- Maybe the tree models were simply too shallow.
- Maybe RF/GB were finding something useful, but our conversion from model prediction → readable rule was throwing it away.
We therefore did a clean rerun rather than trying to reconstruct old model artifacts.
What happened
We ran a fresh self-contained benchmark with:
- 700 cells total
- 500 planted-pattern cells
- 100 historical-background/no-added-effect cells
- 100 association-broken negative-control cells
- 11 configurations per cell
- all models, predictions, rules, configs, logs, etc. saved
- artifacts reopened and verified
- 21,867 manifest entries
The first surprising result was that depth was not the problem.
We tested deeper trees while changing nothing else.
Decision Tree:
depth 2 → validation R² ≈ -0.10
depth 6 → validation R² ≈ -0.32
Gradient Boosting:
depth 2 → validation R² ≈ -0.10
depth 6 → validation R² ≈ -0.22
Random Forest was basically flat across the tested depth range.
So giving the models more freedom did not uncover a hidden signal. In the Tree/GB cases it actually made validation performance worse.
That makes “the trees were just too shallow” a pretty poor explanation.
Then we tested the translation problem
This one was interesting.
Random Forest appears to contain some structure in prediction space. Our internal capture metric was relatively strong for RF, and it had the best rank-correlation behaviour among the tree-based models.
But:
the original RF/GB models themselves had negative validation R².
So the useful-looking structure largely did not survive onto new observations in the first place.
For RF, making the surrogate deeper improved prediction fidelity somewhat (roughly 0.44 → 0.57), so the shallow surrogate definitely wasn't perfectly reproducing the original model.
But the actual candidate-rule recovery stayed around zero.
That makes me think the translation layer is losing information, but also that translation is probably not the primary reason these models failed, because the original models themselves aren't demonstrating stable out-of-sample relationships.
That is the part I'm trying to understand.
Meanwhile, ElasticNet and GAM behaved very differently
They are not perfect, and I am absolutely not claiming we found a winning trading system.
But relative to the tree family, they actually demonstrated recovery.
Across the planted tests we saw things like:
- Additive exact recovery roughly 0.74–1.00 depending on detector/family.
- C_regime_flip around 0.84 in one of the stronger breakout cases.
- B_confluence roughly 0.42–0.50 in breakout for EN/GAM.
- A_zones defeated all five detectors.
Then we did a genuinely untouched sealed evaluation.
On the opening family, nothing survived.
On the breakout family, only the additive relationship survived the sealed test, with roughly GAM 0.50 and EN 0.26 recovery.
So even EN/GAM are absolutely not “solved.” They are just the only two detector families that have demonstrated some ability to recover a planted relationship and preserve at least part of it out of sample.
What I don't understand
This is where I'm hoping someone with deeper ML/statistics experience can help.
Why such a dramatic divergence?
I'm familiar with the basic idea that trees / forests / boosting are more flexible and therefore can have higher variance or overfit.
But that explanation feels incomplete here.
Why does it look more like:
ElasticNet/GAM = some real detection capability
versus
Tree/RF/GB = “finds lots of stuff, then almost everything dies”
rather than a smooth continuum?
My current hypothesis is something like this:
Suppose the real detectable market relationship is relatively weak and mostly additive/smooth.
ElasticNet has a fairly constrained vocabulary for explaining the data. GAM is more flexible, but still mostly builds smooth feature-wise relationships.
A tree can say:
if X > 1.17
and Y < 0.42
and Z > 2.9
then ...
and there are enormous numbers of possible thresholds and combinations it can try.
So perhaps the tree family has many more opportunities to find accidental structure in a noisy training sample.
Random Forest reduces some of the instability by averaging trees, and Gradient Boosting is very powerful at fitting residual structure, but if most of that residual structure is noise, it may simply have more ways to chase it.
In other words, maybe this is not:
“Tree models are bad.”
but:
“The signal-to-noise ratio and structure of this particular problem favour a more constrained model class, because the extra flexibility mostly gets spent modelling sample-specific structure.”
Is that a reasonable interpretation?
A few things make me hesitate
The algorithms themselves are obviously not bad algorithms.
Random Forest and Gradient Boosting are used successfully all over the place.
Scikit-learn's own documentation, for example, makes clear that tree ensembles can be very effective and that model complexity needs to be evaluated empirically rather than assumed to be better or worse. Gradient boosting also has explicit mechanisms such as early stopping because adding model stages can eventually hurt generalization.
So I don't want to make the lazy conclusion:
“RF/GB overfit, therefore they are useless.”
There may be something about my problem formulation that is making them particularly unsuitable.
That is the part I'd really like to understand.
Some questions for people who've actually fought this battle
Have you seen a situation where a tree / RF / boosting model looks materially more capable than a linear or GAM-type model in-sample, but then completely collapses out-of-sample while the simpler models retain some signal?
If so, what usually turned out to be the real cause?
For example, would you investigate:
- too little independent information relative to the number of possible splits?
- weak signal-to-noise ratio?
- strongly correlated / redundant features?
- noisy targets?
- event dependence or overlapping observations?
- the geometry of the feature representation?
- trees trying to approximate an additive/smooth relationship inefficiently?
- too many possible interaction structures?
- instability of threshold selection?
- some issue specific to regression rather than classification?
- something about the way we're turning model predictions back into discrete trading rules?
And importantly:
Is there a diagnostic you'd run that could distinguish “the model class is a poor fit for this problem” from “our implementation / feature representation is handicapping the model”?
One other complication
We also discovered that our false-positive control needs more work.
We built a market-shaped negative control by breaking the association between feature episodes and their corresponding outcome blocks.
The block reassignment itself appears structurally defensible, but the empirical family-level false-alarm rate came out around 0.16 and 0.28 versus our 0.10 gate.
So we're not yet comfortable unleashing the detector on real-market discovery.
I mention that because I'm interested in whether the FPR issue is potentially connected to the tree-model behaviour, or whether these should be treated as completely separate problems.
What I'm NOT looking for
I'm not looking for:
“Just use XGBoost.”
or
“Use a neural network.”
or
“All financial ML is garbage.”
or
“Trees always overfit.”
I already know those are too simplistic.
I'm trying to understand why this particular model-family split happened, and whether there's a known pattern here that experienced ML practitioners recognize.
We've put a fair amount of work into making the experiment itself reasonably controlled, so I'm less interested in generic advice and more interested in the underlying mechanism.
If you've encountered something similar, I'd especially appreciate hearing what finally solved it—or what convinced you that the simpler model was actually the right tool.
Also, if there's an obvious diagnostic I'm missing, please tell me. I'm still learning this stuff, so I may be staring directly at the answer without recognizing it.