I have been developing and backtesting crypto strategies for around five years, testing over 300 combinations in detail. The most important lesson wasn't related to an indicator or a model. It's that every backtest is based on a sample size of one. The market has only ever existed once, and a backtest only tells you how a strategy performed on that one path.
I learned this the hard way. I tested a strategy on TRUMP over just a few days. It looked great. I let it run live and, overnight, it lost 50%. The code was fine. I had simply fitted a strategy to a few days' worth of data from one coin and called it an edge.
What I do differently now:
Out-of-sample performance has to be near perfect, not just positive. I split the history into chronological segments, and a candidate has to perform well in all of them, or very close to it. One bad segment and it's out.
The neighbours matter more than the peak. If a parameter set is excellent but the surrounding values are mediocre, that's overfitting. Every time. A real edge looks like a plateau, not a spike.
If it works on everything, check your code first. At one point, the same strategy worked on every coin I tried. It should have felt like winning. Instead, it felt suspicious. When I investigated the code further, I discovered a look-ahead bias.
After years of this, my live results now closely track the backtests. It took a lot of setbacks to get there.
The part I still haven't solved is this: I still can't predict whether a single entry will end in profit. Not even roughly. The edge only becomes apparent across many trades, and each individual trade still looks like a coin toss to me.
So, my question is: has anyone here managed to get trade-level filtering (or anything like meta-labeling or an ML filter on entries) to work out of sample? Or is the edge only ever visible in aggregate?