AriaS
Well-known member
- Messages
- 397
- Likes
- 124
Hi everyone,
I keep seeing people presenting one single long backtest as proof that their strategy works. That is basically not a test at all, because you use one period to develop a strategy and then expect it to work on another period. That's called a curve-fit. Curve-fits don't work on another curve.
To evaluate whether a strategy actually has an edge, you should perform a rolling walk-forward analysis (WFA). The idea is that a strategy needs to demonstrate statistically meaningful survival across unseen market periods. That is the real challenge, rather than simply fitting a strategy to one long historical sample.
A simplified rough example of a rolling WFA.
You are currently in month 25. Month 26 is the future.
Cycle 1:
The process described above is not performed during each re-optimization. It is done only once, at the stage when you want to confirm whether a strategy has an edge at all.
When you start trading, whether on a paper or live account, your future-in-the-past stage becomes the actual future. The cycle is then repeated whenever you decide that it is time to re-optimize. I do it once every 2 months, or less often.
Example of my own workflow:
My optimization period is 3 months long, but my backward OOS is 3 years long. Yep, during each re-optimization cycle!
Those 3 years are divided between 2 regular OOS periods and 5–6 stress tests. Only if a setup passes those 3 years and additional robustness tests does it get traded live.
Some of you may think of another potential problem: data contamination.
Yes, portions of my backward-validation data were optimization periods at some point in the past — but the fact that my optimizations reach 1.5 million variants makes data contamination statistically virtually impossible. What is the probability that the selected setup was also the selected setup in each of the seventeen 3-month portions within the 36-month OOS, or even in some of them?
In fact, the large search space creates a different problem: with 1.5 million candidates, it is entirely possible to select a configuration that looks exceptional over 3 months simply by chance (curve-fit). That is why the optimization period is short and used only for primary selection. The setup then goes through the real test: three years of unseen data + several robustness tests, such as Monte Carlo, Parameter Stability, etc.
The bottom line is this: a curve-fit cannot pass unseen data, and the probability that all of the unseen data had actually been seen is 1 in 10^105, which is effectively zero - it's vastly smaller than the Planck length written in meters. The highest probability of partial contamination is about 1 in 88,000. But even if that happens, it only reduces the amount of unseen data from 36 months to 33 months.
How do you think — can such a setup be a curve-fit?
Much success!
I keep seeing people presenting one single long backtest as proof that their strategy works. That is basically not a test at all, because you use one period to develop a strategy and then expect it to work on another period. That's called a curve-fit. Curve-fits don't work on another curve.
To evaluate whether a strategy actually has an edge, you should perform a rolling walk-forward analysis (WFA). The idea is that a strategy needs to demonstrate statistically meaningful survival across unseen market periods. That is the real challenge, rather than simply fitting a strategy to one long historical sample.
A simplified rough example of a rolling WFA.
You are currently in month 25. Month 26 is the future.
Cycle 1:
- Months 13–15: optimization in-sample (IS)
- Months 10–12: backward out-of-sample (OOS) validation
- Months 16–17: forward OOS test (future-in-the-past)
- Months 15–17: optimization IS
- Months 12–14: backward OOS validation
- Months 18–19: forward OOS test (future-in-the-past)
The process described above is not performed during each re-optimization. It is done only once, at the stage when you want to confirm whether a strategy has an edge at all.
When you start trading, whether on a paper or live account, your future-in-the-past stage becomes the actual future. The cycle is then repeated whenever you decide that it is time to re-optimize. I do it once every 2 months, or less often.
Example of my own workflow:
My optimization period is 3 months long, but my backward OOS is 3 years long. Yep, during each re-optimization cycle!
Those 3 years are divided between 2 regular OOS periods and 5–6 stress tests. Only if a setup passes those 3 years and additional robustness tests does it get traded live.
Some of you may think of another potential problem: data contamination.
Yes, portions of my backward-validation data were optimization periods at some point in the past — but the fact that my optimizations reach 1.5 million variants makes data contamination statistically virtually impossible. What is the probability that the selected setup was also the selected setup in each of the seventeen 3-month portions within the 36-month OOS, or even in some of them?
In fact, the large search space creates a different problem: with 1.5 million candidates, it is entirely possible to select a configuration that looks exceptional over 3 months simply by chance (curve-fit). That is why the optimization period is short and used only for primary selection. The setup then goes through the real test: three years of unseen data + several robustness tests, such as Monte Carlo, Parameter Stability, etc.
The bottom line is this: a curve-fit cannot pass unseen data, and the probability that all of the unseen data had actually been seen is 1 in 10^105, which is effectively zero - it's vastly smaller than the Planck length written in meters. The highest probability of partial contamination is about 1 in 88,000. But even if that happens, it only reduces the amount of unseen data from 36 months to 33 months.
How do you think — can such a setup be a curve-fit?
Much success!
