Back-testing and Curve fitting

[momothebored]

From my beginner's understanding, the entire concept of the past replicating itself again mechanically in the future is the basis of TA.

Why wouldn't you be able to back-test something and find out what combination of indicators worked best in the past to develop a model?

e.g. I take a look at the weekly and daily charts of a large-cap stock, and plot what would have worked best for the last 10 years.

So i come up with, for example, that if you bought whenever EMA(10d) crossed above EMA(20d) and sold whenever MACDH ticked down, you would have made money 99% of the time. Any value doing this?
(The idea, not the actual combination of indicators)


Curve fitting could arguably be the creation of a new indicator in itself. Instead of a MACD, my new model would be the indicator. Same concept, no?

I'm not promoting the idea, but I'd like to understand WHY it does / doesn't work.

 

[ronblack]

Why wouldn't you be able to back-test something and find out what combination of indicators worked best in the past to develop a model?

Because this introduces data snooping and selection bias. Think about it; if you have a large enough set of indicators and exit strategies, at the end of the day you will find a combination that fits the data and works well (selection bias). You achieve this by repeatedly using the same data (data snooping). Your system is curve-fitted. You may find excellent papers and articles about this in many places. Here are three from the price action lab blog:

Fooled by Randomness Through Selection Bias | Price Action Lab Blog

Curve-fitting and Optimization | Price Action Lab Blog

Trailing Stops and Curve-Fitting in Trading System Development | Price Action Lab Blog

 

[momothebored]

Because this introduces data snooping and selection bias. Think about it; if you have a large enough set of indicators and exit strategies, at the end of the day you will find a combination that fits the data and works well (selection bias). You achieve this by repeatedly using the same data (data snooping). Your system is curve-fitted. You may find excellent papers and articles about this in many places. Here are three from the price action lab blog:

Fooled by Randomness Through Selection Bias | Price Action Lab Blog

Curve-fitting and Optimization | Price Action Lab Blog

Trailing Stops and Curve-Fitting in Trading System Development | Price Action Lab Blog

Hello

Isn't bias the point of the exercise?

You would certainly want to exclude methods that don't work..

 

[Martinghoul]

No, bias isn't the point of the exercise. In this case, bias means that you're observing all sorts of positive outcomes, where in reality they are random. What that means, in turn, is that the backtest results for the strategy are meaningless, as backtested performance won't be realized in live trading.

 

[Lee Shepherd]

The main answer to this is simple:

Try forward testing then trade it with hard earned cash.

If you find a method that passes the first trial - Back testing - then

Forward test it for the same period of time - If this passes then put money down and again trade it for the same period of time using the exact same set of rules.

The amount of money down must be the maximum expected to lose and the trades must be played out equally for the same period of time with no hindsight or hypothetical trades, whether or not you start on a losing streak or not.

In most cases you will find what works on paper does not always work for real. More importantly you will also find methods that work for real but yet do not work for you.

If you have found a method over a period of time, forward test it for a similar period and see how it fares. If it is deemed as working then trade it to the letter and see how you get on.

Lee Shepherd

 

[Spec-K Trading]

From my beginner's understanding, the entire concept of the past replicating itself again mechanically in the future is the basis of TA.

Why wouldn't you be able to back-test something and find out what combination of indicators worked best in the past to develop a model?

e.g. I take a look at the weekly and daily charts of a large-cap stock, and plot what would have worked best for the last 10 years.

Utilizing the information of the past to give us an edge in trading the future is the objective of TA. It doesn't necessarily require the past to replicate itself, but for something predicted to happen ("replica" or different, it doesn't matter) enough to give us an edge, based on our observations ("TA").

If the "model" gives us some kind of probabilistic edge it's a start, but what about all the rest of what goes into a trade, exits and all that? You've got to have it all and applied consistently to know if you've got a per trade profit.

What do you think might happen when you test against a variety of other stock's histories? It's had better be "acceptable" to "very good", or else it's not robust. If it only works on one set of data it is valueless on everything (even that stock in the future, because that stock will change it's behavior to be "out of tune" with your "model"). Why? Probably because what you keyed off of was random chance to have happened at the frequency it did through that data set.

Broad testing is a means of de-optimizing a system to be robust (if it's possible at all given the general structure). It needs to succeed well in forward testing in the face of diverse trade data. By developing a system with training on one stock over ten years, you've done the opposite: you've created a unique system for that data and that data alone (not even the future data of that same trading instrument).

My $0.02.