System Backtesting FAQ

[Tadragh1]

In this thread I would like to establish a FAQ for system backtesting. As I am not definitely the ultimate expert at the topic, all suggestions will be more than welcome. The procedure I would like to follow would be putting the FAQ in this, first post of the topic. The post will be modified when I have time to add more things, according to your suggestions and remarks.

I'd like to start with a few links.

1. Internet Knowledge Sources
These websites provide some good insight on system backtesting, check which statistics people present while testing a system. Good stuff there, especially on the MQL4 website.

EXLENCE - C2 trading system which trades futures
Don't Curve-Fit That System
No-Curve Fitting | TurtleTrader
What the Numbers in the Expert Testing Report Mean - MQL4 Articles
Mathematics in Trading: How to Estimate Trade Results - MQL4 Articles

Cheers,
Piotr

 

[Chorlton]

In this thread I would like to establish a FAQ for system backtesting. As I am not definitely the ultimate expert at the topic, all suggestions will be more than welcome. The procedure I would like to follow would be putting the FAQ in this, first post of the topic. The post will be modified when I have time to add more things, according to your suggestions and remarks.

I'd like to start with a few links.

1. Internet Knowledge Sources
These websites provide some good insight on system backtesting, check which statistics people present while testing a system. Good stuff there, especially on the MQL4 website.

EXLENCE - C2 trading system which trades futures
Don't Curve-Fit That System
No-Curve Fitting | TurtleTrader
What the Numbers in the Expert Testing Report Mean - MQL4 Articles
Mathematics in Trading: How to Estimate Trade Results - MQL4 Articles

Cheers,
Piotr

Hi Piotr,

Thanks for starting the thread.... I shall be taking a look at the links in more detail.....

Chorlton

 

[egro1egro]

Is anybody concerned with data-mining and data-snooping bias at all?

 

[Chorlton]

Is anybody concerned with data-mining and data-snooping bias at all?

Can you explain what the difference is between data-mining and data-snooping, as I've never heard the term "data-snooping" before

Thanks....

 

[Mathemagician]

Data snooping is when you (hopefully accidentally) use future information as part of your trading rule. An example would be buying market on close when the market closes above its 50 day moving average.

Data mining is the process of searching through data looking for patterns. This process creates a bias which positively skews results, often significantly.

jj

 

[Chorlton]

Data snooping is when you (hopefully accidentally) use future information as part of your trading rule. An example would be buying market on close when the market closes above its 50 day moving average.

Data mining is the process of searching through data looking for patterns. This process creates a bias which positively skews results, often significantly.

jj

Mathemagician,

Thanks for the explanation. I was aware of Data Mining but not the other one. Luckily, the software I use allows me to easily check whether "future" data is being used so hopefully I shouldn't encounter any problems with Data Snooping.

 

[Mathemagician]

Mathemagician,

Thanks for the explanation. I was aware of Data Mining but not the other one. Luckily, the software I use allows me to easily check whether "future" data is being used so hopefully I shouldn't encounter any problems with Data Snooping.

There is no substitute for knowing what you're doing. Be careful!

jj

 

[Chorlton]

There is no substitute for knowing what you're doing. Be careful!

jj

mmm .... Valid Point !!!!

 

[egro1egro]

Sorry, for my late reply. I didn't notice that I wasn't subscribed to this thread.

Data-snooping is not look-ahead bias at all!

Data-snooping bias is similar to data-mining bias but appears when someone else performed data-mining and then only reported the final result. It is the most dangerous part, since nobody wants to report about failed attempts.

Data-mining is a useful procedure that can potentially produce profitable strategy, but average performance of such system on historical data is most likely to be higher than in reality (or out-of-sample). For that reason one might want to store results of all runs while optimizing parameters or modifying rules etc. Then the results might be recombined randomly in order to model effect of data-mining bias.
However, when you pick up some "good idea" on a forum, for instance, you wouldn't have all the data from each run - you can only reproduce trading performance on the historical data for just one run. One can try to test on out-of-sample data and the performance can be great as well. Only when you trade in real time the defects will develop. For that reason, some perform a rather cumbersome procedure of "reverse engineering", i.e. try to guess all possible trading rules that were tested before the profitable strategy had been found. For example, if you see a "good system" based on 2 indicators of usual kind (ma and stochastic, for example), then you might want to gather MAs with different periods (including EMA, WMA, KAMA etc.) then add all oscillators, then add all indicators like bollinger bands, then combine them all with different levels for stop-loss and tp... and so on. Seems a lot of work. But you can also reduce the search space since MAs with similar periods (like 11 and 13) are very much correlated. Having collected all these combinations of 2 arbitrary indicators with arbitrary sl and tp levels you run them all on historical data one by one, store the results in a large database. After that history can be randomly scrambled and the whole process repeated on rearranged quotes. After repeating this 1000 times one might get an idea that the "good system" is not so good, since many combinations produced similar results in some versions of randomly scrambled history... This procedure is called Reality Check.
Obviously, the most popular trading software (MetaTrader, MetaStock, TradeStation etc.) do not help traders with measuring data-mining bias. They'd rather people don't know about the danger of data-snooping...

 

[dillonpanthers]

does anyone use CQG for backtesting? how would you rate it?

 

[igual]

over optimization - help

reading through some of the links, people frown upon over optimisation. I am not sure what over optimisation is. Surely systems fit different currencys and different timeframes in different ways so all systems need to be optimised.

I can make my ea produce huge returns over the last two years but it fails over the course of a year five years ago. Does this mean the system has failed or that markets have changed. That said I can make my system perform even better by just backtesting it over 3 months. So again which data is more relevant. Should I use a system live changing the data each week according to the optimisation of the previous 8 or 12 weeks? Or should I base it on 18 months of data or be too scared and not use it at all??

Thanks for your input

igs