Valid Backtesting

[new_trader]

I may have originally posted this in the wrong forum:

I have just read the 10 principles and it mentions choosing a valid test period. I have tickdata for the E-mini S&P500 going back to September 1997 and have backtested my plan right up to 2005. It produced some discouraging results around 1999 and 2000, but still worth trading.

I was encouraged to read that 2-4 years backtesting is more appropriate because the volume of E-mini's contracts were very low prior to 2001. Buy WHY should that matter? Am I missing something? Isn't the price of an E-mini contract related to the much larger S&P 500 which trades huge volumes?

 

[Tuffty]

Low volume may mean data gaps. The instrument if traded may have moved against you and taken out a position in your backtesting but because it never traded at that level (due to low volume) you'll never know.

Also (bit of a side line really) there is a scientific principle that makes the point you can't experiement (ie backtest) on something that if you did it would move the object (market). This is a particular issue where any dealings in an instrument would tend to move the price even with small trades.

 

[Directional]

Isn't the price of an E-mini contract related to the much larger S&P 500 which trades huge volumes?

the price of the emini AND the full size is linked to the cash index. Even accounting for the fact that 5x emini = 1x big S&P, the emini overtook the pit traded contract in comparable volume a couple years back - huge volume is on the screen nowadays, not in the pit

 

[new_trader]

Thanks for the info. So, to clarify the points you have both made in regard to backtesting. Is it not worth backtesting periods prior to 2001 because low volumes may give less realistic results?

The tick data I have shows OPEN, HIGH, LOW, CLOSE. I thought this would indicate actual trades, not just quoted prices.

 

[Tuffty]

You could use the full size contract prior to when there was reasonable volume on the emini for backtesting.

"The tick data I have shows OPEN, HIGH, LOW, CLOSE. I thought this would indicate actual trades, not just quoted prices"

Depends how the data supplier deals with the issue or how your software deals with missing data.

NB: As we're in the first steps section don't forget that 'out of sample' backtesting and the consistency of different periods is the only way (I know of) to not over fit to the data.

 

[new_trader]

I have written my own backtest program in Visual Basic based on the example trading system from this site. I have backtested every year from Sept 1997 to Dec 2005. I am still checking to ensure there are no errors. I have tickwrite and select "Fill Empty Intervals" when creating the tickdata ASCII file. The results of 1999 and 2000 are the worst years so I am wondering if they are valid or not. I don't understand what "out of sample" backtesting means. How can you do this if you system covers all the sample data available? Do you mean testing with a different contract?

 

[Tuffty]

I guess "Fill empty intervals" will just put in data from the prior bar. This may well be an issue for your system testing.

"out of sample" backtesting means proving the system on unseen data as you would when you trade it in real time going forward. Testing a sytem that has been taken from the public domain on possibly the same data the system was designed on will invalidate the "out of sample" testing. This is because the designer could have curve fitted the data....you just don't know. Any system I find in the public domain that I think may be worth looking at I note the date it was first mentioned. The only valid out of sample testing will be any date thereafter. (I basically don't trust anybody else's backtesting methodology when risking my own money.) Of course some people think backtesting is a complete waste of time anyway......it's down to whatever you need to do to get you comfortable about the risk (ie putting money on the line in a system) you are considering.

 

[new_trader]

The system I have backtested is:

http://www.online-futurestrading.com/example_trading_system.htm

The system used data from 2003 & 2004. I have backtested 2005 and it looks good.

"Fill Empty Intervals" does repeat the last bar but it has negligable impact on my results (Assuming the missing data hasn't hit my stops). However, I had a choice of not trading days where all the data isn't available or filling empty intervals. The results weren't dramatically different considering the system makes about 50-60 trades/Quarter.

 

[Tuffty]

2005 is a true out of sample test assuming the parameters were not tweeked after seeing the results or have been updated in 2005 (I think Tim did update them at some point) . Also slippage and dealing costs can affect the results quite a lot.

 

[new_trader]

Tuffy,

I wrote a VB program using the exact setups described and backtested 2005 with my own tick data. I have compared the first 2 quarters of 2003 with the results posted and they are almost identicle. (Slightly different closing prices and some days where I make a trade and they didn't). So my 2005 results are as per the original parameters and they are encouraging.

Slippage and dealing costs would affect any system though. I have tried a few variations of the system with different filters and so far the stock standard "every trade" one comes out best. I can't see how you can avoid the pitfalls of erroneous data/slippage etc with any system you backtest. I suppose there is an element of faith required when you decide to trade it with real money, as I am now doing.

I have tried intuition, trading on economic news and technical levels but have done poorly. I need a simple mechanical system that eliminates gut feeling and this one seems right for me.

The more I backtest the more I believe that there is NO HOLY GRAIL trading system.