HOW many trade are legitamate for Backtesting

[lbranjord]

How many trades would you consider a legitimate backtest? Also, can all the data come from the same year, or are historical charts a neccessity for you?

Thanks!

 

[Adamus]

Hmm, well, I'd say 30 but I haven't proved this to myself yet. Just read it somewhere. Seems like a nice round even number that's not too small and not too big.

Perhaps 42 would have been better.

 

[Adamus]

Just thinking more (surprisingly) and I think the number of trades should relate sensibly to the number of OHCL bars you are backtesting.

To take it to the extreme, 1 trade on a market history of 1 bar would not convince me.

10 trades in 100 bars might be more convincing, especially if repeated and walked forward successfully.

 

[1Pipped]

This might be useful http://en.wikipedia.org/wiki/Checking_whether_a_coin_is_fair

I know trades are different to coins because you might have different probabilities on each separate trade, but it might give you an idea of how many you really would be happy with in a backtest.

 

[Adamus]

That is exactly the kind of calculation that I need to do. I guess I was dumb not to think it could be calculated. Lucky this thread came up.

Only one problem. I have very little idea what all these terms are that the wikipedia article uses.

Seems to me that the second formula is the one to go for - 'estimator of true probability' where you can choose the 'confidence level' you would be happy with - I can comprehend a bit more of the statistics jargon than the first formula :smart:

But I'm not even sure what the big formula for 'estimator of true probablility' calculates, or better said, which symbol represents what we want, i.e. how many trades are legit for backtesting.

It says "any fairness test must only establish a certain degree of confidence in a certain degree of fairness (a certain maximum bias)."

and

"This method assumes that the experimenter can decide to toss the coin any number of times. He first decides on the level of confidence required and the tolerable margin of error. These parameters determine the minimum number of tosses that must be performed to complete the experiment."

so I reckon the minimum number of coin tosses is equivalent to the minimum no of trades required.

But the other parameters leave me a bit !

 

[shadowninja]

Enough to cover all market conditions. What works now might not have worked in 2006-2009.

 

[lbranjord]

I should add that I like to backtest my systems using "worst case scenario". If there is any discrepancy on where the entry or exit would have come, I chose the worst one. If there is any discrepancy on whether a stop loss would have been tapped, it would have.

After all this, I look for at least 50% strike rate and strong profit.

My backtesting has only been for 2008-2009 though in my hourly systems, hence the reason for the thread.

I suppose that I will always backtest however, and adapt as the market changes and notice fluctuations in each pair. Because of that, I think I'll be fine rain or shine.

I wouldn't mind a charting system with 10 years of hourly charts though, so I could try things out in earlier markets.

 

[Mr Flibble]

Just make sure you hide some data from it as you optimise it. I cannot stress enough how important that is. I do about 3 years on an hourly chart but leave off at least 4-6 months optimise then include the newest data to see if it sinks (which it normally does) or swims. I look for a nice line going from the bottom left to the top right of accrued pips and drawdown must be very small if possible.

 

[lbranjord]

Mr Flibble,

I'm not sure I follow how to do this. So leave off the latest 4-6 months?

Also, what charting do you use? Oanda's is notoriously weak, but I still backtest with it since I'm used to it.

 

[nunrgguy]

I'd test with as much data as you can get hold of. My current method has been tested (so far ) with data from 2001-2005. I will not be fully satisfied until I'm up to the present day with it and have then finished forward testing for 3 months.

Going through bar by bar on the hourly is not good enough either btw imo.

The data I use goes down to the minute and is simulated randomly intraminute for ticks. Even this isn't perfect as the software I use fills you at your price whether the market gaps or not. However it slips you on every entry by 4 pips.

Once I've gone through all the data (this is a manual test traded on the 15min, no forward peeking and traded only during the market hours I would actually have traded), the data will be gone through again looking for any gaps. Those trades will then be deleted from the results.

Likewise, my method involves having lots of small losers, small winners, waiting for the runners. In reality I know that plenty of the 1 and 2 pips winners will be slipped and turned into losers. Again a percentage of the small winners will be deleted and likewise a percentage of the largest winners will be deleted before I'm happy with the results.

Now, in theory, so far from 2001-2005 I would have turned 10k into over 200k risking a max of 2% per trade, often much less than 0,5% for certain periods...wow, great. However when done the data results will be put into a monte carlo simulator to see what the worst case scenario could have been. I then take that as the real world likliehood.

 

[tomorton]

Suitable number of backtest example trades depends on what you're trying to prove. If you're trying to prove a TA 'truth' on a certain market or grouping of markets as many examples as you can find - if in the thousands and encompassing bull, bear and ranging market periods, that would be good.

But if you're trying to prove and refine your trade / money management, its the losing trades that are more interesting. Writer Mark Douglas suggests no system is worth using until you have tested it to show what happens if you have 20 straight losers. But he was a daytrader so that figure might be way too easy. Even if your entry signal and trade management together are 90% effective, you can still be hit by such a run. Better if you can go beyond 20 straight losers, assuming 20 still leaves you with an account - test your account to destruction if you can. If you find you just cannot wipe out your capital in backtesting, even if your TA isn't that great, you'd be onto something.

 

[lbranjord]

Good insight Tom. THis may work for me. I actually trade 2 different systems and 4-5 currency pairs are involved. Either way, 20 losing trades would be devastating, but seemingly recoverable. I think I'll look into this and test for the apocolypse.

 

[Adamus]

20?

At what level of confidence was that?

i am still trying to work out how to use that statistics formula for working out the minimum number - you choose your level of confidence (60%, 80%, 90%, 99% etc) and your margin of error and plug the values in. But the examples I've found googling are all for binomial results (coin flips), not multiple various profits / losses.

 

[lbranjord]

Good addition to the thread. I have thought about incorporating confidence levels into my trading systems or at least post-trade tracking. My 2 systems though are such that If i see a signal, there is no discrepancy, I trade it.

I think logging confidence levels would show where a lot of losses come from and maybe even show waves in profit and loss we did not realize exist.

 

[lbranjord]

I'd test with as much data as you can get hold of. My current method has been tested (so far ) with data from 2001-2005. I will not be fully satisfied until I'm up to the present day with it and have then finished forward testing for 3 months.

Going through bar by bar on the hourly is not good enough either btw imo.

The data I use goes down to the minute and is simulated randomly intraminute for ticks. Even this isn't perfect as the software I use fills you at your price whether the market gaps or not. However it slips you on every entry by 4 pips.

Once I've gone through all the data (this is a manual test traded on the 15min, no forward peeking and traded only during the market hours I would actually have traded), the data will be gone through again looking for any gaps. Those trades will then be deleted from the results.

Likewise, my method involves having lots of small losers, small winners, waiting for the runners. In reality I know that plenty of the 1 and 2 pips winners will be slipped and turned into losers. Again a percentage of the small winners will be deleted and likewise a percentage of the largest winners will be deleted before I'm happy with the results.

Now, in theory, so far from 2001-2005 I would have turned 10k into over 200k risking a max of 2% per trade, often much less than 0,5% for certain periods...wow, great. However when done the data results will be put into a monte carlo simulator to see what the worst case scenario could have been. I then take that as the real world likliehood.

Nurgguy,

I'd like to test as far back as you but I am not a programmer and have to test these manually as well. I use Daily and 3hr charts. Where do you get your data (if you don't mind my asking). I'd love to have 3-5 years of 3hr and daily charts to backtest, but I'd need to toss a few indicators on there in order to test my data. Any ideas where I could get my hands on that? Not opposed to paying.

 

[lbranjord]

Curious what you guys see here. Here are the pips made on my trading system throughout the month of January. This is cumulative. So the first day I made 61, the next I made 120 (add to 61 for 180), the next I made 10 (add to 180 for 190). Does the draw down in the middle of the month make this a survivor? Or is getting pulled down to 46 pips something to seriously worry about? I got spreads to deal with too, so I would like to see better numbers I guess...

61
180
190
193
174
209
223
216
217
305
258
223
174
169
214
141
94
46
120
208
229

 

[nunrgguy]

Curious what you guys see here. Here are the pips made on my trading system throughout the month of January. This is cumulative. So the first day I made 61, the next I made 120 (add to 61 for 180), the next I made 10 (add to 180 for 190). Does the draw down in the middle of the month make this a survivor? Or is getting pulled down to 46 pips something to seriously worry about? I got spreads to deal with too, so I would like to see better numbers I guess...

61
180
190
193
174
209
223
216
217
305
258
223
174
169
214
141
94
46
120
208
229

Firstly, the sample size you have is insignificant. The results also mean nothing without knowing actual account % drawdown. You've got a drawdown in pips of 85% but on its own that means nothing.
Yesterday I made +ve pips but the account ended up down...

 

[lbranjord]

Do i need to be trading with a percentage of my capital in order to calculate drawdown? I ask because I like to trade with a certain lot size. I'm not opposed to changing this though

Thanks Nurg

 

[nunrgguy]

However many lots you employ on a trade will be worth x % of your capital yes? You need to know how much you''ve won and lost on each trade. Pips in themselves mean absolutely nothing.

Yesterday's example:
Trade 1 = -27 pips
Trade 2 = +9 pips
Trade 3 = +64 pips
Day total = + 46 pips

So how did the account end up £15 down yesterday?
You're not trading pips, you're trading money.

 

[lbranjord]

Gotcha. Yeah I trade same lot size always so i can measure this easily.