Backtesting - A beginners guide

Nick Best

Newbie
Messages
5
Likes
0
As a beginner, I've started backtesting some initial strategies (using amibroker).

However, I've read contradictory views on the benefits of backtesting trading strategies. I don't want to spend months doing this if it is not worthwhile. I'd love some advice on:

1) The benefits of backtesting
2) The pitfalls and how to avoid them
3) Is it worth the time and effort spent
4) Backtesting software requires a highly programmed approach - is this an inherent weakness of ths approach.
5) What data should I use for backtesting ideas
6) Anything else that's relevant.

Thanks
Nick
 
A few brief thoughts...

1) It can show whether a system would have been viable in the past and hence MAY continue to be so in the future.
It can show your maximum potential loss and maximum consecutive number of losing trades. Would the "max adverse excursion" as Investor/RT euphemistically puts it have wiped out your account or at least rendered you unable to trade due to lack of funds?
Optimisation - you can fine tune stop loss levels, trailing stops, indicator parameters etc. to see which settings would have given the most consistently good results.

2) Curve fitting (over optimisation): it's no use if a system gives a 400% return over one year because you have tweaked it within an inch of its life to fit that particular data series. Look for consistency over several different periods with as much data as possible. e.g. I've tested an hourly bar system over 1,2,3,4,5,6 years (&random chunks in the middle) and would consider that just about enough. Not accounting for commission and slippage. Trying to add more and more rules to a simple system to account for every eventuality - simple is usually best.
Can you handle trading the system in real time without breaking the rules? e.g. if it regularly takes 10 losers in a row, or has a low percentage of winning trades, or makes most of its annual return out of a handful of trades etc.

3) I think so but many would disagree.

4) Not necessarily. Some traders swear by mechanical systems, while others thrive on discretionary ones involving little or no mechnical decisions. It is generally accepted that a mechanical system will make less money than a skilled discretionary trader, but a poor discretionary trader may often lose a lot more than either! One's choice of style comes down to personality, commitment, discipline, experience, knowlewdge time available etc.

5) The data for the instrument you plan to trade. So it's no use backtesting INDU (the Dow index) when you can only trade Dow futures, for instance. Common instruments would include index, commodity and currency futures or individual stocks. If you require lots of data be prepared to pay for it (or ask nicely here!).

I'm sure others can add to this. Our resident experts are probably JonnyT, Kimmrunner and Grey1?

More stuff here -
http://www.moneycareers.com/bonanza.htm
 
Last edited:
Nick

Congratulations on selecting a very interesting bit of s/w!
AB is totally flexible and what with the many ways one can interface to it - there are no programming limits :)
(I've just purchased AB licence and not regretting the move)

I have read elsewhere on T2W that in the end backtesting gives one an idea regarding suitability of a system/method/idea however all that data is past and the present is really all you've got. So perhaps actually paper trading asif for real or better still hookup to a few of the virtual/simulated SB offerings and Alpha/Beta/Ztest your brainchild to exhaustion!

Have a look/signup for some of the following simulator offerings at SB's:
www.tradindex.com
www.capitalspreads.com
www.fxcm.com
www.cityindex.co.uk

"Backtesting software requires a highly programmed apporach - is this an inherent weakness of this approach."
In s/w the weakness is ALWAYS the designer,coder and tester. Since you are all three - at least no communication problems - hopefully! You can get so involved that you cannot see clearly even tho staring you in the face! For instance, just look at the compilers 10,000 line o/p of a first time compile and you'll know what I mean. It's easy to kid yourself that "of course I'm doing it right/on-the-right-track/etc..."
But heck, this forum AND the AmiB's highly active forum (sorry T2W) is great for all the AB related probs and even backtest designs too.

Do you have AmiQuote? should have as comes with full download/else get from AB download page. Just fire up and put 'source' to Yahoo Historical - get a symbol list in the ticker pane and fasten yer seatbelt :)
Oh yeah, make sure the 'Automatic Import' box is ticked!

Backtesting ideas can come from your mind meanderings during all types of activities - reading the books/forums/lookin out the window! Again, search around on T2W and AmiB users forum. All books rave on about method/system xyz, yes? Well then, now you've got the tools to actually program the baby and then probably have a good laf when backtesting LOSS variable blows its size!!!

Also, many websites listing zillions of forumlas/ideas/methods/systems out there.

I'm sure you will have great times ahead doing the WHATIF's with AmiBroker.

Wishing you the best.

Tim
 
Bcktesting can make a lot of sense IMO, it can - at least - be used to test whether your latest brainwave is a load of rubbish... I wake at 2am yelling "Of course! A golden cross of the MACD with Williams %R, why didn't I think of it sooner!' ... fire up the tester, discover it gave 359 consecutive losers in the past year, go back to sleep!
Many people do not think logically - a program is logical, lots of people input test parameters that find A and they think they're seeing B, others think that 'tweaking' is okay (see earlier, very good mail from frugi). Like much in trading, it's a tool that some use well, others misuse and then abuse because they didn't get out what they expected to see.
It's also, I have to admit, something you can misuse by accident - more than one 'scan' has looked good, only for the cold light of day to show that the user has unintentionally shaded the odds to favour the expected outcome.
Dave
 
NB & Frugi
As you are AB users why don't you consider putting together a review of the s/w- we don't have one so far
 
Just got Ami myself, like Nick - however I am shortlisting bits 'n pieces to put together AmiBroker blurb. Just wanna be sure to give fair assessment of what to date strikes me as being a good bit of kit - with shed-loads of untapped functionality waiting to be played around with...
Tim
 
Top