Backtesting - is it a valid concept ?

[TheBramble]

i can say i am a little disappointed about backtesting though .

George, I think we all are when we discover the limitations (pointlessness?) of back-tetsing.

When you realise you're fitting your 'systems' to what was, in the hope you'll capture what is to be...

There's a benefit in looking to the past to decide how the future may pan out (as in not making the same mistake twice), but traders tend to bend this approach right back on itself.

 

[theknifemac]

George, I think we all are when we discover the limitations (pointlessness?) of back-tetsing.

When you realise you're fitting your 'systems' to what was, in the hope you'll capture what is to be...

There's a benefit in looking to the past to decide how the future may pan out (as in not making the same mistake twice), but traders tend to bend this approach right back on itself.

I understand what Curtis is saying but I think it still makes sense to perform backtesting - a system which backtests badly is likely to perform badly, unfortunately the converse is not necessarily true. When looking at the back test results you should see what happens when the parameters are changed slightly, if the results are consistent then the system is unlikely to be over optimized.

As Curtis puts it "just because optimization results in tests that overstate likely future results, this does nto mean that optimization should not be done". It means that the best set of parameters for the system are not necessarily those which produces the highest profit / best statistics, instead you are looking for parameters which put you in a flat area of the optimization results rather than at the top of a peak.

Stew

 

[georgek]

I understand what Curtis is saying but I think it still makes sense to perform backtesting - a system which backtests badly is likely to perform badly, unfortunately the converse is not necessarily true. When looking at the back test results you should see what happens when the parameters are changed slightly, if the results are consistent then the system is unlikely to be over optimized.

thats the point.


Curtis , my english is not so good and i cannot understand 100% what you'are saying..

 

[theknifemac]

thats the point.


Curtis , my english is not so good and i cannot understand 100% what you'are saying..

Don't worry George, your English is a lot better than my Greek ! And congratulations on the football. Yamas !

Stew

 

[georgek]

Haha thanks ! we were great,everyone's crazy about this success here !...

 

[mr_cassandra]

I have and I do

You can get many widely diverse opinions about back-testing but many times the person is just quoting some article they read or commentator they listened to.

I invest with real money using a program I wrote myself over the last 4 years, called the mvp signal system. It contains data and back-testing going back into 1995. During this period I've had all sorts of feedback pro & con but only about 1 in 20 responders had any research background of their own, ie they actually went and crunched large amounts of numbers on their own.
In my opinion, from having been at this awhile, three things are crucial for back-testing to be of value.
1. The back test period should include widely varied types of markets from flat to bull run to a bear market. As regards a bull run, it would be even better if it included a 'bubble run' or as Mr Greenspan puts it, - euphoria. During the past 8 years all of that has been seen in the U.S. Market.
2. The settings, or 'triggers' in the system should be tuned 'down' to the mildest possible parameters.
3. You also have to be careful that the back-tested results have not caused or included some wild trade(s) which make it look great but were probably anomalies.

Last but not least, the system must act the same way and produce the same results when used in real-time. That is the stage I'm currently at. My website explains in more detail and I post the signals in real-time at clearstation.com (my prior posts contain links to these)

Another site doing a system (succesfully) is www.vtoreport.com - 5 day rsi system.

Regards, Steve

I wonder what others think about the concept of backtesting ?
One the one hand it perports to find the best set of parameters for a given method and thereby tune it to the period tested.
If the results are good, then these parameters can then be used with some (?) confidence for live trades.
On the other hand one could argue that backtesting is just curve fitting and therefore simply 20:20 hindsight.

Seems to me that for a backtest to have any validity, it must produce a smoothly increasing equity (profit) curve with no wild girations. This at least demonstrates that the method and parameters found worked fairly consistently over the period tested.
Of course a lot depends on the period(s) selected.
Again to have any validity, I think that a test must either be run over a considerable time period, embracing many different market conditions and yet still remaining consistent, or it must be run on selected market conditions with a view to using the optimised parameters from the test in that type of market.

To disprove the theory of curve fitting, one suggestion is to take a period of time, say 5 years, and run your test over the first half of the data to get some results.
These results can then be run over the second half of the data as if it were the unknown future. i.e. a kind of Forward Testing.

One thing's for sure, the Software vendors must love things like backtesting. What an opportunity to develop and sell a nice technical product, using the math co-processor and the disk drive of your PC to the full. A nice 'gadget'.
Lovely jubbly - at least for the vendors.

Enough from me. What do others think? In particular anyone who has used a backtested system in the real market sucessfully.

Glenn

 

[mr_cassandra]

re forward testing

This is a tough one. I'm on version 21.C as we speak and all of its code is a collection of if-and-or but logic which has been evolving and being improved since day 1. I understand the gist of your thought and agree, but would not know how to apply it.
One other observation, any code change I ever make to the program is always copied back to day one to look for what I call 'bogeymen'. If it can't bring a steady and/or calm reaction to the entire 8 years, its trashed.

Hi Mr Cassadra
Agree with your thoughts.
I've continued experimenting with one system which was tested since June 1998. This covers quite a range of market types.
One modification I have employed is in respect of the Equity curve itself. By applying a simple 30 day moving average to the Equity curve it gives a signal when to stop trading the system (because it has gone into a losing phase) and when to start again.
6 months on and it's still working ok, including a 40-day 'stay out' period.
The system itself is designed to take account or recent volatility (or the opposite) and respond accordingly.
This way it tries to include current conditions and data in the math model, rather than just relying on an indicator etc.

Have you tried what I described as 'forward testing' ?
e.g. Test and refine a system over 3 years and then run it forward over the next 2 years to see if it still works. (These are just example numbers, - the longer the better in both cases imo.)

Glenn

 

[Glenn]

This is a tough one. I'm on version 21.C as we speak and all of its code is a collection of if-and-or but logic which has been evolving and being improved since day 1. I understand the gist of your thought and agree, but would not know how to apply it.
One other observation, any code change I ever make to the program is always copied back to day one to look for what I call 'bogeymen'. If it can't bring a steady and/or calm reaction to the entire 8 years, its trashed.

Re Forward Testing

All I would suggest in your case is that you backtest over the first 5 years to get the system parametrs tuned, and then you use those parameters over the next 3 years data and see how the system performs.
OK this does not utilise the whole 8 year period, and it precedes 9/11/01, however it would be interesting to exclude that particular unfortunate event and see how the system coped with it as well as the remainder of the bear market which followed and the bull run since.
Glenn

 

[mr_cassandra]

Many ways to show it does

The market is a large herd of cattle, stampeding back and forth. Their numbers reach repeating extremes, particularly on the down side. It takes a lot of work to go diligence this. Long ago I ran barrons stats on microfiche and entered into a laptop to define net new lows, year after year.

You'll find that most naysayers of such concepts have not actually done the research themselves, rather are relying on books, articles or comentators they have read.

Glenn,

Before you ask about back testing being of any value , you have to ask yourself another more important question..
DOES PAST CORRELATE tO FUTURE ?

You can only have three answers .


1) yes it does..
2) No it does not
3) it does to some degree
if you think number 3 is the answer then your back testing will be effective to the same degree ..

 

[Naz]

Never mind the past look to the future.If you see it take, manage it, trade it.

To many people get analysis paralysis and hence get no where.

Of course this is just a personal opinion, others will probably think differently.

 

[The General]

There is a fundamental flaw with classical probability which no-one here seems to have realised. This is of course linked to the discussion on backtesting.

Suppose you have 3 systems, each of which (as measured by simple traditional probability) has a 33% success rate. Which one would you choose to trade ?

 

[TheBramble]

Suppose you have 3 systems, each of which (as measured by simple traditional probability) has a 33% success rate. Which one would you choose to trade ?

If I HAD to choose one to trade, it would be that with the lowest risk:reward, lowest potential drawdown - i.e. highest positive expectancy.

But I'm interested, what is "There is a fundamental flaw with classical probability which no-one here seems to have realised"?

 

[JonnyT]

Hi General,

I would look at the drawdowns on all systems and at the rewards.

It will all depend on what the capital requirements are within your risk analysis and how much bang per buck you could reasonably expect to achieve.

But then again I'm seriously flawed.

JonnyT

 

[mr_cassandra]

Drawdown on mvp version 21C

Max drawdown is 8.6% during one long trade which ended in the black when held to completion by system. Any data I post about this system can be firther diligenced at the site and/or my public trades at clearstation.com

regards, Steve

Hi General,

I would look at the drawdowns on all systems and at the rewards.

It will all depend on what the capital requirements are within your risk analysis and how much bang per buck you could reasonably expect to achieve.

But then again I'm seriously flawed.

JonnyT

 

[mr_cassandra]

One siggestion

My opinion would be that you would perform due diligence on all three, and look to determine which one had the most consistent performance as defined below:

1> works thru diverse types of markets
2> provides consistent trades, not based on a few huge hits
3> beats buy and hold by enough margin to justify all this
4> results include all losers
5> taxes and commissions would not reduce its results to less than buy and hold.
6> general timing of trades make sense when diligenced versus the chart of the vehicle; IE I would have liked to buy or sell at X where this system did so.

There is a fundamental flaw with classical probability which no-one here seems to have realised. This is of course linked to the discussion on backtesting.

Suppose you have 3 systems, each of which (as measured by simple traditional probability) has a 33% success rate. Which one would you choose to trade ?