Dangers of back testing and optimisation

[pkfryer]

I've been playing with omnitrader and I think its a very useful tool for experimentation.

Its got a facility for splitting up the test period into back testing and forward testing, with optimisation of signals etc. One thing that its highlighted to me is the dangers of backtesting. A lot of these signals when optimised (to a very limited degree) were giving amazing hit rates and APR during the back testing period. But as soon as it crept over to the Forward Testing period the performance nose dived for most of them. Some of the stock still did well, but 9 times out of 10 these stock were in a really strong trend for the majority of the test and a brain dead monkey would have been hard pushed not to profit!

The optimisation wasn't that great... but even so, it highlighted that optimisation on historic data is a bad idea. Plus it emphasised the fact that backtesting successfully doesn't actually mean a great deal. When I increased the testing period the performance decreased, as the markets had changed behaviour.

I'm wondering whether indicator based systems are really the way to go and whether looking at price patterns directly might not be the best approach. The only problem with this is that it is hard to back test it unless you are willing to manually go through all the historical data one day or tick at a time! In itself not a bad learning tool, but not exactly very fast!

 

[Trader333]

Backtesting is of limited use in my view and I have done it extensively in my earlier days of trading. When trading stocks there is just no way that I could backtest at all because Level II cannot be backtested and yet forms a critical part to my trading approach.


Paul

 

[Glenn]

"I'm wondering whether indicator based systems are really the way to go"
In my experience they are useless. Better to use the raw ohlc data and nothing else imo.
As for volume - no use to me. Others may find it useful.

I think this is the third thread created on this topic. Have a look through the others.
Glenn

 

[theknifemac]

For people wanting to use a system, back testing is valid in the sense that a system which fails backtesting is unlikely to perform well in the future. Unfortunately the converse is not always true !

Cheers

Stew

 

[mr_cassandra]

Another thought

I find a lot of systems at wealth-lab.com are geared to one stock and fail when tried with various major indexes. There may be more merit to trying to base a system on indexes only.

<img src="http://img350.imageshack.us/img350/7020/version21u2ts.png" alt="Image Hosted by ImageShack.us" />

I've been playing with omnitrader and I think its a very useful tool for experimentation.

Its got a facility for splitting up the test period into back testing and forward testing, with optimisation of signals etc. One thing that its highlighted to me is the dangers of backtesting. A lot of these signals when optimised (to a very limited degree) were giving amazing hit rates and APR during the back testing period. But as soon as it crept over to the Forward Testing period the performance nose dived for most of them. Some of the stock still did well, but 9 times out of 10 these stock were in a really strong trend for the majority of the test and a brain dead monkey would have been hard pushed not to profit!

The optimisation wasn't that great... but even so, it highlighted that optimisation on historic data is a bad idea. Plus it emphasised the fact that backtesting successfully doesn't actually mean a great deal. When I increased the testing period the performance decreased, as the markets had changed behaviour.

I'm wondering whether indicator based systems are really the way to go and whether looking at price patterns directly might not be the best approach. The only problem with this is that it is hard to back test it unless you are willing to manually go through all the historical data one day or tick at a time! In itself not a bad learning tool, but not exactly very fast!

 

[mr_cassandra]

The vto 5 day rsi system and Nocona variation

Both are taking a backtest and doing very well in the real world. In fact, I suspect that if most people compared their own brokerage statements to vto since 1997, they might find it a shocker. www.vtoreport.com - rsi link


QUOTE=pkfryer]I've been playing with omnitrader and I think its a very useful tool for experimentation.

Its got a facility for splitting up the test period into back testing and forward testing, with optimisation of signals etc. One thing that its highlighted to me is the dangers of backtesting. A lot of these signals when optimised (to a very limited degree) were giving amazing hit rates and APR during the back testing period. But as soon as it crept over to the Forward Testing period the performance nose dived for most of them. Some of the stock still did well, but 9 times out of 10 these stock were in a really strong trend for the majority of the test and a brain dead monkey would have been hard pushed not to profit!

The optimisation wasn't that great... but even so, it highlighted that optimisation on historic data is a bad idea. Plus it emphasised the fact that backtesting successfully doesn't actually mean a great deal. When I increased the testing period the performance decreased, as the markets had changed behaviour.

I'm wondering whether indicator based systems are really the way to go and whether looking at price patterns directly might not be the best approach. The only problem with this is that it is hard to back test it unless you are willing to manually go through all the historical data one day or tick at a time! In itself not a bad learning tool, but not exactly very fast![/QUOTE]

 

[inflector]

This is a pretty good article on optimization:

http://www.tradingblox.com/tradingblox/optimization_paradox.htm

 

[mr_cassandra]

Back-testing and optimization

This is only an opinion from trading my own program and several years work. At some point you have to make a detrmination between being too optimistic and too pessimistic.

Ie I feel you do need a set of rule which continue to work in bull, bear of flat markets and can't over-tune to match any one type. My work goes back to 1995 so my rule set has been applied to a bubble run (dot com), the nasdaq bear (end of dot com) and all the mkts in between.

During the development of my program I have run into all manner of people with good sounding reasons it won't work, but after awhile I began to nitice that in many cases, I was talking to someone who had never actually checked or worked on a system himself.

A lot of times they were just quoting some learned sound bite they'd heard or read. One widespread mis-belief is that if any system or indicator works reliably, it will become so widely used that it will cancel itself out. Why thats not true imo, detailed below.

I read somewhere recently that any particular strategy will be skewed if it becomes popular enough and enough people start using it.
That is indeed a widespread belief or concept. It is one of many that are often repeated by the financial media, another one is 'you can't time the market'.

I think it does not take into account the huge effect human nature plays in its application. I’ve been developing a computer program that governs my trading since late 2000 and in discussions with other developers and users since then I have experienced many scenarios of human nature that would prevent any system or method from becoming so widespread that it would invalidate itself. Below is a list of factors of human nature that would be involved.

1. The myth that any system or method can catch every point of every move every time. If any system or method had even one loss or drawdown, this would cause X percent of followers to drop off in search of a better one. If this happened on their first trade X would become a larger percent. If there were two losers in a row, I’ve seen enough posts over the years to know there would be a large exodus.
2. Along the same theme, if the system buys or sells and misses any possible further gain, another X percent would become discouraged.
3. Along the way, you will also get people who 2nd guess the system, and thereby miss trades and get discouraged. This also involves media coverage making them discouraged or too optimistic versus what the program is telling them.
4. The 2nd guessing can also take the form of pure disagreement with the system, for instance if it said to buy at a major market low when things looked especially gloomy or vice versa during euphoria periods.
5. How much work will it take also eliminates anther X percent of possible system users. To illustrate this, here’s an example. Early versions of my program took me 5 minutes per day to update. I had several people ask me if there was any way to automate that so they wouldn’t have to spend 5 minutes per day. Think about it.
6. You would also loss another X percent of investors due to ‘the grass is greener on the other side’ syndrome, wherein they would drop off to chase a different system that looked better.
7. another X percent of people out there are religiously convinced it can't be done at all; to the extent they will steadfastly refuse to even look at trade receipts.
8. others feel that dollar cost averaging is the only way to go.
9. Others are all hung up on management fees to the extent that they will ignore a 20% fund with 2% fees in favor of a 14% fund with 1% fees, because a sound bite told them fees are everything.
10. others will get all hung up with macro bear or bull projections and miss all the intermediate moves a system could produce for them, sometimes ending up on the wrong side of an entire run.

In short, based on years of watching people discuss what they felt were problems or drawbacks on many systems discussed here, its my opinion that unless you have a concept that will deliver them consistent profits, no losses and deposit it in their bank account for them, no system or method will ever become so widespread that it invalidates itself.

I’ve seen variants of this idea saying that if any one indocator was all that useful, everyone would follow it and it would become useless. In particular I’ve seen this said about the vix. Yet, it is as useful to my program today as it has been at any other time since the first back-tested trade in 1996.

In part, I think that is because it measures fear levels translated into option premiums and as such is a guage of the crowd. As a mass effect, the crowd is highly consistent, decade over decade.

I've been at this since late 2000 to correct my own gross inability to be a trader by using my career abilities as a programmer and fascination with contrarian indicators into a program which removed 'me' from the picture.

You are right about people slagging it off. Over the years I've seen about 75% of the people who respond just quote some sound bite they once heard saying 'you can't time the market' or must be curve-fit etc etc. When you press them for details, many times it turns out they have done little or no work on their own, just read an article once by a broker who said so. JonnyT has downloaded the data and I gave him the rules and the RSI formula. If I talk to 50 people, I get perhaps 5 like him that actually will do the diligence.

My favorite incorrect stock market euphemism is: if any system or indicator works well then everyone will use it and it will be negated. How sad that one is. Whoever thinks that knows very litte about human nature.
This is a rehash of a post I made months ago why NO system or method will ever become so widespread that it will invalidate itself.

The brief answer is the human condition. Below are several examples.

1. it takes money to make money so if it takes 5,000 how many people did you just rule out?
2. Many people will not stay the course thru drawdowns and/or a losing trade or two, how many more people did you just rule out?
3. How many people will jump ship on a long because todays news scared them, or bail on a short because of some great economic news?
4. How many people will quit after one losing trade?
5. How many will try it for awhile then skitter off to try a more exciting looking prospect?
6. How many will freak if they get a drawdown?
7. How many of them would attempt to 2nd guess the signals then get discouraged even though they fudged it up?
8. How many people will make some excsue to withdraw their money because they want a new sofa? (or fill in the blank)

In short, unless it delivers 10% per month to their account in a bank with zero risk, I say no system or indicator will ever become that widespread because people are people.

This is a pretty good article on optimization:

http://www.tradingblox.com/tradingblox/optimization_paradox.htm

 

[new_trader]

This is only an opinion from trading my own program and several years work. At some point you have to make a detrmination between being too optimistic and too pessimistic.

Ie I feel you do need a set of rule which continue to work in bull, bear of flat markets and can't over-tune to match any one type. My work goes back to 1995 so my rule set has been applied to a bubble run (dot com), the nasdaq bear (end of dot com) and all the mkts in between.

During the development of my program I have run into all manner of people with good sounding reasons it won't work, but after awhile I began to nitice that in many cases, I was talking to someone who had never actually checked or worked on a system himself.

A lot of times they were just quoting some learned sound bite they'd heard or read. One widespread mis-belief is that if any system or indicator works reliably, it will become so widely used that it will cancel itself out. Why thats not true imo, detailed below.

I read somewhere recently that any particular strategy will be skewed if it becomes popular enough and enough people start using it.
That is indeed a widespread belief or concept. It is one of many that are often repeated by the financial media, another one is 'you can't time the market'.

I think it does not take into account the huge effect human nature plays in its application. I’ve been developing a computer program that governs my trading since late 2000 and in discussions with other developers and users since then I have experienced many scenarios of human nature that would prevent any system or method from becoming so widespread that it would invalidate itself. Below is a list of factors of human nature that would be involved.

1. The myth that any system or method can catch every point of every move every time. If any system or method had even one loss or drawdown, this would cause X percent of followers to drop off in search of a better one. If this happened on their first trade X would become a larger percent. If there were two losers in a row, I’ve seen enough posts over the years to know there would be a large exodus.
2. Along the same theme, if the system buys or sells and misses any possible further gain, another X percent would become discouraged.
3. Along the way, you will also get people who 2nd guess the system, and thereby miss trades and get discouraged. This also involves media coverage making them discouraged or too optimistic versus what the program is telling them.
4. The 2nd guessing can also take the form of pure disagreement with the system, for instance if it said to buy at a major market low when things looked especially gloomy or vice versa during euphoria periods.
5. How much work will it take also eliminates anther X percent of possible system users. To illustrate this, here’s an example. Early versions of my program took me 5 minutes per day to update. I had several people ask me if there was any way to automate that so they wouldn’t have to spend 5 minutes per day. Think about it.
6. You would also loss another X percent of investors due to ‘the grass is greener on the other side’ syndrome, wherein they would drop off to chase a different system that looked better.
7. another X percent of people out there are religiously convinced it can't be done at all; to the extent they will steadfastly refuse to even look at trade receipts.
8. others feel that dollar cost averaging is the only way to go.
9. Others are all hung up on management fees to the extent that they will ignore a 20% fund with 2% fees in favor of a 14% fund with 1% fees, because a sound bite told them fees are everything.
10. others will get all hung up with macro bear or bull projections and miss all the intermediate moves a system could produce for them, sometimes ending up on the wrong side of an entire run.

In short, based on years of watching people discuss what they felt were problems or drawbacks on many systems discussed here, its my opinion that unless you have a concept that will deliver them consistent profits, no losses and deposit it in their bank account for them, no system or method will ever become so widespread that it invalidates itself.

I’ve seen variants of this idea saying that if any one indocator was all that useful, everyone would follow it and it would become useless. In particular I’ve seen this said about the vix. Yet, it is as useful to my program today as it has been at any other time since the first back-tested trade in 1996.

In part, I think that is because it measures fear levels translated into option premiums and as such is a guage of the crowd. As a mass effect, the crowd is highly consistent, decade over decade.

I've been at this since late 2000 to correct my own gross inability to be a trader by using my career abilities as a programmer and fascination with contrarian indicators into a program which removed 'me' from the picture.

You are right about people slagging it off. Over the years I've seen about 75% of the people who respond just quote some sound bite they once heard saying 'you can't time the market' or must be curve-fit etc etc. When you press them for details, many times it turns out they have done little or no work on their own, just read an article once by a broker who said so. JonnyT has downloaded the data and I gave him the rules and the RSI formula. If I talk to 50 people, I get perhaps 5 like him that actually will do the diligence.

My favorite incorrect stock market euphemism is: if any system or indicator works well then everyone will use it and it will be negated. How sad that one is. Whoever thinks that knows very litte about human nature.
This is a rehash of a post I made months ago why NO system or method will ever become so widespread that it will invalidate itself.

The brief answer is the human condition. Below are several examples.

1. it takes money to make money so if it takes 5,000 how many people did you just rule out?
2. Many people will not stay the course thru drawdowns and/or a losing trade or two, how many more people did you just rule out?
3. How many people will jump ship on a long because todays news scared them, or bail on a short because of some great economic news?
4. How many people will quit after one losing trade?
5. How many will try it for awhile then skitter off to try a more exciting looking prospect?
6. How many will freak if they get a drawdown?
7. How many of them would attempt to 2nd guess the signals then get discouraged even though they fudged it up?
8. How many people will make some excsue to withdraw their money because they want a new sofa? (or fill in the blank)

In short, unless it delivers 10% per month to their account in a bank with zero risk, I say no system or indicator will ever become that widespread because people are people.

mr_cassandra, an excellent post. I agree with your sentiments because I am going through exactly what you have described but I refuse to deviate from my system.

 

[mr_cassandra]

Congrats, keep going !

Take everyones input as a possible good idea and consider it in your own time in your own way. But focus your main efforts on improving your own system, trying to minimize drawdowns and losses but bearing in mind they are a given in any method or program.
And, bear in mind some of the folks who come out of the woodwork with reasons why your idea won't work have never even tried one themselves.
It took me awhile to notice this but once I caught on, I found a way to cut thru the baloney. If someone says your idea won't work because of xyz, ask them for a constructive suggestion on how to address that issue via an improvement.

If they start blustering and have no constructive idea, you'll then know how much creedence to give their input.

In my experience, about 1 in 10 armchair system critics actually had useful experience of his own.

mr_cassandra, an excellent post. I agree with your sentiments because I am going through exactly what you have described but I refuse to deviate from my system.

 

[arnie]

How much work will it take also eliminates anther X percent of possible system users. To illustrate this, here’s an example. Early versions of my program took me 5 minutes per day to update. I had several people ask me if there was any way to automate that so they wouldn’t have to spend 5 minutes per day. Think about it.



Love your comment mr_cassandra. So many people have got so used to one-click solutions that they can't be bothered to put the work in any more.

Richard Dennis, one of the original Turtle Traders, once commented ".... I always say that you could publish my trading rules in a newspaper and no one would follow them ..."

 

[mr_cassandra]

the human condition

That was just a real world illustration of how idelaist and incorrect that one euphemism about devloping a trading system is. Sure, if everyone was consistent, hard-working, persistent, and never gave up or changed their mind, maybe one idea could become that widespread. But many of our fellow citizens want instant gratification, no hard work, a profit every time etc etc.
Ain't gonna happen.

I agree with Richard Dennis wholeheartedly.

How much work will it take also eliminates anther X percent of possible system users. To illustrate this, here’s an example. Early versions of my program took me 5 minutes per day to update. I had several people ask me if there was any way to automate that so they wouldn’t have to spend 5 minutes per day. Think about it.



Love your comment mr_cassandra. So many people have got so used to one-click solutions that they can't be bothered to put the work in any more.

Richard Dennis, one of the original Turtle Traders, once commented ".... I always say that you could publish my trading rules in a newspaper and no one would follow them ..."

 

[PKFFW]

Can someone please explain the difference between backtesting and forward testing? If you are testing the system on historical data wouldn't it all be backtesting? How can one "forward test" as system using historical data?


Cheers,
PKFFW

 

[new_trader]

See if I'm right!

Can someone please explain the difference between backtesting and forward testing? If you are testing the system on historical data wouldn't it all be backtesting? How can one "forward test" as system using historical data?


Cheers,
PKFFW

PKFFW,

This is what I think the answer is, hopefully I’m right.

When developing a trading system you will use a period of historical data, usually 6 to 12 months worth. I think it’s best to use a sample somewhere in the middle of your data. Once you have a system you think may be profitable, you should then test it on the rest of the data, so-called ‘out of sample’ data. This reduces the likelihood of curve fitting. To test it further, you need to use data you have never seen before, ie/ no historical data. This is I believe is forward testing. It may include paper trading, or obtaining new data that was never used in your back test. If you developed a strategy using data from 1983 to 2000 and it was profitable, and then obtained data from 2000 to 2005/2006 then this new data may count as forward testing, but I’m not sure.

 

[mr_cassandra]

I think you're right on definition

My program is a variation on that theme, in that any code I apply to today is also applied to the entire 9.9 year data base. There is no difference in how it trades ten years ago vs in real life today. I have been doing real-time trades for a couple years now.

PKFFW,

This is what I think the answer is, hopefully I’m right.

When developing a trading system you will use a period of historical data, usually 6 to 12 months worth. I think it’s best to use a sample somewhere in the middle of your data. Once you have a system you think may be profitable, you should then test it on the rest of the data, so-called ‘out of sample’ data. This reduces the likelihood of curve fitting. To test it further, you need to use data you have never seen before, ie/ no historical data. This is I believe is forward testing. It may include paper trading, or obtaining new data that was never used in your back test. If you developed a strategy using data from 1983 to 2000 and it was profitable, and then obtained data from 2000 to 2005/2006 then this new data may count as forward testing, but I’m not sure.