Backtesting

[Rognvald]

As I have noticed an interest in this topic recently I just want to post this link to manual backtesting on ET for the attention of those considering as there seems to br some useful info there


http://www.elitetrader.com/vb/showthread.php?threadid=30988

 

[timsk]

Manual Backtesting
Automated backtesting is a distant far off dream for me. In the meantime, I'll have to make do with the Shanks' pony method. I trade U.S. shares and I'm uncertain as to how many different stocks and over how many months I need to test in order for the results to be statistically meaningful. Any suggestions greatly appreciated.
Tim.

 

[GeneticTrader]

In Design, Testing and Optimisation of Trading Systems by Pardo he suggests using standard error as a measure of precision in the value of a random variable (meaning the value of some function of your trading system like mean profit per trade).

Pardo makes some assumptions in the use of standard error to do with the distribution (spread of values) of the random variable - but let's not go into that for the sake of brevity.

Pardo uses 1/sqrt where n is the number of trades. The standard error gets smaller the larger the value of n, when n = 10 the standard error is 0.316, when n = 20 it is 0.223, and when n = 100 it is 0.1. In other words, the more trades you have the more certain you can be of the result of some function like mean profit per trade. If you multiply the standard error by 100 you get a measure of the percentage of precision and you should use this when stating results and as a guide as to how many trades to backtest.

Let's take an example, you manually backtest 10 trades and you calculate the mean profit to be 30 per trade, you would state mean profit per trade is 30 +/- 31.6% - (the 31.6% is the standard error for n=10 multiplied by 100).

As statisticians like to be conservative you would use 30 - 31.6% and not 30 + 31.6% when calculating how many trades you need to make before you can buy that new Porsche ;-)

In short, for manual backtesting anything less than 10 trades would generally be considered meaningless.

But (you just knew there was one!) having said that, unfortunately there are a whole bunch of other considerations in backtesting and it's hard to give any golden rule without also considering the wider backtesting methodology .

Andy.

 

[mr_cassandra]

re backtesting

I once saw a suggestion that you should also eliminate the 2-3 biggest profits and see how your system looks overall. While this sounds good on the face of it, I question the meaningfulness or appropriateness. IE if your system applies the same rule set year in and year out, and happenes to make big scores shorting in a bear mkt or vice-versa in a bull move, why is it that one would want to take out those good trades?

In Design, Testing and Optimisation of Trading Systems by Pardo he suggests using standard error as a measure of precision in the value of a random variable (meaning the value of some function of your trading system like mean profit per trade).

Pardo makes some assumptions in the use of standard error to do with the distribution (spread of values) of the random variable - but let's not go into that for the sake of brevity.

Pardo uses 1/sqrt where n is the number of trades. The standard error gets smaller the larger the value of n, when n = 10 the standard error is 0.316, when n = 20 it is 0.223, and when n = 100 it is 0.1. In other words, the more trades you have the more certain you can be of the result of some function like mean profit per trade. If you multiply the standard error by 100 you get a measure of the percentage of precision and you should use this when stating results and as a guide as to how many trades to backtest.

Let's take an example, you manually backtest 10 trades and you calculate the mean profit to be 30 per trade, you would state mean profit per trade is 30 +/- 31.6% - (the 31.6% is the standard error for n=10 multiplied by 100).

As statisticians like to be conservative you would use 30 - 31.6% and not 30 + 31.6% when calculating how many trades you need to make before you can buy that new Porsche ;-)

In short, for manual backtesting anything less than 10 trades would generally be considered meaningless.

But (you just knew there was one!) having said that, unfortunately there are a whole bunch of other considerations in backtesting and it's hard to give any golden rule without also considering the wider backtesting methodology .

Andy.

 

[ardhill]

Manual Backtesting
I trade U.S. shares and I'm uncertain as to how many different stocks and over how many months I need to test in order for the results to be statistically meaningful.

Hi Tim

I am maybe not exactly qualified to answer this, so I will just 'throw it out there'.

I have recently signed up with TradeStation, and therefore gotten the ability to back test. Before getting the platform, I had thought about backtesting and read some views and opinions on the subject here on these boards, but came to an different opinion than many others.

I feel that this may be a little controversial, and expect some negative comments but...

You don't have to look too far back in the markets to see that they change. Even the 'personality' of particular stocks change regularly.

So, I thought that most 'systems' should change in line with the markets. It is only a few who find a system that works consistently in all markets over many years. The rest of us have to adapt as our world changes.

Many people apparently want to get a system that works for everything and proves itself over the last 10 years. Though I have been backtesting ideas over a short term and a small number of stocks.

I did this as I wanted a system that would work NOW, in THIS market climate, not necessarily what worked 3 years ago. So, I have been using a two month test period. (I should mention I trade intra day.)

If I get a system that works consistently over the two months, then I will know when it starts to break down and I have to re-think as the 'matrix' has shifted.

I also noticed that some methods work better on different stocks. I have found that one system may give wonderful results with say eBay, but tragic results with Intel for instance.

It has made me think differently about my trading.

Since I trade for two reasons, the joy of the game and for profit, then I can afford to find a system that works well on a particular stock now. I can trade it (or them) until the market changes or I find something better. I enjoy trading so I am always looking out for a better system that suits my trading style.

Of course, what works for me may not work for you. One of the aspects of trading that appeals to me is that there can be many different profitable ways to trade, just find one that suits you and your personality.

 

[mr_cassandra]

Not a negative comment

I think you have a good perspective. Too many people in the ta and systems world get overly locked into their own perspective. Your idea would merely mean that you would have to add another decision factor as to what kind of market you feel you are in, and then act accordingly.

Many people do look for 10 or even 20 years back-test. My own opinion is that if it works in bull, bear and flat mkts, you've covered it all, whether that took 5 years or 15.

In the case of my own system, I tried for an 'all-weather' system which uses the same decision structure in any market. In the 4 years I've been programming it, I've run into all manner of online 'guru's with pre-packaged commentary as to why it couyldn't be done or LOL could be done better. Rarely thought, did any of them offer their own code or precise rule structure with their suggestions.

My work can be found by googling mvp signal system.

Hi Tim

I am maybe not exactly qualified to answer this, so I will just 'throw it out there'.

I have recently signed up with TradeStation, and therefore gotten the ability to back test. Before getting the platform, I had thought about backtesting and read some views and opinions on the subject here on these boards, but came to an different opinion than many others.

I feel that this may be a little controversial, and expect some negative comments but...

You don't have to look too far back in the markets to see that they change. Even the 'personality' of particular stocks change regularly.

So, I thought that most 'systems' should change in line with the markets. It is only a few who find a system that works consistently in all markets over many years. The rest of us have to adapt as our world changes.

Many people apparently want to get a system that works for everything and proves itself over the last 10 years. Though I have been backtesting ideas over a short term and a small number of stocks.

I did this as I wanted a system that would work NOW, in THIS market climate, not necessarily what worked 3 years ago. So, I have been using a two month test period. (I should mention I trade intra day.)

If I get a system that works consistently over the two months, then I will know when it starts to break down and I have to re-think as the 'matrix' has shifted.

I also noticed that some methods work better on different stocks. I have found that one system may give wonderful results with say eBay, but tragic results with Intel for instance.

It has made me think differently about my trading.

Since I trade for two reasons, the joy of the game and for profit, then I can afford to find a system that works well on a particular stock now. I can trade it (or them) until the market changes or I find something better. I enjoy trading so I am always looking out for a better system that suits my trading style.

Of course, what works for me may not work for you. One of the aspects of trading that appeals to me is that there can be many different profitable ways to trade, just find one that suits you and your personality.

 

[timsk]

Thanks!

Thanks for the feedback folks - very helpful.
Andy, welcome to T2W - an excellent 1st post.
Ardhill, no negative comments from me; your approach seems eminently sensible!
Tim.

 

[GeneticTrader]

Target annual returns for share trading?

Tim - thanks for the welcome. I do have a confesion to make, and that is I have never traded a single share in my life!

I'm a member of the GA (genetic algorithm) system developement/backtesting school of thought, and the other school of thought which says i'm not investing my money unless I'm almost certain of a return ! So I'm not trading a single share until I'm happy with my backtesting.

On the subject of returns I'm wondering what you consider to be a good annual rate of return on capital required (trade unit cost + draw down + buffer) for share trading. It seems to me that the market is generally efficient and large double digit returns are difficult for unlevereged share trading in today's markets, am I right?

Andy.

 

[timsk]

Hi Andy,
I'm afraid I'm not in a position to answer your Q. for various reasons - not least because I'm not a consistently profitable trader myself! Also, the answer would be determined to a large extent by the markets you trade; the timescale that you trade them (i.e. day trade, swing trade, position trade etc.) and your risk tolerance. "It seems to me that the market is generally efficient and large double digit returns are difficult for unlevereged share trading in today's markets . . ." Most people on these boards trade via a spreadbet broker, CFD's or some other derivative product. Even those people with direct access accounts trade on margin. In other words, unleveraged share trading is, I suspect, practised by a relative minority of T2W members. However, I agree totally that double digit returns are difficult, especially if you only hold long positions. Anyway, have a look at the thread entitled 'What is your Annual Profit Goal?', it may provide a few pointers. Here's the link, I hope:
http://www.trade2win.com/boards/showthread.php?t=12172
Tim.

 

[mr_cassandra]

Returns on investment

I don’t think there is any method or system that will absolutely guarantee you have a profit every time., even almost certain. If that’s the level of certainty you want, you are probably limited to a bank account or safe deposit box.

That being said, its my opinion that it would be more realistic to seek a system or method, which has more winners than losers, in both count and percents; as well ass acceptable drawdowns.

I also think that many systems can give you annual returns in the double digits, for example the 5 day rsi system from www.vtoreport.com (free).

My own system (not for sale) but publicly posted at clearstation, has far more winners than losers, in count as well as percent. In addition, in both back-testing as well as real-time, makes gains in the double digits every year. MY system can be found by googling mvp signal system.

Regards Steve

Tim - thanks for the welcome. I do have a confesion to make, and that is I have never traded a single share in my life!

I'm a member of the GA (genetic algorithm) system developement/backtesting school of thought, and the other school of thought which says i'm not investing my money unless I'm almost certain of a return ! So I'm not trading a single share until I'm happy with my backtesting.

On the subject of returns I'm wondering what you consider to be a good annual rate of return on capital required (trade unit cost + draw down + buffer) for share trading. It seems to me that the market is generally efficient and large double digit returns are difficult for unlevereged share trading in today's markets, am I right?

Andy.