Why do people not backtest their systems?

[jcl365]

There are many systems described here but it seems almost all are not backtested. I see nowhere some tests or performance statistics. I wonder why people trust their money to untested systems? How can someone know that his system is profitable without having backtested it over at least 4 years? A backtest is no guarantee for a profitable system of course, but not backtesting seems to be a guarantee for an unprofitable system.

 

[Trader333]

It is not possible to back test every approach to trading. For example, how would you back test a "basket trading" or a DOM strategy ?

 

[jcl365]

Hmm, scalping or DOM strategies would be indeed hard to test. But basket trading, or the systems posted here that work on time frames higher than 1 minute and don't use any special intuition for entering or exiting trades, should be relatively easy to backtest IMHO.

 

[startuprob]

As you've said, backtesting is inherently difficult and it's important to know what backtesting is and isn't good for. There is a new startup working on helping people backtest at Backtesting Trading Software for Quant Finance Strategies.

 

[BeginnerJoe]

Why bother back testing when you can forward test on demo ? But both approaches are like testing punches against a dummy that doesn't punch back. Why people believe the market is a dummy that likes to give you money as if it's an ATM I will never know. But let me give you a secret: the market is alive, and not only does it punches, it also kicks. Its kicks are always 100% accurate and straight in the b*lls. Now, you still want to come use the ATM do you ?

 

[startuprob]

Joe, as you've said, neither approach is perfect. Backtesting is faulty because past data isn't necessarily indicative of future activity. Demo trading is faulty because it's a one-way street of information instead of actual trading which is a two-way street. All of that being said, no model is perfect but some models are useful.

 

[Splitlink]

It is not possible to back test every approach to trading. For example, how would you back test a "basket trading" or a DOM strategy ?

Some people love backtesting. Basket trading and DOM strategy? You have given two great ideas to work on over the weekend! Not me, I might add.

 

[BeginnerJoe]

Joe, as you've said, neither approach is perfect. Backtesting is faulty because past data isn't necessarily indicative of future activity. Demo trading is faulty because it's a one-way street of information instead of actual trading which is a two-way street. All of that being said, no model is perfect but some models are useful.

To believe a one way street system is dangerous in a 2 way street world. A head on collisions is guaranteed pretty much immediately. How many can survive a head on collision, and why would anyone knowingly put themselves in that position ?

 

[startuprob]

To believe a one way street system is dangerous in a 2 way street world. A head on collisions is guaranteed pretty much immediately. How many can survive a head on collision, and why would anyone knowingly put themselves in that position ?

Nobody can predict the future, but I'm a big believer in using whatever resources one can to give them an edge - that's why I backtest. The problem with most backtesting platforms is that they're insanely difficult to use - that's why I use QuantBlocks.com

 

[Shakone]

I guess a lot of people don't like backtesting because they don't want to face the truth that their strategy has no edge. Then there are others that can't backtest because of the strategy, and there are others who don't believe it has much value.

I think pretty much everyone does some level of backtesting, even if it is just the flawed scrolling through a chart and seeing type.

If you're going to backtest, why 4 years? Are things now similar to how they have been for the past 4 years? Does 4 years encapsulate all the market conditions?

 

[startuprob]

I guess a lot of people don't like backtesting because they don't want to face the truth that their strategy has no edge. Then there are others that can't backtest because of the strategy, and there are others who don't believe it has much value.

I think pretty much everyone does some level of backtesting, even if it is just the flawed scrolling through a chart and seeing type.

If you're going to backtest, why 4 years? Are things now similar to how they have been for the past 4 years? Does 4 years encapsulate all the market conditions?

I think the period of time is relative on a number of circumstances:

Are you using tick data, or just end of day?
What's the strategy? (is it using a small window or long?)
What's the Max drawdown? (ie Win:Loss Ratio)
etc.

 

[BeginnerJoe]

Nobody can predict the future

I can. I predict in the future, the market is there to make money from you and people like you. In the future, the market will not become an ATM for people to enter PINs on. See ? The future is very predictable.

that's why I use QuantBlocks.com

I reckon the real reason is that there's more money to be made from vedoring than trading.

 

[NVP]

There are many systems described here but it seems almost all are not backtested. I see nowhere some tests or performance statistics. I wonder why people trust their money to untested systems? How can someone know that his system is profitable without having backtested it over at least 4 years? A backtest is no guarantee for a profitable system of course, but not backtesting seems to be a guarantee for an unprofitable system.

The world of trading is full of thousands of ideas and approaches JCL.....so do whatever gets you though the night .and who cares what others think

there is no ultimate right or wrong .....except the market itself

N

 

[startuprob]

The world of trading is full of thousands of ideas and approaches JCL.....so do whatever gets you though the night .and who cares what others think

there is no ultimate right or wrong .....except the market itself

N

Great point - markets don't lie.

 

[BeginnerJoe]

markets don't lie.

Presumably this will be demonstrated to be true by QuantBlocks.com ?

Will I get a commission for the help in vendoring ? I am unable to make my 0.0001 cent today from the market given that the EUR is twisting and turning like someone on PMT. So any alternative source of pennies will be welcomed.

 

[Crassius]

As you've said, backtesting is inherently difficult and it's important to know what backtesting is and isn't good for. There is a new startup working on helping people backtest at Backtesting Trading Software for Quant Finance Strategies.

.

 

[R.Charnock]

There are many systems described here but it seems almost all are not backtested. I see nowhere some tests or performance statistics. I wonder why people trust their money to untested systems? How can someone know that his system is profitable without having backtested it over at least 4 years? A backtest is no guarantee for a profitable system of course, but not backtesting seems to be a guarantee for an unprofitable system.

I think people do backtest, but whether the results mean anything is another matter. If you have plenty of time to spare, try comparing a 'system' with the same thing in reverse (entries, stops, etc, in the opposite direction), then repeat the tests over different time periods. Some will probably give excellent results and some will be disasters. Then pick the best one, trade it live, and prepare for a completely different result.

 

[jcl365]

I think people do backtest, but whether the results mean anything is another matter. If you have plenty of time to spare, try comparing a 'system' with the same thing in reverse (entries, stops, etc, in the opposite direction), then repeat the tests over different time periods. Some will probably give excellent results and some will be disasters. Then pick the best one, trade it live, and prepare for a completely different result.

Testing a system in reverse won't require any time, but normally an asset can not be reversed. For instance the price curves of stocks and stock indices have a sort of sawtooth pattern due to the preference of long trades. So the result of reversed prices are meaningless here. However some assets are indeed reversal invariant, for instance major currencies, and a working system should give similar results then.

 

[JimTru]

1) The human mind is the most wonderful machine on the planet. No computer can match its creative ability to make thousands of observations every minute, learn many lessons from each trade taken and not taken, each stop placed and moved, each tick across many markets, each reaction to unique news events. Most consistently profitable traders that are teachers cannot hope to teach their students all that their own minds are doing through observing and trading, only a hundred or so rules and guildelines, and that's not near enough to guide the student thru to consistent profitability on its own. It would be counterproductive to try to quantify and backtest such an approach.
2) Since the best systems are complex, they are expensive and time consuming to code into a backtest. Most reach a point of diminished returns and the coders/developers give up. Good luck capturing ALL the variables and observations.
3) People who have a working consistently profitable system are very reluctant to give it to anyone to backtest for obvious reasons.
4) Hope is all most students have to cling to, and backtesting represents a threat to that hope. If their precious ideas do poorly, what ever will they do then?
5) Even with all the negative press about subjectivity and gut instinct, most traders still follow their emotions at some point over their rules. Backtesting cannot capture that, and everyone knows it. Backtesting represents the letting go of control to the whims of the Market - something most students never really come to grips with.

 

[Raleigh Lee]

There are some compelling arguments out there for not backtesting, featuring that old adage of "past results are not indicative . . ." At the very least, it is probably best to backtest without any optimization to avoid curve-fitting. However, I just can't bring myself to trade a system without some degree of backtesting. Nevertheless, some of the arguments against it are very persuasive.