Backtesting results v's forwardtesting results

If you have a mechanical trading system that you backtest & perhaps optimise in TS8.3, metatrader, ninjatrader, metastock etc. there are 3 possibilities when you trade the system live -

1. forward test results will be less profitable than backtest results.
2. forward test results will be more profitable than backtest results.
3. forward test results will be about equally as profitable as backtest results.

If you trade/have traded mechanically, how have your backtest & forward test results compared, and what factors do you think influenced the similarity/difference between your BT & FT results.

Thanks.

JTrader said:

If you have a mechanical trading system that you backtest & perhaps optimise in TS8.3, metatrader, ninjatrader, metastock etc. there are 3 possibilities when you trade the system live -

1. forward test results will be less profitable than backtest results.
2. forward test results will be more profitable than backtest results.
3. forward test results will be about equally as profitable as backtest results.

If you trade/have traded mechanically, how have your backtest & forward test results compared, and what factors do you think influenced the similarity/difference between your BT & FT results.

Thanks.

Click to expand...

In general, I've exceeded my* backtest results. Mainy because the backtest was based on selection on fundamental and relative strength criteria that gives out baskets of stocks - my real life entries and exits are then triggered by the use of limit orders, based on recent price action. This isn't factored into the back tests and gives me an advantage....mostly. I also think exceeding a backtest is easier on longer term systems, which I use.

*backtests are from aaii.com and digitalook.com. Not price only which I suspect you've got in mind. I have done some backtesting in excel on various indices and failed to match the results real time.

UTB

Someone made a good point on another trading website -

I also believe that better conviction can only be obtained by forward testing. however even if a system has been proved by forward testing, there is still the nagging suspicion of "how long will it continue working?". in a sense, forward testing is just like backtesting. by the time the forward testing is done, it becomes history. and if I trade a system based on this, I'm still effectively trading history.

Click to expand...

So at some point, you have to risk trading your system with real money, and are exposed to the same risks as if you had not done any demo forward testing, as the system could stop working after successful backtesting followed by successful demo forward testing, just as easily as if you had not done any demo forward testing. Therefore, why not trade with real money right from the start after initial backtests....

JTrader said:

Someone made a good point on another trading website -


So at some point, you have to risk trading your system with real money, and are exposed to the same risks as if you had not done any demo forward testing, as the system could stop working after successful backtesting followed by successful demo forward testing, just as easily as if you had not done any demo forward testing. Therefore, why not trade with real money right from the start after initial backtests....

Click to expand...

Why not indeed. Like getting married after the first date. A bit previous IMO.

Actually this reveals the level of ignorance on this issue.

Rigorous forward testing is a necessary part of forming a successful strategy for a myriad of reasons.

Failure to do so may be painful.

A problem i have with backtesting is that you hear people say you should test on different data samples.
Therefore lets say you test & optimise on the last 1 year of H4 data - lets say approximately 200 trades.
You then apply those optimum parameters to the previous year. The results for the previous year might make a loss.
You now have a system/parameters that made a healthy profit over the last 12 months, but made a loss over the previous 12 months, and whose to know how it will perform in the futures.

You could say - well, the markets evolve, and i don't particularly care what was happening 1-2 years ago, only the last 12 months is relevant IMO.
Or, the instability in the results between the last year & previous year may lead you to have doubts

rols said:

Why not indeed. Like getting married after the first date. A bit previous IMO.

Actually this reveals the level of ignorance on this issue.

Rigorous forward testing is a necessary part of forming a successful strategy for a myriad of reasons.

Failure to do so may be painful.

Click to expand...

I also believe that better conviction can only be obtained by forward testing. however even if a system has been proved by forward testing, there is still the nagging suspicion of "how long will it continue working?". in a sense, forward testing is just like backtesting. by the time the forward testing is done, it becomes history. and if I trade a system based on this, I'm still effectively trading history.

Click to expand...

But you've completely ignored the points made that i quoted, and not explained the value of forward testing.

Sure, foward testing is important, to make sure you have no bugs in a coded strategy for example, but in terms of likely profits once traded live, what is the value of forward testing? as forward testing once passed, becomes history also, and you are still left with needing to dip your toes in the water for real after forward testing, at which point the system could still fall flat on its face AFTER successful BT's & FT's.

correct backtesting is a very valid question, because many platforms do not yield reliable results with regards to realistic fills.

Tradestation is certainly the worst for giving misleading backtesting!

Arbitrageur said:

correct backtesting is a very valid question, because many platforms do not yield reliable results with regards to realistic fills.

Tradestation is certainly the worst for giving misleading backtesting!

Click to expand...

Its all a catch 22 situation really -

We want to know that the system parameters have led to good results in the past, otherwise on what basis would we decide to use them in the future?

But is backtesting meaningless, when the strategy parameters can stop working before or after forward testing?

Is past performance any indication of future performance?

It does all seem rather fickle and flimsy, and its easy to see how minor things can lead to big variations in performance between backtest and live trading results -
eg. Your SL is hit, price goes no further against you then heads 500 pips into profit etc. etc.

However, there is also hope that live forward testing results will be better than the optimised backtest results....

Assume 70% of backtest and then hope to be pleasantly suprised.If you cannot live with 70% then do not trade it. Make sure slip and costs in backtest are realistic and make sure you update them with real results.

The most likely reason you will fall down in reality is deviation from system after a string of draws that convince you it's all gone horribly wrong. Then after you pull the plug the best trade of the year comes along.

Arbitrageur said:

correct backtesting is a very valid question, because many platforms do not yield reliable results with regards to realistic fills.

Tradestation is certainly the worst for giving misleading backtesting!

Click to expand...

I would be surprised if TS8.3 is worse than MT4 for backtest accuracy, as IMO, MT4 is pretty amateurish in comparison - though it is free and can be used with a multitude of brokers.

With regard to realistic fills, this is an issue, especially when you look at shorter timeframes that trade more often. However, being filled at the correct price or not is a two way street in that - yes you may be filled at a less favourable price, but you could also be filled at a more favourable price just as easily, depending on whether price goes for or against your proposed trade direction, after the signal has been generated, but b4 the auto-execution. Does this make sense?

Cheers.

backtesting allows you to develop an idea, but if your idea is complex it is very easy to "curve fit" or over optimise the system so that it works too specifically on the "in sample" data only, and would not produce anything like those results in real life. testing it on "out of sample" data gives you a chance to see how it would have performed on unseen data and give you an indication of whether it is too optimized or not - if it works well on OOS data, then there is a chance it might work well in realtime.

the only test of whether something works or not as a realtime strategy is to test it walking forward.

there are some very specific problems with TradeStation when using historical data that give serious distortions to backtesting performance. I'll elaborate on the problem in another post maybe tomorrow if I have time, with some examples.

Suffice to say, the only way to test performance with TS reliably is to walk it forward. You'll get an idea about the problems with TS when you walk your strategy forward and watch trades plot in realtime, save the performance data, and then refresh the chart - when TS plots the trades from historical data rather than realtime plotted trades, compare the difference!

If its of any interest - I developed a mechanical system for trading digital options on indices and more recently forex. In order to match the money management system in to the trading signals it was necessary to backtest back to the early 1980's on the "senior" indices and since inception for other european indices, and to 2000 on forex which is the earliest period I can find data for.

I have used the system with real money since mid 2003 since which time the IRR has been about 55% - this is not as good as the back testing suggested should be achieved, which I put down to "new" circumstances which had never been seen previously in the market during the back testing period.

However the core of the system which basically measures the probability of an event happening does seem to be pretty robust. The more frequent the signal historically the more reliable it is going forward.

Hi BW - i don't see why a strategy today has to test favourably on data from years back in order to be labelled possibly good. Time moves on, so do the markets and the market conditions. Therefore why does my 2007 strategy need to perform well on 2000/2003/2005 data in order to be labelled good?

Hi arb - re: curve-fiitting - when i tested a few strategies in TS I found that a tell tale sign of curve fitting was fewer & fewer historical trades with each strategy parameter optimised.
In my MT4 backtesting, with my present system, this does not happen, i still get a good/similar number of trades generated after optimisations - which is a good sign.

In TS I also saw how multi-component systems (eg. SL +trailing SL + profit target) were easy to curve fit.

Perhaps a better way to look at backtesting might not be to say look at x months worth of data (which will have different implications depending on timeframe being tested anyway) but to say backtest over the last x number of trades :idea: whether this may be 100, 200, 500 trades etc.

JTrader said:

Hi BW - i don't see why a strategy today has to test favourably on data from years back in order to be labelled possibly good. Time moves on, so do the markets and the market conditions. Therefore why does my 2007 strategy need to perform well on 2000/2003/2005 data in order to be labelled good?

Click to expand...

JT hi there - its just an observation that the shorter the time frame, the easier it is to get any mechanical system to appear to work. For example using simple moving average crossovers to give buy/sell signals works well in certain markets but is disastrous over the long term.

I also took the view that the better a system worked over a long time frame (and therefore in many different market scenarios) the more likely it was perform well going forward. That theory may of course be flawed depending on what style of trading you're are doing - I only analyse daily HLC - to help determine market direction over the next fortnight or so.

I suppose to answer your last point above - if you can be certain that you are not about to encounter the same market conditions from earlier years then there is no reason to concern yourself about 2007 strategy fitting earlier periods. I take the view that as I have no idea what the markets are about to do all I can rely on is price action from prior periods to help determine the most probable outcome.

Does that help or am I talking garbage!?

Baldwreck said:

Does that help or am I talking garbage!?

Click to expand...

Hi BW

i think it all demonstrates backtesting is a subjectve, limited, and complex area, with many methodology possibilities available, and probably no concrete right or wrong ways of doing it.

JTrader said:

Hi BW

i think it all demonstrates backtesting is a subjectve, limited, and complex area, with many methodology possibilities available, and probably no concrete right or wrong ways of doing it.

Click to expand...

Well put like that its a wonder that anyone back tests at all! However if you have a theory to put to the test ,done properly, back-testing will at least let you know if there is any substance to your theory - and I guess it's also dependent on what you feel comfortable with. Any testing of financial markets is merely assessing human behaviour in relation to economic events. Mostly, that behaviour is rational, even though unpredictable.

Data Simulation...

...if you'll test your back test results for a uncertaim future period you have basically 3 methods:

1. you divide your historical data in two parts and use one for the back test and the other for a following "out-of-sample test (that is the classical approach, described in many books...)

2. you test your system setup with other historical data of different markets (dax, ftse, oil, etc. ...). If you're successful - congratulation: perhaps you have found the holy grail!

3. the third possibility is to use random data. Tushar Chande calls this in his book "Beyond technical analysis" data scrambling. I've extended this approach using the original historical data with its properties as a base and generate new synthetic test data using additional random components and customized future market expectations, e.g. optional change of volatilty etc. ... This data simulation proccess step is described in my own book "Monte Carlo Simulation and System Trading" and on my website.

It's clear that there are infinite test possibilities and you can't simulate all thinkable possibilities, but so you get a better feeling for the robustness of your system setup concerning different market scenarios.

bye,
zentrader

I now have ZERO confidence is the accuracy of MT4 backtest/optimisations - for several reasons, they are a complete sham!

They can generate completely different results and optimum parameters for the same set of data, based on -

what the starting parameters were b4 optimisation/backtest
in what order you optimise the parameters

You can also just optimise one parameter such as an indicator with 2 input values. It selects the optimum parameters. If you then set those parameters, then immediately re-optimise those parameters, MT4 will come up with a different set of optimum input parameters - even though the last input parameters were meant to be optimum & you have not done anything since!

Its a bit of a messy & unfunny joke IMO , and seems very much a random process/sequence of anomolies in MT4.

Baldwreck said:

If its of any interest - I developed a mechanical system for trading digital options on indices and more recently forex. In order to match the money management system in to the trading signals it was necessary to backtest back to the early 1980's on the "senior" indices and since inception for other european indices, and to 2000 on forex which is the earliest period I can find data for.

I have used the system with real money since mid 2003 since which time the IRR has been about 55% - this is not as good as the back testing suggested should be achieved, which I put down to "new" circumstances which had never been seen previously in the market during the back testing period.

However the core of the system which basically measures the probability of an event happening does seem to be pretty robust. The more frequent the signal historically the more reliable it is going forward.

Click to expand...

Hey, BW. I've been looking everywhere for this system of yours. could you point me to the link. or describe it a little bit.

Thanks.

overfitting's always in the shadow... is just what my signature says