Article Why Back Testing is Better than Forward Testing

T2W Bot

Staff member
1,441 50
Firstly back-testing should be done as accurately as possible. This means that all the best practices of back-testing should be implemented. Some examples are:

Including reasonable spreads
Avoiding look forward bias
Including any possible forms of commissions if applicable
Using historical data which is as accurate and complete as possible
Ensuring trade logic tested is reasonable and practicable in actual trading
Reducing slippage (although this is normally not a problem with Forex due to its immense liquidity)

With accurate back-testing, characteristics of the trading strategy can be observed and analysed reliably. What this means is the risk and returns profile of the strategy is statistically reliable, and any performance is deemed to be attributed to the strategy and not due to pure chance.
Forward testing is not the most efficient manner to determine risk and returns characteristics of a strategy. A forward test is one price series in time. Backtesting is also one price...
Continue reading...
 
Last edited by a moderator:

NVP

Legendary member
36,535 1,842
Backtesting

interesting and well written piece - but for me sums up everything that frightens me about business modelling (in any arena) and how one can generate wonderful results if you keep playing with data for long enough............thats not how the real world works ...sadly !

still a good piece of work

N
 
  • Like
Reactions: Forexmospherian

Sigma-D

Established member
648 61
Back testing is easier than forward testing, but I'm not sure it's better.

When I ran back-testing last year on my system it was really good. I then traded it on a demo account, off and on for about 4 months. The results were still OK but nowhere near the same returns I got from my theoretical back-test results.

I've been trading live since late January this year using exactly the same system and the results are dire.

Back-testing assumes your trading is totally automated and get in and out at exactly the levels and points calculated by your system and that you sit through the pullbacks without moving your stop to breakeven and that you don't get shaken out by massive single bar moves against you that turn out after they close to be nothing of the sort. It also assumes you're trading 24 hours a day and taking every trade. It is so far from reality I really don't see how back-testing helps at all. It does the opposite if you think about it as it confers a credibility which cannot be achieved in practise.
 
Last edited:

timsk

Legendary member
6,938 1,812
Hi Sigma-D,
Your points are well made and I would agree with you completely - but for the two trump cards that back testing has up its sleeve . . .

Firstly, if your methodology doesn't come up with the goods when back tested, then it's got zero chance of magically making money when forward tested on demo or, heaven forbid, traded for real on a live account. This point could be discounted completely if it was as quick to forward test as it is to back test. Unfortunately, it's not. This leads to . . .

The second trump card in favour of back testing: speed. If your back test is automated - then it's lightning fast. Even if it's performed manually, it's going to be infinitely faster than forward testing in real time. If one starts with the assumption that the methodology being tested isn't profitable and will need revising or scrapping, what would a trader rather do: spend X amount of time arriving at that conclusion via a back test - or spend two, three, five or perhaps ten times as long via a forward test in real time? In the long run, back testing will save time.

For these two reasons, I recommend traders always back test their methodologies. I agree completely that the forward test is unlikely to produce results as good as the back test and, when traded live, the results are likely to show yet a further drop off in performance.
Tim.
 
Last edited:
  • Like
Reactions: Forexmospherian

patong

Active member
213 12
Back testing is never as good as forward testing. If you back test the mathmatics are already calculated. However with forward testing with the correct program the math resembles more of a live market condition, where as back testing does not. Its that simple.
 

I'vereadthemall

Member
74 9
Howard Bandy can teach us a thing or two about good back-testing practice which also incorporates forward-testing too.

Using Amibroker (although I am sure all other good backtesting software does the same) we can easily incorporate a walk-forward optimization as part of our back-tests.

We can set the parameter for optimization, set the in-sample periods and out-of-sample periods and then tabulate the out-of-sample test runs to give us our expected returns when taking the strategy live.

My brother is in the process of testing candlestick patterns so I am lucky enough to have a pre-written strategy that can be used to show an example;

I have started the first In-sample period as 01/01/95 - 01/01/96.

The first out-of-sample period would then be 01/01/96 - 01/01/97.

The next In-sample period would be 01/01/96 - 01/01/97.

The next out-of-sample period would be 01/01/97 - 01/01/98.

And so on.

The parameter that is optimized in-sample is the lookback period for a Rate of Change filter. I.E, we only take signals if the ROC > 0, but we optimize during the in-sample period how long a lookback we use in the ROC calculation during the following out-of-sample period.

The objective function (the metric that we want our optimization to maximise) will be the CAR/MDD.

In summary....Whatever ROC lookback period > 0 produces the best CAR/MDD from the most recent in-sample run, we use in the next out-of-sample run.

At the end of all test runs, we can tabulate the entire out-of-sample results and in doing so we have achieved a simulated walk-forward test on nearly 20 years of data, in less than a couple of hours.

Depending on what the most recent in-sample optimization shows to be the best loockback for ROC, we use those settings to go live.

After a year of live trading, we optimise again and trade the new settings live, and so on until our live results differ greatly from previous out-of-sample runs.

Obviously the above assumes that the out-of-sample results of the walk-forward optimization are profitable. Otherwise, we wouldn't trade the strategy.

It also assumes that you haven't repeated the process multiple times in a bid to find a good system - because each time you do so, the out-of-sample results become more and more 'in-sample'.

If you are interested, the signal that I tested was an inverted hammer if it made a 3 day low and had a green real body, the entry was a break of the inverted hammer high on the following day.

All positions were held for 10 days. Max open positions was 10 and the amount of equity risked per trade was 10%.

Tests were run on a database that included all currently listed stocks and ETFs on the NASDAQ, NYSE or AMEX.

The variable that was optimized in-sample was the ROC(C,X) where X = the number of days lookback for the ROC indicator.

In Amibroker the entire strategy was written as;

////////////////////////////////////////////////
SetOption ("MaxOpenPositions" , 10 );
SetPositionSize (10, spsPercentOfEquity);
SetTradeDelays(0,1,0,0);

LongUpperShadow = H - Max(O,C) > (H - L)*0.67;
rng = abs((C-O)/O);
SmallRealBody = rng < 0.003 AND rng > 0;
DownTrend2 = L < Ref(LLV(L,3),-1);
rngy = H-L;
lowerShadow = Min(O,C) - L;
shaven = lowerShadow < rngy*0.1;
RealBodyGapDown = Max(O,C) < Min(Ref(O,-1),Ref(C,-1));

InvertedHammer = smallRealBody AND shaven AND realBodyGapDown
AND longuppershadow AND downtrend2 AND O < C;

Buysetup = invertedhammer
AND Close > 5
AND MA (C,20) > 1
AND MA(Volume,20) > 100000
AND ROC(C,Optimize("rocLB",10,10,250,10)) > 0;
Sell = 0;
Buystop = Ref(H,-1) + 0.01;

Buy = Ref(Buysetup,-1) AND Cross(H,Buystop);
BuyPrice = Max(Buystop,Open);

ApplyStop (stopTypeNBar,stopModeBars,9,False,3);

PositionScore = 100 + Volume;
/////////////////////////////////////////////////

While there are still inherent dangers with the above approach, (mainly that we might do multiple walk-forward optimizations until we find a profitable strategy out-of-sample - and in doing so muddy the out-of-sample data), I still feel that this is the best approach to back-testing that your average retail trader can implement.

The attached images show the system performance stats and equity curve...Of note is the 59% win-rate and less than 10% average market exposure.

If you are interested...the current best ROC lookback is 160.
 

Attachments

foroom lluzers

Veteren member
3,612 135
This is the most useless article and discussion.I won't waste time even reading 1 sentence.

Price action doesn’t replicate itself over time

First of all, market prices are chaotic, and. The patterns observed on any day’s trading do not replicate themselves at other times. As a result, the attempt at testing today’s market action on the basis of yesterday’s patterns will never lead to results that can be duplicated over the long term.

A trader lost his entire account , after market volatility increased 200 % and back tested patterns changed.


http://www.trade2win.com/boards/educational-resources/223150-most-back-testing-useless.html
 
  • Like
Reactions: piphoe