Swing & Position Trading Day Trading & Scalping Why Back Testing is Better than Forward Testing

Firstly back-testing should be done as accurately as possible. This means that all the best practices of back-testing should be implemented. Some examples are:

• Including reasonable spreads
  
• Avoiding look forward bias
  
• Including any possible forms of commissions if applicable
  
• Using historical data which is as accurate and complete as possible
  
• Ensuring trade logic tested is reasonable and practicable in actual trading
  
• Reducing slippage (although this is normally not a problem with Forex due to its immense liquidity)
  

With accurate back-testing, characteristics of the trading strategy can be observed and analysed reliably. What this means is the risk and returns profile of the strategy is statistically reliable, and any performance is deemed to be attributed to the strategy and not due to pure chance.

Forward testing is not the most efficient manner to determine risk and returns characteristics of a strategy. A forward test is one price series in time. Backtesting is also one price series in time. There is no significant difference – let’s say you forward test for 3 months to make conclusions, won’t it become a 3 month back test at the point of conclusion?

In addition, backtesting can be done over multiple random price series for a test of robustness. This might be done viaMonte Carlosimulation or Boot strapping.

Next, a test of 5 years to cover a full market cycle is required. This cannot be achieved by forward testing. If you are running a strategy on forward testing and it generates 10 trades over a period of 2 months, that’s hardly statistically significant to make a reliable conclusion about the risk and returns characteristics of the strategy. These 10 trades could be due to pure chance. Many more trades are required to be statistically significant and this might be years of forward testing. Back testing is much more efficient.

Having said that, the purpose of forward testing is to test for operational issues such as whether your IT infrastructure is sufficient and stable for live trading, whether the broker is able to fill your order properly, and whether a forex trading software has any bugs etc. You will also get a feel of the robot’s behaviour and also gain confidence that the robot operates as it should.

Back Testing Tip 1: Time Frame
Choice of a suitable time frame must be aligned to the type of strategy to be traded.

Using a shorter time frame (e.g. 5min, 15min) introduces more noise into the price series making it difficult to find the trend. It is more suitable for scalping, short term arbitrage, mean reversion type of strategies in general. Shorter time frame responds quickly to price changes, results in shorter holding periods, and likely more trades are generated. Statistically the more trades generated, the closer actual mean returns match “population mean returns” or “theoretical mean returns”.

In simpler coin toss analogy, even when toss of the coin is statistically 50-50, there can be 10 heads in a row for 10 tosses of the coin. However when we toss 1,000 times, the proportions of heads should lean towards 50%. This same concept applies to trading. We need more observations of trades to determine the actual risk and returns of the system. Shorter time frames are more sensitive to execution quality where slippage matters more.

Using a longer time frame (e.g. weekly) puts more emphasis on the trend and less emphasis on other components such as mean reversion. Longer time frames also react slower to price changes. Longer time frame is not as sensitive to execution quality and slippage rarely matters.

Back Testing Tip 2: Testing
Good trading strategies must be based on sound logic or economics. This should be the base case for trade method ideas generation. Backtesting is then the only method to validate trade ideas generated. Backtesting is often misunderstood to be a tool to “discover” trading strategies by scanning a massive combination of techniques or indicators. The base case for backtesting is actually to validate ideas to seek empirical evidences about a hypothesis (in this case to determine if a strategy is statistically deemed to be profitable beyond mere chance).

That said, data mining techniques to discover strategies are available. However, we need to be aware to discount data mined strategies’ results to account for a positive bias in data mined results. Backtesting is also to test how robust a strategy is. The strategy should work over different situations such as different price regimes, in a fairly similar or predictable fashion. Backtesting also gives us an indication of expectations, both in terms of risks and returns.

Back Testing Tip 3: Performance Monitoring
After a system is trading live, it is entering a new phase of life. You should at regular intervals monitor trading results and compare with expectations (both risk and returns) of your backtest, to determine if the system is working properly.

However you should definitely not micro manage your system. A good rule of thumb is to monitor the system on a monthly basis and not on a trade by trade basis! So long as the expectations are reasonably similar between actual trades and backtested performance, the system should be performing normally. It is very important to recognise that statistically, trades would be subjected to strings of wins or losses. Do not be overly alarmed.

One simple method to monitor is to check the actual drawdown of your live system and compare it to your maximum drawdown of the backtest. The actual current drawdown exceeding the maximum drawdown tested is a red flag that the system is not performing as well and market conditions may have changed.

At this point it is important to take note that backtesting should be done over a full market cycle (for example 5 years). It is pretty meaningless to backtest for 2 months and expect the live trades to follow the expectations (risk and returns) of the 2 month backtest.

Jay Ng can be contacted at StreetPips.com

 

[NVP]

Backtesting

interesting and well written piece - but for me sums up everything that frightens me about business modelling (in any arena) and how one can generate wonderful results if you keep playing with data for long enough............thats not how the real world works ...sadly !

still a good piece of work

N

 

[Sigma-D]

Back testing is easier than forward testing, but I'm not sure it's better.

When I ran back-testing last year on my system it was really good. I then traded it on a demo account, off and on for about 4 months. The results were still OK but nowhere near the same returns I got from my theoretical back-test results.

I've been trading live since late January this year using exactly the same system and the results are dire.

Back-testing assumes your trading is totally automated and get in and out at exactly the levels and points calculated by your system and that you sit through the pullbacks without moving your stop to breakeven and that you don't get shaken out by massive single bar moves against you that turn out after they close to be nothing of the sort. It also assumes you're trading 24 hours a day and taking every trade. It is so far from reality I really don't see how back-testing helps at all. It does the opposite if you think about it as it confers a credibility which cannot be achieved in practise.

 

[timsk]

Hi Sigma-D,
Your points are well made and I would agree with you completely - but for the two trump cards that back testing has up its sleeve . . .

Firstly, if your methodology doesn't come up with the goods when back tested, then it's got zero chance of magically making money when forward tested on demo or, heaven forbid, traded for real on a live account. This point could be discounted completely if it was as quick to forward test as it is to back test. Unfortunately, it's not. This leads to . . .

The second trump card in favour of back testing: speed. If your back test is automated - then it's lightning fast. Even if it's performed manually, it's going to be infinitely faster than forward testing in real time. If one starts with the assumption that the methodology being tested isn't profitable and will need revising or scrapping, what would a trader rather do: spend X amount of time arriving at that conclusion via a back test - or spend two, three, five or perhaps ten times as long via a forward test in real time? In the long run, back testing will save time.

For these two reasons, I recommend traders always back test their methodologies. I agree completely that the forward test is unlikely to produce results as good as the back test and, when traded live, the results are likely to show yet a further drop off in performance.
Tim.

 

[patong]

Back testing is never as good as forward testing. If you back test the mathmatics are already calculated. However with forward testing with the correct program the math resembles more of a live market condition, where as back testing does not. Its that simple.

 

[I'vereadthemall]

Howard Bandy can teach us a thing or two about good back-testing practice which also incorporates forward-testing too.

Using Amibroker (although I am sure all other good backtesting software does the same) we can easily incorporate a walk-forward optimization as part of our back-tests.

We can set the parameter for optimization, set the in-sample periods and out-of-sample periods and then tabulate the out-of-sample test runs to give us our expected returns when taking the strategy live.

My brother is in the process of testing candlestick patterns so I am lucky enough to have a pre-written strategy that can be used to show an example;

I have started the first In-sample period as 01/01/95 - 01/01/96.

The first out-of-sample period would then be 01/01/96 - 01/01/97.

The next In-sample period would be 01/01/96 - 01/01/97.

The next out-of-sample period would be 01/01/97 - 01/01/98.

And so on.

The parameter that is optimized in-sample is the lookback period for a Rate of Change filter. I.E, we only take signals if the ROC > 0, but we optimize during the in-sample period how long a lookback we use in the ROC calculation during the following out-of-sample period.

The objective function (the metric that we want our optimization to maximise) will be the CAR/MDD.

In summary....Whatever ROC lookback period > 0 produces the best CAR/MDD from the most recent in-sample run, we use in the next out-of-sample run.

At the end of all test runs, we can tabulate the entire out-of-sample results and in doing so we have achieved a simulated walk-forward test on nearly 20 years of data, in less than a couple of hours.

Depending on what the most recent in-sample optimization shows to be the best loockback for ROC, we use those settings to go live.

After a year of live trading, we optimise again and trade the new settings live, and so on until our live results differ greatly from previous out-of-sample runs.

Obviously the above assumes that the out-of-sample results of the walk-forward optimization are profitable. Otherwise, we wouldn't trade the strategy.

It also assumes that you haven't repeated the process multiple times in a bid to find a good system - because each time you do so, the out-of-sample results become more and more 'in-sample'.

If you are interested, the signal that I tested was an inverted hammer if it made a 3 day low and had a green real body, the entry was a break of the inverted hammer high on the following day.

All positions were held for 10 days. Max open positions was 10 and the amount of equity risked per trade was 10%.

Tests were run on a database that included all currently listed stocks and ETFs on the NASDAQ, NYSE or AMEX.

The variable that was optimized in-sample was the ROC(C,X) where X = the number of days lookback for the ROC indicator.

In Amibroker the entire strategy was written as;

////////////////////////////////////////////////
SetOption ("MaxOpenPositions" , 10 );
SetPositionSize (10, spsPercentOfEquity);
SetTradeDelays(0,1,0,0);

LongUpperShadow = H - Max(O,C) > (H - L)*0.67;
rng = abs((C-O)/O);
SmallRealBody = rng < 0.003 AND rng > 0;
DownTrend2 = L < Ref(LLV(L,3),-1);
rngy = H-L;
lowerShadow = Min(O,C) - L;
shaven = lowerShadow < rngy*0.1;
RealBodyGapDown = Max(O,C) < Min(Ref(O,-1),Ref(C,-1));

InvertedHammer = smallRealBody AND shaven AND realBodyGapDown
AND longuppershadow AND downtrend2 AND O < C;

Buysetup = invertedhammer
AND Close > 5
AND MA (C,20) > 1
AND MA(Volume,20) > 100000
AND ROC(C,Optimize("rocLB",10,10,250,10)) > 0;
Sell = 0;
Buystop = Ref(H,-1) + 0.01;

Buy = Ref(Buysetup,-1) AND Cross(H,Buystop);
BuyPrice = Max(Buystop,Open);

ApplyStop (stopTypeNBar,stopModeBars,9,False,3);

PositionScore = 100 + Volume;
/////////////////////////////////////////////////

While there are still inherent dangers with the above approach, (mainly that we might do multiple walk-forward optimizations until we find a profitable strategy out-of-sample - and in doing so muddy the out-of-sample data), I still feel that this is the best approach to back-testing that your average retail trader can implement.

The attached images show the system performance stats and equity curve...Of note is the 59% win-rate and less than 10% average market exposure.

If you are interested...the current best ROC lookback is 160.

 

[foroom lluzers]

This is the most useless article and discussion.I won't waste time even reading 1 sentence.

Price action doesn’t replicate itself over time

First of all, market prices are chaotic, and. The patterns observed on any day’s trading do not replicate themselves at other times. As a result, the attempt at testing today’s market action on the basis of yesterday’s patterns will never lead to results that can be duplicated over the long term.

A trader lost his entire account , after market volatility increased 200 % and back tested patterns changed.


http://www.trade2win.com/boards/educational-resources/223150-most-back-testing-useless.html