Differences between backtests done by several software

[kojinakata]

Hi everyone,

Does software play a significant role in the quality of the backtest? MT4 is capable of doing backtests and it is free but I am wondering if other software like Amibroker or Multicharts offer additional quality and maybe other capabilities that I cannot think of. Am I missing something while using MT4? Any info on the topic is appreciated. Thank you.

 

[malaguti]

Hi everyone,

Does software play a significant role in the quality of the backtest? MT4 is capable of doing backtests and it is free but I am wondering if other software like Amibroker or Multicharts offer additional quality and maybe other capabilities that I cannot think of. Am I missing something while using MT4? Any info on the topic is appreciated. Thank you.

When you say quality of the backtest, its only as good as the parameters or functionality of the software. I've performed manual backtests many times especially as alot of strategies are more subjective but I've tended to place more reliance on the software generated ones.
But it needs to be realistic. Ie a backtest that takes account of overall market trend is far better than an individual backtest on one instrument. One that can also "prioritise" based on a ranking, as its highly likely you may have more than buy signal on different instruments at the same time.
One that can backtest against a whole portfolio and one that can recognise different timeframes.
All of these are the qualities I look for, and lastly but also just as important is one that is easy to configure. Design your own exit strategies, partial exits and scale in or out of positions
They all play quite a large role I've found in assessing the quality of the strategy I intend to use

On the flip side, with backtesting software there is a tendency to add more and more parameter, to get the results better, leaning to optimisation
I never optimise, however curvefitting becomes inevitable the more you tweak something.

But no doubt for me, software plays a large part in the ability to perform these functions

 

[kojinakata]

Thank you for your reply malaguti.

I don't optimize any of the parameters I do backtest on to avoid curvefitting. I plan to do forward testing to my strategies as well to measure their performance better. I just wondered whether MT4 had inferior backtesting qualities because it is free. I think MT4 cannot backtest strategies that incorporate indicators from different timeframes, but I am not sure. Maybe that is the advantage of the paid backtesters.

Can you recommend any backtesters that are free? Any of them capable of walk-forward testing? I downloaded M1 data from a free source and would like to give my strategies a try in a free application. I am a student so anything free, I will take it .

 

[malaguti]

Thank you for your reply malaguti.

I don't optimize any of the parameters I do backtest on to avoid curvefitting. I plan to do forward testing to my strategies as well to measure their performance better. I just wondered whether MT4 had inferior backtesting qualities because it is free. I think MT4 cannot backtest strategies that incorporate indicators from different timeframes, but I am not sure. Maybe that is the advantage of the paid backtesters.

Can you recommend any backtesters that are free? Any of them capable of walk-forward testing? I downloaded M1 data from a free source and would like to give my strategies a try in a free application. I am a student so anything free, I will take it .

ProRealTime is free and you can backtest. Not sure how difficult it is vs MT4, I've never got into metatrader. good luck though

 

[richieforex]

Backtesting has limitations and is subjective.

If you use software there is often a variety of limitations and assumptions it makes in terms placing trades, how the trades are filled and the calculation of profits and losses.

I am not saying you should not back test but you must be aware that the results often don't replicate future performance.

 

[kojinakata]

Backtesting has limitations and is subjective.

If you use software there is often a variety of limitations and assumptions it makes in terms placing trades, how the trades are filled and the calculation of profits and losses.

I am not saying you should not back test but you must be aware that the results often don't replicate future performance.

I suppose that these limitations are the same when doing Walk Forward Analysis. Can you elaborate the limits and assumptions that these softwares have? Is taking slippage into account one of them? How can one identify these limitations and assumptions? What actions can a trader take to overcome these limitations and assumptions?

 

[malaguti]

I suppose that these limitations are the same when doing Walk Forward Analysis. Can you elaborate the limits and assumptions that these softwares have? Is taking slippage into account one of them? How can one identify these limitations and assumptions? What actions can a trader take to overcome these limitations and assumptions?

Richie is quite right.
Assumptions:
could indeed be slippage and assumes that you are able to buy at that price. some sotware are based on the last closed price rather than the next open
Even if its next open there may not enough liquidity at that price.
stops also subject to slippage, whereas software just takes your stop loss
Also assumes you have the equity to take every trade, and even the time to take every trade. likelihood is you won't on both counts
assumes you follow the strategy whereas you the human will take over

many good software allows an element of "forgiveness" and can assume a %ge of slippage, can trade the next open, can take account of equity etc etc but will never take the human out of the equation

 

[kojinakata]

Richie is quite right.
Assumptions:
could indeed be slippage and assumes that you are able to buy at that price. some sotware are based on the last closed price rather than the next open
Even if its next open there may not enough liquidity at that price.
stops also subject to slippage, whereas software just takes your stop loss
Also assumes you have the equity to take every trade, and even the time to take every trade. likelihood is you won't on both counts
assumes you follow the strategy whereas you the human will take over

many good software allows an element of "forgiveness" and can assume a %ge of slippage, can trade the next open, can take account of equity etc etc but will never take the human out of the equation

Any free&paid (free is better of course) software that might enable to assume % of slippage for simulating low liquidity and close-open difference? The aim to develop a mechanical/algorithmic trading strategy so the human is out of the equation from the start, I suppose.

 

[Solas0077]

I recall that TS2000i made some assumption regarding the relative position of open to high and low and that affected any exits triggered during a bar. I also recall getting quite different results from other platforms that did not employ such scheme. I find Amibroker quite accurate if, and only if, the AFL is programmed correctly will all settings. Free backtesters are unlikely to produce reliable results.