Differences between backtests done by several software

This is a discussion on Differences between backtests done by several software within the Trading Systems forums, part of the Methods category; Hi everyone, Does software play a significant role in the quality of the backtest? MT4 is capable of doing backtests ...

Reply
 
LinkBack Thread Tools Search this Thread
Old Mar 13, 2015, 11:14pm   #1
 
9 Posts
Joined Mar 2015
Differences between backtests done by several software

Hi everyone,

Does software play a significant role in the quality of the backtest? MT4 is capable of doing backtests and it is free but I am wondering if other software like Amibroker or Multicharts offer additional quality and maybe other capabilities that I cannot think of. Am I missing something while using MT4? Any info on the topic is appreciated. Thank you.
kojinakata is offline   Reply With Quote
Old Mar 13, 2015, 11:55pm   #2
 
malaguti's Avatar
Joined Nov 2009
Quote:
Originally Posted by kojinakata View Post
Hi everyone,

Does software play a significant role in the quality of the backtest? MT4 is capable of doing backtests and it is free but I am wondering if other software like Amibroker or Multicharts offer additional quality and maybe other capabilities that I cannot think of. Am I missing something while using MT4? Any info on the topic is appreciated. Thank you.
When you say quality of the backtest, its only as good as the parameters or functionality of the software. I've performed manual backtests many times especially as alot of strategies are more subjective but I've tended to place more reliance on the software generated ones.
But it needs to be realistic. Ie a backtest that takes account of overall market trend is far better than an individual backtest on one instrument. One that can also "prioritise" based on a ranking, as its highly likely you may have more than buy signal on different instruments at the same time.
One that can backtest against a whole portfolio and one that can recognise different timeframes.
All of these are the qualities I look for, and lastly but also just as important is one that is easy to configure. Design your own exit strategies, partial exits and scale in or out of positions
They all play quite a large role I've found in assessing the quality of the strategy I intend to use

On the flip side, with backtesting software there is a tendency to add more and more parameter, to get the results better, leaning to optimisation
I never optimise, however curvefitting becomes inevitable the more you tweak something.

But no doubt for me, software plays a large part in the ability to perform these functions
malaguti is offline   Reply With Quote
Thanks! The following members like this post: kojinakata
Old Mar 14, 2015, 1:02pm   #3
 
9 Posts
Joined Mar 2015
kojinakata started this thread Thank you for your reply malaguti.

I don't optimize any of the parameters I do backtest on to avoid curvefitting. I plan to do forward testing to my strategies as well to measure their performance better. I just wondered whether MT4 had inferior backtesting qualities because it is free. I think MT4 cannot backtest strategies that incorporate indicators from different timeframes, but I am not sure. Maybe that is the advantage of the paid backtesters.

Can you recommend any backtesters that are free? Any of them capable of walk-forward testing? I downloaded M1 data from a free source and would like to give my strategies a try in a free application. I am a student so anything free, I will take it .
kojinakata is offline   Reply With Quote
Old Mar 14, 2015, 5:59pm   #4
 
malaguti's Avatar
Joined Nov 2009
Quote:
Originally Posted by kojinakata View Post
Thank you for your reply malaguti.

I don't optimize any of the parameters I do backtest on to avoid curvefitting. I plan to do forward testing to my strategies as well to measure their performance better. I just wondered whether MT4 had inferior backtesting qualities because it is free. I think MT4 cannot backtest strategies that incorporate indicators from different timeframes, but I am not sure. Maybe that is the advantage of the paid backtesters.

Can you recommend any backtesters that are free? Any of them capable of walk-forward testing? I downloaded M1 data from a free source and would like to give my strategies a try in a free application. I am a student so anything free, I will take it .
ProRealTime is free and you can backtest. Not sure how difficult it is vs MT4, I've never got into metatrader. good luck though
malaguti is offline   Reply With Quote
Thanks! The following members like this post: kojinakata
Old Mar 26, 2015, 8:09am   #5
Joined Mar 2015
Backtesting has limitations and is subjective.

If you use software there is often a variety of limitations and assumptions it makes in terms placing trades, how the trades are filled and the calculation of profits and losses.

I am not saying you should not back test but you must be aware that the results often don't replicate future performance.
richieforex is offline   Reply With Quote
Thanks! The following members like this post: kojinakata
Old Mar 26, 2015, 5:12pm   #6
 
9 Posts
Joined Mar 2015
kojinakata started this thread
Quote:
Originally Posted by richieforex View Post
Backtesting has limitations and is subjective.

If you use software there is often a variety of limitations and assumptions it makes in terms placing trades, how the trades are filled and the calculation of profits and losses.

I am not saying you should not back test but you must be aware that the results often don't replicate future performance.
I suppose that these limitations are the same when doing Walk Forward Analysis. Can you elaborate the limits and assumptions that these softwares have? Is taking slippage into account one of them? How can one identify these limitations and assumptions? What actions can a trader take to overcome these limitations and assumptions?
kojinakata is offline   Reply With Quote
Old Mar 26, 2015, 10:23pm   #7
 
malaguti's Avatar
Joined Nov 2009
Quote:
Originally Posted by kojinakata View Post
I suppose that these limitations are the same when doing Walk Forward Analysis. Can you elaborate the limits and assumptions that these softwares have? Is taking slippage into account one of them? How can one identify these limitations and assumptions? What actions can a trader take to overcome these limitations and assumptions?
Richie is quite right.
Assumptions:
could indeed be slippage and assumes that you are able to buy at that price. some sotware are based on the last closed price rather than the next open
Even if its next open there may not enough liquidity at that price.
stops also subject to slippage, whereas software just takes your stop loss
Also assumes you have the equity to take every trade, and even the time to take every trade. likelihood is you won't on both counts
assumes you follow the strategy whereas you the human will take over

many good software allows an element of "forgiveness" and can assume a %ge of slippage, can trade the next open, can take account of equity etc etc but will never take the human out of the equation
malaguti is offline   Reply With Quote
Old Mar 27, 2015, 9:40pm   #8
 
9 Posts
Joined Mar 2015
kojinakata started this thread
Quote:
Originally Posted by malaguti View Post
Richie is quite right.
Assumptions:
could indeed be slippage and assumes that you are able to buy at that price. some sotware are based on the last closed price rather than the next open
Even if its next open there may not enough liquidity at that price.
stops also subject to slippage, whereas software just takes your stop loss
Also assumes you have the equity to take every trade, and even the time to take every trade. likelihood is you won't on both counts
assumes you follow the strategy whereas you the human will take over

many good software allows an element of "forgiveness" and can assume a %ge of slippage, can trade the next open, can take account of equity etc etc but will never take the human out of the equation
Any free&paid (free is better of course) software that might enable to assume % of slippage for simulating low liquidity and close-open difference? The aim to develop a mechanical/algorithmic trading strategy so the human is out of the equation from the start, I suppose.
kojinakata is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Similar Threads
Thread Thread Starter Forum Replies Last Post
Why backtests are useless, EAs are flawed and their parameters are bad [DISCUSS] Darwin-FX Trading Systems 55 Nov 29, 2013 9:09pm
Free tick data backtests of your EA with 99.9% modeling quality babyjake1961 Trading Systems 0 Jul 27, 2013 10:00pm
Statistical analysis, Backtests & automated trading : Which way to go? JossBeaumont Trading Software 17 Jul 19, 2013 4:29am
Metatrader backtests flawed oildaytrader Trading Software 19 Nov 4, 2009 9:57pm
Long chart history for backtests.. dbinvest Forex 0 Dec 20, 2007 1:01pm

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)