Trader333
Moderator
- Messages
- 8,746
- Likes
- 1,025
It depends on what you are trying to back test. I went through a phase of back testing everything to pieces and I used Tradestation 2000i to do this. I don't back test any more but I still use the same software which, even though others are telling me is old and limited in use, is still one of the best on the market in my view. It also works with Vista which is a big bonus for a package written back in the last millenium.
Paul
Paul
Chorlton
Established member
- Messages
- 693
- Likes
- 48
Hi Paul,
"It depends on what you are trying to back test"
Can I ask why this would make a difference?
Out of interest, I trade Stocks & use MS for my charting but have recently purchased TradeSim for backtesting as it seems to be one of only a few that can perform Monte Carlo Analysis.
I was just interested in what others were using in comparison......
blackcab
Established member
- Messages
- 523
- Likes
- 51
This may not be a useful reply as I'm not yet trading the 'stuff' that I'm developing. I trade UK stocks with play money but I'm in the long process of developing my own system for a different instrument, and I'm some way from going live yet.
To your question, for system exploration and backtesting I'm writing my own code. It's fairly low level - just plain C with no libraries, APIs or fancy pants like that - and just reads historical intraday data from a .csv file.
The beauty of writing your own code is the freedom it gives you, including possibly the potential to explore things in ways that the standard packages don't support. Then again I haven't used the standard packages, so they might do everything for all I know. Doing it yourself also forces you into the data and the analysis which I feel has benefits of its own. I also just enjoy it.
The downside is the time it takes and the occasional difficulty in implementing your ideas.
Chorlton
Established member
- Messages
- 693
- Likes
- 48
Hello Blackcab,
Interesting approach although probably one I wouldn't choose for a number of reasons. One obvious reason is that I don't think I would enjoy it so I would find the process quite painful although you, on the otherhand don't have that problem!!!
Another more important point though is what elements are you coding in? For instance, are you taking into account money management / position sizing when you backtest and is it designed to test a complete portfolio (of many stocks) rather than on an individual stock by stock basis?
I know of some systems that when backtested on individual stocks can return a specific profit but when applied to a complete portfolio of stocks shows a much lower return.....
Regards,
Chorlton
blackcab
Established member
- Messages
- 523
- Likes
- 51
Yes, it can take money management, sizing, scaling in/out into account as necessary. Because it's open ended, and works by simulating a trader bar by bar as opposed to applying an equation to whole datasets, it can simulate arbitrary strategies, e.g. reserving Tuesday mornings for long-only trades at reduced exposure except where Monday's high was touched more than once and the previous week was low volatility, and the clocks went back within the last 10 days, and volume/range/close profile of the last 20 bars was x or y, etc., etc. You get the idea.
I'm looking at a futures contract not stocks so I'm not taking portfolios into account, but that would clearly be an issue if you were. If you're going technical and wanting to backtest mechanical systems, I'd tend towards a more powerful and flexible system that gives you the option to explore in depth, even if what you end up using is a simple, intuitive KISS-type system. At least you can test it properly. I'm not sure what the main options are there - Tradestation, Easy Language, MetaStock, maybe?
TS2000i still rules with world! A truly excellent piece of software.
It's amazing Paul that this program works with Vista although it was developed 10 years ago or maybe more even when XP was not around.
Alex
Good question.
For quick testing of simple Metastock is the best.
For anything more complicated, there is nothing better (still) around than TS2000i
For much more complicated stories, Wealth-Lab is an good package you can also use to do portfolio back testing.
For anything more complicated, you must develop your own code.
I have used and done all of the above. Just avoid optimization, fitting and multivariate systems. These break down really fast in actual trading. Simple ideas are the best.
Alex
TWI
Senior member
- Messages
- 2,552
- Likes
- 263
I use TS8 with Rina portfolio evaluator/ portfolio stream add ons for portfolio analysis. Can do monte carlo with that and other money management stuff.
Although it can be frustrating and annoying at times writing and implementing in EL it is easy to learn and there are large archives of code offering help with the more tricky problems.
Some people have a big probllem with TS and I suspect if I was someboody who had spent my life writing my own code I would also have issues with it but all I can verify is that it has allowed me to take my proprietary trading ideas, quantify them, test them and ultimately lead me to a set of systems which have been proven robust and profitable with real money over a number of years.
Although it can be frustrating and annoying at times writing and implementing in EL it is easy to learn and there are large archives of code offering help with the more tricky problems.
Some people have a big probllem with TS and I suspect if I was someboody who had spent my life writing my own code I would also have issues with it but all I can verify is that it has allowed me to take my proprietary trading ideas, quantify them, test them and ultimately lead me to a set of systems which have been proven robust and profitable with real money over a number of years.
Chorlton
Established member
- Messages
- 693
- Likes
- 48
Hi Alex,
Good Advice which I already try and follow.
On the subject on optimization, I believe it does have a use but one must remember to always carry out the backtest (after optimizing) on an out-of-sample selection of data.
IMO Where people go wrong is not carrying out the last step......
Trader333
Moderator
- Messages
- 8,746
- Likes
- 1,025
If an idea is not profitable without any optimization then you can be certain that it wont be afterwards regardless of how good the results are made to look. I have seen many offerings time and again, and some that are still commercially available, that are clearly optimized back tested systems that always fail to deliver.
Paul
Paul
Chorlton
Established member
- Messages
- 693
- Likes
- 48
I do use ProRealTime for day to day monitoring but as I trade a portfolio of stocks I find Prorealtime (like most other backtesters) of limited value as you only have the opportunity to back on one specific stock at a time...........
This is not meant to be a negative comment towards ProRealtime thou, as I think it is excellent as a charting tool especially as it is free!!!!
Chorlton
Established member
- Messages
- 693
- Likes
- 48
Hello Paul,
I agree with what you say but IMO it depends on your definition of optimisation.
For Example:
If you have a moving average cross-over type system and after backtesting the initial idea you decide to change the specific values of the MA's to offer a more favourable result then this is basically curve-fitting to previous data and IMO is what you are referring to above.
However, if your system relies on some form of specific pattern search and you decide after testing that there might be a better alternative this could be classed as optimisation of the overall system. The same logic could be applied to your money management rules such as which TYPE of trailing stop to use. In the search for a more ROBUST set of rules it could be argued that we are optimising.
As Howard Bandy wrote in his book on System Development, "...Only a system based on truly random entries and exits would not be the result of some form of optimisation"
Kind Regards,
Chorlton
Correct. Optimization means adjusting variables to minimize or maximize some function, in this example maybe net profit, profit factor, success rate, or anything else you may choose.
Optimized systems are doomed to fail and they often fail faster than you thought.
I will not agree with you for whatever my opinion worths. You are talking here about different systems not optimization in the usual sense. One could say this is a loose use of the term but again you are not using the variables of the original system you are introducing new variables and you may have to optimize those in a strict sense.
Alex
My two euros worth.
I use a self devised optimised mechanical trading system that is based on back testing as far back as I can find reliable statisics on 7 indices and (more recently) 8 currency pairs. It has been profitable for 4 years now and although the success rate in live trading is not as great as in the back tests (currently 61% v 68%) it seems to be pretty robust and consistent. One of the most useful aspects of the backtesting was to put the money managment element through the mill.
So I would disagree with those who say that it can't be done - the fault that I believe most back-tests fall in to is that they are not done over a sufficently long time period and on enough different instruments. Lets face it, any mechanical system is simply trying to rationalise human behaviour relative to price and that will be the same on the FTSE as it is on Gold,Oil or the Euro, albeit measured in different ways, IMHO.
I use a self devised optimised mechanical trading system that is based on back testing as far back as I can find reliable statisics on 7 indices and (more recently) 8 currency pairs. It has been profitable for 4 years now and although the success rate in live trading is not as great as in the back tests (currently 61% v 68%) it seems to be pretty robust and consistent. One of the most useful aspects of the backtesting was to put the money managment element through the mill.
So I would disagree with those who say that it can't be done - the fault that I believe most back-tests fall in to is that they are not done over a sufficently long time period and on enough different instruments. Lets face it, any mechanical system is simply trying to rationalise human behaviour relative to price and that will be the same on the FTSE as it is on Gold,Oil or the Euro, albeit measured in different ways, IMHO.
zentrader22
Member
- Messages
- 62
- Likes
- 1
Chorlton,
I'm using a full programming language called "PowerBasic" www.powerbasic.com for development of my software including trading systems (but for this you need some programing skills). Former i used Metastock which is easy to use and if you search the tool with the best cost/benefit ratio in this area you should try Amibroker www.amibroker.com .
Additionally I use my monte carlo simulation product ( www.zentrader.de/html/monte_carlo_simulator1.html ) for stresstests of conventional backtests and especially for data simulation issues (´= test your system with synthetic data to get a scenario under changed market conditions).
bye,
zentrader
Chorlton
Established member
- Messages
- 693
- Likes
- 48
Firstly, Thanks to all who have contributed.....
ZenTrader,
Can you elaborate on the above especially those phrase in BOLD? To get a scenario under CHANGED market conditions sounds interesting.....
Thanks....
zentrader22
Member
- Messages
- 62
- Likes
- 1
Chorlton,
I've written a short paper concerning to this topic on my website:
http://www.zentrader.de/dataSimulation_e.pdf
Only my two cents...
bye,
zentrader
zentrader,
Excellent white paper. I think full sample back testing followed by Monte carlo simulation is the better way to go.
Alex
