Tradestation (or any other type of...) backtesting, optimising, curve-fitting

[JTrader]

Hi

I can see how its possible to curve fit strategies if working with a limited data history - perhaps 2 months of data, and working between 1-60 minutes timeframes for backtesting and optimising strategies.
But I would think that it is not possible to curve fit a system through back-testing - optimisation as long as you are using a big enough history of data. For example, if working with 1.5 years of 1-60 minute EUR/USD data to backtest strategies and optimise the parameters with, how could a person curve fit a system to this large amount of data? surely it's not possible, and any strategy that can withstand a large history of data and be profitable at the end of it must surely be seen as a robust system that has/is withstanding the test of time (???)

Cheers

jtrader.

 

[fxmarkets]

you would be surprised, but I think one of the best ways is if you have a few thousand bars of data is to break that down and test from different perionds, ie. test on first 1000 bars, test on the next 1000 bars, then the next 1000 bars. etc. rather than the whole at once. See what results come out at.

But using your creativity programming may certainly throw up many ideas along the way .

 

[Tuffty]

What I try and do is replicate real life as far as possible i.e. curve fit and optimize as much as possible on a sample of the data to get your system & parameters, then test those out on unseen data. Re-optimise on the seen data and repeat. The consistency of results on unseen data is what counts. I'm happier in the results if the unseen data has all market conditions in it (bull, bear, flat, hi volatility, low volatility).

 

[Directional]

the best way to avoid curve fitting is to test and optimise your code on say a years worth of data, and as long as the sample data produces a statistically significant number of trades, ie: >200 you then use "out of sample" data, such as the next years worth of data - without changing or optimising any parameters.

This sort of testing allows you to see if you are in fact curve fitting - if your system is over optimised, it will produce great results in the sample period, but suck on the out of sample period.

If the performance results stay similar on in sample and out of sample, you have a decent indication how it would perform on the out of "sample data" created by the markets walking forward.

this of course assumes you correctly account for the costs of doing business - commissions, and the generally accepted (on TS forum) 1 tick per side slippage, and are not using limit orders (for which TradeStation is notorious for producing unreliable backtest results)

 

[blackcab]

I agree with the replies so far -- it's a mistake to do anything other than developing a system that works on one sample and only when you're happy with it, testing it on out-of-sample or 'unseen' data. There are variations that work on three sets of data as well.

To answer the original question, you can curve-fit any sample of data regardless how large; it just depends on the capacity of your function. Complex formulae that you 'train' or 'evolve' by computer such as neural nets are prone to curve fitting, but the reasons why are usually overlooked or misunderstood, at least on these boards.

Neural nets by themselves are just function approximators and aren't inherently 'good' or 'bad' - it's the fact that you can easily give them more 'nodes' than is necessary, i.e. give them too much capacity, that effectively lets them use the spare capacity not as generalising processing units but as simple memory. Train a 1,000-node net on a sine curve function and it will probably end up using those nodes to remember the shape of the curve rather than deriving the sine function, and so will have no predictive power. Train a 1,000,000-node net on three years of 5-minute NQ data and it will end up just remembering the patterns which is pretty useless. Train a 100-node net on that data, though, and it might just uncover some useful generalities.

The principle isn't confined to neural nets; I just used the term as that's what I know about.

 

[JTrader]

Hi

If you have 2 years of 1-minute+ intraday data, I can see the value in testing your system against the first year, optimising the parameters, then testing the system against the second years data and seeing if the results are comparable. You could then optimise the system on the second year of data, test the system over the full two years worth of data with both the year 1 and year 2 optimised parameters and see which are most favourable, optimise the parameters again with 2 years data, and look at how the results from the new optimised parameters compare to the year 1 and year 2 optimised paraameters tested on 2 years data..............Are you with me the possibilities are diverse..................

Having thought about it, I can see how some intraday strategies can be curve-fitted against large amounts of historic intraday data. If an intraday strategy trades infrequently (maybe 2-3 times a week) and only when a complex setup arises on the chart - curve fitting is possible. I would think that if you have an intraday strategy that trades maybe 2-3 times a day on average, and you test it over 2 years intraday data, curve fitting would be much more difficult to simulate. Therefore the more often that a strategy produces trades the more difficult it would be to curve fit it ????

Quick question. Roughly how long would TS8.1 take to back-test 2 years of intraday data with a strategy on a 5-minute timeframe? Is the time affected by your Internet connection speed? And roughly how long would TS8.1 take to optimise 2 years of intraday data with a strategy on a 5-minute timeframe?

Cheers

jtrader.

 

[JTrader]

Hi

I now have a system that I am fairly happy with. It produces long and short entries with reversals of the entry criteria leading to sellshorts and buytocovers. It also has a stop-loss. In backtesting in seems to just about break even, But this is without a profit target, percent trailing stop, dollar trailing stop, breakeven stop etc. applied.

I have optimised the main entry critieria only (CCI based) length of period used, and entry points on a scale of 0-50 & 0 - -50, and this aspect seems stable. Looking at it on the chart, most trades present the opportunity for profit, small - large, with smallish stop-losses in place.

The reason i have decided to stop testing it with profit targets etc. and going down the road of optimisations is because it seems once i get hung up on backtest results and trying to improve them, i am automatically starting to curve fit my system - evident because as the results improves, the number of trades within the period tested is reduced, and just because these parameters were the best in the past, doesn't mean they will be the most profitable in the future, and may actually limit the potential of the system.

I intend to use this strategy to produce entry signals, a mechanical stop-loss is also in place, a mechanical exit (and reversal) is also in place (sellshort or buytocover, but this aspect would rarely come into play, as a discretionary profit target, or the stop-loss are more likely to be hot first). I will manage the profit taking aspect of the strategy on a discretionary basis, and hope that i will be able to return good profit levels as are evident on the historical charts.

 

[Trader333]

James,

If you are intent on using a mechanical system then it is essential that it shows profit without any optimisation whatsoever. I have, (in the past), done a lot of work in this area and those who are successful in using mechanical strategies all said that the system has to be profitable in a raw format. If it is then you can play with optimisation but are advised not to do so before.

You will undoubtedly have seen many commercially advertised systems that show fantastic profits on a historical basis but, "Remarkably" seem to fail when users buy the system and try and use it in realtime. The reason is almost always because an idea is taken and optimised to past results and some of these jokers are charging thousands for their lame systems.

I hope this helps


Paul

 

[JTrader]

You will undoubtedly have seen many commercially advertised systems that show fantastic profits on a historical basis but, "Remarkably" seem to fail when users buy the system and try and use it in realtime. The reason is almost always because an idea is taken and optimised to past results and some of these jokers are charging thousands for their lame systems.

Thanks Paul for the info, all critical feedback is appreciated.

Before really getting stuck into using tradestation I was not able to fully comprehend, how optimisation can and does lead to curve-fitting. I can see that you can only optimise a strategy so far, before it will not optimise any further, but it is easy to turn an unprofitable strategy into a profitable strategy, albeit with remote and perhaps unstable parameters that i would not consider beforehand. I perhaps don't want to get drawn any further into this trap, and also be drawn into a situation where my mechanical system limits the potential of results that could be obtained by a discretionary exit.

I have also experienced how hard it is to positively backtest several intraday strategies with seemingly sensible parameters. But then you optimise the parameters, end up with parameters that you would not have considered beforehand, and they are profitable. But then the strategy becomes unrealistic. eg. a 50 pip stop-loss on a 2-minute chart!

With the CCI strategy that I have mentioned, it is not a complete mechanical system, more of a semi-mechanical system producing mechanical entries that will lead to managed trades, but with a maximum stop-loss in place.
I have optimised the CCI LE & CCI SE, in order to find parameters that give the entries the best chance of leading to the most profitable discretionary exits, if this makes sense. But again, the most profitable results have the least trades generated. But I am happy with the average number of trades generated per day.

Because I am no longer intending to trade this strat on a fully mechanical basis, I am not paying close attention to the backtest results of the bare bones system. I can see on the historical charts that the majority of entries lead to opportunities to clip big enough profits, with usual pullbacks that would not hit my stop-loss.

Scaling out of posiitions in order to take a profit is possibly what I'll do....eg. 15 pips profit - exit 25% of position, 30 pips profit - exit another 25% etc.

Hopefully it will work in reality!

Cheers.

 

[firewalker99]

question in general

At what point does your backtesting becomes curve-fitting or optimizing instead of truly testing a strategy. Suppose I found a setup that holds a 50% win ratio. Included in my setup is a fixed stop of 1 point. If I'd use the setup with a 3 point stop the setup would have a 60% win ratio, of course with greater risk. For the moment I'm defining parameters on sample data for about two months. That's not a lot, I know, but I was wondering if going back to a couple of years data is very useful as markets change over time (but what time?). The next phase is comparing my setup with another two months of unseen data so that every part of the market cycle is included. Is there a reason to assume seasonal or cycliclal components that would render data analysis of a couple of months more or less useful than other months? I've calculated that I have to test about three to fourhundred "elements" to have results that are statistically significant up to 5% error. [elements] could be a [signals] or [days]. One possible answer could be that I had to use as much data as possible in order to be confident of the results of the backtesting. However, I'd like not to relate my personal "feeling" of confidence in this matter and only rely on the numbers.

In essence, what would you define as curve-fitting, where is the line between optimizing your strategy or just looking for the optimal elements that are part of the best possible setup?

 

[Tuffty]

firewalker99

For me the questions you pose give the reasons as to why systematic trading is just as artistic as discretionary trading.

I'd like not to relate my personal "feeling" of confidence in this matter and only rely on the numbers

In my experience you can only get a certain amount of confidence from the numbers. Depending on a persons character it may not be enough to put money on the line. If one has to trade a system then the personal "feeling" of confidence (in my view) is extremely important and shouldn't be ignored (as long as the trader doesn't end up fooling themselves ie just choosing stats that back up their idea).