Best Practices for Optimizing

[hobiecatlob]

I would like to get some opinions on the best practices for optimizing a strategy.
I have been optimizing my strategy on about 5-6 years of data. I am using mostly 60 minute charts. I am using the genetic optimizer in TradeStation and I am using all the data (not excluding any data for out of sample testing). I have also set the stress test size to 3.

My concern is that after optimizing on the last 5-6 years, I test a period prior to that to see how it does on the out of sample data and it performs poorly. So, clearly I have curved fit the strategy to the 5-6 year period.

Now, for an ETF such as SPY for which there is almost 20 years of data, how much data would be best to use in the optimizing process? If I used the entire period to optimize then I have the benefit of optimizing over a great deal of different market types. On the other hand, one could argue that the markets have changed over that period so most recent data would be best to optimize on as it will be most similar to current markets.

Should I optimize over that entire period or is using the most recent years more advisable? Should I be using the WFO?

Any suggestions or feedback would be appreciated.

Thanks in advance!

 

[intradaybill]

It depends won hat kind of optimization you are performing. If you optimizing entries and exits then this will lead to a highly curve-fitted system and failure. I think you are asking the wrong question IMO. The issue is not for how long to optimize but what to optimize. This is an excellent blog article that I found very revealing.

As far as WFO, it is a dubious process IMO. Amibroker had this feature long before TS. I never used it because an optimized system is an optimized system regardless of whether that is done in forward steps or in any other way.

Try using trading systems that allow the least amount of optimization like it is explained in that article.

 

[Hotch]

I'm pretty sure best practice is to do out of sample testing, which you've decided against, why?

 

[tickmaster]

If you find a strategy that works well in backtest over many years (“Strategy A”), you can be more comfortable that you’ve found something that could work well over a broader variety of market conditions.

If you find a system that backtests well over a recent period of time (“Strategy B”), you may have something that ‘s better in tune with the market now than Strategy A, but you’ll have to monitor more closely whether current conditions change to the extent that Strategy B ceases to work. Out of sample tests with Strategy B may help you get a better sense of what it is about the current conditions that make Strategy B work now.

Having said that, two things:

-everything intradaybill writes about keep it simple (stupid), and don’t overfit/curvefit applies; experience of price action, and understanding your market (specifically the aspect of its structure that makes your strategy work in the first place), will be your best guide as to whether you’re overfitiing or not.

- don’t blindly use historical bar data without understanding where it’s come from. After all, whatever else you do, your backtests will only be as good as the data you use; garbage in, garbage out. You’re using TradeStation historical data? You can run a simple test over a week or two to compare historical 60-min bars for SPY with real time SPY tick data from your TS software (NB – the tick data must be realtime, not historical). Save the real time tick data (can TS do this? If not use free NinjaTrader connected to TS to record this for the test), and at the end of one/two weeks then download from TS the historical 60-min bars for the same period. If you compare in detail the real time highs and lows of the ticks each 60-minutes, you’ll often get a significantly different price action (because of “bad ticks” more likely making their way into historical data).

http://tickmaster.webs.com

I would like to get some opinions on the best practices for optimizing a strategy.
I have been optimizing my strategy on about 5-6 years of data. I am using mostly 60 minute charts. I am using the genetic optimizer in TradeStation and I am using all the data (not excluding any data for out of sample testing). I have also set the stress test size to 3.

My concern is that after optimizing on the last 5-6 years, I test a period prior to that to see how it does on the out of sample data and it performs poorly. So, clearly I have curved fit the strategy to the 5-6 year period.

Now, for an ETF such as SPY for which there is almost 20 years of data, how much data would be best to use in the optimizing process? If I used the entire period to optimize then I have the benefit of optimizing over a great deal of different market types. On the other hand, one could argue that the markets have changed over that period so most recent data would be best to optimize on as it will be most similar to current markets.

Should I optimize over that entire period or is using the most recent years more advisable? Should I be using the WFO?

Any suggestions or feedback would be appreciated.

Thanks in advance!