Turning the System "Off"

[Chorlton]

Hello StratOpt,

Thanks for your comments. IMO This is a really interesting topic...

One thing that you can do is to test this idea out across a wide variety of different symbols some of which you may have never used before to test. [\QUOTE]

Good Idea. My current testing environment is the FTSE250 so I guess I could try it out on the FTSE350. I'm reluctant to try it out on another Mrkt (such as the US) as I don't believe the strategy will work as well. Saying that I do believe that it is relatively robust on the current Mrkt.

You may also decide that what you have done is not going to affect the strategy in real life and trade upon it with a secondary stop loss level that would determine the end of the idea. [\QUOTE]

Not too sure I follow you here. Surely, the whole idea is that the new filter WILL to some degree affect the strategy in Real-Time. I probably just misunderstand your comment here... Sorry.

To a certain extent whenever we are doing strategy developement we are doing so with a knowledge of what the market have been doing and it is very easy to build in a certain bias to the strategy based upon this. This is why it is mandatory that you set aside a certain length of time for watching the strategy work in real life before plopping any money on it and then start slow when you do. [\QUOTE]

Good Advice, but unfortunately regarding your comment in BOLD I'm not too sure how I could really do this. I say this as I have specific position sizing rules which means that I need a specific minimum amount of starting capital for the system to trade profitably, and this is the amount that I am currently planning to use. I guess the only option would be for me to simply paper trade it.

This is even after you perform a proper walk forward test that has an adequate efficiency of walk forward result. The ideal thing, IMO, is to have a set of different systems that you are using to trade with currently, and then you are constantly in the development and testing phases with new ideas. This will allow you to easily set that time aside to watch the systems in real time while you are actually trading using other systems that have already been through this process. In time you will run into a tough decision process and that is the fact that you have too many systems and now have to decide which of them to use at any given time. [\QUOTE]

Again Good Advice but the System I am developing will be my first one, so I won't have this luxury.

Hence the importance in carrying out sufficient backtesting as well as ensuring that all other factors which could affect the System have been considered and catered for.....

 

[StratOpt]

By affecting the strategy, I mean that it is going to be a parameter that is conducive to overfitting, an adverse affect. Affecting = changing the dynamics of the strategy to the point where it will fail simply due to being overfitted. There are some parameters that do not do much to push a logic into an overfit where some parameters such as the lengths of a moving average can do so with near miraculous ability.

In my world of trading a person is not trading everything on a single strategy idea. One will want to spread the contracts around across multiple strategies on multiple markets. Starting out slow to me means that you do not go to overweighting the importance or potential of a strategy until it has proven itself and then you may want to increase the weighting of it in turn by doing a position sizing method so that the strategy 'earns" the extra weighting. If I was starting in the strategy development world and had enough capital to do so, then I would wait until I had maybe a half dozen working strategies before I would start trading. In the beginning of my trading career, I waited until I came up with a dozen strategies that were all proofed before I started actually trading on them. It took me a whole year to get to that point.

As to position sizing rules and such, I do not backtest a strategy using any position sizing. I treat each strategy on a single contract basis. Any position sizing that I would do will come later. If one has the ability to position size then they also have the ability to trade multiple strategies on multiple symbols which is a form of position sizing in and of itself.

 

[dcraig1]

By affecting the strategy, I mean that it is going to be a parameter that is conducive to overfitting, an adverse affect. Affecting = changing the dynamics of the strategy to the point where it will fail simply due to being overfitted. There are some parameters that do not do much to push a logic into an overfit where some parameters such as the lengths of a moving average can do so with near miraculous ability.

In my world of trading a person is not trading everything on a single strategy idea. One will want to spread the contracts around across multiple strategies on multiple markets. Starting out slow to me means that you do not go to overweighting the importance or potential of a strategy until it has proven itself and then you may want to increase the weighting of it in turn by doing a position sizing method so that the strategy 'earns" the extra weighting. If I was starting in the strategy development world and had enough capital to do so, then I would wait until I had maybe a half dozen working strategies before I would start trading. In the beginning of my trading career, I waited until I came up with a dozen strategies that were all proofed before I started actually trading on them. It took me a whole year to get to that point.

As to position sizing rules and such, I do not backtest a strategy using any position sizing. I treat each strategy on a single contract basis. Any position sizing that I would do will come later. If one has the ability to position size then they also have the ability to trade multiple strategies on multiple symbols which is a form of position sizing in and of itself.

A partially related question - if using an optimizer ( either for the purpose of optimizing or for getting a handle on the robustness of a strategy) do you think there is value in looking at the shape of the optimization surface ?

 

[StratOpt]

Yes, there is most definately value in viewing the optimization landscape. All of my optimizers have the ability to detail out the optimization landscape.

 

[Mathemagician]

As to position sizing rules and such, I do not backtest a strategy using any position sizing. I treat each strategy on a single contract basis. Any position sizing that I would do will come later. If one has the ability to position size then they also have the ability to trade multiple strategies on multiple symbols which is a form of position sizing in and of itself.

While I totally agree with everything else you've said, this is one place where our approaches diverge. To my eyes, evaluating everything on a single contract basis works well over short periods of time but breaks down over longer time frames. This is because the distribution of returns on a single-contract basis is non-stationary. This means that certain time periods are emphasized and others are given a lesser weight in the backtest. This leads to biases and performance that is geared towards a certain type of market at the expense of other types. One can apply position sizing in such a way as to normalize the returns over time so that this does not happen. Obviously, this approach makes sense to me but might not make sense to others and may not even apply if one is doing things in certain ways. That is, after all, what makes a market!

jj

 

[StratOpt]

Yes, this is an area where you and I will differ.

 

[jm99]

some quality posts there stratopt

One thing that you can do is to test this idea out across a wide variety of different symbols some of which you may have never used before to test.

lets say the strategy is properly tested (sufficient sample sizes, walk forward etc) among 10 uncorrelated (as possible) markets. The strategy performs as expected in 7 markets, but poor in 3 markets. How would you go about evaluating this? Would you expect a robust strategy to perform equally well in all markets? or is it another layer of curve fitting to only trade the markets that performed well and excluded the under performing markets??

 

[StratOpt]

I would not expect that a strategy with a single set of parameters is going to work great against all things it is tested against. I would expect that a robust one would work great on many and decent on many and poor on some. What I do not want is a strategy that works only on one thing and that is it. The fact of life for many people is that they do not have the funds to be able to trade correctly in a multiple market method so this means that they are going to have to select one out of a list of symbols the system is applied to and then trade that one symbol. If this was the case, I am going to pick the symbol that does the best over all of my testing in the out periods. Part of the testing process is to see how the strategy performs against multiple symbols even if you are only going to trade one. You test on one check on others. In a multiple strategy multiple market scenario you will often find that adding in a relatively poor performer will actually improve your overall portfolio.

 

[TWI]

I do not see that there is any problem with strategies that are trading a specific market and optimised for such. There may be very specific inefficiencies in certain markets which can be exploited for long periods of time. I run systems which are very specific and also more generic algorithms which I still optimise for component markets as I find this produces better returns overall than a general set of parameters. So long as you find what you consider to be a stable set of values in a specific market then there really is IMO an advantage to this.
As for systems that fail, the ones which I have had fall over have been those that exploit short term characteristics that have subsequently changed as price and volatility have changed. I had an intraday crude system that was a star performer for years and worked on a fixed stop and profit target but started to fail towards the end of last year due primarilly to the increased volatility of the market at higher prices. Was getting stopped in where it shoudln't and stopped out in the noise. A problem with systems that start to fail is recognising that they are failing. Generally by the time you pull something that has made great returns for years because it has blown up, you already gave back far more than you should. I do not see an ideal solution other than ensuring that your portfolio is diversified to the extent that failure of one or two components is not going to create too great an impact on your overall returns. I think the way to avoid this situation in the first place is to ensure that there is not too much static calculation and by that I mean that a lot of values are calculated based on market volaitility rather than having constants. However I do recognise that you can create some great short term systems using static values that can return good money for extended periods, I just think these are the ones you need to monitor more closely.

 

[StratOpt]

The optimum thing is to have multiple systems in place and a continuous development process so that you do not wait for a system to fail, but you generate new strategies that just out perform the older ones and they take their place. An evolution of sorts. Nothing wrong with optimizing to fit a single symbol if that is what you are you are going to trade. You may as well try to get the best you can, but this still does not mean that during the development phase of the strategy that you do not employ multiple market testing on the idea.

 

[jm99]

I would expect that a robust one would work great on many and decent on many and poor on some.

re testing on multiple markets.......with a robust strat that perform as above for example is it acceptable, in your opinion, to optimize the inputs on the poorer performing markets to improve the performance or is this just another way of overfitting

 

[StratOpt]

If you are going to be trading that market, then yes you could optimize on that market to try and improve it just so long as you adhere strictly to a walk forward testing process and then watch the real time results for a bit of time too. If you are not going to be trading that particular market that it didn't do so well on then you are only spending time where you could be spending it elsewhere more productively. Your testing process isn't about finding the absolute best set of parameters for the in period, it is about finding the most realistic expectations of the strategy and the process used to trade it for real time. Simply optimizing something to see how well it could do over the testing period serves perhaps to give one a false sense of optimism, but does little for expanding your bank account

 

[Chorlton]

Hello StratOpt,

I've been reading all the replies to my original question including your comments about Walk-Foward Testing and consequently I would welcome your thoughts on the approach which I was planning to take.

Currently I am developing a system which will trade the FTSE250. I am currently backtesting it on my in-sample data range from Jan1998 to Dec2004 inclusive. I chose this range as it included a Ranging Period, a Bear Period and (more recently in this testing period) a Bull Period.

Following completion of any optimization which I am performing over this date range, I intend to walk-forward the system as follows:

1. Shift the testing period forward by 1 year (ie. change the Range Period from 1999 to 2005 (same overall period of 7 years, but now includes 1 year of Out-of-Sample data) and re-test.
2. If the system remains robust & profitable, move directly onto step 4 otherwise step 3.
3. Re-optimize the System based on this new data range. (This last year of Out-Sample data now becomes In-Sample)
4. Shift the testing period forward by 1 year again (This time from 2000 to 2006) and re-test.
5. If the system remains robust & profitable, move directly onto step 7 otherwise step 6.
6. Re-optimize the System based on this new data range.
7. Shift the testing period forward by 1 year again (This time from 2001 to 2007) and re-test.
8. If the system remains robust & proftable, move to step 10, otherwise step 9.
9. Re-optimize the System based on this new data range.
10. If NO re-optimisation took place over the last 2 backtests, then switch the system on and Trade LIVE. If any re-optimization took place over the last 2 backtests, then the System should then be Paper-Traded for a period of 3 months before going LIVE.

Would this be a practical solution?

regards,

Chorlton

 

[StratOpt]

The process that you describe is a sound one. For your in period length of study you just need to use enough data to gain no less than 30 trades within the testing period. More trades is better in this case. I have typically found that a good ratio of in period to out period will be around 12 parts in period to one part out period. This means that your out period will be about one month per year of in period. So if you are using 6 years of in period, a good place to start would be at 6 months of out period. The out period length will determine your periodicity of re-optimizations. There is such a thing as an optimum in period to out period ratio.

If you get to the end of your out period steps and your walk forward efficiency rating is high enough to warrant that this strategy does indeed pass the walk forward analysis, and the results of the out period are adequate to suit your trading needs, e.g it makes more money than you could get with an alternative investment, then at this point your strategy is ready to go live. I would watch the system run live for a period of time to make sure that it is behaving properly when facing live data and that all trades are being executed as desired. When I was satisfied with this then I would go through my optimization process up to current date and then I would begin to follow my optimization process going forward in time. The walk forward analysis, when using enough numbers of walks, will tell the tale of strategy do or strategy die.

For the example I gave above that would mean that I would use those parameters chosen by my particular fitness function for the next 6 months of time at which point I would then perform a re-optimization of the strategy following the exact same procedure that I did before. Same granularity of parameter steps. Same fitness process. Same end selection process. You may end up with the same parameters from one step to the next, but that is fine. It just means that what you are doing is still the best thing to do. It is a routine to get into.

 

[Chorlton]

Hello StratOpt,

Firstly, Thanks for taking the time to reply!!!

The process that you describe is a sound one. For your in period length of study you just need to use enough data to gain no less than 30 trades within the testing period. More trades is better in this case. I have typically found that a good ratio of in period to out period will be around 12 parts in period to one part out period. This means that your out period will be about one month per year of in period. So if you are using 6 years of in period, a good place to start would be at 6 months of out period. The out period length will determine your periodicity of re-optimizations. There is such a thing as an optimum in period to out period ratio.

Wouldn't this depend on the timeframe for the strategy being employed and the fact that the walk-forward test included both previous "in-sample" and "out-sample" data?? For example, if the walk-forward test only included 6 months of "out-sample" then IMO this would not be enough considering the strategy is Weekly.

If you get to the end of your out period steps and your walk forward efficiency rating is high enough to warrant that this strategy does indeed pass the walk forward analysis, and the results of the out period are adequate to suit your trading needs, e.g it makes more money than you could get with an alternative investment, then at this point your strategy is ready to go live.

Probably a daft question but what would you compare this with? Bank % ROR over the same period or a "Buy & Hold" type strategy? If its the latter, then how does one calculate this?

I would watch the system run live for a period of time to make sure that it is behaving properly when facing live data and that all trades are being executed as desired. When I was satisfied with this then I would go through my optimization process up to current date and then I would begin to follow my optimization process going forward in time. The walk forward analysis, when using enough numbers of walks, will tell the tale of strategy do or strategy die.

Sorry, but I'm confused now. After watching the system take live signals, then (assuming that everything was still ok), wouldn't you commit real money to it??

Instead, why would you go back and yet again optimize.? Surely, by doing so this invalidates the last step and results in further over-fitting of the system?

I've probably misunderstood your method and would welcome further clarification.


Thanks,

Chorlton

 

[StratOpt]

Bank interest, T Bills, money markets, etc are some things that you will compare to. Holding stocks and bonds long term, real estate and appreciable items are some others. If your trading strategy can't out perform those then you have to question why you are going to be trading it.

Assuming that your strategy has already passed through the walk forward in good shape then your final step will be to re-optimize up to current and then begin trading. You are not in the danger of over fitting because your strategy and process of testing has already shown you that you have not overfitted or else it would not have passed the walk forward testing. So, at that point you will optimize up to include the most recent data to get your strategy up to date and then you will begin to follow the same process used in the testing in live trading.

Don't confuse optimization with overfitting. While one can certainly lead to the other this is not necessarily the case and will likely not be the case if you follow sound testing procedures without variance. The walk forward testing will show you if you are overfitting or not. Also, just because you have a strategy that fails your first walk forward testing does not mean that the strategy is a failure. It could simply be that the fitness and selection process that you are evaluating is one that is highly prone to being able to find an overfit with that particular trading logic. Using Highest Net Profit over a period is an example. If one were to use a different fitness criteria or simple selected a net profit value that was a Nth value down the list each time, then the strategy may actually do much better.

 

[jm99]

stratopt, in your experience, what % of strats that pass the backtest and walk forward continue to perform as expected in real time??

 

[StratOpt]

I personally expect to gain a very high percentage of what is shown during the out periods of a properly executed walk forward testing from all strategies that pass through the process. I say properly executed testing, because there are some things that you can do that will make your testing improper. One of the ones that I see people do the most is that they will optimize their strategy and use no costs of trading and then when they test it they get a good result but then in real life they realized that there is no such thing as a free lunch when it comes to commissions and slippage and their strategy begins to falter from testing. Another is that the testing that they do uses limits and in real life those limits did not get filled and the trades get missed. The worst of them all that I come across, speaking of TradeStation testing, is somebody will run their strategy on a large tick count tick chart, use no costs, and then use profit targets, trailing stops, and trade very fast. This is a near perfect recipe for failure.

I also believe that strategies have a shelf life. I also believe that the faster your strategy trades the shorter its life span is going to be. This is why you must use not only a strategy stop loss, but also a system wide stop loss and this is the orginal purpose of this topic is how to switch the system off.

 

[Chorlton]

Bank interest, T Bills, money markets, etc are some things that you will compare to. Holding stocks and bonds long term, real estate and appreciable items are some others. If your trading strategy can't out perform those then you have to question why you are going to be trading it.

Assuming that your strategy has already passed through the walk forward in good shape then your final step will be to re-optimize up to current and then begin trading. You are not in the danger of over fitting because your strategy and process of testing has already shown you that you have not overfitted or else it would not have passed the walk forward testing. So, at that point you will optimize up to include the most recent data to get your strategy up to date and then you will begin to follow the same process used in the testing in live trading.

Don't confuse optimization with overfitting. While one can certainly lead to the other this is not necessarily the case and will likely not be the case if you follow sound testing procedures without variance. The walk forward testing will show you if you are overfitting or not. Also, just because you have a strategy that fails your first walk forward testing does not mean that the strategy is a failure. It could simply be that the fitness and selection process that you are evaluating is one that is highly prone to being able to find an overfit with that particular trading logic. Using Highest Net Profit over a period is an example. If one were to use a different fitness criteria or simple selected a net profit value that was a Nth value down the list each time, then the strategy may actually do much better.

StratOpt,

As always, Thanks for your comments!!