testing systems on index data or continuous contract

[belflan]

I've been testing a few YM systems on TS8.4. and have noticed huge differences between results using the dow index and @YM.D (continuous contract).

I'm sure I must be doing something wrong. but can't think what..

can anyone help
thanks

 

[dcraig1]

IMHO, there is no point in testing trading systems for non tradeable instruments (possibly unless those instrument(s) are an input to a system for a tradable instrument). Futures are generally more spiky than their index for a start. Can also be above or below fair value.

I suspect that it is much easier to construct a system for an index than for a futures contract on that index.

In general live index data is much more infrequent than live futures data (unless you constrcut your own fast cash index from the constituent stocks).

 

[Glenn]

Glen
These are my own views after 11+ years of trading:-
a. There's no point in what people call proper backtesting no matter what instrument you use. (i.e. using a contiguous data set) The results are an illusion in terms of the future. Markets are not technical all the time, so any results you get are meaningless. They just happen to look pretty and satisfy the software vendors.
The best you can do is to identify specific portions of a chart and see how your algo performs there. Then you need the ability to read the market on the (live) day and decide what to tell your algo to look for.
b. You should always take your signals from the underlying. Use the Dog, not the Tail. Do you want to bet on the horse or bet on the other punters ?
c. There is no single definitive way to produce a continuous futures chart because of the differences at expiry. But if you trade futs you should be taking the signals from the underlying anyway and therefore you don't need continuous futs charts.

Glenn

 

[belflan]

IMHO, there is no point in testing trading systems for non tradeable instruments (possibly unless those instrument(s) are an input to a system for a tradable instrument). Futures are generally more spiky than their index for a start. Can also be above or below fair value.

can see your point, I'll set up another strat that takes its signal from the index but trades the YM, & see what that looks like.

I suspect that it is much easier to construct a system for an index than for a futures contract on that index.

this seems to be true for almost everything i've tried on the dow



thanks for the reply dc, Btw
belflan

 

[belflan]

Glen
These are my own views after 11+ years of trading:-
a. There's no point in what people call proper backtesting no matter what instrument you use. (i.e. using a contiguous data set) The results are an illusion in terms of the future. Markets are not technical all the time, so any results you get are meaningless. They just happen to look pretty and satisfy the software vendors.
The best you can do is to identify specific portions of a chart and see how your algo performs there. Then you need the ability to read the market on the (live) day and decide what to tell your algo to look for.

do you not believe a 'stable', 'robust' system that has been back tested, out of sample tested and then forward tested could continue to be profitable in the future..

b. You should always take your signals from the underlying. Use the Dog, not the Tail. Do you want to bet on the horse or bet on the other punters ?

c. There is no single definitive way to produce a continuous futures chart because of the differences at expiry. But if you trade futs you should be taking the signals from the underlying anyway and therefore you don't need continuous futs charts.

agree, I'll set it up so the strat takes signals from the index but trades YM. (for back testing and out of sample I'll have to use the continuous YM chart, but I'll use actual ym for forward test.)

thansk for the reply glenn
belflan

 

[Glenn]

do you not believe a 'stable', 'robust' system that has been back tested, out of sample tested and then forward tested could continue to be profitable in the future..

agree, I'll set it up so the strat takes signals from the index but trades YM. (for back testing and out of sample I'll have to use the continuous YM chart, but I'll use actual ym for forward test.)

thansk for the reply glenn
belflan

I was trying to qualify the word backtesting. If you run tests on (say) several years continuous data, then my view is that it's a waste of time for the reasons I mentioned.
The same applies to forward testing. e.g. If you have a range-trading algo and you test it in a trending market what do you think will happen ? .. and vice versa.
You need to test a trend-trading system in a trending market and test a range trading system in a ranging market. That means dividing your historical data up into a number of different chunks.

Not sure what you mean by stable and robust. Systems can appear to be that in hindsight, but that means nothing imo.

If you are going to use a mixture of underlying and continuous futs charts for testing then again I think you are wasting your time. Just throwing a lot of data at it won't help.

If you are hoping that your algo will, by itelf, determine the market conditions on the day, and trade them accordingly then I really do think you are being too optimistic. That's how the Hedge Funds came unstuck - no "adult supervision".

Unfortunately I guess I'm saying things that you don't want to hear.
Glenn

 

[belflan]

I was trying to qualify the word backtesting. If you run tests on (say) several years continuous data, then my view is that it's a waste of time for the reasons I mentioned.
The same applies to forward testing. e.g. If you have a range-trading algo and you test it in a trending market what do you think will happen ? .. and vice versa.
You need to test a trend-trading system in a trending market and test a range trading system in a ranging market. That means dividing your historical data up into a number of different chunks.

Not sure what you mean by stable and robust. Systems can appear to be that in hindsight, but that means nothing imo.

If you are going to use a mixture of underlying and continuous futs charts for testing then again I think you are wasting your time. Just throwing a lot of data at it won't help.

If you are hoping that your algo will, by itelf, determine the market conditions on the day, and trade them accordingly then I really do think you are being too optimistic. That's how the Hedge Funds came unstuck - no "adult supervision".

Unfortunately I guess I'm saying things that you don't want to hear.
Glenn

I though this would be worth investigating after reading a Ralph Vince book on portfolio management and after reading the threads\posts of Mathemagician and TWI of this BB who seem to have success with trading a Diversified portfolio of mechanical futures systems.

But your comments and experiences with back testing are certainly making me think twice about going down this route as I have alot of respect for your views.

 

[Adamus]

if you do carry out that strategy and fail, it doesn't mean it's not going to work for someone else.

and if you don't carry out that strategy but make your money some other way, you've got no experience to judge it by.

the reason why I will 'throw data' at a mechanical trading system is because my trading systems do not rely on the market being a particular shade of whatever to have an edge.

There are ways of categorising market behaviour that lie beyond the narrow definition of ranging and trending.

 

[Glenn]

There are ways of categorising market behaviour that lie beyond the narrow definition of ranging and trending.

I used Trending and Ranging as examples of market types, but I did not say that they were the only ones.

Glenn

 

[StratOpt]

The proper backtesting and forward testing of strategies is the most critical aspect of strategy development. It is also the method that will gain or save you the most money. If you have a strategy that tests well then this is no guarantee of future success, but a strategy that does not pass the testing is going to fail in real time with near certainty. I will whole heartedly disagree with anyone who states that this sort of testing is a waste of time.

You can't realistically use the index as a proxy to the futures on daytrading types of strategies that have a lot of trading, but for slower trading strategies that hold positions for a longer period of time you can get a fair approximation to reality by using the index.

 

[jm99]

I've been testing a few YM systems on TS8.4. and have noticed huge differences between results using the dow index and @YM.D (continuous contract).

I'm sure I must be doing something wrong. but can't think what..

can anyone help
thanks

whenever i've had this in the past, it has been due to the index not accurately reflecting gaps

 

[jm99]

whenever i've had this in the past, it has been due to the index not accurately reflecting gaps

....looking a bit closer at the charts u posted though this does not appear to be the issue. yr system looks like it not consistently generating like of like signals eg on 29th ss generated for cash, but not futures.....without looking at the code its impossible to have any useful input unfortunately

 

[belflan]

....looking a bit closer at the charts u posted though this does not appear to be the issue. yr system looks like it not consistently generating like of like signals eg on 29th ss generated for cash, but not futures.....without looking at the code its impossible to have any useful input unfortunately

the signals are generated by a very simple strat (i can post the code if you like, it's a standard ts code). with the same settings for both YM and the index. i was expecting slightly different entries\exit, but for the overall results to be simular.. this is why i'm thinking I'm doing something fundamentally wrong in TS.

 

[jm99]

the signals are generated by a very simple strat (i can post the code if you like, it's a standard ts code). with the same settings for both YM and the index. i was expecting slightly different entries\exit, but for the overall results to be simular.. this is why i'm thinking I'm doing something fundamentally wrong in TS.

ok, if u want to post the code or let me know which strats and the settings (assuming u mean u r using a ts in built strat) i'll take a look

 

[omnpmh]

look at this:

http://www.automated-trading-system...ht-to-know-about-continous-futures-contracts/