Back-testing

[JTrader]

Hi

Is it better to backtest using tick data in minute-by-minute format, or 1 minute data in OHLC format?

What are the advantages of one data type over the other?

Under what circumstances is it good to choose one over the other type of data?

I am thinking specifically about data for backtesting spot forex, futures forex and emini futures.


Many thanks

jtrader.

 

[Doctor Leo]

As far as I know, Tradestation simulates the price movements within bars if you have the initial data in OHLC format and force the program to test tick-by-tcik. So the vast majority of disappointments on real trading after testing the strategy in this manner comes from the desribed issue. I wouldn't recommend backtesting with tick resolution. Moreover, for spot market I would strongly recommend using only close values for entries since usage of stop and limit orders is really quite obscure. And if you mean trying to backtest any scalping strategy where every minute or even tick datum is sensitive, then Omega is not the best place to do it.
Everything above is only IMHO for sure.

 

[JTrader]

Thanks

Doctor Leo

I intend to back-test my system/s and trade between probably the time intervals of 5-30 minute charts - not scalping - but intraday momentum trading.

Cheers

jtrader.

 

[Doctor Leo]

Thanks

Doctor Leo

I intend to back-test my system/s and trade between probably the time intervals of 5-30 minute charts - not scalping - but intraday momentum trading.

Cheers

jtrader.

Me advice remains the same: do not try to test with tick resolution. Use OHLC values, and you'll be satisfied with real trading. And as to 5-minute timeframe, be very careful, for it's almost impossible to imitate some real market situations, as slippage for instance, and if you're going to trade spot forex it's a tough task to backtest a 5-minute-based strategy properly.

 

[JTrader]

Thanks

Doctor Leo

are you suggesting that 5 minutes is too small a time interval to back-test on and that I should look to go higher?

When would tick data be useful for back-testing? for back-testing on scalping strategies only, that would be traded on a tick chart only? or also on a low time interval (say 1 minute) chart?

Cheers

jtrader.

 

[ardhill]

When I started testing on Tradestation I was disappointed when the live trading did not show the same profit that the back testing did.

After a while I discovered the problem...

My strategies were for intra day trades which had a trailing stop. As I had not told Tradestation to test by the tick, it used the complete bar and the actual live results differed quite a bit as they are formed tick by tick and not on complete bars.

Now, I tell TS to test tick by tick. You can still do this even when you stipulate the bar length you want to use eg I can test on 5 minute bars, but have tradestation look at the bars tick by tick.

It takes longer, but what does that matter when it is the difference between a profitable strategy and a looser.

So my advice...

If your strategy may include data which is small enough to be influenced by the tick values within a bar - then have your back testing program look at the chart tick by tick.

 

[JTrader]

ardhill -
So my advice...

If your strategy may include data which is small enough to be influenced by the tick values within a bar - then have your back testing program look at the chart tick by tick.

Thanks ardhill.

Is it only possible to do this if you are using tick data?
or can i get ts2000i to look at the chart tick by tick if the data is in 1-minute OHLC format? What data format must the data be in to back-test?

Many thanks

jtrader.

 

[ardhill]

You should be able to use OHLC bars if you wish on whichever time frame suits your trading - 1 min, 5 min, whatever.

I do not use 2000i but am a brokerage client of Tradestation, so I use Tradestation 8.

I can't tell if the procedure is the same in 2000i, but for me to get TS to look at a chart tick by tick for testing I...

Set up the chart in whatever way I wish, for example 5min OHLC bars and set my strategy up that I wish to back test.

Then I right click and choose 'Format Strategy'. The strategy window opens. I click on the 'Properties' button. Then under the tab 'General', the section 'Back-Testing Resolution', I tick the box to 'Use Look-Inside-Bar Back-testing', then select 'Tick' as the inside value.

Click OK, then close and the data is downloaded into my machine. After that when I run a performance report, it has taken into account the tick movements in my 5 minute bars.

 

[Doctor Leo]

When would tick data be useful for back-testing?

Only when you have REAL tick data and do not force Omega to emulate the price behaviour. I cannot agree that really good results may come from backtesting on such small movements in tradestation, although I have several strategies based on tick data, but I use quite different software to analyze them and for me it was completely impossible to translate the code into Easy Language.

 

[JTrader]

Thanks guys .

How is it possible to back-test/analyse OHLC time interval data on a tick-by-tick basis? Won't it be the case that there will inherently not be individual tick prices, but just 4 individual prices for the OHLC?

Thanks again

jtrader.

 

[bdholmes]

I backtest around 4 years of intraday data in 2000i. I found that tick-backtesting took too long so I backtest using 1 minute bars instead. I found results to be very similar, so until I can afford one of those Cray supercomputers, this will do fine.

 

[JTrader]

ardhill -
Then I right click and choose 'Format Strategy'. The strategy window opens. I click on the 'Properties' button. Then under the tab 'General', the section 'Back-Testing Resolution', I tick the box to 'Use Look-Inside-Bar Back-testing', then select 'Tick' as the inside value.

Hi

I thought that this facility (tick the box to 'Use Look-Inside-Bar Back-testing', then select 'Tick' as the inside value.) was available in 2000i - but it does not seem to be available by following your instructions from TS8. Under - properties - backtesting settings - strategy testing resolution can either be set to tick or minute or day.

This option does not appear in the format strategy window. Perhaps it is elsewhere??

Thanks again

jtrader.

 

[JTrader]

Tradestation 2000i help files -
TradeStation can evaluate a strategy as each tick is collected. However, when you create a chart using data that has already been collected (historical data), TradeStation must make assumptions about price movement. Therefore, TradeStation implements a feature that enables you to specify a strategy testing resolution so that it does not need to make assumptions.

For example, when plotting a 5-minute chart, you can specify that TradeStation back-test your strategy using 1-tick bars, 1-minute bars, or 5-minute bars.

{ ** © 1987, 1999 Omega Research, Inc. ** }

Hi

it seems that ts2000i can do look inside the bar/tick-by-tick back-testing.

My one remaining question is - how can inside the bar/tick-by-tick assumptions be made of OHL&C say 1 minute data in back-testing? Because there are no individual ticks - just OHL&C prices for each 1 minute period. :?:

Many thanks

jtrader.

 

[JTrader]

Another ts2000i backtesting question!

Tradestation 2000i help files -

If you collect and store only daily data, you will only be able to specify a data compression of daily, weekly, or monthly. If you use, collect and store tick data, you’ll be able to specify any data compression.

{** © 1987-1999 Omega Research, Inc. **}

The first sentence makes sense - as it would if it said that If you collect and store only 5 minute data, you will only be able to specify a data compression of 5 minutes and above.

This question kind of refers to the opposite of my question in my preceeding post..............

With regard to the second sentence - Can tradestation compress/use tick-by-tick data into/or in the same way as time interval data, so that it can be used in the same way as 1 or 5 minute OHLC data for example?


Thanks again

jtrader.

 

[ardhill]

Hi JTrader

I think that this is what you are asking...

The charts are formed by your OHLC 1 minute bars and your indicators are based on the same, but if you were watching the chart live, you would see that a bar is made up of various price movements that can sometime be quite erratic. A 1 min bar may be 20c from high to low and have the close 1c from the high, but during that 1 min, the price may have fluctuated the full 20c up and down until the 1 minute was up and the bar was formed from the 'snapshot' and the beginning and end of the 1 minute.

What I believe Tradestation does, is when looking at the price movement, not indicators et all, but the price movement, sees the erratic nature of the formation of the bar. This is why the difference I had when I first noticed the difference in my trailing stop.

I may have had a 15c trailing stop that looked like it should have been fine on the example above if you had looked at it after it had formed (the 20c bar with close 1c off the high). But if the tick movement showed that the price jumped from open to the high, down the 20c to the opening price again, before build back up again, then my 15c trailing stop would be triggered in that bar in real time.

I hope that make sense - it does in my head

 

[JTrader]

ardhill
but if you were watching the chart live, you would see that a bar is made up of various price movements that can sometime be quite erratic. A 1 min bar may be 20c from high to low and have the close 1c from the high, but during that 1 min, the price may have fluctuated the full 20c up and down until the 1 minute was up and the bar was formed from the 'snapshot' and the beginning and end of the 1 minute.

What I believe Tradestation does, is when looking at the price movement, not indicators et all, but the price movement, sees the erratic nature of the formation of the bar. This is why the difference I had when I first noticed the difference in my trailing stop.

Thanks ardhill

I understand that this is what ts2000i can do with real-time data - tick-by-tick evaluation of OHLC time interval bars.

However, when talking about purely historical data in say 1-minute OHLC format ts2000i cannot evaluate the tick action within a bar - as it only has the OHLC prices available.

Does it guess in which order the OHLC were hit when back-testing a strategy with historical OHLC time interval data for the purposes of evaluating whether the stop-loss has been hit? if so, do you know how?

Many thanks

jtrader.

 

[fowkesp]

jtrader,

In my experience, you should be fine using 1-min OHLC raw data and setting the strategy back-testing resolution to "1 minute". This is on the assumption you are using 5 to 30 min bars on the chart and using stop and/or limit orders for entry and exit.

When using OHLC bars, TS2000i has to make an assumption as to what order the prices occurred during the bar, and this assumption fairly obviously can affect the outcome of back-testing a strategy. If you think about it though, this only really matters if your entry and exit both occur during the same bar. If the entry and exit are in different bars, then it doesn't matter what order the prices actually occurred.

So, as long as your chosen trading strategy is unlikely to both enter and exit within a single 1-minute bar, you will gain very little by choosing to use tick resolution. It is of course, quite possible to use raw tick-data and specify a 1-minute back-testing resolution. It's much faster than using tick-resolution, and in my experience gives almost exactly the same results.

When you actually run the strategy live, you will of course be using tick-data resolution for the strategy, whatever you specify in the back-testing resolution box.

Paul

 

[JTrader]

fowkesp -
In my experience, you should be fine using 1-min OHLC raw data and setting the strategy back-testing resolution to "1 minute". This is on the assumption you are using 5 to 30 min bars on the chart.

Thanks fowkesp

Why is this on the assumption you are using 5 to 30 min bars on the chart?

Do you know exactly how ts2000i makes assumptions on the order of price movements within a OHLC 1-minute bar with purely historical data?

I'm not exactly sure how ts2000i can compress purely tick data into OHLC bars, given that tick data is presented in bid and offer, or mid price format?? But I guess the solution to this confusion is to stick to OHLC data!

Thanks again

jtrader.

 

[fowkesp]

Thanks fowkesp

Why is this on the assumption you are using 5 to 30 min bars on the chart?

Do you know exactly how ts2000i makes assumptions on the order of price movements within a OHLC 1-minute bar with purely historical data?

I'm not exactly sure how ts2000i can compress purely tick data into OHLC bars, given that tick data is presented in bid and offer, or mid price format?? But I guess the solution to this confusion is to stick to OHLC data!

Thanks again

jtrader.

jtrader,

Sorry, what I really meant was that I was assuming you were using chart bars that were a fair bit longer than the raw data available. If you are using 1-minute raw data, then any, chart from 5 min upwards gives identical results in the testing I have done. If you drop down to 1 or 2 minute charts, then I have found that I can get different back-testing results depending on whether I specify tick, or 1 or 2 minute resolution. Not a lot of difference, mind you, but it's there.

I don't believe TS2000i supports bid/offer distinction in back-testing - a bit of a shortcoming that can be got around by specifying realistic amounts in the commission and slippage boxes. You must at least fill in the commission box, and I encourage you to put in a figure that is over the top, because many strategies that at first sight appear profitable are nothing of the sort once you account for commmision and slippage. If you use optimisation, then this can catch you out badly. Always specify a commission or slippage figure and over-estimate it by a fair bit before optimising. This way the optimisation will not home in on the strategies that only work if the slippage and commission costs are negligible.

The TS2000i Help files have a good explanation of the assumptions made about price movements within a bar if the raw data is unavailable - Omega call this this "bouncing tick" method, and you can override the amount of "bounce" somewhere or other. However, if you use Stop and Limit orders, and you always enter and exit on separate bars, this is of no consequence. The price movement during a single bar must be less than your Stop or Profit Target for this to work, of course.

Paul

 

[Trader333]

The way I have found TS2000i to work on backtesting (and in realtime situations) is that it will only trigger entries, stops or other requirements at the close of the bar and this is where the problem lies. If you had a stop order at a set price, then price action may go straight through it and the trigger to exit would only happen once the bar had closed. This is the problem with using time intervals as opposed to tick data with TS2000i. I have not done too much work using tick data so I am unable to say quite how it would differ.


Paul