The text below is copied from a PM I recently sent to mmillar, followed by his response. This should prove useful to anyone contemplating what software to utilise for backtesting their strategies.
quote:
--------------------------------------------------------------------------------
Simon wrote on 09-06-2003 08:35 PM:
Martin,
With reference to your 'updated' review of eSignal, would you mind elaborating on its unsuitability for backtesting? I have recently taken out a subscription to this service & although I have no immediate need for a backtesting facility, I do anticipate having a requirement for it in the future.
Furthermore, I noticed in your 'profile' that you have previously backtested strategies with eSignal, yet why no longer? Has the service been reduced/downgraded, are the strategies you are studying too advanced to be tested on eSignal, etc.? Your thoughts would be greatly appreciated.
Many-thanks in anticipation.
Regards,
Simon
Re: eSignal
Hi Simon,
Sorry I didn't get back to you sooner - I've been away for a few days. This is a long one.
eSignal is a great real time data feed and a great charting package - in fact I think it is the best - but backtesting is just a big no-no. I wasted several months learning efs and backtesting strategies before I realised what the true picture was. I am now seriously pissed off with eSignal. I am still using them for real time data but will probably switch to TradeStation as soon as they release their next version.
1. You can currently only backtest on 60 days of historical data. This is shortly to be increased to 100 days. This is nowhere near enough for backtesting. eSignal's plan is to eventually get to 2 years of backtest data but they wont buy data in, and they only started collecting data in Feb 2003, so it will be 2005 at the earliest before this 2 years of data becomes available. Compare this with TradeStation 7.x which currently allows 10/20 years of backtesting, or TradeStation 2000i which allows unlimited backtesting.
2. Time Zone problems. I am sure you are aware of the general issue about changes in Time Zones. In the UK we change from GMT to BST (and back again) a week before the US make their equivalent change. It means that instead of the US opening at 2.30pm UK time it opens during the changeover week at 3.30pm. You can have problems if you have time conditions around your trading such as ‘Don’t trade before 3.00pm’ or ‘Close at 8.30pm’. Data providers are supposed to normalize this data so you don’t get these problems but eSignal don’t. It means for two weeks per year eSignal will produce rubbish backtesting results.
3. Data holes. If there is data missing from an instrument, for whatever reason, it is ‘technically impossible’ to backfill that data. As an example on the EuroStoxx50 (ET M3-DT) on the morning of Monday 19th May 2003 there is 40 mins of data missing. I asked why this was and if eSignal could backfill it. They told me their servers had crashed that morning – which is fair enough, it happens to the best of us – but that they could not fix it. I have no idea how often this issue happens but for them not to be able to fix it is unacceptable to me.
4. 'Zero Volume Bar' problem. I regard this problem as a major issue, but eSignal seem to regard it as a minor technical glitch that they have no plans to fix. If you bring up three Advanced Charts – one for $COMPQ, one for ‘YM #F’ and one for ‘ES #F=2’, set them all to 5 min bars. Create the following function and apply it to all three charts.
function main() {
return close("$COMPQ");
}
Go back to 29th April at 3.00pm. According to the $COMPQ chart the $COMPQ closed at 1462.21. According to the YM #F chart the $COMPQ closed at 1458.69. According to the ES #F=2 chart the $COMPQ closed at 1521.27.
Although eSignal has the correct data (1462.21) they are unable to return it correctly in all instances. When you backtest one symbol against another it means the results you get are basically random!
To backtest a system and then use it for trading you have to have 100% confidence in what you have done. Because of the above issues I have zero confidence in eSignal. Maybe the issues I have are big, maybe they are small, but with no confidence in eSignal’s data I cannot have any confidence in the systems I have produced. I have had to junk all the work I did with eSignal over the past few months and am currently rewriting everything in TradeStation’s EasyLanguage.
HTH. If you have any other questions feel free to ask.
Cheers
Martin
quote:
--------------------------------------------------------------------------------
Simon wrote on 09-06-2003 08:35 PM:
Martin,
With reference to your 'updated' review of eSignal, would you mind elaborating on its unsuitability for backtesting? I have recently taken out a subscription to this service & although I have no immediate need for a backtesting facility, I do anticipate having a requirement for it in the future.
Furthermore, I noticed in your 'profile' that you have previously backtested strategies with eSignal, yet why no longer? Has the service been reduced/downgraded, are the strategies you are studying too advanced to be tested on eSignal, etc.? Your thoughts would be greatly appreciated.
Many-thanks in anticipation.
Regards,
Simon
Re: eSignal
Hi Simon,
Sorry I didn't get back to you sooner - I've been away for a few days. This is a long one.
eSignal is a great real time data feed and a great charting package - in fact I think it is the best - but backtesting is just a big no-no. I wasted several months learning efs and backtesting strategies before I realised what the true picture was. I am now seriously pissed off with eSignal. I am still using them for real time data but will probably switch to TradeStation as soon as they release their next version.
1. You can currently only backtest on 60 days of historical data. This is shortly to be increased to 100 days. This is nowhere near enough for backtesting. eSignal's plan is to eventually get to 2 years of backtest data but they wont buy data in, and they only started collecting data in Feb 2003, so it will be 2005 at the earliest before this 2 years of data becomes available. Compare this with TradeStation 7.x which currently allows 10/20 years of backtesting, or TradeStation 2000i which allows unlimited backtesting.
2. Time Zone problems. I am sure you are aware of the general issue about changes in Time Zones. In the UK we change from GMT to BST (and back again) a week before the US make their equivalent change. It means that instead of the US opening at 2.30pm UK time it opens during the changeover week at 3.30pm. You can have problems if you have time conditions around your trading such as ‘Don’t trade before 3.00pm’ or ‘Close at 8.30pm’. Data providers are supposed to normalize this data so you don’t get these problems but eSignal don’t. It means for two weeks per year eSignal will produce rubbish backtesting results.
3. Data holes. If there is data missing from an instrument, for whatever reason, it is ‘technically impossible’ to backfill that data. As an example on the EuroStoxx50 (ET M3-DT) on the morning of Monday 19th May 2003 there is 40 mins of data missing. I asked why this was and if eSignal could backfill it. They told me their servers had crashed that morning – which is fair enough, it happens to the best of us – but that they could not fix it. I have no idea how often this issue happens but for them not to be able to fix it is unacceptable to me.
4. 'Zero Volume Bar' problem. I regard this problem as a major issue, but eSignal seem to regard it as a minor technical glitch that they have no plans to fix. If you bring up three Advanced Charts – one for $COMPQ, one for ‘YM #F’ and one for ‘ES #F=2’, set them all to 5 min bars. Create the following function and apply it to all three charts.
function main() {
return close("$COMPQ");
}
Go back to 29th April at 3.00pm. According to the $COMPQ chart the $COMPQ closed at 1462.21. According to the YM #F chart the $COMPQ closed at 1458.69. According to the ES #F=2 chart the $COMPQ closed at 1521.27.
Although eSignal has the correct data (1462.21) they are unable to return it correctly in all instances. When you backtest one symbol against another it means the results you get are basically random!
To backtest a system and then use it for trading you have to have 100% confidence in what you have done. Because of the above issues I have zero confidence in eSignal. Maybe the issues I have are big, maybe they are small, but with no confidence in eSignal’s data I cannot have any confidence in the systems I have produced. I have had to junk all the work I did with eSignal over the past few months and am currently rewriting everything in TradeStation’s EasyLanguage.
HTH. If you have any other questions feel free to ask.
Cheers
Martin
