IB's Data Quality

Currently the IB data packet sent out at best 5 times a second only includes the last price in that 1/5th of a second. The downside of this approach is that when each tick is constructed it only includes the last price within the tick period (as low as 1/5th of a second). This means that a prior higher or lower price might occur but be ignored.

So your highs and lows are suspect for each bar especially if there are a lot of prices printed in a short period. It seems that this problem is biggest with markets like ES and HSI.

Some solve this problem by using another data feed which IMHO weakens IB's franchise with its customers. I would prefer to see IB improve their data feed. Don't get me wrong, I don't want it to be slowed down at all, but including the extra data need have no significant effect on the data.

My suggestion is that the data include another field. If the last price is the only price then thats all thats needed. If its not then the existing price field should include the high and the other field should include the low. The Sierra Chart, QT etc will still need a modification to include the improved data feed but will not be any worse even if they are not "fixed."

What do others think?

What does IB think?

Some solve this problem by using another data feed which IMHO weakens IB's franchise with its customers

Click to expand...

You must be joking, IB are a brokerage first and a data provider very much second. They are pretty much the largest in the World with a highly competitive product. I cannot understand anyone who is serious about trading trying to avoid paying for a quality datafeed. Any person serious about trading needs to tools to do the job and attempting to do this on the cheap will put whoever does at a disadvantage in my view.


Paul

Sounds reasonable Paul but I trade HSI.

The normal "quality" datafeed, esignal, is anything but quality for HSI. So, I'd like to see IB improve.

I've also noticed the charting programs increasingly bad-mouthing ib data and encouraging US users in particular to move to a bunch of alternative brokers who provide supposedly better data feeds etc than IB. As Woodie has done with his uncounted hordes. So I'd think IB would be interested in improving this aspect of their operation.

Some might suggest that getting the basics right was more important than adding another damned bell or whistle to TWS???

Cheers

Kiwi

I was under the impression that data is transmitted 'on change' , and this appears to be confirmed from my observations (ES only).

II use the API to collect data and auto trade, and have never experienced a situation where a stop or limt is triggered and the price has not appeared in the data stream. In addition, published top/bottoms appear to correspond with the data stream.

the following data indicates the "last price" transmitted in the data stream appears to confirm this as there are more than 5 prices fed win the second. (Ignore the 25 - thats my internal ID for ES)

date Time Price
2006-08-14 16:58:38 25 1281.75
2006-08-14 16:58:38 25 1282
2006-08-14 16:58:38 25 1281.75
2006-08-14 16:58:38 25 1282
2006-08-14 16:58:38 25 1281.75
2006-08-14 16:58:38 25 1282
2006-08-14 16:58:38 25 1282.25
2006-08-14 16:58:38 25 1282
2006-08-14 16:58:38 25 1282.25
2006-08-14 16:58:38 25 1282
2006-08-14 16:58:38 25 1282.25
2006-08-14 16:58:38 25 1282

I am sure that IB will provide a definitive answer.

Which application is using the data could the limitation be the interpretation of the data?


Regards
Kevin

On IB's own board this explanation is given:

Generally speaking, IB only sends a message when something changes. So if a trade occurs at the same price and size as the previous trade, IB sends neither price nor size messages (but it may send a volume message).

When the size has changed, but not the price, it sends only the size message.

When the price has changed, but not the size, it sends both price and size messages.

When both the price and the size have changed, it sends both the price and the size messages, and then sends the same size again. I've no idea why, but that seems to be the way it is.

This description applies to index futures. For stocks, it seems to be altogether more bizarre, and I haven't been able to make any sense of what it sends.

I should also point out that IB sends price data at regular intervals. For index futures, the interval seems to be 300 milliseconds, and for stocks it seems to be 200 milliseconds, though this may have changed since I investigated it in early 2004. At the end of each interval, it sends data for the last trade that occurred during that interval, if any. It compares the prices and sizes against the last ones sent. So suppose the last price and size notified were 1187 and 5, and volume was 22841. Then during the next interval there are two trades, one at 1187.25 for 10, and another at 1187 for 10. At the end of that interval, IB would notify only a size of 10 and a volume of 22862, because the price is still the same as the previous price notified. The fact that a trade occurred at 1187.25 doesn't get reported. Note however that the volume has increased by the amount of all the trades during the period.

The great benefit of this scheme is that the total load on IB's market data servers is well-defined, no matter how busy the markets become. As a result, IB's data feed keeps pretty much exactly in step with the market at all times. Compare this with other feeds that send every price, like eSignal. At busy times, like when there's a sudden drop in the DAX or the S&P and all the other index futures follow, there's a huge surge of data and quite a noticeable lag happens as their servers struggle to pump it all out. I've noticed two occasions this week where my eSignal feed has been a good 10 seconds behind the IB feed.

The disadvantage is that because the IB feed misses out some prices, your 1-minute or 5-minute bars (or whatever timeframe you use) might occasionally be slightly inaccurate in their highs and lows. Whether this is a big issue depends on your trading strategy.

Here is an example with HSI. I don't trade ES but have been assured that the same problem exists with ES.

The green data is backfilled from IB's server and includes all highs and lows. The red data is the real time data obtained from TWS and is overlaid with the same scale on top of the backfilled data so you can see the differences.

Its not an application issue as you'll get the same result using TWS charting, Sierra Chart, Quotetracker or Ensign. The pc clock is "accuate" within 100ms and wouldn't explain the variation in peak highs and lows even if it was a problem.

Day before yesterday I had a bad day with the TWS freezing and only the ftse/dax futures updating.Yesterday again the YM was frozen at various periods when my chart was showing price and volume change.The TWS would then 'unfreeze' and jump to the chart price.I've never had this before (When it's lasted so long).

I and a friend in Singapore also had some problems on Friday. Not sure if they were related.

One last request for IB (one I gave up on previously): The backfill still doesn't include transaction information so Tick Charts don't work. I just saw another newbie to IB's plaintive request for help ...

IB, if you should chose to improve the data quality so that bar highs and lows re correct, or even if you don't, could you please add transaction numbers to backfill data?

Anyone who thinks that either

- accurate data including bar highs and lows is important; or
- backfill data should include tick info so tick charts work.

Please register and vote at: http://www.interactivebrokers.com/e...1&ib_entity=llc

jonnyy40 said:

Day before yesterday I had a bad day with the TWS freezing and only the ftse/dax futures updating.Yesterday again the YM was frozen at various periods when my chart was showing price and volume change.The TWS would then 'unfreeze' and jump to the chart price.I've never had this before (When it's lasted so long).

Click to expand...

Jonnyy40 & Kiwi,
You may need to increase the size of your on board RAM, if you have less than 2MB this maybe contributing to your system freeze ( IMHO )
Ive had similar experience sometime ago, and adding more RAM solved it !...........Good luck

Thanks wg ... only 512k

Kiwi said:

One last request for IB (one I gave up on previously): The backfill still doesn't include transaction information so Tick Charts don't work. I just saw another newbie to IB's plaintive request for help ...

IB, if you should chose to improve the data quality so that bar highs and lows re correct, or even if you don't, could you please add transaction numbers to backfill data?

Anyone who thinks that either

- accurate data including bar highs and lows is important; or
- backfill data should include tick info so tick charts work.

Please register and vote at: http://www.interactivebrokers.com/e...1&ib_entity=llc

Click to expand...

Hi Kiwi
The link says "Page not found."
Tried to find your poll using the User Forum and my account, but got nowhere.
Am I doing something wrong ?
Glenn

I've got 1.25Gb and that should be enough IMO.

Thanks Wheezergeezer,

And if you'd like to have the highs and lows of each bar correct (suggestion 1035) or the backfill enhanced to include transaction number so that tick charts still work with backfill data (suggestion 1044) please go and vote at:

http://www.interactivebrokers.com/en/general/poll/poll.php?ib_entity=llc

Glenn, I checked and it seems I posted the link after I had registered. This one should work.

jonnyy40 said:

I've got 1.25Gb and that should be enough IMO.

Click to expand...

He means level 2 cache (nothing to do with Level 2 in trading), not RAM, although I doubt that would have anything to do with the problem.

128kb (everest home edition-what a usefull thing) but it doesn't show current utilisation % so I assume it's not critical?I don't know what it is anyway.

Guys,
I can only say what happened in my experience, I increased from 1MB of RAM to 2mb , It solved my problem, at the time I had 2 monitors, TWS, and sierrachart along with L2, thats a lot of data coming in, More RAM is better..........

Why did it only happen on those 2 days?On another note I had a supposed -1000 euro position on my ideal position/time in force window.I rang up IB europe and he said ignore it our technitians are sorting the problem.It just seems that lately the rock steady nature of TWS is slipping for some reason.I haven't had any bad freezes since those 2 days.According to everest my memory utilisation when running 4 monitors,my charts and TWS is usually 51%.

Thanks to all here who voted on these two issues.

IB are now working to include transaction information in the backfill so we can use it for tick charts and have scheduled implementation of data packets that don't miss any highs and lows! Thanks IB.