Data Integration

[cozgenc]

Hello,

I am trying to build a small home based algorithmic trading system. One of the challenges I am facing so far is to combine the data pool for cleaned and standardized historical data with the real time data feed. There are issues like

1) When is end-of-day? At what point the data should be moved to historical data pool?
2) How should strategies refer to historical data and real-time data? Through a single API or multiple APIs?
3) How protect trading strategies from stock-split or futures' backwardization adjustments?

Thanks in advance

 

[NetTecture]

Hm:

1: when no more trades arrive. You can maintain ametadatabase of trading hours, though, and you may need to - so you can know when EOD is in the trading algorythm (and close positions before).
2: name me one reason for multiple API's I have one in the system I build, just different parameters.
3: Not at all. They need to take that into account. One thing I really hate here is classical systems: they are ignorant. A good strategy runner would take things like futures delivery dartes into account, know when to change futures and put that into the simulation / his behavior (i.e. changing actively). Stock splits - are totally transparent, just put up a "corrected" price. Not that stock splits are neutral - basically in 2:1 the price goes down, but the shares up, so the value is the same. You can just half all prices historically and be done with that

 

[Adamus]

1: all exchange-traded markets have market closing times defined by the exchange.

 

[cozgenc]

1: all exchange-traded markets have market closing times defined by the exchange.

That's really not my point. How good is to mix and match EOD of Nikkei and EOD of CBOT in a trading strategy? There is probably 9-10 hours of difference. Maybe one should come up with a lowest common denominator approach to EOD, a fixed time for all exchanges, which would reflect the most accurate prices at that point in time.

 

[cozgenc]

Hm:

1: when no more trades arrive. You can maintain ametadatabase of trading hours, though, and you may need to - so you can know when EOD is in the trading algorythm (and close positions before).
2: name me one reason for multiple API's I have one in the system I build, just different parameters.
3: Not at all. They need to take that into account. One thing I really hate here is classical systems: they are ignorant. A good strategy runner would take things like futures delivery dartes into account, know when to change futures and put that into the simulation / his behavior (i.e. changing actively). Stock splits - are totally transparent, just put up a "corrected" price. Not that stock splits are neutral - basically in 2:1 the price goes down, but the shares up, so the value is the same. You can just half all prices historically and be done with that

Suppose that you have 1 API. How would you calculate Moving Average of last 120 minutes for example when half of the data is in your deep storage and half is coming from the live data feed? Are you going to keep a temporary buffer for live data? How do you standardize timestamps between your deep storage and live data if your deep storage and live feeds are from different brokers, for example?

No, stock splits are not transparent. Your risk management module, if you have any, should warn you constantly on invalidated strategies that refer to absolute prices in case of splits, dividend adjustments, or futures' contract rolls. Combine that difficulty with the not yet adjusted deep storage for the previous day with the new values coming through the live data problem.

 

[Adamus]

That's really not my point. How good is to mix and match EOD of Nikkei and EOD of CBOT in a trading strategy? There is probably 9-10 hours of difference. Maybe one should come up with a lowest common denominator approach to EOD, a fixed time for all exchanges, which would reflect the most accurate prices at that point in time.

I'm not getting your point. If the market is closed, the price will be the same as it was when it closed.

Are you basing your trading signals for one market on activity in other markets?

 

[NetTecture]

He probably is. Stocs going up in Asia indicate a possible strong stock market in the US later on. You see that pretty often. As such, market close in Asia may be an indication for market opening in the US.

 

[cozgenc]

He probably is. Stocs going up in Asia indicate a possible strong stock market in the US later on. You see that pretty often. As such, market close in Asia may be an indication for market opening in the US.

Exactly. 24h futures are more indicative however.

 

[GammaJammer]

That's really not my point. How good is to mix and match EOD of Nikkei and EOD of CBOT in a trading strategy? There is probably 9-10 hours of difference. Maybe one should come up with a lowest common denominator approach to EOD, a fixed time for all exchanges, which would reflect the most accurate prices at that point in time.

There is no simple answer to that precise question, but a good rule of thumb is US dominates in equities imho. Unfortunately if you just take that as a black and white rule you'll get carried out. Risk appetite on a global, consolidated basis tends to surface wherever the market moving events surface first. And in this current climate, global risk events are trumping regional fundamental events to a fair extent, so in practical terms it's a bit of a toughie.

Sorry to be the bearer of bad news. I understand your question, it just might not have an answer imho.

GJ

 

[Adamus]

There are whole books on this subject - it's essentially what currency traders do, except with fundamentals. Very interesting.

 

[dcraig1]

That's really not my point. How good is to mix and match EOD of Nikkei and EOD of CBOT in a trading strategy? There is probably 9-10 hours of difference. Maybe one should come up with a lowest common denominator approach to EOD, a fixed time for all exchanges, which would reflect the most accurate prices at that point in time.

The only rational thing to do is to keep your time stamps as Universal Time (GMT). You can if you wish adjust them during processing to (exchange) local time using time zone data of the exchange.

It really doesn't make a lot of sense to pretend that Tokyo closing time == Chicago closing time. That doesn't reflect the real world. Not to mention the fact that you will inevitably (possibly accidently) introduce "future knowledge" into backtesting you might do.

You really need to think hard about your time series class, because you will have to live with any design decision you make - possibly for a long time to come.