Fxx's macro algo adventure

[FXX]

Inspired by sharky and something I have been wanting to do. I am tight for time at the moment but want to start slowly researching what's needed and an overall plan.

This isn't going to be a walk in the park. I am not looking to apply basic price rules. What I am looking to achieve is a natural language processor that processes real-time news feeds and economic data and process that through a rule system. The first objective would be to target swing trades as I am unwilling to spend a fat chunk of money monthly on premium APIs. If I can build something that is abke to process and apply rules successfully then I will consider looking at intraday trading.

First steps:

1) build code to process forexlive rss feed
2) build code to process eikon data (still assessing if I can do this with my subscription. I might have to buy an addon)
3) build code that can execute and manage trades not using mql4. It will need to using an api or a library that bridges with mt4
4) research natural language processing and deep learning.
5) build code that can access tradingeconomics.com for historical data
6) build trading rules engine based on my existing profitable strategy


Once I have all these I can look to putting them all together. I will be using Azure, streaming, in mem processing, various languages and big data tech.

 

[Trader333]

Looks good and I would think that one of the biggest issues will be the deep learning of real time news so do you have an idea of how you will go about it ? Either way I will be following your project with interest which I hope goes well

 

[FXX]

Looks good and I would think that one of the biggest issues will be the deep learning of real time news so do you have an idea of how you will go about it ? Either way I will be following your project with interest which I hope goes well

Yep I do. I have commercial experience with a platform called apache ignite, an open source software. It is an in memory platform that is a database but has a ton more features including real-time streaming and machine learning libraries but is also compatible with various languages and other tech like spark which has its own deep learning libraries.

Everything will run in memory using streaming so will be pretty quick even though I don't need the speed at the moment but maybe later down the road if I look at intraday.

Check it out on the apache site.

 

[FXX]

Update

So Eikon is ruled out with the Xenith subscription not having access to the api. Refinitiv said they don't sell addons and said I should move to the professional edition which is well over a grand a month.

Tradingeconomics offers what I need but their api is also over a grand.

Several other vendors have APIs but they are not real-time so won't work.


Fxstreet has a free api that I will have a look at. I am only looking at swing trading at the moment so I don't need real-time but near it will suffice. So it is looking like this will be my option.


Additionally I am building a twitter client to access economic data and market commentary which will be, next to the calendar api, the basis of the data I will be consuming.

Next step is building a scoring system based on entity and importance. For example

Headline: US major indices open mixed

Key words
US
Indices
Open
Mixed

The scoring process will need to use this to score risk sentiment (one of many inputs to risk sentiment I'll be adding)

Another example:

Headline :US January prelim Markit services PMI 54.2 vs 54.0 expected

The logic will need to identify currency, indicator, value versus expected. This is the sort of headline that could trigger a trade but the system will need to account for many factors hence I am building a scoring system across many metrics. A trade would be valid if there is sufficient parameters operating within a given scope. I will make this adjustable without needing to open the code.

Focus at the moment is to work on the api and twitter feeds. Once I have access to the data I can start developing the system so right now it's data collection and classification that's the main focus.

 

[FXX]

Current thinking of how the control flow of the algo will work so here is version 1:

1. Each day a process runs and collects the days scheduled releases from the calendar.
2. Red category events are sent to a queue.
3. another process launches a review of the relevant currency ( main algo logic goes here )
4. the output of no. 3 produces a score card which is sent to another (separate) queue
5. another process is running and monitoring data feeds and processing them. The objective here is to keep a live score card of things like market sentiment, risk mood, upcoming major events (i'll create a scoring system for it).
6. another process will be taking items off queue from step 4 and matching it with the live score card ( logic here yet to be determined). This is where i will get my directional bias, counterpart matching (ill have to write a matching engine) . This should produce the foundation for a trade plan. The plan will be updated based on market data triggers.
7. when the release happens, if there is a deviation and the trade plan has either a buy and sell plan available, execute the relevant plan.
8. another process will be reviewing the live release and reviewing if it affects any active trades ( a whole lot of more logic here and yet to be determined)

yep it is a sh!t load of work... i do enjoy a good project and do this sort of thing for clients all the time, although not trading related projects.

none of this of course possible unless i can get my hands on the data, so back to it. I can't believe the staggering prices for economic releases in real-time through an API. There are absolutely no options for the average man. Perhaps a business opportunity there the more i think about it.

 

[Panic!]

Hi FXX, instead of trying to get economic releases through an API, try just scraping headlines or twitter. Most of the info you want is in the headline anyway.

Also, there are some affordable options via Interactive Brokers: https://interactivebrokers.github.io/tws-api/news.html

I would be careful with trading the news in an automated way though. I've never seen it work. I dabbled with it myself, but decided instead to just halt trading in an underlying in which there was news released.

 

[FXX]

very belated update but I have made a start on this although my approach is different. I am building an algo that processes central bank communication using NLP and my own logic and formulas. This algo will be extended to also process news feeds but for now, i am planning on building indices for every central bank and central banker.

It has been an amazing experience and i have found this to be a lot doable than previously assumed. I am writing it all in python. My next step is to train the model and then i am going to take all the central banks statements and create an index for the fed. I will do this for each central bank and banker that is active in commentary, before i look at building an indicator that plots this in mt4 or the api can be consumed separately.

I will post a chart of the index next to the dxy in time

 

[FXX]

very belated update but I have made a start on this although my approach is different. I am building an ago that processes central bank communication using NLP and my own logic and formulas. This algo will be extended to also process news feeds but for now, i am planning on building indices for every central bank and central banker.

It has been an amazing experience and i have found this to be a lot doable than previously assumed. I am writing it all in python. My next step is to train the model and then i am going to take all the central banks statements and create an index for the fed. I will do this for each central bank and banker that is active in commentary, before i look at building an indicator that plots this in mt4 or the api can be consumed separately.

I will post a chart of the index next to the dxy in time

 

[FXX]

update

the US economy index has moved on leaps and bounds. I was so impressed by the sentiment of policy statements that I took it further.

the index constituents are going to be in phase 1

1) central bank statements
2) interest rates
3) key economic data


in phase 2 I will be adding news feeds
here is a chart of the dollar index and the index with just sentiment and rate. it is only half the equation as I need to do the same with other economies so that I can find the divergence and automate trades. I am building an api for the index and will follow suit with meta trader indicators, then automation.

 

[Trader333]

update

the US economy index has moved on leaps and bounds. I was so impressed by the sentiment of policy statements that I took it further.

the index constituents are going to be in phase 1

1) central bank statements
2) interest rates
3) key economic data


in phase 2 I will be adding news feeds
here is a chart of the dollar index and the index with just sentiment and rate. it is only half the equation as I need to do the same with other economies so that I can find the divergence and automate trades. I am building an api for the index and will follow suit with meta trader indicators, then automation.

Looks good so far and you are clearly enjoying it

 

[FXX]

Looks good so far and you are clearly enjoying it

I wasn't expecting it to be so enjoyable. the process was daunting but after exploring it with the tools and getting really promising results I am confident this is going to be a success. I have an azure cosmosdb backend and building python api. I had to purchase economic data feeds but that's expected. in the next week I will provide an update with the inclusion of economic data. Once I am at that stage I'll begin automating the entire process and moving onto the next economy.

planned indexes

fed index
ecb index
boc index
rba index
rbnz index
boe index
boj index

also looking at central bank speeches index for each central banker. this will show when a dovish central banker turns hawkish - anecdotes for the main indexes.

for meta trader I will build an indicator that shows each side of a currency pair overlayed on the same chart. this should show divergences and ultimately trading opportunities. the eventual goal is automated trading but I need to build the indices first and ensure they reflect a true reflection of sentiment and fundamentals. I am looking forward to eventually including real-time news which I suspect will be the biggest challenge although calling upon lessons learned from central bank sentiment processing.

stay tuned for updates.

 

[FXX]

update

I need some more work on the central bank statement parsing and I need to shift into more granular changes in the date axis as I am rolling everything into a single month.

This is the correlation between the DXY and this algo. Note that there is no price input to this, it is all economic data and central bank sentiment. Where are those technical junkies telling us that fundamentals are rubbish. Look at the general correlation and i am not done with it i have more work to do. Here are charts covering 2016 to the end of 2019

 

[FXX]

update

been a lot of changes that's been made and here is a breakdown of elements that will form the index as well as elements that will exist next to it like the balance sheet and rates. There will be a chart that is comprised of just the index but for each economy. There is still work to be done but I am well over halfway to being done with the formula.

by the way here are the indicators involved

consumer

1. unemployment
2. job numbers
3. housing (starts, new builds etc)
4. sentiment
5. retail sales

business

1. sentiment
2. service and manufacturing PMI (ism also for the U.S.)
3. inventories
4. productivity
5. industrial production

inflation

1. base inflation
2. core inflation
3. producer prices
4. expectations

interest rates
central bank balance sheet

 

[FXX]

update

been a while so apologies

I have moved into the world of AI as a component of the indices I am building. so just to recap I am building indices that encapsulate fundamental data which includes economic data, central bank meeting minutes and speeches, and finally news. to teach the ai models for each central bank I have downloaded 20 years of statement and minutes and busy scoring the content to train the models. I have about 40k sentences to score so its a lot of work and when done Ill be able to process central bank content in real time producing a sentiment score.

This will be a component of the indices alongside economic data and news. I have about 6 months of work ahead before I can build the new indices.

 

[FXX]

update


having spent several months now on AI I have been dissatisfied with the results. Not only is it laborious manually scoring central bank minutes and speeches, I have scored 10k so far. The test results yielded surprising scores that shouldn't happen so I went back to the drawing board.

after several weeks of work I have formulated a superior strategy, built various tools and an algo which instead of requiring sentence scoring, it's a series of configurations that operate on about 7 rules in the algo. I still have to setup the configuration rules and have done 5 already out of about 15 planned. each rule focuses on a specific subject like inflation, businesses investment, consumer and business sentiment,etc, etc, you get the idea.

so it is still processing central bank publications but can also accommodate speeches and rules can be shared amongst central banks or only focused on one.

testing is fast as well, I can process 20 years of minutes (20k sentences) in 4 seconds and if needed I can add threading so that I am processing multiple rules concurrently.

the results have blown me away. not only is this algo running circles around the AI's I have tested including:

Watson AI
Google AI
Azure AI
AWS AI

, it is also handling complex sentence scenarios which AI falls short on. Sentences that have contrasting statements that confuse an AI.

it still takes a while to configure rules as you have to isolate areas of the economy in their content and work through the language around that configuring the rule. So far its been a 2 day process per rule and this involves about 50 test cycles and reviewing matches and sentences that don't match. The concept behind the algo, which I won't go into detail here as this is my IP, is the language around a subject matter has a limited number of ways it can be said. each rule consists of around 100 words or phrases that function around a rule work flow. In all the years I have been researching this, I have never found anything like what I am doing here. All cases I have found in the professional and academic world operate on extracting key phrases. This strategy falls short when you have complex sentences such as

core inflation has been picking up recently, but is still well below the committee's 2 percent objective.

a key phrase approach will not have rules that handles this type of scenario and therefore their correlation not as close as it could be. Here is analysis of a fed index done by a business using key phrases against the SnP and Russell

there is a fairly strong correlation and yet my results are handling far more complex scenarios and in the test results are extracting and scoring sentiment on a specific subject with over 90 percent accuracy.

next update will be in some weeks after I have enough rules to test overall sentiment and build the indexes. I will likely have to have additional isolated rules for some central banks as their language is very diff like China and Japan. the beauty of this system is this is very easy to do and maybe 3 months of work to have 8 major central banks configured. I was excited about AI but the results I have seen with this are on a whole new level.

later..

 

[FXX]

update

things had been progressing well unitil I built a review feature that takes rules as pairs, say consumer spending (positive and negative rules), and runs them side by side for review. The results for this showed I have a gap in my logic for past, current, and future tense. There are central bank statements that can discuss the past in a negative tone and the future in a positive tone.

So now I am working this new problem to solve where I have to identify matched rule content by its respective tense. I am still conceptualising the problem and solutions and hope that I am able to define a process that can accurately categorise content. My current thinking is to place greater emphasis on future tense as this is the area where market expectations are set for future policy.

I am also toying with the idea of incorporating economic data into the logic as a means of confirmation. I already have tested logic and plans for economic indices based on economic data,it is the reason I got into this journey in the first place. But including economic data into the central bank algo might prove to be a good choice.

 

[FXX]

update

I have made a productivity enhancement. Up until now I have been building rules, looking at the results and then looking at charts to see when the rule applied. This has been very inefficient so I have sourced market data, loaded it to my database and built code that generates charts for it. I then integrated the output of the algo so that I can view a rule overlayed on a chart. The feature will help me validate price correlations at a rule level.

The algo logic is done, and I have built a Ui that allows me to rapidly build rules. I have 20 years of statements to process (25 to 30k sentences) and it takes 5 seconds to run the algo over that dataset and produce a chart. Now I can fine tune the rules and within 5 seconds have a visual of how that rule showcases a fragment of sentiment correlation with price.

I have spent a great deal of time since my last update rewriting the algo to be more efficient at rule building by removing duplication of config. This new feature will go a long way to helping me build, test and repeat forall the rules much quicker than before. I can now get back to building rules.

 

[FXX]

update

I have been working on the economic data indices and have found something I wasn't expecting, and didn't know about so going to share it.

there is an inverse correlation between foreign fx reserves and the country's fx movements.

the blue line is the USD going back to the year 2000 and the orange line is foreign fx reserves in the US.

now you too have been enlighten, enjoy and you're welcome.

oh and ignore the first data point for the foreign fx reserves, its due to an average calculation and it null so it isn't a real data point.

also it isn't inverse in this image because i inverted it.

 

[Trader333]

there is an inverse correlation between foreign fx reserves and the country's fx movements.

An interesting observation thank you.

 

[FXX]

An interesting observation thank you.it's a strong

found something on this, a study of the Chinese ccy, some fancy maths in their analysis as well is some interesting findings. I highlighted what interests me most for my algo.

quote from the conclusion:
"There is an equilibrium relationship existing between foreign exchange reserve and
RMB exchange rate in the long-term. Foreign exchange rate has the negative
relationship with foreign exchange reserves. On the contrary, total volume of import
and export trade and GDP have the positive relationship with foreign exchange
reserves.

Through Granger Causality Test, we found that foreign exchange rate is NOT Granger
cause of foreign exchange reserves, but foreign exchange reserve is Granger cause of
foreign exchange rate. This result proved that the causality between foreign exchange
reserves and foreign exchange rate is unidirectional. Any changes of foreign exchange
reserves would lead to the fluctuation of RMB exchange rate but not vice versa."