Adventures in python and algorithmic trading

Wow, what a great Thread, too bad I was already gone when it started, but better late than never J

I am going to post few random thoughts to help motivate my friend Sharky to complete this, because we all know what my other friend Warren is saying:

"If you don't find a way to make money while you sleep, you will work until you die."

But first, few disclaimers, I am a paranoid dude and it is completely work related, up until recently and for many years I was leading a team supporting major security infrastructure pieces for large telco, so I will not give you any personal details just because I believe there are a lot of crazies out there who have nothing better to do but look for trouble…

Second, I do not believe in sharing trading strategies, I do not believe any successful hedge fund will share their trading strategy, quite the opposite, they will make you sign NDAs up to the wazoo and they will enforce it even at the slightest discretion just to protect their IP, just gigle hedge fund developer email code jail and you’ll find more than one example… I believe as other traders trade the same stuff, it will be arbitraged away as more people will start trading against the system…

3rd I do not believe in platforms as a service like quontopain where you put your code on their servers, again because their developer can copy your strategy, there are a big hacker target for the same reason, if there are successful strategies, a hedge fund can buy them up and etc… just too many attack vectors for something you can build yourself, which would be my next post on some ideas…
 
Mandatory groups for everyone who wants to do this, become a member, it is free and there will be a solution for any problem you might face with IB there:

https://groups.io/g/ibcalpha - essentially on how to make the auto login to the headless Interactive Brokers' Trader Workstation and Gateway to be able to use the API, one thing to note all new accounts have 2FA by default which you will need to acknowledge the login on your phone . the old grandfathered accounts are being converted to 2FA as well, so once a week you’ll have to re-login to the system

https://groups.io/g/insync very good python framework for IB

https://groups.io/g/twsapi - the general IB API user group

IB are also working on a REST API, but few years ago when it became available was very crude and you still need the headless Interactive Brokers' Trader Workstation and Gateway installed and configured, have to check to see if is usable, but the gateway is working and it is very robust piece of software…
 
Brilliant to have some input on the thread, definitely a great motivator!

Thanks for the links Quantt, I've added these to the list of useful resources on the first post of my thread.
 
Quantt said:
Mandatory groups for everyone who wants to do this, become a member, it is free and there will be a solution for any problem you might face with IB there:

https://groups.io/g/ibcalpha - essentially on how to make the auto login to the headless Interactive Brokers' Trader Workstation and Gateway to be able to use the API, one thing to note all new accounts have 2FA by default which you will need to acknowledge the login on your phone . the old grandfathered accounts are being converted to 2FA as well, so once a week you’ll have to re-login to the system

https://groups.io/g/insync very good python framework for IB

https://groups.io/g/twsapi - the general IB API user group

IB are also working on a REST API, but few years ago when it became available was very crude and you still need the headless Interactive Brokers' Trader Workstation and Gateway installed and configured, have to check to see if is usable, but the gateway is working and it is very robust piece of software…
Click to expand...

Just to be clear it's my understanding that once you have 2FA you can't / don't need to use ibcalpha since you can set up the gateway to reset daily automatically but you will still need to do the weekly relogin.

Rob
 
globalarbtrader said:
Just to be clear it's my understanding that once you have 2FA you can't / don't need to use ibcalpha since you can set up the gateway to reset daily automatically but you will still need to do the weekly relogin.

Rob
Click to expand...
Yes, either way you will need to do 2FA once a week, so up to you if you want to use IBC or not
 
Again, this is not a financial advice, as I can give them to accredited investors only and I would not confirm or deny I am doing any of this, because I am a paranoid SOB, as you might know by now, but here we go with few more random thoughts on this:

The infrastructure part: renting Linux or Unix VPS are very cheap, I have a unix sysadmin background so for me it is easy, but you can find online guides on how to secure the servers and just follow the instructions, even the on the link you posted by Sharky there is a section with basic security:

https://dimon.ca/how-to-setup-ibc-and-tws-on-headless-ubuntu-in-10-minutes/

You can rent as little as $5 per month and it will work for this, I would rent them in a data centres close to where IB has entry points, you can find on a website the current, but they have New Jersey, Chicago, etc and you can configure every TWS Gateway to talk to the closet IB API server, not mandatory, but why not, it will eliminate the possibilities where portion of the internet can go down, and we have seen this few times in recent years…

I am not using containers/kubernetes and all the new DevOps orchestration stuff on purpose, because when you are building the container, you are essentially getting the pieces from untrusted sources J

The data part: you will need data for both trading and back testing the more data the better, even the same data from multiple data sources, so you can compare, I would run a separate program just for checking the data to make sure it is clean… Also collect your own data and store it locally for future use, you can never have enough data…

The backtesting part: for me this is crucial, the important part is to test as far back as possible and at least 1 market cycle, as we are not crashing yet I wold say we are still in a bull market, so you need to go back to around 2007, I would go even one more to before the dot com crash…

I am not doing options, because it is hard to find good data so far back and also it is a leveraged product and I do avoid leverage when I can help it…

The program structure part: consider using different modules executed from cronjobs as opposite to one monolith program, easy to manage, change and maintain, which reminds me, when configure the unix servers make sure you configure the time protocol to sync with a good time source so your server time is correct, few seconds diff is not a problem, but it can drift and few minutes will be a big problem…

Run more than one server and make sure you get the same results everywhere :)
 
Sharky said:
This week I've been reading a couple of books by Andreas Clenow that I'm really enjoying. For a start they're highly readable, and void of fluff, so it absolutely feels like time well spent. The two books are Following the Trend and Stocks on the Move, the first is focussed on trend following futures, the latter on momentum trading stocks...
Click to expand...
Just a heads-up that Andreas Clenow released a second edition of Following the Trend a couple of months ago:
https://www.amazon.co.uk/Following-Trend-Diversified-Managed-Futures-dp-1119908981/dp/1119908981/

Since I've already got the first edition, can't really justify splashing out on the second, but if you don't already have it, is definitely worth a look.
 
Quantt said:
Wow, what a great Thread, too bad I was already gone when it started, but better late than never J

I am going to post few random thoughts to help motivate my friend Sharky to complete this, because we all know what my other friend Warren is saying:

"If you don't find a way to make money while you sleep, you will work until you die."

But first, few disclaimers, I am a paranoid dude and it is completely work related, up until recently and for many years I was leading a team supporting major security infrastructure pieces for large telco, so I will not give you any personal details just because I believe there are a lot of crazies out there who have nothing better to do but look for trouble…

Second, I do not believe in sharing trading strategies, I do not believe any successful hedge fund will share their trading strategy, quite the opposite, they will make you sign NDAs up to the wazoo and they will enforce it even at the slightest discretion just to protect their IP, just gigle hedge fund developer email code jail and you’ll find more than one example… I believe as other traders trade the same stuff, it will be arbitraged away as more people will start trading against the system…

3rd I do not believe in platforms as a service like quontopain where you put your code on their servers, again because their developer can copy your strategy, there are a big hacker target for the same reason, if there are successful strategies, a hedge fund can buy them up and etc… just too many attack vectors for something you can build yourself, which would be my next post on some ideas…
Click to expand...
Yes, the idea of buying a strategy for 10$ which will make you consistently profits would mean we would all be rich. When too many follow the same signal it will stop working. Keeping the code on own server is important, but in many cases the broker will discover your success and start copying, unless hiding what one is doing. Hedge funds pay 20-25% for working signals.

The original post is using Python. But you cannot use Python for algorithmic development, you will never finish even passing the learning curve of try and error, since Python means cloud back-testing, or very slow back-testing, adding the slow coding process. But even using blitz fast C++ back-testing, like on MT4/5, is not possible today, unless you have a team of pro developers, since it is not operationally possible to develop algorithmic trading by programming, in any language. Why Quantopian had to close, 45.000 developers could not develop anything useful - should wake you up from even the deepest sleep, but many didnt wake up and continue on that same path.

The major enemy of trading, is the trader himself, the disability to listen. Either you are smart and find new ways learning fast what is possible and not, and why, or you spend many years of pursuing an impossible idea/narrative, closing your ears and eyes, getting nowhere.

In general financial trading changed dramatically about 10 years ago. I think cause algorithmic trading exploded at that time. Discretionary pro traders had a hard time, today all have a hard time no matter trading style, since markets have become very efficient. So you need to think new thoughts, or get out do something else :)

As a result, forums like this, before full of activity now dead, since there is nothing new to talk about, many traders gave up, combined with censorship on what you could say in the forum, often cause the forum became commercial.
 
Last edited:
Hey Sharky - just wanted to say thanks for this. I just started diving into this when I realized my trading strategy would be too cumbersome to do manually as often as I want to do it. The resource lists you have here are great and I'll be referring to them regularly - this thread is now on my bookmark bar! Looking forward to continued updates...
 
TickCOM said:
Yes, the idea of buying a strategy for 10$ which will make you consistently profits would mean we would all be rich. When too many follow the same signal it will stop working. Keeping the code on own server is important, but in many cases the broker will discover your success and start copying, unless hiding what one is doing. Hedge funds pay 20-25% for working signals.

The original post is using Python. But you cannot use Python for algorithmic development, you will never finish even passing the learning curve of try and error, since Python means cloud back-testing, or very slow back-testing, adding the slow coding process. But even using blitz fast C++ back-testing, like on MT4/5, is not possible today, unless you have a team of pro developers, since it is not operationally possible to develop algorithmic trading by programming, in any language. Why Quantopian had to close, 45.000 developers could not develop anything useful - should wake you up from even the deepest sleep, but many didnt wake up and continue on that same path.

The major enemy of trading, is the trader himself, the disability to listen. Either you are smart and find new ways learning fast what is possible and not, and why, or you spend many years of pursuing an impossible idea/narrative, closing your ears and eyes, getting nowhere.

In general financial trading changed dramatically about 10 years ago. I think cause algorithmic trading exploded at that time. Discretionary pro traders had a hard time, today all have a hard time no matter trading style, since markets have become very efficient. So you need to think new thoughts, or get out do something else :)

As a result, forums like this, before full of activity now dead, since there is nothing new to talk about, many traders gave up, combined with censorship on what you could
Click to expand...
I'd fundamentally disagree that you can't use Python for algorithmic development - Rob Carver is a case in point. I would agree it's too slow for high frequency trading, and probably day trading, but you're better off leaving that to the hedge funds and institutions anyway. For swing/position trading, it's more than adequate. Python is the go to language for data science and it has all the open source libraries you'll ever need.

I also don't agree forums, per se, are dead.. there's been a massive expansion of social media and so whereas before trading discussion was concentrated to a few nice forums, this one included, it's now migrated to youtube, twitter, reddit, discourse and facebook(?). There are still just as many, if not more, people interested in learning to trade; and for the average person, probably even more confusing to know where to start as there's almost too much information out there, often conflicting, and difficult to differentiate the wheat from the chaff.

jamoli85 said:
Hey Sharky - just wanted to say thanks for this. I just started diving into this when I realized my trading strategy would be too cumbersome to do manually as often as I want to do it. The resource lists you have here are great and I'll be referring to them regularly - this thread is now on my bookmark bar! Looking forward to continued updates...
Click to expand...
That's awesome, glad it's been of help. These days there are so many good resources both online and in books for every aspect of trading. The problem is it's easy to quickly get overwhelmed and just give up.
 
As far as my adventures are going, I'm still making slow progress.

On the one hand I've hired a part time python developer through upwork who I've just started to pair up with in building out some new tools for T2W, it'll help me improve my own python. We've started with a simple project - a compound interest tool, which will be available on the site perhaps even later today - super simple, but a good way to start, by getting into a development workflow using git and the hosting sorted. The next python project will be an online version of Rob Carver's Interactive Broker's UK tax calculator, anyone who uses Interactive Broker's in the UK knows that when it comes to submitting your tax returns, there's no report on IB you can simple run off to calculate your CGT. I'll certainly find this tool useful for my own trading in IB, and I'm sure others will too.

As far as getting pysystemtrade up and running, I've got as far as backfilling the futures contract data from a bar charts premium account, using https://github.com/bug-or-feature/bc-utils - you're limited to 100 downloads a day so it's taking a bit of time to backfill hundreds of contracts. I then plan to run the system on my own linux box using a paper trading account on IB.
 
For those who may be interested, FreeCodeCamp have published a 3 hour YouTube course on Python and Algorithmic Trading. To quote the course text:
"You will learn how to design an Unsupervised Machine Learning Trading Strategy. You'll also learn how to leverage Sentiment Analysis and intraday trading strategies using the GARCH Model."

As it is only 3 hours, this is a good way to get an overview of this subject and here is the link:

https://www.freecodecamp.org/news/learn-algorithmic-trading-using-python
 
You must log in or register to reply here.

Similar threads

C
Yfinance
Replies
0
Views
1K
cassiopeia
C
H
Proposal - Open Source Trading SW - Open AI Desk
Replies
0
Views
2K
hANSIc99
H
McQuant
QuantConnect LEAN platform
Replies
2
Views
2K
frankhu
frankhu
S
Automated trading using MATLAB?
Replies
1
Views
3K
raghu
R
B
Python and Interactive Brokers
Replies
5
Views
3K
TickCOM
T
Top