Adventures in python and algorithmic trading

Have you decided on what options strategy you may use in broad terms such as directional, volatility etc ?

Too early to say, I'm assuming directional will be easier from an algorithmic standpoint, but volatility adds another dimension which makes it interesting.

I have put my project on hold for a while. Busy developing a site centralbanktrader.com as a trader fundamental research portal that traders can use to reduce their bigger picture analysis overhead. Will get back to my own automation maybe in a couple of months. How is yours coming along sharky

Oh congrats on starting your own website. Mine is coming along, I'm going to post an update now...
 
Okay, apologies everyone for the radio silence. It's been 3 weeks since my last update, and I really wanted to update the journal weekly. Hopefully the more I post the more it'll become second nature.

So, what progress have I made in almost a month? Well, I've definitely made some progress, so that's the good news. A few milestones worth sharing. I'll start with the Options trading side of things, before talking about the Python side on follow-up post..

1. I've already made a bunch of live options trade in my Interactive Brokers account. Yes, I know that I *should* paper trade for several months first to get to grips with the platform and develop a profitable trading style, and I could do that, BUT the reality is I'm in a situation where time is super precious and motivation is super important. I certainly don't intend to lose a whole bunch of money, but I know that trading for real will be a much greater motivator to keep learning and improving than a paper trading I don't really care about. I'm fortunate enough to have £5000 of risk money to put into Interactive Brokers and it's not going to have a material impact if I lost the lot. Rule one, don't risk what you can't afford to lose.

2. Trader Workstation, the platform that IB provides, takes some getting used to. For a start, it's lousy on a Mac, I've given up trying to use it, and instead I'm using a spare laptop that has Windows 10 installed. Tip: if you do use it on a Mac, turn off any sort of Windows Manager like Moom, that's handy for snapping windows to the corner of the screen and generally tiling your programs. I had to get in touch with their support and they suggested the fix, and it worked, otherwise TWS was just consuming all my available ram and ended up freezing. I think because the whole thing is written in Java, it's painfully unresponsive, compared to the native applications we're used to. Anyway, I'm getting my head around how it all works, and it's certainly powerful and efficient, once you know where things are.. hence I'll solider on, knowing it'll be worth the effort.

3. Patience is a virtue. If you're not day trading (I'm not) then don't open and close a multi-legged option trade on the same day (I made a mistake on the expiry of one of the legs so had to close the trade), you'll be tagged a Pattern Day Trader, and will have to request for that honor to be removed. It'll take a few days (is it 5?) before you can open any more trades, unless you happen to have $25,000 to credit your account with. Also withdrawing funds, take 3 business days.. I have no idea why, they're happy for me to fund my account and start trading with it on the same day.

4. As mentioned in 1. above I really prefer not to lose a bunch of money whilst I'm learning the ropes, so I figured the best route to start with was effectively mirroring the options trades of already established traders. I stumbled upon Steady Options (www.steadyoptions.com) after a bit of googling and they offer a free 10 day trial of their Steady Options plan ($125/month afterwards) and signed up. I liked the fact they're a community forum site, very much like T2W, and even if you don't sign up you can access a whole bunch of forums (obviously not the ones where trades are announced and tracked). I appreciate the value add of a community and there's plenty of discussion on their that goes into the reasoning behind the trades, so if you've got the time, there's a lot you can learn.

5. My options book shelf is expanding, I've bought a few extra books, namely: Options as a Strategic Investment (McMillan) and the accompanying Study Guide, they're not cheap but there heavily recommended, and I can see why (tip buy it from the Amazon marketplace, from whoever's selling it new or second hand, rather than direct, although you won't get it next day of course!). The other book I picked up was Options, Futures and Other Derivatives (Hull) which also is very well regarded. A little bird tells me there's an Indian edition listed on Amazon that's basically the same book but a lot cheaper; but I wouldn't know anything about that :/

6. Finally it's totally worth checking out the free CBOE website and mobile app (iOS and Android) for education, there's everything you need right there. I guess it all depends how you prefer to learn, we're all different and some people get more out of books, videos, online courses or in person workshops/1:1 training). I like to use different sources to reinforce my learning, but that's just me.

That's the main points that come to mind for Options trading, I'll aim to post about the Python side sometime this weekend.
 
On the Python front, I've made less progress, but there are a few things to report.

I got QTPYlib up and running and working following the installation instructions here:
https://qtpylib.io/docs/latest/install.html

And set-up their suggested simple MA-crossover strategy described here:
https://qtpylib.io/docs/latest/algo.html#simple-ma-cross-over-strategy

I immediately ran into three issues though:
  1. The strategy would only for about 11-12 secs before it all blew up. This was a simple fix , I needed to add
    Code:
    ibport=4002
    in the strategy code to use the right port.
  2. The web frontend dashboard looked weird, all the styles were missing, but I found a fix for it here in Github issues: https://github.com/ranaroussi/qtpylib/pull/99 - basically I needed to edit the
    Code:
    reports.py
    file manually for the library itself (note a fix was correctly merged in 5 days ago, so this shouldn't be a problem anymore).
  3. The paper trading account mirrors the trading permissions (I think) and data feeds you have for your live Interactive Brokers account. I didn't have market data yet for Crude Oil Futures (CL on Nymex) so the strategy couldn't work without any data, and hence wouldn't make any trades. I subsequently set up the right data for US Securities, Options and Futures (note, those USD fees below are also waived if monthly commissions hit $30.)..

Screenshot 2019-02-18 at 18.30.11.png


One other thing to mention with IB is you can only be logged in to the API AND open up TWS on your desktop at the same time. There's any easy workaround though and that's creating creating a secondary user for the same account in Account Management.

Next steps then are:

At the same time I want to get better at Python coding in general, and for that, I've got a couple of ideas..
 
From their Meetup page it says 6:30-8:30pm. They recommend getting a python development environment setup locally on your laptop and also some suggested reading material to be familiar with (mostly 'pandas'). I've attached the docs with further instructions and you can find their github repo with everything here: https://github.com/LondonPythonForTrading/GettingStarted
 

Attachments

  • Python for Trading Installations.docx
    18.6 KB · Views: 418
  • Python for Trading Prerequisites.docx
    15.6 KB · Views: 399
Last edited:
I couldn't think of anything worse than pandas for trading. Its a wonderful library but heavy on memory. I once had to process 1tb of data on my work pc with 6gb of memory. Pandas would fall over within a few minutes. I moved onto spark and processed the entire lot much faster and without crashing.

The more I think of it the more I must admit that python is too slow for an algo unless you don't need speed.

Also, have a look at the spark phthon dataframe implementation. It has the same functionality (mostly) and is far more memory efficient.
 
What is your preferred language ?

I don't have a preferred it's always a case of the best one for the job

I know

Java
C#
Vb. Net
Python
Scala
Bit of R
Java script
And others that aren't important
 
I don't have a preferred it's always a case of the best one for the job

I know

Java
C#
Vb. Net
Python
Scala
Bit of R
Java script
And others that aren't important
I must admit that learning Java has always appealed to me.
 
I must admit that learning Java has always appealed to me.

Scala is the hardest but the most powerful language I have ever used. The nice thing is you can access java libraries with it too so quite handy. Functional languages are super if you have the time to learn them.
 
The more I think of it the more I must admit that python is too slow for an algo unless you don't need speed.

From my limited understanding, unless your algo trades on tick data, Python should do the job just fine. So, for example, if your strategy works on 1 min data, you'll have no problems. Personally, I don't intend to day trade intraday. So speed really isn't a concern for me.
 
I'm off to Eastbourne this weekend, so I thought I'd get my update in today instead.

First off, motivation is still good, and I feel like I'm making progress. The journal is helping a lot, as I'm not just accountable to myself. I think the biggest danger ahead is just the more you learn the more you realise you really know nothing and the changes of giving up are greatly increased. It's the classic Dunning-Kruger effect.

This week I've been reading a couple of books by Andreas Clenow that I'm really enjoying. For a start they're highly readable, and void of fluff, so it absolutely feels like time well spent. The two books are Following the Trend and Stocks on the Move, the first is focussed on trend following futures, the latter on momentum trading stocks. Each give specific rules for a trading system that replicate with surprising success what the majority of hedge funds are doing, through diversification that captures the whole market. Whilst the former doesn't have much practical utility for the retail trader like you and me as it requires capital in excess of $1m given you're trading potentially 50 instruments across 5 markets. A question that springs to mind is whether you could trade instead with the e-minis, although I don't believe there are enough alternatives to diversify across (either alternatives don't exist, or volume is so low the spreads are unfavourable), in order to make the strategy work. On the stocks side, the strategy trades weekly, and requires scanning the stock constituents of the S&P500, which is ripe for algorithmic trading to do the heavily lifting work. Anyway, what I like is that Clenow presents a complete strategy, the theory behind it, the backtesting, even a year by year look at how it would actually feel like to trade it, then further details about some practical considerations around doing it in practice (data, position sizing, risk management etc).

On the options side, I continue to follow the Steady Options trades, I allowed (or to be more exact let slip!) my free trial rollover to the monthly plan, so I intend to continue following their trades on my live IB account for now. I also picked up a copy of Options for Rookies that I've seen mentioned, more than once ,as a great beginners guide for Options (quick tip: on Amazon it looks like it's free if you have Kindle Unlimited, except it's not in the UK, as it's restricted to the US only). I prefer physical books anyway. My trading library is expanding exponentially, and it was already pretty large before (let's hope they do more than just collect dust). I also traded my first weekly credit spread on ZB (US T-Bonds) last week, and that's been an interesting ride, it'll expire today, but just the experience of following the trade day by day, and the effect of the underlying movement has been an eye opener. This directional option trades (as opposed to the volatility plays on Steady Options) is what actually got me interested in the momentum books I mentioned above, as ultimately to have any change of success you need to be able to identify the trend (and of course manage the position thereafter).

On the algorithmic side, I hit a bit of a wall with QTPYLib, through no fault of the software itself, but just it felt like I was spending more time on getting it up and running that I needed to. I planned to set it up on a virtual server to run 24/7 using the tutorial here: https://dimon.ca/how-to-setup-ibc-and-tws-on-headless-ubuntu-in-10-minutes/ - but given I'm time poor I just felt I was getting enough bang for my buck; and probably I should look at what other options are out there.

So I spent some time research the current state of algorithmic trading, to see what are the main options, in particular with a focus on using Python to code up your system. Here's what I found out.

You've basically got three approaches; and I think for anyone starting out, who isn't necessarily all that tech-savy; it's a bit of no-brainer. The approaches are:
  1. ONLINE approach - Use a fully online platform (everything is done in your browser, nothing to install locally) which comes with everything you need (inc. solid data)
  2. DIY approach - Use an open-source framework/library that you have to install yourself on your own computer or server (baremetal or cloud) and f
  3. DESKTOP approach - Use a desktop application that you run yourself but it's more likely you'll have to use a proprietary language, or if not, it's unlikely you can use Python.
1. ONLINE

So, if you just want to get up and running with minimal fuss, and focus on the strategy and get back-testing, then go with option 1. right now, I don't see any reason why you wouldn't. From my research there are two main players, Quantopian and Quantconnect - the big difference is that Quantopian you can't live trade through the online platform (although you can through the DIY approach since they've open-sourced the backtesting/trading engine, it's called Zipline so that avenue isn't entirely blocked, also see IBridgePy as another way to do it).

With Quantconnect you can trade through the platform your strategy, for a small monthly fee ($20+ for the server costs) on Interactive Brokers, FXCM, Oanda and GDax (cryptos). Quantconnect have also open-sourced their engine, so you can run the algorithms on your own setup, it's called Lean.

Both platforms let you code in Python, in fact Quantopian only supports Python whereas Quantconnect also offers C# and has F# (in beta). You may ask what their business model if they're giving this all away for free, well the idea is that you can share your strategy easily and the best ones get funded and they take a cut from the proceeds. However everything is private unless you choose to share your strategies. I signed up for both and Quantopian comes across as a more mature platform, the $50m they've received in venture money no doubt has helped (Quantconnect is on $1m). I plan to spend more time on both, although I think Quantopian is easier for the average joe, as they have a lot of good introductory material:
Outside of the big two, there are a couple of other offerings that I'll probably play around with, time permitting:
2. DIY

I've already mentioned both of the two online platforms have outsourced their backtesting/trading engines, so they're both worth checking out:
Another framework I've heard good things about is Backtrader. Apparently it's quite a feature complete Python backtesting and live trading framework that can do live trading and has a solid community behind it. However, I've heard it, doesn't have a "portfolio first" approach, so you're out of luck with the Clenow strategies I mentioned earlier, which is a big minus for me.

Then there's QTPyLib that I've previously mentioned, as well as another couple of options:
Reddit has a fairly complete list here, if you really want to exhaust all options:.

3. DESKTOP

I won't talk much about these, as none of these (to my knowledge) lets you uses Python, but I include some of them (by no means an exhaustive list) to give a more complete picture as to the options out there for algorithmic trading (in no particular order):
Final comment, I spotted yesterday this topic on the Quantopian community forums: https://www.quantopian.com/posts/stocks-on-the-move-by-andreas-clenow - it was started 3 years ago, but you can see Andreas Clenow posting about the very strategy he outlines in his Stocks on the Move book and shares the code to implement it in Quantopian. The great thing about these online platforms is the ability to share strategies, which you can then just 'clone' and editing yourself (if you're a techie, think github for trading systems). I intend to delve into this further in the coming weeks.
 
Last edited:
@Sharky Regarding the approach by Andreas Clenow, it is similar to that used by Grey1 although he did it on an intra-day basis and with huge and ongoing consistent success. The key was trading with a basket of stocks all at the same time and with a different number of shares based on the individual stocks volatility. So what would happen is that around 12 stocks would be traded at once and after the overall position hit a target profit the position for all stocks was closed at the same time. This was all done using algos and what was noticeable is that some of the portfolio would be losing but the overall position made the target profit on around 17 out of 20 trading days per month. So as the overall success of this relied on a number of diverse instruments being traded simultaneously (similar to that of Andreas Clenow), I am not sure how you would mimick it with E-minis, so how would you envisage it working ?
 
Just to clarify, you started off by saying Grey1's system relied on a basket of stocks, but at the end you said it relied on a diverse set of instruments?
 
Thanks for the clarification. Yeah it's very early days and I'm not really at the stage of honing on any particular strategy. Clearly my account size will be a big factor. Couple of points to note: cost of borrowing has never been cheaper (I already have a margin account and could borrow further), and by selling options, or at least credit spreads, I can generate income to trade with. It feels like absolute minimum to start on any of this, a potential expensive hobby, would be $5K but ideally $10K. $100K would be a more sensible minimum if I planned to trade full-time and $1m is when things really start to get interesting.
 
I'm off to Eastbourne this weekend, so I thought I'd get my update in today instead.

.............................................
This week I've been reading a couple of books by Andreas Clenow that I'm really enjoying. For a start they're highly readable, and void of fluff, so it absolutely feels like time well spent. The two books are Following the Trend and Stocks on the Move, their both focussed on momentum trading, the former on futures, the later on stocks. Each give specific rules for a trading system that replicate with surprising success what the majority of hedge funds are doing, through diversification that captures the whole market. Whilst the former doesn't have much practical utility for the retail trader like you and me as it requires capital in excess of $1m given you're trading potentially 50 instruments across 5 markets. A question that springs to mind is whether you could trade instead with the e-minis, although I don't believe there are enough alternatives to diversify across (either alternatives don't exist, or volume is so low the spreads are unfavourable), in order to make the strategy work. On the stocks side, the strategy trades weekly, and requires scanning the stock constituents of the S&P500, which is ripe for algorithmic trading to do the heavily lifting work. Anyway, what I like is that Clenow presents a complete strategy, the theory behind it, the backtesting, even a year by year look at how it would actually feel like to trade it, then further details about some practical considerations around doing it in practice (data, position sizing, risk management etc).
...............................................
....................................

A really excellent and most interesting thread – I hope you will be able to continue to find the time to keep us updated.

Ref the Clenow books - they too are very worthwhile: especially the one on stocks for retail traders like myself. My methods are somewhat similar to his and having read his book ages ago I'd forgotten how clearly and expertly he states his principles and methodology. It's so simple! – And I'm convinced that's the key to success, though my experience is that applying a simple system successfully is far from easy – the human element easily wrecks it. So more power to your automation.
 
The more I delve into algorithmic trading, the more I find it's a pandora's box. No doubt many quickly reach the same conclusion that's there's just way too much to think about and give up without ever having placed a single trade; OR they jump in feet first with the minimal of knowledge, get burnt and fall back to whatever their comfort zone in trading is. So I'm keeping that in perspective, and trying not to let my mind explode. Small goals and wins along the way is what keeps us sane.

This week I've been narrowing my focus to gaining familiarity with the two main online algorithmic platforms, Quantopian and Quantconnect. Which basically means getting comfortable with navigating the sites, understanding how they work and what resources they offer to get up to speed. I want to try and build on my progress, rather than pivot from one shiny object to the next. So my plan is to stick with the two strategies outlined in Andreas Clenow's books which I mentioned on my last update; and attempt to get these coded and working in both platforms. I plan to do this first with Quantopian and then at least one, with Quantconnect to start paper-trading through Interactive Brokers. Why Quantopian? Well Quantopian seems to lead the way in terms of it's community, which means it's attracted some heavy weights in the algorithmic world who have contributed in the past in some degree:
Note: If anyone can suggest other algorithmic traders, practitioners, authors, commentators to follow, please do share your list here. It's easy to find them on Quantopian either searching on their site or a bit of googling.

This means that those two strategies are already available (in some way, shape or form) on their site and I already shared the link to the strategy in Stocks on the Move on my last post. If you look over that thread in their forum, there are various versions shared, so I'll need to spend some time understanding how they compare to the vanilla rules laid out in the book.

I also found a simple adaption of the one from Following the Trend in their excellent Overview/API Reference here, of all places: https://www.quantopian.com/help#sample-futures-trend-follow - this one requires more work to get it into shape, but it's the strategy that appeals to me more. When it comes to coding, there's no shortcuts, you just have to dig into the code, line by line, and cross-reference with the API documentation to be able to understand what's going. I like to print out the code and the annotate it, but whatever works for you.

One complication I've found when cloning some of the strategies on Quantopian, is that they launched a new version of their API, the aptly named API 2, back on April 2nd, 2016. So it's quite possible strategies you clone from before then will not work out of the box without some minor changes. For more info check out:
In terms of this week's ready material, I've been reading Building Winning Algorithmic Trading Systems by Kevin Davey. I'd heard it was a good primer, and to it's credit it's very readable, which when you're starting out is exactly what's needed. I think the Amazon review are mostly spot on, in that it'll give you a flavour of some of the areas of concern, but really is lacking in substance. Definitely one for the newbie.

On the trading front, I've continued to follow most of the volatility options trades from Steady Options (SO), in my live IB trading account. Between the trade on ZB and the ones on SO I'm up about £200, on a £5000 account, so that pays for all the books I'm reading!

I'll say it's been a bit hit and miss following the trades on SO, as their just posted in their forums. You can get email alerts, but unless you can monitor it all day, it's easy to miss one.
 
Top