Best Thread Algorithmic trading experiences

[johnsue]

yes i use cycle analysis theory but looking to model a code based on statistics such as after 3 days of up or down days then go opposite on fourth day as this is a high prob trade, another one being favourites in sports betting lose 1/3 games or races.

i would like to eventually have a statistical model of some kind

 

[adamscj]

looking to model a code based on statistics such as after 3 days of up or down days then go opposite on fourth day as this is a high prob trade,

Not really. A quick analysis of the FTSE since 1984 shows that if you had traded the anti-trend after 3 up days, you would have been right 49.6% of the time and wrong 50.4%. Even worse news is that if you had put on £1 per point, you would be down over £930.

Remember, the trend is your friend

 

[robster970]

Rob,
I was interested to read your thread. After 6 years of writing algo trading apps for an investment bank & subsequently systematic hedge fund, I'm having a go at doing it myself. I've written an application that connects to various data providers, collects & stores prices and then generates a daily signal based on various technical & statistical indicators I've designed. The resulting trades are then transacted automatically. There are loads of things to think about and I'm happy to discuss in detail if you want to contact me, but here are a few "pointers":

- Always start with a hypothesis. The worst thing you can do is crunch some past data and try to produce a trading rule - that way you'll just end up overfitting your data and losing big time going forward. Start with an idea which includes a justification (e.g. a dummy example might be "I think the FTSE always goes up the day after it has fallen more than 100pts, as people think it has been oversold") and then analyse the past data to see if it fits.

- Understand risk. As a medium/long term trader I never trade a fixed percentage of my capital but instead vary my position based on volatility. Once my system has generated the appropriate signal for a contract, the final step is always to multiply by my own volatility adjustment factor, with the result being the position I want to hold. That way, I bet smaller amounts when volatility is higher (as an example, volatility of the ftse has been averaging about 3 times as much in Dec2008 compared to Dec2007, so for the same signal coming out of the model I would only take on a position 1/3 the size).

- Don't under-estimate technology risk. You can have the best model in the world but it means nothing if your program or hardware is unreliable as Murphy's law says you will suffer a failure just when it will hit you hardest. I've developed a whole series of tests for my code and whenever I make changes I always re-run the tests to make sure nothing has broken. I also host my code on a dedicated VPS as my home internet connection is unreliable.

Hope that gives you some things to think about, let me know if you want to discuss more.

Great post. Will definitely be in contact. Just for the record:

i) It took me about 4 weeks of watching 20 stocks movements in FTSE to come to a broad hypothesis but my understanding of TA is quite amateurish at the moment so the code I am developing keeps changing as I spot other problems (i.e. I discoved yesterday how ranging can kill me quite quickly). I'm not trying to get data to fit at the moment, it's still too early for me to say. If anything the broad hypotheses I started with are being refined and will continue to be refined until I can either ditch them of use them.

ii) Risk and Money Management. These are areas I know nothing about right now and as the more experienced posters pointed out at the beginning of the thread I should get my act together and understand this.

iii) I know about technology failure and its catastrophic effects. Really I do. I work in high performance system development so running db's in memory in cluster configurations, server affinity, 3 tier architectures, load balancing etc all come naturally. The app uses MySQL dB which currently has about 25k rows in the database written in php (yes somebody still uses it - I couldn't be bothered to run JBOSS/Tomcat for J2EE) and apache currently in one box. When it's ready for production, they will get seperated into 3 boxes with a UPS and dedicated circuit to an ISP. I can't think of anything worse than it going down.

I'll pm you some questions;

It's quite interesting to note that there is a little community of us who can build this stuff and the thread has flushed us out of the woodwork.

I was also considering starting my 'analysis' object using FFT's as johnsue is doing.

 

[pedro01]

For what it's worth here are my thoughts.

Did I read that one member has worked on a method of fetching price data from Yahoo finance ? If so - WHAT ON EARTH FOR???? This will not help in the slightest.

First thing you need to do is to figure out your strategy - when you will go long, when you will go short, when you will not trade and when you will exit. Once you have done this, you may or may not have an entry/exit technique that you can apply to historical data. If the strategy is simple, then you may find it is easy to apply to historical data. If your strategy involves many different timescales and different data streams (e.g. NYSE Tick, The dow jones, put/call ratios etc), then you will find it hard to test historically. At the end of the day, if your entries & exits are not good, money management etc. will not help in the slightest.

Next you need to figure out how to prove your strategy works. If historical testing is out of the window, you may still be able to prove some parts of your strategy historically but not all. If historical testing is out, then you need to write your strategies and forward test.

In terms of software, Tradestation is OK as long as your strategy is not too complex, if it is complex, then it gets tricky. I have not tried Ninja Trader yet but having become pretty adept at coding for Tradestation, I am considering either moving to another platform or writing my own trade management API to plug into Tradestation.

If you have never heard of Tradestation or Ninja Trader, then you need to look at these tools and others of their ilk as they take a huge burden off the developer that assumes they need to start from the ground up. These tools can also give you the historical data you need once you are hooked up to your broker.

Ultimately coding will not get you around the fact that you need a good set of entry & exit rules.

 

[robster970]

For what it's worth here are my thoughts.

Did I read that one member has worked on a method of fetching price data from Yahoo finance ? If so - WHAT ON EARTH FOR???? This will not help in the slightest.

That's me. Although the rest of your post if good advice for a newbie like me, I don't understand why there is no value in me picking up price information, storing it in my own db and then analysing it thus allowing me to develop strategies with entry and exit criteria.

If started this thread off to get advice. If I'm doing something daft, pls explain why it is daft rather than just telling me it is without qualifying so.

 

[pedro01]

That's me. Although the rest of your post if good advice for a newbie like me, I don't understand why there is no value in me picking up price information, storing it in my own db and then analysing it thus allowing me to develop strategies with entry and exit criteria.

If started this thread off to get advice. If I'm doing something daft, pls explain why it is daft rather than just telling me it is without qualifying so.

First of all - picking up price data off Yahoo will not give you anything like good data.

Secondly - go check out Tradestation & the kind of historical data you can get using their tools. I can look at a chart and get price data by volume (number of shares per bar), ticks (number of ticks per bar), Intraday (number of minutes per bar), daily, weekly & monthly. I get open, close, high, low for the bar as well. I can analyze that price data in my programs, I can analyze indicator results in my programs as well as tap into the huge amount of open source code that is out there.

Lets's say you get daily prices off Yahoo & you want to do analysis of 5 min charts - what will you do ?

You could go to Yahoo every minute and then do lots of crunching when you need a higher timeframe above that. There is no reason to do this as it is grunt work that people like Tradestation have done for you. They have also taken care of the time zones, market hours etc. etc. Even if you ask Yahoo for a price every minute, you won't get one, nor will you get OHLC data.

You need to focus on your value-add, in this case, you are not adding any value over & above what is currently out there at a much higher level of sophistication than you can hope to achieve on your own. Rather, you need to leverage what is already there and add your own layer of smarts to come up with a winning system.

You don't get points of the market for hard work, so don't re-invent the wheel, just subscribe to the data service. You can always get the tick data into your own database if that's what you want but I reckon a few hours on the Tradestation web site will send you down a different path.

Cheers

Pete

 

[robster970]

So, to summarise what I've picked so far:

1) If you're going to write code that does the legwork of data collection then make sure it's granular enough to analyse at whatever level you want and timeframes you need.

2) There isn't a real need to do (1) as there is lots of stuff out there such as Tradestation that already does the grunt work - therefore use it.

3) If you're going to spend time coding, do it in the smart area by coming up with a hypothesis and then back testing it thoroughly before using it for real.

4) Learn about risk and money management and apply it into your application.

5) Before doing anything for real, make sure your networking and computer infrastructure has redundancy built into it as the consequence of failure can be dire.

6) When you move from paper trading to doing it for real, small amounts only to learn how to be profitable. If you've back and forward tested your code thoroughly over a lengthy period of time then this should occur as a consequence of thorough prep.

7) There will be enormous value if you've traded yourself and are therefore in a position to take over from the machine should something weird happen.

8) There are plenty of helpful people in this forum with tons of experience of the markets - don't ignore the good advice.

 

[rols]

So, to summarise what I've picked so far:

1) If you're going to write code that does the legwork of data collection then make sure it's granular enough to analyse at whatever level you want and timeframes you need.

2) There isn't a real need to do (1) as there is lots of stuff out there such as Tradestation that already does the grunt work - therefore use it.

3) If you're going to spend time coding, do it in the smart area by coming up with a hypothesis and then back testing it thoroughly before using it for real.

4) Learn about risk and money management and apply it into your application.

5) Before doing anything for real, make sure your networking and computer infrastructure has redundancy built into it as the consequence of failure can be dire.

5) When you move from paper trading to doing it for real, small amounts only to learn how to be profitable. If you've back tested your code thoroughly over a lengthy period of time then this should occur as a consequence of thorough prep.

6) There will be enormous value if you've traded yourself and are therefore in a position to take over from the machine should something weird happen.

7) There are plenty of helpful people in this forum with tons of experience of the markets - don't ignore the good advice.

Er one other slightly important point usually missed by most most wanted to milk the markets via an ATS

LEARN TO TRADE FIRST

Otherwise you are wasting your time.....

 

[robster970]

Er one other slightly important point usually missed by most most wanted to milk the markets via an ATS

LEARN TO TRADE FIRST

Otherwise you are wasting your time.....

That's a given imho - it's what I meant in (7) although re-reading it, it wasn't explicit

 

[johnsue]

i would not bother with backtesting if i was you. Backtesting does not mean a thing. Forward testing is far better indication but only an indication still of course, but imo better and more reliable than backtesting. if your code has a positive expectancy over 6-12months or a full cycle then thats a good indication code works and can then be traded.

 

[hansdampf]

How is backtesting worse than forward testing? Just look at backward testing as forward testing started a few months ago.

 

[adamscj]

Whilst I agree that it doesn't always make sense to reinvent the wheel, I personally trust the software I've written more than a 3rd party app (especially when I'm putting my own money on the line).

Although I do use some 3rd party providers, for FX pairs I collect and store data directly from an api (GAIN) as I know these prices are actually tradeable. I store every tick in realtime - although currently my strategies only use daily data I know I've got the tick data to fall back on if I need it.

Personally I like the fact that I know the code inside out and have all the relevant logging & debugging information that I need as well as performance tuned to my set up. This is obviously especially important when it comes to automatic execution.

 

[robster970]

Personally I like the fact that I know the code inside out and have all the relevant logging & debugging information that I need as well as performance tuned to my set up. This is obviously especially important when it comes to automatic execution.

I know exactly what you mean. However given my complete inexperience in everything so far I'm not going to ignore Pedro's advice and am going to have a look at Tradestation before I ditch my fetch application which works pretty nicely.

 

[dcraig1]

A bunch of technical papers by John Ehlers:

eMINIZ.com - Technical Papers

 

[pedro01]

Whilst I agree that it doesn't always make sense to reinvent the wheel, I personally trust the software I've written more than a 3rd party app (especially when I'm putting my own money on the line).

Although I do use some 3rd party providers, for FX pairs I collect and store data directly from an api (GAIN) as I know these prices are actually tradeable. I store every tick in realtime - although currently my strategies only use daily data I know I've got the tick data to fall back on if I need it.

Personally I like the fact that I know the code inside out and have all the relevant logging & debugging information that I need as well as performance tuned to my set up. This is obviously especially important when it comes to automatic execution.

Are you going to set up your own brokerage too ?

How do you handle data corrections from your provider ? What do you do when you get a disconnect and can't collect the data real-time ?

I run a software company and for the life of me, I can't imagine why anyone would do what you are doing because you don't 'trust' a company that specializes in this field. You will never have the richness or reliability of data that Tradestation has in your home grown system.

This is a bit like writing your own word processor because you don't trust Microsoft.

 

[pedro01]

So, to summarise what I've picked so far:

1) If you're going to write code that does the legwork of data collection then make sure it's granular enough to analyse at whatever level you want and timeframes you need.

2) There isn't a real need to do (1) as there is lots of stuff out there such as Tradestation that already does the grunt work - therefore use it.

3) If you're going to spend time coding, do it in the smart area by coming up with a hypothesis and then back testing it thoroughly before using it for real.

4) Learn about risk and money management and apply it into your application.

5) Before doing anything for real, make sure your networking and computer infrastructure has redundancy built into it as the consequence of failure can be dire.

6) When you move from paper trading to doing it for real, small amounts only to learn how to be profitable. If you've back and forward tested your code thoroughly over a lengthy period of time then this should occur as a consequence of thorough prep.

7) There will be enormous value if you've traded yourself and are therefore in a position to take over from the machine should something weird happen.

8) There are plenty of helpful people in this forum with tons of experience of the markets - don't ignore the good advice.

Maybe a bit pedantic but number 6 should change. You should be using a demo account as opposed to paper trading. This way you get to run your strategy live but without risk.

 

[dcraig1]

Are you going to set up your own brokerage too ?

How do you handle data corrections from your provider ? What do you do when you get a disconnect and can't collect the data real-time ?

I run a software company and for the life of me, I can't imagine why anyone would do what you are doing because you don't 'trust' a company that specializes in this field. You will never have the richness or reliability of data that Tradestation has in your home grown system.

This is a bit like writing your own word processor because you don't trust Microsoft.

I think you are overstating your case a little. There are a lot of deficiencies in many of the offerings, no doubt including TS. Which is not to deny that it may well be a prohibitive amount of work to roll your own and completely beyond the capabilities of other than experienced developers.

IMHO the whole idea of attaching scripts to charts and monkeying around with global variables is just plain horrible. eSignal is just as horrible. Anything that is chart centric is just plain wrong. Of course there a quite a few alternatives, but it can be very time consuming evaluating packages only to find out that you have hit some sort of immovable object in your path. If you do your own you will never hit an immovable object, but the time and effort may be enormous.

The choice is not obviously simple, and depends on what you are doing.

 

[pedro01]

I think you are overstating your case a little. There are a lot of deficiencies in many of the offerings, no doubt including TS. Which is not to deny that it may well be a prohibitive amount of work to roll your own and completely beyond the capabilities of other than experienced developers.

IMHO the whole idea of attaching scripts to charts and monkeying around with global variables is just plain horrible. eSignal is just as horrible. Anything that is chart centric is just plain wrong. Of course there a quite a few alternatives, but it can be very time consuming evaluating packages only to find out that you have hit some sort of immovable object in your path. If you do your own you will never hit an immovable object, but the time and effort may be enormous.

The choice is not obviously simple, and depends on what you are doing.

Agreed - the global variable thing is horrible in TS but all the stuff you don't have to do makes up for that in my opinion.

My current 'obstacle' in TS is doing a scale out. Ihave to remove my stop loss, sell part of the position and re-apply the stop loss. I am using macro orders, not strategies and the issue I have is that my scale out trade sometimes fails as the order to cancel the stop loss has not yet been executed. It's a pain in the ass and there are work-arounds. In any case, I'd rather be dealing with this end of things than working on a database for trade data which is a long way from adding indicators & placing trades.

 

[dcraig1]

Agreed - the global variable thing is horrible in TS but all the stuff you don't have to do makes up for that in my opinion.

My current 'obstacle' in TS is doing a scale out. Ihave to remove my stop loss, sell part of the position and re-apply the stop loss. I am using macro orders, not strategies and the issue I have is that my scale out trade sometimes fails as the order to cancel the stop loss has not yet been executed. It's a pain in the ass and there are work-arounds. In any case, I'd rather be dealing with this end of things than working on a database for trade data which is a long way from adding indicators & placing trades.

As long as the "work arounds" don't involve such convoluted logic that it becomes very difficult to ascertain their correctness. Which is usually an indication that there is something fundamentally wrong or inadequate underneath somewhere. I know this is a way of life for small software houses doing application software, but sometimes it comes back and bites you. It is a very hard call to know when to change horses sometimes. I've certainly seem this dilemma a few times.

 

[alan5616]

Most retail traders are undercapitalised, and are also purely technical traders. These factors both point them towards trading short (usually way too short) timeframes GJ

I would agree that undercapitalisation is a key ingredient for retail traders failing to succeed. However, I disagree with you on the subject of short time frames.

We live (and trade) in interesting times. Many markets are experiencing some of the highest volatility that has ever been recorded. As a result of this, markets with high volatility can be traded, more successsfully, on a short time frame (1min/5min).

If I was trading longer time frames (daily, four hourly, hourly) at present, I would be experiencing a roller-coaster ride that I don't want to be on. By day trading on a short time frame, I can exploit this current volatility. I would rather take x pts., per day, from a 5 min chart, than get whipsawed by a daily chart that doesn't have a clear trend at present.