A small matter of programming

[rnicoll]

Up 3.41% before margin, after commission, since Sunday night. With a margin of 50:1, and 5 currencies "in play" dropping the margin to an effective 10:1, that means about 34% profit in 2 and a half days.

Can I just say, w00t?

Now, someone's going to point out it's got a few trades lucky, to have made this much. That's kind of the point though, this system looks for opportunities and takes them. Most of the time it does okay-ish, but if the conditions are right it makes a fortune.

Downside is, the GBP has collapsed vs USD. That's not so good for me.

 

[Adamus]

I don't understand all this complikated maffmatix that you use in forex.

wot's a margin of 50:1 in plain terms that a futures trader can grasp?

 

[rnicoll]

I don't understand all this complikated maffmatix that you use in forex.

wot's a margin of 50:1 in plain terms that a futures trader can grasp?

Margin of 50 times my stake. I put in £1, I get to trade £50. Is there a clearer shorthand I should use, I figured ratio notation was a good way of describing it?

 

[rnicoll]

Now down to 0.29% up over the week. Admittadely, still profitable by Thursday is a new best for my stuff, but given it hit nearly 4% up it's somewhat depressing. Tried trading by hand again. Am now down £1,500. I can deal with that, what is getting me is I don't know WHY it's not working.

Feeling a bit lost. The code seems to make money reliably over time, in my own backtesting (which uses actual bid/ask prices to ensure spread is correctly calculated, for example), but it's not making nearly as much as my theory says it should. The real issue here is "over time" - if it loses too much at once, I go bankrupt and fail to continue trading to make it back again.

Lack of information is killing me. I don't know how much is reasonable to expect to make, how many trades per day is reasonable, whether I'd be better trying index futures instead of currencies, etc. etc. etc. and the cost to try each theory is high. Without my own trading being successful I have no working theories to code, either.

Very frustrated. Any advice on how to work on my by-hand trading would be appreciated. I'm open to most theories, but by choice would generally go for swing trading on index futures or currencies.

 

[Adamus]

I wouldn't trade manually, if I were you. If you programmed your trading strategy and it's not working, what makes you think you can do it better than the computer?

I'd be interested to see your back-test analysis and what the stats are telling you. That should give you a better idea of whether you should stick with what is probably a winning system or whether you've just curve-fitted your parameters or some other mistake.

I've got an idea what you're suffering from. I personally am physically scared that the same thing will happen to me when I put my trading program into action. All that time and energy invested out the window (and probably money at that point too)! Gives me a tightening sensation in the bottom of my stomach.

 

[rnicoll]

I wouldn't trade manually, if I were you. If you programmed your trading strategy and it's not working, what makes you think you can do it better than the computer?

The fact there's significant evidence of people making money doing this by hand, and not so much by automated trader, suggests to me that at the very least it's the fastest path to something that reliably makes money.

I'd be interested to see your back-test analysis and what the stats are telling you. That should give you a better idea of whether you should stick with what is probably a winning system or whether you've just curve-fitted your parameters or some other mistake.

I've got an idea what you're suffering from. I personally am physically scared that the same thing will happen to me when I put my trading program into action. All that time and energy invested out the window (and probably money at that point too)! Gives me a tightening sensation in the bottom of my stomach.

I've attached the backtests for my previous system, as it to be honest comes out a lot more positively than my current approach, on everything except EUR/USD. I suspect these aren't going to cover the stats you want to see, if you tell me what would help I'll modify the report generation code.

All backtests are done on per-minute bars, with spread simulated accurately (but not slippage), and exclude commission (about 3% on most of these, over the period covered).

 

[Adamus]

The fact there's significant evidence of people making money doing this by hand, and not so much by automated trader, suggests to me that at the very least it's the fastest path to something that reliably makes money.

Maybe you're right. I guess they're called 'bot traders' like the bots that play online poker and so on. I think that though is a different discussion that's probably not what you want here in your Trading Journal.

Suffice it to say IMHO you should be able to turn on an automated trader and "for a limited time only" make good profits while the suitable market conditions prevail. You just need to have the stats from your back-testing that tell you when the market conditions have shifted into and out of your system's happy-trading-zone.

I suspect these aren't going to cover the stats you want to see, if you tell me what would help I'll modify the report generation code.

total profit
total trades
# wins
# losses
sum all profits
sum all losses
avg winning trade
avg losing trade
largest win
largest loss
max drawdown (dollars)
max losing streak (time)
rate of return
equity curve (correlation to 45 degree upward slope!)
% of 6 month periods that were profitable
% of 12 month periods that were profitable
% of 18 month periods that were profitable
geometric mean trade (taking into account distribution of returns)
geometric per annum
optimal f and terminal wealth ratio (Ralph Vince stats)

If you want to do a good graph, then run your PNL through a randomizer and produce a bell curve of your maximum drawdown, showing 1 and 2 Standard Deviations. (I haven't done this yet, but you use java right? You just keep applying Collections.shuffle() to your ArrayList of PNL.)


In terms of what you said in an earlier post about being unhappy with the performance and not knowing where to go from here, I think that having stats like these give you that knowledge.

It's not that you can't trade without them, as obviously you can and do, and so do many highly successful traders, but you have the system that can generate these sorts of stats easily.

I mean, you've got an amazing resource there. You've built an automatic trading program! It's the stuff of sci-fi almost. The algorithm that your automatic trading program uses is secondary - if you stick at the research, I'm sure you can find something to slot into the program that will kick ass. Like Markus BSD said in another thread, all it takes is a 'simple, robust system' and pretty soon with compounding, you'll be making serious returns. Just make sure you don't program in any artificial consciousness!

And another thing that impressed me, if you can handle the adulation is that you are actually putting this system live already, so you can get a very good handle on the slippage and other costs involved per trade. That certainty will help you enormously in back-testing - currently I've got no idea whether I should be applying £5 or £50 as my commission and slippage in my trading program.

Well - enough splurging all over your trading journal! Do you do this full time or do you do 9 to 5 somewhere?

 

[rnicoll]

Maybe you're right. I guess they're called 'bot traders' like the bots that play online poker and so on. I think that though is a different discussion that's probably not what you want here in your Trading Journal.

Well... I hear of people with automated traders, but frankly most of them either seem to be doing the heavy lifting (as it were) for a human trader, or attempting to scalp tiny bits here and there. Problem with scalping is it makes your data feed, connection speed and general connectivity much more of an issue. To me, I see it as much more sensible to work with the constraints you're presented, rather than fighting them head on.

So, a lot of people worry about slippage, and being able to get a trade quickly enough. Me, I see it as better to use limit orders for entry/exit, skipping both issues completely. Make sense?

Suffice it to say IMHO you should be able to turn on an automated trader and "for a limited time only" make good profits while the suitable market conditions prevail. You just need to have the stats from your back-testing that tell you when the market conditions have shifted into and out of your system's happy-trading-zone.

Ah, we're on the same page, excellent I'm a big believer, and becoming an even bigger believer, that an automated trader basically changes the nature of trading. Instead of requiring me to concentrate all the time, means I trade by development of strategies well suited to the market as it is _now_.

(big bucket of stats cut - I'll work on those soon)

If you want to do a good graph, then run your PNL through a randomizer and produce a bell curve of your maximum drawdown, showing 1 and 2 Standard Deviations. (I haven't done this yet, but you use java right? You just keep applying Collections.shuffle() to your ArrayList of PNL.)

All profits and losses from the sample period, yeah? Will add it to the to-do list

In terms of what you said in an earlier post about being unhappy with the performance and not knowing where to go from here, I think that having stats like these give you that knowledge.

It's not that you can't trade without them, as obviously you can and do, and so do many highly successful traders, but you have the system that can generate these sorts of stats easily.

Yeah... when I say I'm not sure where to go from here, it's in terms of "So many options, so little time"!

I mean, you've got an amazing resource there. You've built an automatic trading program! It's the stuff of sci-fi almost. The algorithm that your automatic trading program uses is secondary - if you stick at the research, I'm sure you can find something to slot into the program that will kick ass. Like Markus BSD said in another thread, all it takes is a 'simple, robust system' and pretty soon with compounding, you'll be making serious returns. Just make sure you don't program in any artificial consciousness!

It's nothing actually that revolutionary. NinjaTrader have been doing this for years, and there are related projects such as JBookTrader (which I'm merging my work into at the moment).

Also, are you sure about the consciousness? Neural nets do look like a promising approach...

And another thing that impressed me, if you can handle the adulation is that you are actually putting this system live already, so you can get a very good handle on the slippage and other costs involved per trade. That certainty will help you enormously in back-testing - currently I've got no idea whether I should be applying £5 or £50 as my commission and slippage in my trading program.

Oh, it's not live for long at a time, most of the time I run it with Interactive Brokers' paper-trading account, doing market entry/exit by market orders. Not fantastic, but provides a reasonable idea of how it would behave under real conditions.

What are you thinking of trading, and I can probably help you with commission & slippage figures?

Well - enough splurging all over your trading journal! Do you do this full time or do you do 9 to 5 somewhere?

This is just hobby stuff for me, I'm a full time computer science researcher 9-5. Normally that's not a very hands-on role, but in my case I'm primarily a developer/sys-admin crossover. My areas of speciality are multi-threaded/distributed systems, and web applications, but I can do most stuff. AI's a weakness, alas, because neural nets really do look promising...

Mostly though, don't worry about my ability to code stuff, as long as I can get my head around the algorithm

 

[rnicoll]

A bit time challenged right now, but here's how I'm doing with the new stats so far.

Geometric mean... erm, doesn't apply to negative numbers. I can do profits and losses separately?

 

[Adamus]

Hi R

would you include a time frame with those stats as well? I see you've named the PDF accordingly but it would be cool to have it in the same table. Perhaps trades / day would be better so you don't have to think about what were trading days etc.

I notice your max drawdown was zero. It would be good to adjust your code for that - your max drawdown can only ever be as small as your biggest loss - and that's highly unlikely unless it's a really short trading period.

Regarding the Ralph Vince stats such as geometric mean, it is only a valid stat if you actually make a profit. It's not interesting and possibly impossible to calculate if you lose overall. You need to decide on your risk level for that - something that Ralph Vince (in Mathematics of Money Management) naively sets to the same as your biggest loss in your PNL - which is obviously never going to be the biggest loss in the future.

Hope that all makes sense.


ps the forex mkts are closed Sat/Sun aren't they? And the trading hours? Is it truly 24 hours or is there a big lull between NY close and Tokyo open? As far as I think I know, not much goes on in the West Coast & NZ time zones Just trying to work out how many days that is from 01 to 22 Oct. By my rough calculations though your system's doing 1.5 trades per hour.

 

[vincentlynch]

been fully enjoying this thread.

 

[rnicoll]

Good grief, has it really been a month?

One thing after another has kept me busy the last couple of weeks, in no small part having to give sessions at a couple of workshops for the day job.

Currently trying to get time to finish some changes I started to JBookTrader, and then the next stop is more work on my own platform. Basically it's going to get a re-write into a semi-automated system, where it continually provides feedback to the users with options to intervene. Lets take this weeks results so far (attached), as an example. I think the USD is overvalued, and therefore would choose not to enter any trades that are long the USD. The auto-trader didn't make this decision.

The auto-trader, after commission but before margin, is 1.91% up this week. Without the long USD trades, it would be 2.16%. By excluding those long USD trades, I can increase the profit and decrease the risk and commission.

Secondly, the auto-trader finds it hard to judge good entry prices, quite happily trying to trade a massive spread at times. Having someone be able to manually tweak entry and exit prices should also significantly improve profitability.

I'll keep you updated as I make progress.

 

[ziduan]

Could you just remind me what software/Platform is this you are using?

Thanks

My general feeling is that the longer time scale, the more fundamentals matter, and therefore the less easy technical analysis is. Commission is calculated by NinjaTrader for me, that's not an issue (except that it "helpfully" doesn't do it by default). Slippage's a bigger issue, and is basically why I'm after per-tick data for use in my back testing, as it's the best way to be sure.

Remembered I had a second Windows box with NinjaTrader, so here's a bunch of results, plus the strategies used to generate them. All are done on per-minute data since the start of the year, with the default parameters included in the strategy code. "Mind the gap" is a bit odd, and you need to sure to give it market open/close times for the underlying market of the future it's used on, and ensure that "Exit on close" is unchecked, as it trades the market close/open gap.

If you get any of them to work in reality, do tell me

 

[rnicoll]

Could you just remind me what software/Platform is this you are using?

Thanks

Custom build application, written in Java. Uses the Interactive Broker's API for data feed and trading, and MySQL as backend database. I do occaisional source code releases, but can't find where the most recent is, right now, and don't have time to track it down.

 

[ziduan]

Custom build application, written in Java. Uses the Interactive Broker's API for data feed and trading, and MySQL as backend database. I do occaisional source code releases, but can't find where the most recent is, right now, and don't have time to track it down.

You downloaded the histrical intra-day tick data into the MySQL DB, then ran back-test using your own algorithm and order managements?

The resutls are very impressive!!!

 

[ziduan]

Custom build application, written in Java. Uses the Interactive Broker's API for data feed and trading, and MySQL as backend database. I do occaisional source code releases, but can't find where the most recent is, right now, and don't have time to track it down.

But why couldn't you do this on their platform? Do they not have the programming environment on the platform?

 

[samuelhogg]

How is it going with the trading? I am enjoying your thread.

Think C++ would be a good place for a non programmer to start?

 

[rnicoll]

But why couldn't you do this on their platform? Do they not have the programming environment on the platform?

No, it's intended to be done through the API they provide. Their platform still provides actual connectivity to Interactive Brokers, and I can use it to check what orders are being executed, while my code primarily handles the automation (somewhat similar to NinjaTrader; much less powerful, but does what I need and I know the quality of the code).

 

[rnicoll]

How is it going with the trading? I am enjoying your thread.

Think C++ would be a good place for a non programmer to start?

Oh good grief no!

Erm.

So, not really? I'd recommend F#, as I think functional programming is ideal for this; I didn't use F# because when I started, I hadn't heard of it, but am seriously considering moving over. F# will work with anything you can find a C# API for (even if I haven't got far enough into the book to understand how, so far). If not F#, then Java or C#. C++ is an old, old language, whose primarily benefit (speed) is minor these days, and you'd get more from a good book on algorithms.

We're recommending Algorithm Design: Foundations, Analysis and Internet Examples: Michael T. Goodrich, Roberto Tamassia: Amazon.co.uk: Books for learning algorithms, at the moment, apparently. For F#, I'm reading my way through Amazon.co.uk: Foundations of F# (Expert's Voice in .Net): R Pickering: Books . Can't readily recommend any books on Java, I haven't read one in over a decade...

 

[rnicoll]

This week's trades attached. Bottom line, 0.33% down.

That's not the important bit. The important bit is the last trade. Lost 0.44% in 46 seconds. Hell, my stuff works off minute bars, that's insane. It also highlights something I think is currently killing my system; it's far too willing to hop in with high spread.

Lets imagine I knocked 1 pip of spread off each trade (entry and exit). 92 trades, 1 pip each way, 184 pips. My system is trend-reversal, so it shouldn't be too hard to do this, either, as it involves putting limit orders directly into the path of the market.

Secondly, not only does this mean a lot less loss, it means that the stop-losses are just a little bit further out. I could be wrong about this, but I'm hoping that means less being stopped out, and more trades ending up profitable/near profitable. Will it actually? Hard to tell, but as soon as I have time I'll code it up and we'll see!