The definitive test

[KIMMRUNNER]

I am fascinated by automatic trading systems, and have spent years analysing them - concluding as most have , that most of what is written in standard technical trading texts may be interesting, but it doesnt work & you begin to wonder why the same myths keep getting propagated on! .

I have also found a variety of trading systems that do appear to work over long periods of testing:

These vary from systems that work on 65% of trades.

Looking at comments made on these boards there is often the criticism that even if historic data does bear out - the accusation is made that it is over-optimising

So the question is:
What is the definitive test for an automatic trading system?

Lets just stick to a single index - the trades being made only on the basis of historic close and high/low: what does a system have to demonstrate to survive the sceptic tests?

- some elements of this are -
must work in bull market as well as bear: - so I think must prove over 5 years data.

The response to a number of identified events must be random 50/50 - eg no trading system can predict 9/11, the capture of Saddam: the date of ENRON collapse etc. Need to pick 10 events and prove that the response can only be random- if indeed the system is active that day

In my view the capital employed is a major issue: so in deciding what size of position to trade on margined trading - the system must measure its own volatility. Some systems show very small accumulated losses from the maximum capital balance on the way to greater peaks, others need lots of capital. I actually prefer the
accumulated wealth from single point trading net of all spreads.

But this clearly doesnt reflect the way a trader would trade: increasing positions with capital. But what is the maximum position allowed for a standard test, in terms of measured capital risk?


I have several systems that choose to trade very infrequently - trases weeks or months apart but when they do it is with near 100% success.:, where trades placed more or less daily seem to only achieve 66% : so what is the criteria that defines BEST in this regard?


SO in summary it seems to me the rules need to be defined before anyone can say they have a winning system: I would also like to see a standard performance measure for a trading system.

ANy views?

 

[TWI]

I have done some work in this area and it seems to me the only test is on realtime data as historicl simulations may well be curve fitted and the curve fitting can be very well done so that it continues to look very stable over all variables.
This may not be exactly what you are asking but I think that as far as construction is concerned there is a fairly well defined criteria for testing systems and it is based on the splitting of data into 3 parts. the first is used for the system construction to test the idea, the second to refine the system and the third to walk forward test the system as if it is real-time. If the system performs well on the third set then it may be assumed to be reasonably stable. The actual process through these 3 steps is too complex to go into in one mail. Just the data integrity issue is bad enough at least when it comes to Futures. I am sure there is a good amount written about it however and maybe somebody could suggest a good site/books on the subject.

 

[KIMMRUNNER]

Thanks for that

The 3 portions of data is an OK way to develop systems,
but is not so good as a performance test, and it cant be used to work out whether a 3rd party system has been overoptimised.

Seems to me that there needs to be a plausibility test along the lines I suggested.

Here is the crux of the problem

I attach a spreadsheet which shows the capital balance increase of a notional spreadbet account over 5 years, ( actually the spreadsheet is reversed so day 1 is top) form trading a single point of FTSE net of all spreads:

ie it has done 15 points a day average. Notice the flat portion of the graph is the aftermath of 9/11 and for 4 months after it failed to rise: the markets were in turmoil and less predictable.

Notice that it rarely loses cumulative more than £200 before rising to new highs.

It would be easy to show a gain of £1m over the period by increasing the stake in with capital balance or £100000, or more or less any number. Obviously the greater

There needs to be a standard to measure the performance of this against any other scheme on the basis of proving that random behaviour must stay random!- ie cataclysmic events cannot be predicted: & otherwise a standard basis for example for stake per point increase with capital on the basis of capital cover relative to voltatility of the capital increase.

An APR defintion for trading schemes!

So what do I need to prove?

 

[Glenn]

A good discussion topic.

"..the rules need to be defined before anyone can say they have a winning system:"

Are you assuming that there is such a thing as a winning automatic trading system ?
Imho this alone is a very big assumption which must be proven before trying to look any further into rules etc.

There are plenty of people who can produce a backtested system which has given good results (although they may not disclose the method ). But that proves nothing.
I too have such a system which has produced 27,000 points allowing for commissions and slippage since June 1998, including walk forward testing. At the moment it is long in a down market.
I have learnt that applying some simple TA (Supp/Res) has prevented me entering losing positions generated by the system, so currently I am flat instead of 120 points adrift.
I remain to be convinced about the merits of fully automated systems and expect that a lot of time needs to pass in order to 'prove' any system. Until that happens, rightly or wrongly I will continue to override the system as I see necessary.

At a practical/psychological level, long term systems generate another quandary. It is easy to look at a 5 year equity chart and see the big rise over time. But when you look at the detail you can see that there a long flat periods and losing periods, some of which can last for months.
It is very different when you are actually trading them in realtime.
And if you happen to start trading a system during one of the losing periods, it can be rather disconcerting

Perhaps the acid test is whether a system will work on a randomly generated dataset.

Glenn

 

[KIMMRUNNER]

IMHO your reply hits the nail on the head:

"if there is such a thing" is more or less a statement that the answer as to whether there an automatic trading scheme can work lies in philosophy rather than sicence:

a very similar argumenmt occurs in theoretical physics as to whether the basic 'laws' are actuallly nothing of the sort, and just happen to be simple equations which fit most of the observed data most of the time:but because they are not fundamental, cannot be expected to fit other data as yet unobserved, or cannot be expected to fit any of the data for all time. Science cannot get to the answer of that - it is in the realms of philosophy.

All technical analysis has a basic presumption that the herd behaviour of investors can give rise to prediction of more likely directions of future movement from historic price movement.

So the moment TA is accepted, the presumption is that automatic trading is possible:

Your data looks very like one of the constituent parts of the trading system for which I enclosed data, in this one a second check is made automatically against two other forms of TA which then inhibits trading - if it does not agree with the basic trading model.

Is there any chance you can enclose the Excel values for the data you have graphed? I am curious - in as far as I can see the humps and hollows are in similar places: to this one of my schemes.

Also - please confirm this is a amximum of single £/point trading of FTSE?

It is straightforward enough to produce trading schemes - effectively based upon candlestick patterns which have 100% success rate, but the patterns do not occur frequently: for all that the statistical significance is such that the patterns can hardly be dismissed as random chance.

Moving forward , lets pick a standard of the following
and see who can come up with the best result on the following basis.

(a) Trading decisions can only be made at closing price of FTSE, on the basis of all historic high, low and close.

(b) On each day the position can only be £1 long, £1 short or
no position.

(c) Every time the position changes from by £1 in any direction costs £1.5 spread - so for avoidance of doubt going from £1long to £1 short costs £3, or 0 to £1 long or short costs £1.5 - which are the effective D4F rates

(d) the model must trade for 6 years from 1/3/98 to present to ensure it includes bull and bear markets.

(e) The trading system must be an automatic function that can include any amount of maths or logic but for the sake of argument must be simple enough to be encoded in excel spreadsheet without recourse to VB macro.

(f) Produce the notional accumulating balance of the trading scheme. Track the maximum to date value of that balance for each day in a separate column. Then define "minimum capital employed" as the maximum difference between the daily balance and that tracking maximum, adding in £50 for a notional TR of a point of spreadbet.

(g) The winning scheme is the return of final balance tomin capital employed.

I think that to compare like with like it is necessary to produce such a set of rules .

I enclose the calculation for the scheme I enclosed

It is easy yo conclude that an infrequent trading scheme with 100% performance will beat almost any other scheme, because the capital employed is only £50.00 - there are a variety of candle patterns that trade maybe 5-6 times per annum that do well on this rating.

WOuld anyone like to join in and beat the 700% per annum or so produced by this scheme according to those rules?

Instinctively the schemes I prefer of all are hedge trading schemes of pairs of indexes: the main problem with doing this is you cannot use market close for the calculation because all of the markets close at diffferent times: so collecting relisable data for all of the indexes at (say) 4.30 is a serious problem.

US Market open figures are hopeless - you cannot trade at them.
Mytrack figures have significant errors too.

 

[KIMMRUNNER]

Oops

Did the sum too rapidly, thought it sounded wrong!

See new spreadsheet: is 185% p/a compound for 6 years.

Amazing the difference a bracket in the right place makes..

 

[Glenn]

Quote "Is there any chance you can enclose the Excel values for the data you have graphed? I am curious - in as far as I can see the humps and hollows are in similar places: to this one of my schemes.
Also - please confirm this is a amximum of single £/point trading of FTSE?"

Calculation uses £1 per point at all times. It's not the FTSE, so direct comparison with your data is not really worthwhile imo. However if you still want the equity data I can post a link.

Quote "So the moment TA is accepted, the presumption is that automatic trading is possible:"
But an automated system may not be able include code to cover all the future possibilities. Some TA cannot be coded easily or at all. e.g. How do you code for a major supp/res level which may be from months ago ?
'Accepting' TA could either mean blind faith or an acceptance that it appears to provide an edge some or most of the time.
I prefer to work with the latter thinking and to 'interfere' with a system rather than using it as an act of faith.
Only time will tell whether such interference improves or spoils the results, or that the system works well enough if left alone.

Glenn

 

[TWI]

I am in little doubt that a successful, fully automated trading system is a reality. One example would be Aspect Capital who have well over 2bln under management and run everything via a fully automated system. Their trading floor is very impressive with different traders covering different market sectors. They have screens that tell each trader what he has to buy in terms of price and volume over the next x minutes. The traders job is to execute at as good a price as possible and preferably better than the price shown on the screen. One of the premises of their success seems to be diversification. I think that it is far harder to construct a long term consistent winner for one market than it is for a diversified portfolio.

 

[Thirteen]

wot bout DE Shaw? big wall st firm/bank who do nothing but technology. all their systems are mechanical so although profitable, they aint outstanding despite their obnoxious attitude/website

 

[growltiger]

Oops

Did the sum too rapidly, thought it sounded wrong!

See new spreadsheet: is 185% p/a compound for 6 years.

Amazing the difference a bracket in the right place makes..

Still not quite right, I think. Your 185% figure includes the capital, so it should actually be 85% compound. That is still quite an impressive result, over nearly 6 years. What I cannot see, from this thread, is any indication (however carefully wrapped up) of what sort of system is meant to have produced the numbers.
Was I missing something?

Philosophically, I am inclined to think that a rules-based system that has generated a statistically significant edge is no worse off than any other statistically-justified proposition. So I don't think that the whole quest is a wild-goose chase. The problem, however, is whether the sort of historical series you have produced is actually evidence of a robust system, or just prolonged and outrageous good fortune (although I admit the strength of support for the system must increase with the length of the series).

 

[Vaco]

For me the only true test of if a system is successfull in all market conditions is to see what results it would give you on another index say for example the ftse 250 or even a non uk index, after all the primary index you are looking at may very well behave like one of the other indexs' one day or another.

 

[Glenn]

6 months later .....

Referrring to the chart posted earlier in this thread (post #4), here is a chart of what has happened since, from Feb 2004 to tonight, without any interference.
Attempts at interference have proven worthless. In fact the overall profits were reduced in all the approaches tried. The only saving grace of interference appears to be that trading is curtailed during some of the longer losing streaks. But by the time you start trading the system again, it usually has a higher equity than when you stopped. There may be ways I haven't tried yet, but can't thing of another at the moment.

This method includes allowance for costs, so the chart shows the net profit (in points), with no compounding (i.e. assumes the same stake per point throughout).

Who knows whether it will continue to work, but it's been doing ok as you can see.

In the spirit of the thread, does anyone else have a similar update on their own previously backtested system ?
Glenn

 

[blackcab]

Are the systems you're talking about hand-written or generated/evolved by an automatic process? I'm working on 'evolving' neural nets to produce signals from a few years of daily OHLCV on stocks and indices.

 

[Glenn]

Are the systems you're talking about hand-written or generated/evolved by an automatic process? I'm working on 'evolving' neural nets to produce signals from a few years of daily OHLCV on stocks and indices.

If your system will always continue to evolve, then I suppose the questions is what results has it achieved during a period of time since it was at an earlier evolutionary stage ?
If you are expecting it to evolve to a point where you effectively freeze it, then the measurement would start from there.
Glenn

 

[The General]

FWIW, I tried a spot of data mining a while ago. I had a databse of technical indicator levels and asked the program to find me 10 wins in a row. i.e. the combination of indicator values that gave me a historical win rate of 100% based on 10 (or more) observations. The system duly obliged.

The thing you are missing is that in the case of the example I give and that of optimisation, 10,000 tries to find ten moves (say) in the same direction is more inevitable than not (to come up randomly). Hence such research is futile.......

 

[stevet]

Question
"What is the definitive test for an automatic trading system?"
Answer
10 years real time trading

but a lot of money is made out of sytems that only work for months and when they stop working - plug in another one

 

[The General]

Lads (and Lasses),

By definition, we should only use systems that have performed well in the recent past. Why ?

hink abt it - everyone is keen to jump on the Nasdaq when it starts rallying a coupla years ago, but the 15 yrs up until that point were go no-where. Hence, intuitively, we are drawn to markets that are performing well now...

So why not apply the same ideals to systems !?

 

[Tuffty]

Question
"What is the definitive test for an automatic trading system?"

In my ‘philosophical’ view I have never come across an automatic trading system. They always appear to be implemented by a person who decides to trade it.

You are seeking a definitive test to see if they work and I assume from this a definitive black or white answer being yes it's a winning system or no it isn't. From my ‘philosophical’ view I can only see grey being it may or may not work.

 

[stevet]

the bulk of automated trading is risk based and is 100% full automation and an incredibly high percentage of all daily trading volume

 

[JTrader]

If you're a mechanical trader that doesn't believe strongly in the value of backtesting (due to issues with data intergrity, slippage etc.) and would only have faith in a strategy after forward testing it in real-time, on what basis would you decide upon the parameters of your strategy?

Would you backtest the strategy and optimise the parameter settings, and then forward test it - thinking that these are the parameters that are more likely to be successfu in real-time?
Or, would you ever simply devise a strategy that looks like it may be a winner, and simply start forward testing it alone from the start, without looking at backtesting and optimisation?

Thanks

jtrader.