BACK TESTING - who does it

[commanderco]

Just interested to see who back tests their systems and by what means.

I used to back test mechanically, but I have given it away and now test manually
to enable me to interpret the prevailing market mood.

I find the market mood to too complex for my programming skills and so
it is back to mark IX eyeball.

It is all in the placement of stops in anycase.

 

[frugi]

I have much the same attitude as you by the looks of it commander.

I used to backtest and indeed trade mechanical systems but soon gave up as I didn't like the blind methodology or the results. (No offence to those that use them effectively, of course.)

A system which requires the mighty human brain to distinguish and contextualise patterns cannot, AFAIK, be mechanically backtested. However there is an immense amount of value to be had from manually going back over old charts to see what the best parameters for stops and targets would have been, given prior subjective confirmation of patterns and signals. Having spent the (sober) part of Easter eyeballing Dec and Mar YM futures I now have a jolly good idea o where to place a safe stop and realistic scaling out targets for one of my setups so I'm happy.

Of course the market now knows how much work I put into this and will adjust accordingly :cheesy:

 

[commanderco]

I have much the same attitude as you by the looks of it commander.

I used to backtest and indeed trade mechanical systems but soon gave up as I didn't like the blind methodology or the results. (No offence to those that use them effectively, of course.)

A system which requires the mighty human brain to distinguish and contextualise patterns cannot, AFAIK, be mechanically backtested. However there is an immense amount of value to be had from manually going back over old charts to see what the best parameters for stops and targets would have been, given prior subjective confirmation of patterns and signals. Having spent the (sober) part of Easter eyeballing Dec and Mar YM futures I now have a jolly good idea o where to place a safe stop and realistic scaling out targets for one of my setups so I'm happy.

FRUGI

I am glad you are happy
Might I enquire as to which part of Easter was declared "sober"

 

[TWI]

Mechanical backtesting is a very useful tool, it just has to be used correctly, that is, not to find highest returns but rather to establish stability. Why would anybody not want to use functionality like this?

 

[frugi]

Might I enquire as to which part of Easter was declared "sober"

"s" thru "r", give or take a pint

 

[FetteredChinos]

Ithe mighty human brain

speak for yourself.

im still at the mighty acorn stage.....

 

[frugi]

I speak for all of us Fettered, least of all myself

 

[FetteredChinos]

Mechanical backtesting is a very useful tool, it just has to be used correctly, that is, not to find highest returns but rather to establish stability. Why would anybody not want to use functionality like this?

seconded.

 

[commanderco]

Mechanical backtesting is a very useful tool, it just has to be used correctly, that is, not to find highest returns but rather to establish stability. Why would anybody not want to use functionality like this?

TWALKER
That is my point.
I could never teach the machine to input everything that I could see on the charts and
give me the sort of stability that I could see.
Hence I was getting an equity curve that looked like a drunk.

Now I am satisfied with manual testing, I ask myself why bother with mechanical
systems.
I still use mechanical triggers for my components to make certain that I am picking them all up.
The human mind still has a tendency to see only what suits it at times.

As I say to my Wife.
"I just want to win Miss World and work with children"

 

[TheBramble]

I don't backtest at all. I employ a handful of basic strategies, tried and tested through use & time as my main trading tools. My complete 'live' set of strats change over time with only a couple being genuinely constant (no tweaks). New ones are being researched all the time, but only on a forward/onward testing basis.

The window of usability of anything other than my vanilla-flavour strats (and probably your vanilla-flavour strats too ) are too narrow IMHO to consider backtesting as anything other than an initiation to curve-fit and optimise an instant in time that has a high probability of being unique, all factors considered.

Backtesting was however, an extremely useful device in the early days to give me more screen time and exposure to instruments and the markets.

 

[Trader333]

I went through a stage of backtesting to death and it was useful in my development but it is not something that I would do again now.


Paul

 

[commanderco]

I have much the same attitude as you by the looks of it commander.

I used to backtest and indeed trade mechanical systems but soon gave up as I didn't like the blind methodology or the results. (No offence to those that use them effectively, of course.)

A system which requires the mighty human brain to distinguish and contextualise patterns cannot, AFAIK, be mechanically backtested. However there is an immense amount of value to be had from manually going back over old charts to see what the best parameters for stops and targets would have been, given prior subjective confirmation of patterns and signals. Having spent the (sober) part of Easter eyeballing Dec and Mar YM futures I now have a jolly good idea o where to place a safe stop and realistic scaling out targets for one of my setups so I'm happy.

Of course the market now knows how much work I put into this and will adjust accordingly :cheesy:

FRUGI
Straying off the subject for a minute, I would be interested in your thoughts on the YM.

I trade CME IMM and am looking to expand into e minis for more food. ( YM,ES,NQ)
The currencies have become highly colineated as the DX becomes the driver and
the fishing is not what it used to be.
Mind you, I now have more hooks on the line than before, so one must not grumble.

Muchas gracias

 

[Splitlink]

Mechanical backtesting is a very useful tool, it just has to be used correctly, that is, not to find highest returns but rather to establish stability. Why would anybody not want to use functionality like this?

I try to find out how much, in points, that I would lose if I placed stops in certain spots, to help me to place them correctly. How much I would have made is for dreamers. In any case, a maximum of six months is enough for me and more often three.

Split

 

[commanderco]

I try to find out how much, in points, that I would lose if I placed stops in certain spots, to help me to place them correctly. How much I would have made is for dreamers. In any case, a maximum of six months is enough for me and more often three.

Split

Agreed Split,
I use a method called RC ( rat cunning)
It has a component called CMA ( cover my ****)

Nowhere do I consider the number of points I will make.

I used to dream of profit points until I realised that my dreams
costing me money.were

 

[Markus_H]

IMO backtesting can be a very powerful tool if used correctly.

The problem is that many trader over-use the functions provided by the different backtesting software packages and think "more is better". Many so-called "system developers" try to imply that the longer you backtest the better and more robust your system will be. That's not always true.

Let me use the e-mini S&P as an example. In 2000 the average daily range was 100-150 ticks per day; in 2004 it was only 40-60 ticks per day. If you backtest any trend-following daytrading system in the e-mini S&P you will see that it worked perfectly until 2002 and then suddenly falls apart. It seems that there are no more intraday trends. That's not surprising as the daily range of the e-mini S&P decreased by more than 50%.

What happened?

There are a couple of reasons. Probably the most important one is the introduction of the "Pattern Day Trading Rule" in August and September 2001by the NYSE and NASD: "If a trader executes four or more day trades within a five business day period then he must maintain a minimum equity of $25,000 in his margin account at all times." Because of this rule made traders stopped daytrading equities and started trading the e-mini S&P future instead.

Look at the sudden increase in volume in the e-mini S&P in the beginning of 2001:

Many of these stock daytraders used methods to scalp the market for a few penny. Using the e-mini S&P they suddenly had a much higher leverage, paying less commissions, and their methods were extremely profitable.

Unfortunately these scalping methods kill an intraday trend almost instantly, making almost every trend-following approach fail.

Another reason for the dramatic change of the market was the introduction of the automated strategy execution in TradeStation. In 2002 TradeStation's customers who were using this feature increased by 268%. Overbought/Oversold strategies became very popular and when the market made an attempt to trend these strategies immediately established a contrary position.

Conclusion
When backtesting you need to know these things. It's not enough to just run a system on as much data as possible; it's important to know the underlying market conditions.

In non-trending markets like the e-mini S&P you need to use trend-fading systems, and in trending markets like commodities you should use trend-follwing methods.

And that's when "clever" backtesting helps you:
If your backtesting tells you that a trend-following method worked in 2000-2002, but doesn't work in 2003 and 2004 then you should not use this strategy right now.And vice versa: When you see that a trend-fading method produced nice profits in 2003, 2004 and 2005, then trade it.

I haven't yet seen a strategy that works in all market conditions: trending and non-trending. Usually a strategy works very well in ONE market condition (e.g. trending) and produces small losses in the OTHER market condition. That's why you need to alter trading strategies.

And THAT'S where backtesting can help you.

Just my two cents...

Markus

 

[FetteredChinos]

good stuff markus.

i found the same, and the only way i have been able to get anything to work consistently is to combine both trend following and trend fading..in the same strat. it backtests well all the way back to 1994.

i use multiple positions, enter my trades counter-trend, and exit with trend-following criteria (ie when a trend has just been established)

kind of in reverse to most classical methods. but hey, it is the most stable thing i have come up with over the past decade...

 

[Tuffty]

Good post Markus. It is interesting looking at the constantly changing market structure (ie regulatory rule changes and the impact on the markets). I also think volatility was just mean reverting from a period where it was historically high so any one trading intra day won't make as much as their fixed cost base doesn't change when volatility changes.