Backtesting - is it a valid concept ?

Rognvald

Established member
916 15
I didn't say I was - only that if there is a gap between perceived value and price then the price will change accordingly. Expectation/sentiment is a substantial element of price. That is my opinion - it may not be yours.
 

Glenn

Experienced member
1,040 118
jon
The supply demand situation you describe is similar to the analysis of PnF charts.
e.g. double and triple tops and bottoms etc.
Dorsey provided stats on the success rate of various such patterns in his first edition of Point and Figure Charting. Note these have been withdrawn from his later edition.

Not wishing to go off on too much of a tangent here but I believe he didn't adhere to his own medicine i.e. Long patterns work better in bullish markets, so use them in those conditions, and vice versa. Going long on a breakout in a bear market is somewhat risky etc.
So looking at an individual stock chart is not enough. The prevailing market at any time has a great deal of influence.
"75% of the risk in any stock is in the market and sector" - Dorsey.

Grey1
Agree the problem of market costs. Suggests aiming for longer term positions, larger moves, fewer trades.... (and consequently needing bigger b*lls.)
As regards correlation of test results to the future, I think there is a danger in looking at individual stock charts for the reasons I gave above.
Indexes are a different matter because they tend to encapsulate most of the sentiment at any time. But even they can be swung by large caps, a rush into/out of techs or utilities etc.

I suppose I'm saying that backtesting needs to have more than one layer in order to take account of the available information.
You can test drive a car with bald tyres in the dry and get consistent results. You buy the car and then it rains........
Glenn
 

Grey1

Senior member
2,186 178
Quote "You can test drive a car with bald tyres in the dry and get consistent results. You buy the car and then it rains........ "

LOL.. :cheesy: :cheesy: :cheesy: :cheesy:
 

BBB

Experienced member
1,071 3
I think time frame has a lot to do with backtesting validity.

If you are backtesting an intra day system the results would probably be less reliable due to the manipulation that goes on. A daily or weekly system may have more valid results as price is harder to manipulate over the longer time frame, and therefor you get a better picture of the true demand and supply and the sentiment/participants reactions to the market conditions.

I have never tested this idea, its just my feeling (personally, I don't care much for backtesting at all due to my trading style).
 

mr_cassandra

Well-known member
349 36
A couple observations

Basically everything everyone you know is doing has been derived from a back-test of some sort, whether they realize it or not.

That being said, from my experience with my own system, I’ve found the following to be true of back-testing.

1. It only proves that your method worked during the time you checked and only for the market events which occurred during ‘that’ time’
2. The longer the period of the back-test, the better. It would be great if it covered enough market types to include a bull market, flat market and a bear market.
3. I’m only a few cycles into real-time trading with my systems version 21 and can tell you from experience that what has derailed each prior version was an event or market anamoly which was ‘not’ programmed in and/or occurred in the prior several years back-test.


Glenn said:
I wonder what others think about the concept of backtesting ?
One the one hand it perports to find the best set of parameters for a given method and thereby tune it to the period tested.
If the results are good, then these parameters can then be used with some (?) confidence for live trades.
On the other hand one could argue that backtesting is just curve fitting and therefore simply 20:20 hindsight.

Seems to me that for a backtest to have any validity, it must produce a smoothly increasing equity (profit) curve with no wild girations. This at least demonstrates that the method and parameters found worked fairly consistently over the period tested.
Of course a lot depends on the period(s) selected.
Again to have any validity, I think that a test must either be run over a considerable time period, embracing many different market conditions and yet still remaining consistent, or it must be run on selected market conditions with a view to using the optimised parameters from the test in that type of market.

To disprove the theory of curve fitting, one suggestion is to take a period of time, say 5 years, and run your test over the first half of the data to get some results.
These results can then be run over the second half of the data as if it were the unknown future. i.e. a kind of Forward Testing.

One thing's for sure, the Software vendors must love things like backtesting. What an opportunity to develop and sell a nice technical product, using the math co-processor and the disk drive of your PC to the full. A nice 'gadget'.
Lovely jubbly - at least for the vendors.

Enough from me. What do others think? In particular anyone who has used a backtested system in the real market sucessfully.

Glenn
 

Glenn

Experienced member
1,040 118
Hi Mr Cassadra
Agree with your thoughts.
I've continued experimenting with one system which was tested since June 1998. This covers quite a range of market types.
One modification I have employed is in respect of the Equity curve itself. By applying a simple 30 day moving average to the Equity curve it gives a signal when to stop trading the system (because it has gone into a losing phase) and when to start again.
6 months on and it's still working ok, including a 40-day 'stay out' period.
The system itself is designed to take account or recent volatility (or the opposite) and respond accordingly.
This way it tries to include current conditions and data in the math model, rather than just relying on an indicator etc.

Have you tried what I described as 'forward testing' ?
e.g. Test and refine a system over 3 years and then run it forward over the next 2 years to see if it still works. (These are just example numbers, - the longer the better in both cases imo.)

Glenn
 

georgek

Newbie
7 0
Feel So Lucky



I'm into FUTURES DAY TRADING , E-minis.

I've traded for a couple of months,like everyone,lost money,no system,no method,no discipline.
It's been 2 or 3 months now,that i started developping a SYSTEM.Based on MFL(Metastock Formula Language) and backtested on 5-min intraday data.

I can't say more than i 've seen.

When i reached at a point where i thought my system was good (always testing for a month or 2 of intraday data) i went back to the past and tested it LIKE IT WAS(with no changes) for another different period of time.

The results were bad.There was no profit,there was loss.I Optimized my system for this period,and the parameters changed,now there was profit.

In the next days(the following week) i'm planning to start trading with that system.
What i'm hoping?
What i'm hoping is : Trade with the last good system parameters that i have.Trade for a week.
Have profit similar to the one i had for the past.Re-calculate my parameters at Friday and continue trading at Monday with the new ones.
Then wait till 1 month (22 trading days) finishes and then see if i'm positive or not.

I can't say that i'm optimistic , i wasnt before, and now,with what i've read , i'm more scared of what could happen.I trust my system though.I trust what i've created.I hope it will go well in trendless and trend days..

I'm looking forward to your comments

ps:feel lucky because i found this forum,especially this topic.
 

Glenn

Experienced member
1,040 118
Hi Georgek
In my opinion :-

Optimisation is a problem. I think it is better to have a system which is not optimised.
Optimising the past does not mean it will work in the future.
If you find a system which works, you also need a way of knowing that is has stopped working. That is why I have been using the 30-day average on the equity curve.

Backtesting needs a lot of historical data, not just a couple of months.

You say "I trust my system though. I trust what i've created."
Why do you trust it if it doesn't work ?

Glenn
 

theknifemac

Well-known member
340 0
Glenn said:
Hi Georgek
In my opinion :-

Optimisation is a problem. I think it is better to have a system which is not optimised.
Optimising the past does not mean it will work in the future.
If you find a system which works, you also need a way of knowing that is has stopped working. That is why I have been using the 30-day average on the equity curve.

Backtesting needs a lot of historical data, not just a couple of months.

You say "I trust my system though. I trust what i've created."
Why do you trust it if it doesn't work ?

Glenn
I think optimisation is fine AS LONG AS the optimised parameters picked work fine when they are changed - i.e. if you look at your optimisation results as a surface you want the point you have picked to be in the centre of a plateau rather than a peak. This increases the chances of you having found an edge which will persist in the future rather than a curve fitted system which will perform terribly in the future.

Cheers

Stew
 

georgek

Newbie
7 0
Thank you both for your answers.

Well,the thing is , i trust my system because it really has some profit in a 2-month period.

2 months of INTRADAY data is fine , thinking that 2 months of 24/24 ,5-min candles..
are a lot of candles!

This is a complex system,changing methods that it enters markets (depending on trend - trendless days)
and i've optimized every single period and line of it...

Thats why i trust it
I'm sure thinking of what theknifemack told....My optimizations maybe need to be chosen more carefully.

Still,i am not optimistic.
 

sidinuk

Established member
624 5
The market characteristics change dramatically over time. Even with intraday data you need to look at at least 2 years to make the backtest worthwhile.
 

frugi

1
1,827 126
Perhaps this problem can be mitigated by diversification?

Run several robust systems concurrently for equal stakes. Perhaps each trades a different instrument by a different method on a different timescale from the others.

If any system enjoys a number of consecutive wins equal to the "max consecutive wins" indicated in the backtest, then discard the system until it has suffered a similar number of consecutive losers. While a system is inactive one could add to the stakes of the others to compensate.

This goes against the conventional wisdom of running your winners and cutting your losers, but, as most successful systems are trend following, a run of consecutive losers would suggest prolonged consolidation after which you could reasonably expect a nice big trend, and vice versa. Though remember that markets consolidate for much longer than they trend, on average.
 
Last edited:

georgek

Newbie
7 0
well i can say i'm a little confused by this information mr frugi!
i'll try to read again and again more carefully
 
 
AdBlock Detected

We get it, advertisements are annoying!

But it's thanks to our sponsors that access to Trade2Win remains free for all. By viewing our ads you help us pay our bills, so please support the site and disable your AdBlocker.

I've Disabled AdBlock