Backtesting - is it a valid concept ?

Glenn

Experienced member
1,040 118
I wonder what others think about the concept of backtesting ?
One the one hand it perports to find the best set of parameters for a given method and thereby tune it to the period tested.
If the results are good, then these parameters can then be used with some (?) confidence for live trades.
On the other hand one could argue that backtesting is just curve fitting and therefore simply 20:20 hindsight.

Seems to me that for a backtest to have any validity, it must produce a smoothly increasing equity (profit) curve with no wild girations. This at least demonstrates that the method and parameters found worked fairly consistently over the period tested.
Of course a lot depends on the period(s) selected.
Again to have any validity, I think that a test must either be run over a considerable time period, embracing many different market conditions and yet still remaining consistent, or it must be run on selected market conditions with a view to using the optimised parameters from the test in that type of market.

To disprove the theory of curve fitting, one suggestion is to take a period of time, say 5 years, and run your test over the first half of the data to get some results.
These results can then be run over the second half of the data as if it were the unknown future. i.e. a kind of Forward Testing.

One thing's for sure, the Software vendors must love things like backtesting. What an opportunity to develop and sell a nice technical product, using the math co-processor and the disk drive of your PC to the full. A nice 'gadget'.
Lovely jubbly - at least for the vendors.

Enough from me. What do others think? In particular anyone who has used a backtested system in the real market sucessfully.

Glenn
 

Grey1

Senior member
2,186 178
Glenn,

Before you ask about back testing being of any value , you have to ask yourself another more important question..
DOES PAST CORRELATE tO FUTURE ?

You can only have three answers .


1) yes it does..
2) No it does not
3) it does to some degree
if you think number 3 is the answer then your back testing will be effective to the same degree ..
 

Rognvald

Established member
916 15
One more question
"and for how long" which I suppose is really a subset of "some degree"
 

Glenn

Experienced member
1,040 118
Grey1
"DOES PAST CORRELATE tO FUTURE ?"

Agree the question, hence my thought about Forward Testing as a means of finding out.
But are you suggesting that one's degree of belief in backtesting is only a guarantee of good backtests, rather than good future results ?

Rognvald
"and for how long" ?
Again I'm thinking that forward testing may help to determine this to some degree i.e. within the bounds of available historical data.

One way or another most traders are looking for something which works enough of the time to enable money management to produce a profit.
So how do they find something which works 'enough of the time' ?
And yes, how will they know when it stops working (apart from the obvious) ?

Glenn
 

JonnyT

Senior member
2,560 22
Hi Glenn,

Solve that and you've solved the worlds problems.

In short there is no answer.

JonnyT
 

Grey1

Senior member
2,186 178
Glen,

Quote " are you suggesting that one's degree of belief in back testing is only a guarantee of good backtests, rather than good future results ?"

yes Glen, you are right,, people go to a huge length to back test a strategy and optimise various parameters such as Win/Loss ratio, Sharp Ratio, net profit not thinking to what degree the result of the back tested can be trusted for future performance,.

If you forward test you soon be disappointed .. In fact you soon realise that very best systems donot give you more than few % edge if forward tested.. This edge is often gets over ridden by the market costs ... This is why Trading is such a tough game..
 

barjon

Legendary member
10,705 1,809
Grey1 said:
Glenn,

Before you ask about back testing being of any value , you have to ask yourself another more important question..
DOES PAST CORRELATE tO FUTURE ?

You can only have three answers .


1) yes it does..
2) No it does not
3) it does to some degree
if you think number 3 is the answer then your back testing will be effective to the same degree ..

Grey

There is one lesson of history that will, with certainty, repeat in
the future - if demand outstrips supply, prices will rise and vice
versa.


Some ta, broken support becoming resistance for example, has an
underlying supply & demand logic that is both sound and readily
identifiable. However, in a lot of ta (and indicators in particular)
it is extremely difficult to identify underlying supply & demand
logic (that's not to say it's not there tho' - it might just be very
obscure :) ).

imo if your method has an identifiable and sound supply &
demand logic supporting it and it has worked when backtested,
then the chance of it repeating in the future is quite bright. If you
can't find the logic then, however well it's worked in the past, the
chance is just a chance.

good trading

jon
 

Grey1

Senior member
2,186 178
barjon

Lets say British Airways Q1 is greater than expecation in year 2003 and as a result the demand increases and price goes UP

Now what would you say BA shares would do if Q1 in 2004 is even better.... Will the price go UP or DOWN ?

What a dilemma ?
 

barjon

Legendary member
10,705 1,809
grey

go on, you're kidding me :) you're not really interested in WHY
it's in increased demand are you?

your chart will be telling you if demand is increasing and the price
moving - all you've got to decide is when and if you're going to
take advantage of it. Course your chart may also be telling you
that the supply is going to increase quite soon, so you may wait
to take advantage of that.

Still a dilemma of course :cheesy: :cheesy:

jon
 

Grey1

Senior member
2,186 178
Barjon,

Quote "go on, you're kidding me you're not really interested in WHY
it's in increased demand are you? "

No I am not but I am intrested in your answer for Q1 2004 do you think the price will go up or down ..?
 

Rognvald

Established member
916 15
Price includes all factors known to the market and these include expectation. Therefor if the expectation is that they will do as well year on year and results are in line the price will be largely unchanged as that factor is built into the price. If the results exceed expectation there will be a rise - if less then a fall (all other things being equal)
 

Grey1

Senior member
2,186 178
Rog,

INTC had a better than expectation result out and the price fell.. Stock B can have a lower expectation and market might see it as a good news..

ONE IS NOT CLEVER ENOUGH TO KNOW THE EFFECT OF NEWS ON STOCK PRICE

The BA example I gave was on 2000-1999 when Q1 result was bad and price increased while there was an optimistic earning report for Q3 and the price fell sharply..
 
Last edited:

barjon

Legendary member
10,705 1,809
grey

sorry I thought it didn't matter! It might go up or it might go down
whatever it does the chart will say what's happening.

Absolutely agree with your in bold capitals reply to Rog

jon
 

Grey1

Senior member
2,186 178
Agree barjon,

Like any thing else this phenomana has been studied by few researcher to gain an edge in the market and still no one has come up with any solution why mass behaves differently under the same circumstances how ever the post event analysis of the reaction to comets is easily explianed i nterms of supply and demand..

Now my point in the back testing question was to demonstrate that the back tested result of a strategy only to some degree correlates to the future .. The correlation is not a strong but a loose type .. ( I have avoided to use the term "of the Hinged type .. This gives me the room to defend technical Analysis "


look at this link

http://ideas.repec.org/p/nbr/nberwo/7330.html
 

barjon

Legendary member
10,705 1,809
grey

I hadn't meant to give the impression that technical analysis
needs defending. It's just that I find some of it (maybe the
curve fitting variety you talked about earlier) to be a bit iffy when
the underlying supply & demand logic is not apparent.

I think we're on the same side?

jon
 
 
AdBlock Detected

We get it, advertisements are annoying!

But it's thanks to our sponsors that access to Trade2Win remains free for all. By viewing our ads you help us pay our bills, so please support the site and disable your AdBlocker.

I've Disabled AdBlock