Backtesting - is it a valid concept ?

[The General]

There is a fundamental flaw with classical probability which no-one here seems to have realised. This is of course linked to the discussion on backtesting.

Suppose you have 3 systems, each of which (as measured by simple traditional probability) has a 33% success rate. Which one would you choose to trade ?

 

[TheBramble]

Suppose you have 3 systems, each of which (as measured by simple traditional probability) has a 33% success rate. Which one would you choose to trade ?

If I HAD to choose one to trade, it would be that with the lowest risk:reward, lowest potential drawdown - i.e. highest positive expectancy.

But I'm interested, what is "There is a fundamental flaw with classical probability which no-one here seems to have realised"?

 

[JonnyT]

Hi General,

I would look at the drawdowns on all systems and at the rewards.

It will all depend on what the capital requirements are within your risk analysis and how much bang per buck you could reasonably expect to achieve.

But then again I'm seriously flawed.

JonnyT

 

[mr_cassandra]

Drawdown on mvp version 21C

Max drawdown is 8.6% during one long trade which ended in the black when held to completion by system. Any data I post about this system can be firther diligenced at the site and/or my public trades at clearstation.com

regards, Steve

Hi General,

I would look at the drawdowns on all systems and at the rewards.

It will all depend on what the capital requirements are within your risk analysis and how much bang per buck you could reasonably expect to achieve.

But then again I'm seriously flawed.

JonnyT

 

[mr_cassandra]

One siggestion

My opinion would be that you would perform due diligence on all three, and look to determine which one had the most consistent performance as defined below:

1> works thru diverse types of markets
2> provides consistent trades, not based on a few huge hits
3> beats buy and hold by enough margin to justify all this
4> results include all losers
5> taxes and commissions would not reduce its results to less than buy and hold.
6> general timing of trades make sense when diligenced versus the chart of the vehicle; IE I would have liked to buy or sell at X where this system did so.

There is a fundamental flaw with classical probability which no-one here seems to have realised. This is of course linked to the discussion on backtesting.

Suppose you have 3 systems, each of which (as measured by simple traditional probability) has a 33% success rate. Which one would you choose to trade ?

 

[The General]

Ok, rephrase. You have 3 systems each with a historical probability of 60% (here we assume when we get a winner, we win 1 pound and when we lose, we lose 1 pound). Now, as defined by classical historical probability, all 3 systems have positive expectancy.

You need a way of determining which one system will give the best chance of a winner on the next trade. How do you do this ? All 3 have the same historical success rate ?

 

[TheBramble]

Ok, rephrase. You have 3 systems each with a historical probability of 60% (here we assume when we get a winner, we win 1 pound and when we lose, we lose 1 pound). Now, as defined by classical historical probability, all 3 systems have positive expectancy.

You need a way of determining which one system will give the best chance of a winner on the next trade. How do you do this ? All 3 have the same historical success rate ?

Whichever one has had the most recent longest string of contiguous losses.

There is a fundamental flaw with classical probability which no-one here seems to have realised. This is of course linked to the discussion on backtesting.

You certainly know how to build tension. Come on General - what is it then? Any fundamental flaw with classical probability that the combined might of modern science has yet to discover (let alone us lowly life-forms here on t2w) has got to be a news-worthy event...!

 

[The General]

Champ,

Your answer is wrong I'm afraid and somewhat akin to the martingale roulette system - we have had 20 black come up so I will keep on betting on red.

In terms of what I believe is wrong with classical probability studies, if you can answer the question then you have got it. Think about it. Take it to the extremes and say that you HAD to choose between 3 systems each with a 2% chance of the next trade being a winner. If you think like that, it should become clearer.

 

[mr_cassandra]

Sounds like a question for a statistics teacher.

How would you apply this question and answer to your real world investing?

Ok, rephrase. You have 3 systems each with a historical probability of 60% (here we assume when we get a winner, we win 1 pound and when we lose, we lose 1 pound). Now, as defined by classical historical probability, all 3 systems have positive expectancy.

You need a way of determining which one system will give the best chance of a winner on the next trade. How do you do this ? All 3 have the same historical success rate ?

 

[mr_cassandra]

mvp v21C sell short signal imminent

real-time postings at www.clearstation.com mdy, uipix boards, among others.

real-time results to date vs buy and hold posted at my web-site

I've come to find over my 4 years programming work, that only results matter, theories are just talk, until applied in real-life.

Champ,

Your answer is wrong I'm afraid and somewhat akin to the martingale roulette system - we have had 20 black come up so I will keep on betting on red.

In terms of what I believe is wrong with classical probability studies, if you can answer the question then you have got it. Think about it. Take it to the extremes and say that you HAD to choose between 3 systems each with a 2% chance of the next trade being a winner. If you think like that, it should become clearer.

 

[frugi]

Ok, rephrase. You have 3 systems each with a historical probability of 60% (here we assume when we get a winner, we win 1 pound and when we lose, we lose 1 pound). Now, as defined by classical historical probability, all 3 systems have positive expectancy.

You need a way of determining which one system will give the best chance of a winner on the next trade. How do you do this ? All 3 have the same historical success rate ?

According to classical probability they all have an equal chance so it would not make any difference which one you chose. Take 3 coins, load them slightly so heads comes up 60% of the time, alwyas bet on heads, toss them at the same time and pick one at random without looking. Whichever one you picked would make no difference over the long term.

However trading systems are not strictly like tossing coins, because most of the successful ones are trend following. Thus I believe theBramble is right to find the one that has just suffered the largest number of consecutive losers, as trend becomes more and more likely as a consolidation matures. This is unlike the head/tails scenario where the chance of either is always 1 in 2 regardless of the result of the past x flips.

Therefore, if permitted, I would probably place a third of my capital in each and trade them simultaneously to smooth out the returns.

 

[sidinuk]

why can you only choose one system to trade? It makes more sense to trade all 3 and spread the risk