TheBramble’s Trading 201

[Vrothdar]

I should point out that using a geometric sequence like that isn't the correct way of testing your losing streak. You should use the cumulative frequencies for the tests however calculating the geometric sequence illustrates how it all works well enough for my previous post.

 

[TheBramble]

The number of weeks the system has been traded isn’t relevant, on it’s own, to any of the other information given.

It's going to be very relevent to where I'm going with this.

Great post and don't have sufficient time right now to properly review/respond. Will do so later.

 

[Vrothdar]

It's not relevant to any of the statistical stuff though unless you were completing a consistent number of trades per week. In that situation, however, the number of trades is simply a function of the number of weeks. Knowing the number of weeks adds no extra information over knowing the number of trades.

Where it is useful is looking at whether or not a living could be made from a system with your starting capital. Deciding whether or not a system is/could be/has been successful doesn't require a time frame, just a number of trades.

 

[TheBramble]

It's not relevant to any of the statistical stuff though unless you were completing a consistent number of trades per week. In that situation, however, the number of trades is simply a function of the number of weeks. Knowing the number of weeks adds no extra information over knowing the number of trades.

Where it is useful is looking at whether or not a living could be made from a system with your starting capital. Deciding whether or not a system is/could be/has been successful doesn't require a time frame, just a number of trades.

So you don't believe there is any significant difference in a system and it's associated statistical data that's run 10 trades to that which has run 100 trades? Or one that has run any given number of trades over say 1 week, and one that's run that same number over a much longer period of time?

You'd be equally confident running either?

You don't think 'time in the market' counts for anything?

 

[Vrothdar]

Perhaps I spat out a few waffles in the last post.

Yes the time period is important however it's not important as far as the actual calculations are concerned. It gives perspective to your statistical data/results but it wont change the results.

I hope that's a little clearer.

 

[Mr. Charts]

Perhaps I spat out a few waffles in the last post.

Yes the time period is important however it's not important as far as the actual calculations are concerned. It gives perspective to your statistical data/results but it wont change the results.

I hope that's a little clearer.

Errmm.........this surely depends on the sample size.
10 trades = results not significant
100 trades = possibly significant
1000 trades = probably significant
10000 trades = highly significant

But there are other factors........like does the sample take place over a sufficiently long period of time including different market types e.g. trending, ranging, grinding........how do structural changes in the market affect the results.......are they operator influenced.....what happens in Black Swan situations ?
And many others.
Richard

 

[Vrothdar]

I've just thrown a quick spreadsheet together that generates binomial and geometric distributions for different success rates.

Thought it might be of interest to this thread so I've attached it here.

The only cell you should change is the grey one, if you change any other cells (including the number of trades) the calculated values and graphs will be inaccurate.

 

[Vrothdar]

Errmm.........this surely depends on the sample size.
10 trades = results not significant
100 trades = possibly significant
1000 trades = probably significant
10000 trades = highly significant

But there are other factors........like does the sample take place over a sufficiently long period of time including different market types e.g. trending, ranging, grinding........how do structural changes in the market affect the results.......are they operator influenced.....what happens in Black Swan situations ?
And many others.
Richard

That's pretty much what I was trying to say. You wouldn't use the time period as part of your calculations but you do need to consider the number of trades used in the calculations and time span of your source data to decide how much weight you feel the result holds.

 

[TheBramble]

I think these last few posts are admirably highlighting the differing perspectives between pure X and applied X.

The markets are not statistically random in that any system, regardless of its structure, will not perform exactly the same at all points in time. By that I mean, 1 million trades executed between January and March will not necessarily yield the same distribution or profile or results of calculations as 1 million trades executed between January and December.

Binomial and Guassian isn't where the reality of the markets are at.

 

[Mr. Charts]

Market moves:
80% sentiment
20% maths

Richard

 

[Mr. Charts]

....and that's imput into market moves....
R

 

[TheBramble]

The number of trades is only relevant for comparing your actual results to a statistical models on which your system is based.

The number of trades makes a great deal of difference in the early stages of any system, which is typically (and quite erroneously in my view – TBD) when the most focus will be on the system stats rather than its ‘physical performance’ in the market.

{Remind me to enlarge upon ‘Physical Performance’ at a later stage}

The statistical data produced by the normal performance calculations for any new system will change enormously. A point I made very early on in this thread. All you ever have from any system is the cumulative result of all trades with variances often damped down to just a few mean averages. The variances (standard deviations) are incredibly important and they’re the very things we’re drowning out with meaningless averages.

From trade number 1 through trade N, you’re going to have a set of data that will be becoming (if it’s a relatively stable system in terms of profitability) closer and closer to the average performance of the system. But that’s the thing, the actual physical performance defines the average, not the other way round.

Your system not only could but pretty much always will look very different after trade 10 than it did at trade 5 or than it will at trade 20, 50, 100….Which is why a lot of good systems get the shove before their time (before they’ve hit their metre) and a lot of crappy ones get their awfulness masked by the a random distribution of over-luck in the beginning.

We need to compensate for reality in our hypothetical models.

 

[Vrothdar]

The markets are not statistically random in that any system, regardless of its structure, will not perform exactly the same at all points in time. By that I mean, 1 million trades executed between January and March will not necessarily yield the same distribution or profile or results of calculations as 1 million trades executed between January and December.

The movement of the market itself isn't really what's being tested though rather the accuracy of a set of indicators in predicting the market movement. I'd want to be using these tests to determine if my set of indicators are still a good predictor of market movement or whether the downturn I've just experienced is so far removed from my expectations based on previous performance that I don't believe my indicators are accurate enough predictors anymore.

 

[TheBramble]

But your system, indicators, performance calculations don't work in isolation from movements in the markets....

 

[Vrothdar]

The number of trades makes a great deal of difference in the early stages of any system, which is typically (and quite erroneously in my view – TBD) when the most focus will be on the system stats rather than its ‘physical performance’ in the market.

{Remind me to enlarge upon ‘Physical Performance’ at a later stage}

The statistical data produced by the normal performance calculations for any new system will change enormously. A point I made very early on in this thread. All you ever have from any system is the cumulative result of all trades with variances often damped down to just a few mean averages. The variances (standard deviations) are incredibly important and they’re the very things we’re drowning out with meaningless averages.

From trade number 1 through trade N, you’re going to have a set of data that will be becoming (if it’s a relatively stable system in terms of profitability) closer and closer to the average performance of the system. But that’s the thing, the actual physical performance defines the average, not the other way round.

Your system not only could but pretty much always will look very different after trade 10 than it did at trade 5 or than it will at trade 20, 50, 100….Which is why a lot of good systems get the shove before their time (before they’ve hit their metre) and a lot of crappy ones get their awfulness masked by the a random distribution of over-luck in the beginning.

We need to compensate for reality in our hypothetical models.

Is that not the perfect argument for back testing a system and then making statistical comparisons against those results? Say I backtested over Jan - Dec. I'd compared my cumulative actual results against expectations drawn from backtested data. So for my first 100 trades I'd be doing 2-tailed tests against a bi-nomial and geometric distribution based on the long term success rate of the backtested (p) data with n = 100.

The things I've been suggesting are only ever valid when either using a system that can be backtested* or comparing recent performance to past performance of a system that has been traded for a large number of trades.

Early statistical analysis on a system when you've no valid backtested data is, as I think you're suggesting, pretty meaningless.

*My feeling on this is that the system has to be completely 100% mechanical and use technical analysis only for this kind of back testing to be plausible.

 

[Vrothdar]

But your system, indicators, performance calculations don't work in isolation from movements in the markets....

I'm still struggling to grasp why that's an issue? Testing whether the relationship between movements in the market and accuracy of indicators has changed doesn't require the movements in the market to be random in nature.

If the movement of the market after a particular signal has occurred was random we'd have exactly a 50% hit rate (over time). The result of a trade after a particular signal is not random, it's biased like a dice. The statistical tests allow us to tell whether the amount of bias has changed or is no longer there.

 

[TheBramble]

Is that not the perfect argument for back testing a system and then making statistical comparisons against those results?

That’s the argument that’s normally put forward for back testing. Which is why I rant so relentlessly about the pointlessness of that particular form of smoke & mirrors.

Forward testing is the only reliable method for evaluation of systems.

Early statistical analysis of any system with or without (but especially with) backtesting is meaningless.

 

[TheBramble]

I'm still struggling to grasp why that's an issue? Testing whether the relationship between movements in the market and accuracy of indicators has changed doesn't require the movements in the market to be random in nature.

If the movement of the market after a particular signal has occurred was random we'd have exactly a 50% hit rate (over time). The result of a trade after a particular signal is not random, it's biased like a dice. The statistical tests allow us to tell whether the amount of bias has changed or is no longer there.

To be clear about what I'm saying. Running any system over X periods of time is not going ot be as effective a measure of the system's performance as running it over 2X or 10X or 100X or 1000X periods. Would you agree with that in principle? If not, it's my turn to struggle in understanding what special power you believe the statistical process has over the markets.

 

[Vrothdar]

That’s the argument that’s normally put forward for back testing. Which is why I rant so relentlessly about the pointlessness of that particular form of smoke & mirrors.

Forward testing is the only reliable method for evaluation of systems.

Isn't it better to back test a system, decide that as it's been successful in the past it's worth trying the future and then doing statistical anaylsis to determine whether or not the system is working now in the same way that it was during the period you backtested over? If you just pluck a system out of the air, with no back testing, and then start trading it you've no benchmark (the historical performance) against which to gauge your current/future performance. Without a benchmark you've no way of deciding how successful your system is? Sure you might get lucky and make some money, but if you're losing money how do you know whether or not it's because your system is crap of because of the natural variance in performance? Without a benchmark you've no possible way of knowing.

Early statistical analysis of any system with or without (but especially with) backtesting is meaningless.

Why is it less meaningful with backtesting? If I trade a system for a month and perform 100 trades then compare my actual results to my expected results based on past performance of my system I can tell whether or not my system is still working. If I just trade for a month, performing 100 trades and then try to evaluate my method without anything to compare it to I'll be in exactly the situation you described earlier: I'll be more likely to ditch a system that would work long term too quickly and less likely to ditch a system that wouldn't quick enough.

 

[Vrothdar]

To be clear about what I'm saying. Running any system over X periods of time is not going ot be as effective a measure of the system's performance as running it over 2X or 10X or 100X or 1000X periods. Would you agree with that in principle? If not, it's my turn to struggle in understanding what special power you believe the statistical process has over the markets.

I agree with you there. The larger your sample the more representative of the long term results your calculations will be. To do any meaningful statistical analysis you'd need to compare the actual performance of the last 100 (for example) trades to the expected performance of those last 100 trades based on the actual performance of the 10000 (again, for example) trades preceding the last 100.

If the actual performance of the last 100 trades is too far removed from the expected performance of the last 100 trades based on the actual performance of the 10000 before then you'd be in a position to say that the system no longer works (or works better than before).