The Perils of Prediction

[DaveJB]

Where do you begin with this ? Do you postulate the hypothesis that chaos is an integral element in prediction ? And if so, what is the root core of this hypothesis, if so ?

To rearrange the order as I think it makes more sense.... this (chaos) isn't hypothesis, it's a theory (there's a significant difference between these terms) that is wildely applied in other systems to explain 'what happens'. The history of it I'm only on nodding terms with, but one part of it is this 'linear v non-linear' question. I'm more familiar with it in a physics setting, I know that there are equations that I can use to decide what a system will do for any given inputs - for example I can take a known spring and calculate how hard it will pull for any given extension... knowing the start conditions I can confidently and accurately predict the outcome, perhaps more importantly though if I'm slightly out on my input measuring I will find my result is only slightly different to what I predicted. The spring might pull a little harder, but not outrageously so.
If I make a double jointed pendulum arm (I saw this demonstrated recently) I have a non-linear system - releasing the arm from one point causes the arm to gyrate as it spins slowly to a halt... put the arm to what you believe to be the same point and a radically different set of gyrations results. A tiny change in input causes a radical change in output.
Non-linear systems exhibit this behaviour - significant changes in results despite the input conditions being virtually identical.

Anyone trying to predict the market via computation is going to run their choice of inputs into som sort of algorithm. The market is non-linear - it's not a simple graph, showing regular repetition, so I'd be pushed to accept it is linear. So assuming you got your algorithm/equation right for calculating the future price you still have a significant problem ahead of you... slight imperfection in the input data will cause the prediction to bounce all over the place. (This will look random, but it isn't - it's entirely predicted by your formula).

This deviation from what actually occurs will become worse at an accelerating rate as time passes - longer predictions are far less likely to be accurate than short term ones. They simply can't help but do so - all non-linear systems share this 'flaw'.

My input here on this subject is because I know that many people (including myself) try to use a PC to analyse markets using all sorts of ideas. While we are vaguely aware of limitations in our software and perhaps have an idea that it's better in some cases than others, we are often unaware there's actually information 'out there' in non-finance land that can inform and guide our program development and expectations. Some of our problems have been solved in other fields already.

If you accept Chaos has application to the markets (and it's applicable to just about everything else in creation, so it would be surprising to be able to say it isn't in this one special case), then it follows that prediction via an equation is going to be problematical. However, if you don't look too far ahead then the number of possible outcomes diminishes - you might end up with say 4 target prices for the Dow at 250 point intervals and 'commonsense' tells you which one to go with.... so the problems aren't necessarily insurmountable.

Where do you begin - the $64m question! Personally I think your best bet would be to arrange a distributed computing system of like minded individuals looking to share the outcome. (No point doing all this in public, solving the conundrum and failing to make a profit thereby <g>). I'd then look at deciding possible inputs, then simply (!) run as many combinations of them through a number crunch as possible, comparing the output prediction to reality.... ie hit and miss looking for the equation. There are probably better ways, and you might be able to reduce the number of tests by eliminating some obvious no-hopers. A bit like Omnitrader's approach, in a way, but you are looking for the parameters in your equation.

Dave

 

[SOCRATES]

Dave, let us start at your first paragraph above, because it presents an interesting angle of enquiry. How do you view the hidden coefficient in all of this which is time, if for the purpose of the excercise for arguments sake time were to be considered non linear ? Then "what happens" assumes a value of its own, relative to where it is placed on the time line. This "what happens" now presents itself as a fly in the ointment because its very nature is now disguised, if "what happens" is used as the core matrix for the idea, then possible outcomes relating to this matrix cannot be effectively quantified or qualified on the basis they can be the result of cause, effect or neutrality. On this basis I agree that prediction on this basis (which we shall call the Physical Route) would be problematical and very difficult indeed to encapsulate within a formula. And additionally the number of attempts would be huge so perhaps this route is not the best one, numbercrunching or processing power disregarded ~ additionally there is the problem of scale not only in time value considerations but also in terms of price progression.

 

[ducati998]

JB,

The market is non-linear - it's not a simple graph, showing regular repetition,

I would argue that it does show regular repetition.
The form is one of..........regular, irregularities.

Somewhat similar to a regular, irregular heartbeat. The principal is one of "continuity"
The Market has shown regular bouts of optimism, indifference, and pessimism, of irregular time periods.

cheers d998

 

[SOCRATES]

Listen Ducatti, I am not interested in discussing this with you, I am interested in the view of Dave JB, whose views on the matter are pertinent as a result of him being a physycist unlike you. I am interested to hear the view from the point of view of physics and not anatomy or physiology, please. Don't interrrupppt again, otherwise we will continue this discussion via PM. That's it.

 

[wasp]

What with this being a public BB, surely you should be doing this via PM if you don't want others to contribute?

 

[SOCRATES]

We are currently discussing this from DaveJB's point of view which is that of a scientist.
No sooner does a discussion narrow down to becoming interesting than it is derailed by
unsuitable or irrelevant commentry relating to some other discipline.

Now I do not view all of this via Physics, or Anatomy or Physiology myself, however, but I am interested to see the discussion progress in the structure it is at present until it ends and then anyone else can interject from whichever point of view they like, please.

 

[DaveJB]

Hi Soc,
okay - as I know you've mentioned this non-linearity of time I'll try to frame a cogent answer... like most people I appear to experience time as linear and unidirectional but will grant that there's nothing to insist that it isn't going the other way for one... and certainly elapsed time is relative to the observer (this has been proved repeatedly in the past century and is demonstrated daily by CERN).

I'm not actually sure you need to take account of a non-linear time, what you'd have is a prediction for point in time x.... you MIGHT have to figure out what time x then was on a normal everyday clock <g> I guess the most likely outcomes of allowing time to vary would be to make the computations even more difficult to tie down (a practical consideration that you might consider insurmountable), or you might have a prediction that was accurate but with limited knowledge of when for... not ideal, but knowing the price WILL go past a specific point within the next X days is still of use.

Overall I'm inclined to think that such considerations as chaos render prediction over extended periods extremely difficult, and I'd imagine practically impossible...I think it's much easier to see what has happened most recently, and react to it.

D998 - I'd agree that markets often repeat some behaviours, sometimes they do this extremely regularly for some time, but a linear system would chart out like a sine wave.

Dave

 

[DaveJB]

Ah,
posts 84+ seem to have appeared while I was drafting my reply. I think Ducatti's comment was fair enough, actually, and deserved a reply. I disagree with him as it happens, but that's all part of life's rich tapestry.

On a prediction thread something out of the science field seemed relevant, or at least of likely interest - especially to those looking to produce their own computer help. It's not a private thread, there's no reason I can see for anyone to keep off the grass as long as they're not being abusive or attempting to sabotage it.

Dave

 

[SOCRATES]

Hello Dave,


Thank you for your reply. I find it very interesting. It occurs to me that there may be a harmonious relationship existing between what is considered chaos and non linear time in contradistinction to disharmony when attempting to link chaos to linear time itself.

I find that non linear time is able to slice through what we have already discussed in the past and its quality of fluctuation in terms of what can best be called alternation of speed is a contributory factor to being able to percieve "down the pipe". This manifestation of percieving is denied via the linear time mode, as it imposes sort of "milestones" which immediately are incompatible with the concept.

Actually, the milestones themselves become clear if you are not conciously looking out for them.

I would agree with what you conclude in your final paragraph but one, but my experience suggests it is more due to scale rather than abundance of data.

Additionally, I do not view any of it as a peril, but of a rare and natural faculty, not developed in mainstream, and for this reason, and from this point onwards, not suitable for further discussion in a public forum.

 

[ducati998]

I would argue that it does show regular repetition. The form is one of..........regular, irregularities.

JB,

D998 - I'd agree that markets often repeat some behaviours, sometimes they do this extremely regularly for some time, but a linear system would chart out like a sine wave.

I disagree with him as it happens,

Overall I'm inclined to think that such considerations as chaos render prediction over extended periods extremely difficult, and I'd imagine practically impossible...I think it's much easier to see what has happened most recently, and react to it.

Assuming that you accept that Markets exhibit "continuity" and fluctuate, then "prediction" over extended periods.........( would need to agree what "extended" could be defined as ) would incorporate the past, or historical data, that would then suggest an outcome in the future, if not the exact timing of said event.

Which, leads directly to the two opposing camps, with their conflicting philosophies.....
Those who believe that "Timing" the market is a profitable undertaking, and,
Those who believe that "Pricing" the market is a far more profitable undertaking.

Cheers d998

 

[DaveJB]

Soc,
I'm certainly in the 'time's a fascinating subject in its own right' camp, I'd be wholly incapable of tying variable time (ie time that is non-linear, or exists as perhaps discrete 'moments' that can be stacked in different ways) into chaos in any meaningful manner - I'm definitely getting into the area of guessing at this point <g> I can handle chaos at a beginner's level, and I'm probably a wee bit more able in the area of time, but merging variations in both at the same time (no pun intended) would be Nobel prize work I suspect!

I'm perfectly happy to look at either seperately of course, from the scientific view of it all - we know so little about time for sure that the fields pretty wide open still <g>

D998 - Agreed on the camps, except there are a number of other camps as well... I'm actually in the 'I don't predict either of them camp', I just think "whatever just happened" often continues for long enough to show a profit by climbing aboard. Being able to spot when the chart doesn't look right can be handy too.... if you smell a rat sometimes it's because there's a rat present, and staying out is a good move <g> (I leave it to the experts to smell said rodent, determine what's really going on, then trade with the smart money.... I seldom manage to get that right and have decided to avoid trying to be too clever!) I do find that spotting the turn is not necessarily a lagging indicator, but I won't bore you with the technical side of that.

Accurate prediction is, I think, extremely difficult in comparison to spotting what's happened so far, so I tend to concentrate on the latter, as I could be a very old man before I got anywhere near solving the former.
Dave

 

[SOCRATES]

The difficulty is that Physics does not reveal all in regard to this. And I am not prepared to continue to discuss this delicate topic as I have observed the presence of a creature that has just come out from underneath a stone.It would be more prudent to continue this discussion in private, just you and me, later perhaps, when you are ready.

 

[ducati998]

JB,

D998 - Agreed on the camps, except there are a number of other camps as well... I'm actually in the 'I don't predict either of them camp', I just think "whatever just happened" often continues for long enough to show a profit by climbing aboard. Being able to spot when the chart doesn't look right can be handy too....

And without wanting to be pedantic about it..........................this is really the timing camp.
But to get back on topic, and bring up some other points.

Anyone trying to predict the market via computation is going to run their choice of inputs into som sort of algorithm. The market is non-linear - it's not a simple graph, showing regular repetition, so I'd be pushed to accept it is linear.

So assuming you got your algorithm/equation right for calculating the future price you still have a significant problem ahead of you... slight imperfection in the input data will cause the prediction to bounce all over the place. (This will look random, but it isn't - it's entirely predicted by your formula).

This is why I initially queried your definition, or assertion on linear, or non-linear. I hold that the market for securities is linear, of a regular, irregular pattern.

Therefore, in regard to calculating the future price......( market pricing, as opposed to market timing )........there are two inputs, that based on the present system employed by Wall St ( but different from historical methods ) and they are;

1.....Earnings per share
2.....Arbitrary Multiplier...........15 x earnings, 50 x earnings etc.

The earnings per share are a fact, historically......( no-one lies, or distorts via GAAP ),
therefore the problem becomes one of projection of earnings into the future.

And, then the truly qualitative, applying the multiplier, based on quality of earnings, trend of earnings, and every other prejudice you care to think of.

Or,
You can have your inputs, based upon MARKET PRICE only, crunching the numbers, to project an ENDPOINT based on TREND of prices..............Market Timing method, and I believe you are saying that this is a "non-liear" method. "That a tiny change in input causes a radical change in output"

Now, if I understand correctly, a LINEAR model is easier to PREDICT, than is a NON - LINEAR model. If that is true, then it makes logical sense to utilise a linear model.

I understand that you do not accept the Market is linear, however, that is a topic for further discussion.

cheers d998

 

[DaveJB]

No argument there,
Science is often completely in the dark, and quite frequently adamant that black is white - a great deal of supposed fact is nothing more than guesswork, supposition, and belief... often misplaced.

I'm happy to PM at your leisure, I pop in a little less frequently than once over but check several times a day after late afternoon so delays ought not to be overly bothersome.

Dave

 

[SOCRATES]

OK, anytime.

 

[DaveJB]

D998 -
the chart itself shows that the market isn't linear. A linear function wouldn't just have repeating periods on it, it would be as obvious and regular as a sine wave. Linear and non-linear functions have readily apparent shapes to their graphs, there is no way in creation that a typical market chart is a representation of a linear function.... it's obvious at a glance.

You can't produce a linear model to predict a non-linear market. Your model has to be an accurate model of the market, or it can't predict it. The market provides us with a graph of price action, but it gives us no clue at all as to the equation that could be used to determine the points charted. We have price on the Y axis and time on the X, the shape of the graph tells us it is a non-linear function we are charting.... the trick is to work back from the result to find out what the calculation that produced it was. The difficulty is that we only know that the equation has non-linear terms in it ('volatility times number of days squared' perhaps, as one example of what a non-linear term might look like) and you can arrive at a known answer by a great many routes - if I tell you the answer to the question is '12' then there are a great many calculations that might have led to that answer.

What makes it easier is that we have a lot of 'answers'... the price series for our chosen market - consequently many possible routes to the equation can be ignored, as only the correct equation (when we get to testing it) will come up with the price series correctly every time... when we feed our inputs in for any day in the past the equation will forecast the chosen number of periods ahead correctly.

As Soc and I have discussed briefly, the computing requirement to check this would be somewhat large... it would take a fair while, and frankly I suspect it'd take a year of number crunching just to start getting an idea of where to look for the answer....

Dave

 

[TheBramble]

but I suppose we wouldn't enter a trade at all if we didn't have some expectation that it would move in our favour. So isn't that prediction of a sort?

Maybe we're really talking about the difference between a "prediction" based on the statistical probability of X following Y and one that isn't?

We can never trade the past, only a future which is always unknown.

As interesting as the diversion into physics and quantum time fields has been (seriously) the original intent of Jon's post has been somewhat left hanging in the dry desert air...

Taking a trade (as an experienced trader) is not a prediction. It is a commitment to an outcome which is belived to have a higher probability than any other - the "statistical probability of X following Y"...

It is based on price and volume, indicators, Gann, Gilmore, Pythagoras, Fibonacci, Support & Resistance and the whole pantheon of traders tools.

But it is a probability - not a prediction.

If the trade goes with us - we hang on; If it goes against - we get out.

We're not interested in displaying the correctness of our expectations to others - or ourselves.

A prediction is:- "It (X, Y or Z) going to go down/up/sideways".

A Trade is:- "If it does X then I'll do A; if it it does Y I'll do B; if it does Z I'll do C or I'll do nothing".

The Perils of Prediction are in the NEED to predict - which is nothing to do with trading...

 

[ducati998]

JB,

the chart itself shows that the market isn't linear. A linear function wouldn't just have repeating periods on it, it would be as obvious and regular as a sine wave. Linear and non-linear functions have readily apparent shapes to their graphs, there is no way in creation that a typical market chart is a representation of a linear function.... it's obvious at a glance.

Thats fine, I suspect I have simply reversed the terminology, the market, by my definition follows a series of highs and lows over an undefined timeframe.

This is what I refer to as "regularly, irregular"
If that is "NON - LINEAR" thats fine, the name is unimportant.

That would seemingly put us on the same page, occupying different camps.
You feel that in a non -linear market, very small changes in the input data, result in very wide variations in the outcome..........

The market provides us with a graph of price action, but it gives us no clue at all as to the equation that could be used to determine the points charted. We have price on the Y axis and time on the X, the shape of the graph tells us it is a non-linear function we are charting....

I however posit, that the input data can be calculated accurately enough, so as not to effect a large variation in the outcome.

the trick is to work back from the result to find out what the calculation that produced it was. The difficulty is that we only know that the equation has non-linear terms in it

Just so.

As Soc and I have discussed briefly, the computing requirement to check this would be somewhat large... it would take a fair while, and frankly I suspect it'd take a year of number crunching just to start getting an idea of where to look for the answer....

Well, maybe yes, maybe no. Dependant on the methodology employed I suppose.

BRAMBLE,

Taking a trade (as an experienced trader) is not a prediction. It is a commitment to an outcome which is belived to have a higher probability than any other - the "statistical probability of X following Y"...

And this while true, really must ask, if based on "probability" how reliable is the calculation, or statistical data underlying that probability.

It is based on price and volume, indicators, Gann, Gilmore, Pythagoras, Fibonacci, Support & Resistance and the whole pantheon of traders tools.

Which I would argue are a total waste of time. That would be a minority opinion for sure, but for many hundreds of years the vast majority believed the earth to be flat.......they were all wrong.

We're not interested in displaying the correctness of our expectations to others - or ourselves

However I am. ( certainly to myself )

The Perils of Prediction are in the NEED to predict - which is nothing to do with trading...

Very true, but I don't trade.
Cheers d998

 

[TheBramble]

BRAMBLE,

Quote:
Taking a trade (as an experienced trader) is not a prediction. It is a commitment to an outcome which is belived to have a higher probability than any other - the "statistical probability of X following Y"...


And this while true, really must ask, if based on "probability" how reliable is the calculation, or statistical data underlying that probability.

Depends on the individual trader. And how long they've been trading whatever system/method they're trading. Of course.

If I've been flipping a coin for 10 years and going Long on Heads; Short on Tails and making an 80% W:L - that would qualify as "an outcome which is belived to have a higher probability than any other".

Quote:
It is based on price and volume, indicators, Gann, Gilmore, Pythagoras, Fibonacci, Support & Resistance and the whole pantheon of traders tools.


Which I would argue are a total waste of time. That would be a minority opinion for sure, but for many hundreds of years the vast majority believed the earth to be flat.......they were all wrong.

Well - they're not ALL wrong ALL the time. My point was (and is) as per my coin-flipping example above (no, I don't really flip coins) that it doesn't matter which system/voodoo/indicator you use - if it gives you a statistically, and more importantly, financially viable edge - it's viable. Period.

 

[SOCRATES]

Your post above, number 97.


I must comment this is very well explained Bramble, and should leave no one in doubt of what is required.