Curve fitting

[fundjunkie]

All,
Can someone explain to me what "curve fitting" is? I keep reading about it, have a feel for what it is, but am not exactly sure? Is it something done knowingly or accidentally? I ask because I was reading a book this week that referred to over elaborate strategies being vulnerable to curve fitting...


Thx in Advance,
D

 

[SOCRATES]

Curve fitting is a variant of forcing.

It occurs when someone wishes to see a result in a situtuation that in reality does not exist.

This desire to percieve a result, even if incorrect, is catalysed by emotional input, and not cold reasoning.

 

[the blades]

All,
Can someone explain to me what "curve fitting" is? I keep reading about it, have a feel for what it is, but am not exactly sure? Is it something done knowingly or accidentally? I ask because I was reading a book this week that referred to over elaborate strategies being vulnerable to curve fitting...


Thx in Advance,
D

Hello Fundjunkie,

By using lot's of parameters (ie lots of indicators, constants, rules like don't trade on X and Y day of the week), you can force a situation where you obtain a great looking system (low drawdowns, high win% etc), yet when forward tested doesn't work.

To avoid this, a system should be forward tested paper trading over a future period, or developing the system using data up to, say 2 years ago, then seeing how it would perform using the last 2 years data which wasn't used to develop the system.

There are no hard and fast rules. Sidinuk publishes results from a system which avoids trading on Thursday - purists may say this is a curve fit type rule, yet the system works well I believe. Some commodity trading systems use lots of parameters and do "work". However, the fewer parameters used and the more markets any given system works on, the more confidence you can have that a system isn't curve fitted.

Search these forums and you'll find some good info on this from more experienced and accomplished traders than me.

UTB

 

[Trader333]

In simple terms Curve Fitting is used by system vendors to qualify how great their system has performed in the past but to get this they will skew many of the parameters. The implication being that what has happened in the past will continue in the future and in every case I have come across it always fails. There are many systems (including very highly priced ones advertised in newspapers), that have used curve fitting to make the ridiculous claims of how profitable you will be when you buy their system and I will leave you to work out what is likely to happen if you do go ahead and buy



Paul

 

[fundjunkie]

In simple terms Curve Fitting is used by system vendors to qualify how great their system has performed in the past but to get this they will skew many of the parameters.

I see the logic. An implication is that this is done dishonestly and going by the apparent cynicism of many of the posts on these boards, widespread. Right?

The implication being that what has happened in the past will continue in the future and in every case I have come across it always fails. There are many systems (including very highly priced ones advertised in newspapers), that have used curve fitting to make the ridiculous claims of how profitable you will be when you buy their system and I will leave you to work out what is likely to happen if you do go ahead and buy

So how do I spot curve fitting, in my own or others' systems? As I'm weighing up buying a serious trading application for analysis, backtesting and money management I'll be testing others systems, ammending them and esting my own. I don't want to delude myself - the investment in expensive tools will be wasted otherwise.


Thx in Advance,
D

 

[Trader333]

So how do I spot curve fitting, in my own or others' systems?

In your own case, I suggest that any methods or systems you develop should be tested without any optimisation and if it is profitable in a "raw" state then you have a reasonable chance of having something worth further development.

For any commercially available systems always ask for a free trial lasting at least 30 days which you should compare with the backtested results they have claimed. If it is curve fitted then it is highly unlikely that the results in realtime will match those of the past in terms of performance.

You can also ask for a money back guarantee but most of these are difficult to get based on the experience of other T2W users.


Paul

 

[Tuffty]

The way most people seem to use the term curve fitting in my mind really means over fitting. Every trading system (mechancial or discretionary) is curve fitted i.e. the trade entry and exit decisions are based on market behaviour repeating itself.

Over fitting is where a system has been build around historic data to fit it exactly rather than to identify repeatable behaviour.

There are different methodologies available to distingwish between the two. The main one being 'out of sample' testing. This means testing entry and exit rules on unseen historic data - the main drawback with this is that if you change the system rules once you haven't got any unseen historic data as you've already seen the results from it and may now be subject to over fitting if you design the rules to fit the data you've seen.

Best of luck, Tuffty

 

[QQQShort]

As others have said in this thread, out-of-sample testing is a very good method for checking the robustness of a system.

Varying parameters can also be effective, not for optimization, rather to check for robustness. If the parameters can be varied in a range around the initial set without signficantly affecting results, then your system is less likely to be over fit.

 

[Craig66]

"The way most people seem to use the term curve fitting in my mind really means over fitting. Every trading system (mechancial or discretionary) is curve fitted i.e. the trade entry and exit decisions are based on market behaviour repeating itself.

Over fitting is where a system has been build around historic data to fit it exactly rather than to identify repeatable behaviour."

Well said, every TA based automated system is a curve fit of some kind. Looking at the equity curve & drawdown will tell you if it is an over fit.

 

[fcs]

Curve-Fitting explanation using the 5 Parameter Logistic as an example

All,
Can someone explain to me what "curve fitting" is? I keep reading about it, have a feel for what it is, but am not exactly sure? Is it something done knowingly or accidentally? I ask because I was reading a book this week that referred to over elaborate strategies being vulnerable to curve fitting...


Thx in Advance,
D

Hi,

Here is a post that I have written explaining the 5 Parameter Logistic (5-PL) curve fitting regression model.

It is based on bioassay analysis but the concept is exactly the same. You use a set of "standards" with known values that you use to "fit" a curve. With this curve, you can now interpolate or extrapolate values for "unknown" points.

Let me know if you would like to see an example and I'll try to cook something up that will explain this more clearly.

 

[nevis]

Over fitting is about learning irrelevant data. This happens when there is not enough data to learn from or too many parameters to model the data with.

Backward curve fitting which is about learning the data with future insight is bad.

Curve fitting however can be good as long as the learning process has some generalization features built in. I use piecewise curve fitting and switching.

 

[DDI]

If after one year of trading/observing your strategy, you realise of you add 1 or 2 more conditions (not parameter adjustments) you would have eliminated up to 40% of your losing trades but only 10% of your winning trades, would that be considered curve fitting?

 

[Adamus]

it makes no difference that you actually traded that year live, if that is what you are implying. You are still taking one year of results in a backtest scenario and adding conditions to your system. It sounds as if you are likely to curve fit your system. Of course you can't say for sure until you forward test it and even then if it works, it's not proven, you might just be lucky. You do ten forward tests and you're likely to find a few that work, curve-fit or not.