the futility of trading system optimisation

Say you developed a trading system.

It has 5 variables, and each variable can reasonably have 5 values. If you were going to do an optimisation run, you'd have to run the simulation 3125 times to test them all against each other.

Say you had data from days 22 to 52. The world began on day 1 but you don't have that data. You want to optimise it over days 22 to 52 and then trade it with the best results from 52 onwards.

My theory is that all the equity curves of all the results from the optimisation run if you could look at the whole time period from day 1 to day 80 are pretty much the same - OK, they will look different and go up and down in different places, but essentially the most you can ever make with your particular trading system, from the lowest point to the highest point in any of the equity curves, is generally the same.

Some optimisations might make it at the start, some might make it at the end, some might make nothing or lose completely, but look what happens when you choose the optimal values from the results over your smaller optimisation window:

Here's a chart I prepared earlier. It shows the best result from the optimisation run over the time period from 22 to 52.



Do you see what I mean? if you optimise your trading system variables like this, you'll end up picking optimisation values that give you that maximum - guaranteed to go straight into drawdown.

Tell me I'm wrong.

If you end up with the same result each time regardless of variable settings, I would contend that you probably have chosen the wrong variables to test or your have a system which just goes with the market. The idea of optimization is to ensure that the system is picking out the trades you're after in the most efficient fashion. What those trades are depends on the focus of the system.

If you are into that sort of thing, one of the more interesting uses of optimization is to look at the optimization surface to see what effect changes in system parameters have. How sensitive to changes is the system? Are the any step responses etc.

In reality there is no optimum to be found other than for a time limited input data set.

optimisation without any walk forward is a waste of time....

Rhody Trader said:

If you end up with the same result each time regardless of variable settings

Click to expand...

I meant over the full history, not just the optimization window.

Also, not the same result, but just a ceiling on the max profit, which is reached by several of the optimisation results. So if your optimisation makes the max profit, you can't carry on being profitable in the following trading attempt.

All your various settings may come up with the same max profit, but do they have the same drawdowns and comparable risk characteristics? You need to think about risk-adjusted return, not just absolute gains and losses. You may find certain settings offer smoother equity curves.

Oh absolutely, I need a nice equity curve more than a high return. I'd happily swap a smooth equity curve for backtesting returns 10 times greater.

I need to look at 'risk-adjusted' returns as a stat to use - which one exactly do you mean?

It's up to you. Some people use the Sharpe Ratio. Some folks look at % drawdowns. Some look at other things. It all comes down to what best suits your needs.

Adamus,
Finding a system that is not curve fitted but will take money from the market isn't easy as all systems have a degree of curve fitting in them otherwise they'd just produce random trades.
The methodology of trying 3125 goes without any 'walk forward' period is very highly curve fitted. It produces no statistical evidence that you're onto something if it just happens to make a profit. However, if it "guarantees a drawdown" then there's your system, just put the opposite trades on that the system tells you to and you've got a winning system. Unfortunately I'm afraid I'll have to tell you that you're wrong there too as you can't even curve fit a drawdown.
PS: If your testing does shows a guaranteed draw down please do let me know what the system code is.

dcraig1 said:

In reality there is no optimum to be found other than for a time limited input data set.

Click to expand...

Just saw this comment, skipped it before. What do you mean? Do you mean that the optimum result from a back-test will be irrelevant in an immediately following walkforward?

Rhody Trader said:

It's up to you. Some people use the Sharpe Ratio. Some folks look at % drawdowns. Some look at other things. It all comes down to what best suits your needs.

Click to expand...

Whose needs does the Sharpe Ratio suit? People keep banging on about how market returns are not normally distributed, but that's the whole basis of the Sharpe Ratio. I don't want to walk blindly into my own mini credit crunch. Or is it just that there's no other way of getting such a succinct number?

Tuffty said:

The methodology of trying 3125 goes without any 'walk forward' period is very highly curve fitted. It produces no statistical evidence that you're onto something if it just happens to make a profit.

Click to expand...

I'm doing walk-forwards too. In the original message I guess I forgot to mention it and from the way I introduced the topic I guess it looked like I wasn't looking at walk-forwards.

My walk-forwards in this case are the period after the peak in the equity curve, stupidly. And what I was trying to say was that the equity peak tends to be at the end of the optimisation window, due to the optimisation process.

I guess what I am saying in a ham-fisted way is that optimisation in this case is curve-fitting and useless. As my walk-forward showed.

Adamus said:

Whose needs does the Sharpe Ratio suit? People keep banging on about how market returns are not normally distributed, but that's the whole basis of the Sharpe Ratio. I don't want to walk blindly into my own mini credit crunch. Or is it just that there's no other way of getting such a succinct number?

Click to expand...

Sharpe is not based on the normal distribution seeing as standard deviation is not distribution linked. StdDev is merely looking at the variation from the average. Furthermore, when using Sharpe you're looking at your trading performance, not market returns. You're talking apples and oranges.

Rhody Trader said:

Sharpe is not based on the normal distribution seeing as standard deviation is not distribution linked. StdDev is merely looking at the variation from the average. Furthermore, when using Sharpe you're looking at your trading performance, not market returns. You're talking apples and oranges.

Click to expand...

re: market returns - sorry, I meant trading performance, not market performance.

I thought that my trading profits could vary hugely as soon as I traded through a black swan (or perhaps 'into a black swan', if it didn't go my way).

If I understand the situation from what you say, that makes no difference to the Sharpe Ratio?

Academics may argue otherwise, but the concept of Sharpe is to look at average return over the volatility of that return (standard deviation). It really isn't related to distributions if you are comparing like underlying data sets.

Everyone optimises, even those who don't use mechanical systems. E.g. a discretionary trader might use the 1wk/4wk moving average crossover as a guide ---> optimisation.

The idea about looking at a 3D surface as you vary two variables is a good one, gives you an idea if your method is unnaturally sensitive to one variable.

Another method I find useful is to backtest starting from random dates.. rather than just assume the last 10 years of data give you a "good" curve, start it 8 years ago. Or 3.

It's really not hard to write a mechanical system that makes money in the past and is highly likely to make money in the future. A breakout system in FX going with the trend usually works. The difficulty comes in executing the strategy in real-time, and handling the psychological effects.

Rhody Trader said:

Academics may argue otherwise, but the concept of Sharpe is to look at average return over the volatility of that return (standard deviation). It really isn't related to distributions if you are comparing like underlying data sets.

Click to expand...

"like underlying datasets"? But the datasets could be anything - how are you going to know when you only have a Sharpe Ratio? When a potential investor looks at a Sharpe Ratio from my futures trading, he might compare it with a stock investor, or an option writer. I don't know what my returns look like in comparison but I wouldn't want to back an option writer like Victor Niederhoffer.

I guess I shouldn't worry. I can just pump out the Sharpe Ratio and let them worry about it (or not as the case may be.....)

meanreversion said:

The idea about looking at a 3D surface as you vary two variables is a good one, gives you an idea if your method is unnaturally sensitive to one variable.

Another method I find useful is to backtest starting from random dates.. rather than just assume the last 10 years of data give you a "good" curve, start it 8 years ago. Or 3.

It's really not hard to write a mechanical system that makes money in the past and is highly likely to make money in the future. A breakout system in FX going with the trend usually works. The difficulty comes in executing the strategy in real-time, and handling the psychological effects.

Click to expand...

I like the idea of backtesting from random dates, although I'm currently doing the Pardo thing of 2 years optimisation, 6 month walk-forward.

The 3D parameter landscape thing is good too, I do it when possible but often my optimisation is based on swapping over code snippets which are functions wrapped up with a parameter in there but always 13.

Perhaps I should open the code up a bit more and parameterise it instead of using just 13. Some stuff, i.e. ATRs in the main code, have the parameters but are generally only 3,5,8 and 13.

I like to keep things short term and squeeze lots of trades in, for the sake of getting a bigger sample and also to increase compounding effects when in action.

Not sure yet whether I'm painting myself into a corner like that.

Adamus said:

"like underlying datasets"? But the datasets could be anything - how are you going to know when you only have a Sharpe Ratio? When a potential investor looks at a Sharpe Ratio from my futures trading, he might compare it with a stock investor, or an option writer. I don't know what my returns look like in comparison but I wouldn't want to back an option writer like Victor Niederhoffer.

Click to expand...

I'm not talking about you comparing your result to someone else's. I'm talking about comparing the performance of one trading system, or system configuration, against another. That's the point of this thread, right? You can use Sharpe because presumably you're testing the systems/variables over the same underlying data to be able to judge comparable performance.

I think you do have to be careful with straight optimisation and only looking at summary results. It's important to do things like look at the performance broken down by period.

Personally, as I've become more experienced with backtesting, I find that a combination of brain and optimiser gives much more interesting results./ For example, I might use the optimiser to produce statistical results of random entry on hypothesis X, then use it to switch conditional A on and off and compare that to the random results. To tell you something interesting about conditional A. Similarly, when optimising a value, say 1-20, and getting the best three results as 19,2,18; not only does that scream curve fitting, but it also gives you extra information about how that type of parameter relates to your hypothesis.

I like developing mechanical systems that have human input built in. For example, I like to work out what I think the value of a parameter should be for a particular market, and run with that for a while without optimising. If you find you were way out, you learn something about market characteristics.

Rhody Trader said:

I'm not talking about you comparing your result to someone else's. I'm talking about comparing the performance of one trading system, or system configuration, against another. That's the point of this thread, right? You can use Sharpe because presumably you're testing the systems/variables over the same underlying data to be able to judge comparable performance.

Click to expand...

Aha. OK I'm with you then. I have two objectives - certainly got to compare trading systems, but I'm also putting together a prospectus to get a couple of people to hand over their moolah, and they asked for Sharpe Ratio too. Also asked for VaR, the fools.

Xeno said:

I think you do have to be careful with straight optimisation and only looking at summary results. It's important to do things like look at the performance broken down by period.

Personally, as I've become more experienced with backtesting, I find that a combination of brain and optimiser gives much more interesting results./ For example, I might use the optimiser to produce statistical results of random entry on hypothesis X, then use it to switch conditional A on and off and compare that to the random results. To tell you something interesting about conditional A. Similarly, when optimising a value, say 1-20, and getting the best three results as 19,2,18; not only does that scream curve fitting, but it also gives you extra information about how that type of parameter relates to your hypothesis.

I like developing mechanical systems that have human input built in. For example, I like to work out what I think the value of a parameter should be for a particular market, and run with that for a while without optimising. If you find you were way out, you learn something about market characteristics.

Click to expand...

I'm not that far advanced yet. Still struggling to find something that fits my risk requirements.

For a while I was working on entries only and basing my system on exiting at MOC. After a while it got interesting enough to contemplate actually deploying. But I just didn't find anything that was reliable enough - I don't think I'm looking for the Holy Grail, I don't demand massive returns, low returns would be fine. Just reliable on a 6 month time frame.

Adamus said:

- I don't think I'm looking for the Holy Grail, I don't demand massive returns, low returns would be fine. Just reliable on a 6 month time frame.

Click to expand...

There are many who would say that is a holy grail ;-)