Why backtests are useless, EAs are flawed and their parameters are bad [DISCUSS]

Darwin-FX

Junior member
35 0
not necessarily Numbertea, I could back test my system up to 2011 and then forward test on out of sample data, which would be 2012 and 2013 say. If it still stacks up, then I'm good to test on demo. Thats how I prefer to back test which is why I disagree completely with the original thread.

What you do there is out-of-sample testing, which means you have 1 dataset to optimise your strategy and then 1 dataset to test your optimisation approach.

This is ok, better than a backtest at least.

But are you aware of the fact that you judge the future performance of your strategy based on only 1 datapoint? (in-sample as "past data", out-of-sample as "future data" => 1 datapoint to judge live trading)

A good Walk Forward Analysis can give you at least ~100-150 datapoints to judge from. The approach stays the same, but the results are more reliable as you view a larger part of the picture :)

So even if you disagree with my post, its a matter of fact that a WalkForwardAnalysis is just the way you do it - but one step further :)

Perhaps this convinces you to give it a try, at least ;)

-Darwin
 
Last edited:

Tyger

Junior member
20 1
Hi Tyger, an example of an adaptive strategy, is just what I've mentioned..a genetic algorithm. These can be back and forward tested, however I'm afraid they dont fit nicely into an excel formula. You need to program these things, so when you google excel and genetic algorithm you now get into the realms of programming. Add ons to excel that utilise the very primitive language of visual basic.
So we have exactly what Shakone has described and something that can't fit into your excel formula (or else we wouldn't need add ons and the use of a programming language). So, still Tosh?
Thats the only point I'm trying to make.

Mea Culpa
I was wrong. I should have said formula (or algorithm) instead of Excel formula. What I'm trying to say is:
The scientific approach: Evidence based analysis using testable methods and objective evidence, rigorously and skeptically examined.
Any other approach: Theories built on untestable propositions, anecdotal evidence and intuitive analysis.
Tell me more about GAs (or EAs). Are they not just enhanced versions of Excel's solver function? In other words, sophisticated data-mining tools?
 

Tyger

Junior member
20 1
Curious about something that is tosh?

You make a declaration, that fixed strategies can't be successful for long (declaring your belief). I ask you whether adaptive strategies have the same flaw. You then ask for an example (requesting info) of an adaptive strategy but suggest that it is snakeoil (stating your bias). I answer, but don't give you the answer you want (feedback). Because the answer doesn't satisfy you, you then say that you think it's a load of tosh (dismissive), but ask me to feel free to prove you wrong (manipulative). Now you're all ears (requesting info again) and admit you might be wrong (humble).

So from declarations of certainty, to requesting info, to stating bias, to being dismissive, to being manipulative and back to being curious and open to being wrong.

And you think that there is no such thing as an adaptive strategy? :LOL:

I don't believe that the direction and timing of market movements will fit easily into a formula in excel for you, but that's just my belief and shouldn't stop you trying. I encourage testing everything you can. Lots of things are backtestable, some things aren't. Does that automatically mean those that aren't are bad? I wouldn't go as far as the OP to say backtesting is useless, but forward testing is more useful to me.
I'm envious of your social skills. I don't understand most of what you said on that topic, but that's typical of people on the autistic spectrum. If I offended you then I apologise unreservedly.
 

Shakone

Senior member
2,458 665
I'm envious of your social skills. I don't understand most of what you said on that topic, but that's typical of people on the autistic spectrum. If I offended you then I apologise unreservedly.

Not offended at all, and no need to apologise.

So would you concede that you apply adaptive strategies to all kinds of situations in your life, that aren't statistically verified on historical data, or is it still something that doesn't exist? Looks like I've failed in my attempt.
 
Last edited:

trendie

Legendary member
6,879 1,430
I am still curious about how a walk-forward is any different to testing on out-of-sample data, and how being adaptive is anything other than having just another layer of test clauses to trigger a trade.

I know I am being thick, but still would like to see an example to illustrate.
 

numbertea

Well-known member
257 9
I am still curious about how a walk-forward is any different to testing on out-of-sample data, and how being adaptive is anything other than having just another layer of test clauses to trigger a trade.

I know I am being thick, but still would like to see an example to illustrate.

I am not sure if this would satisfy what you are asking for in an 'adaptive strategy' but I use the market volatility before 9:30am to give my system a volatility level that then will affect which algorithms will be used during my trading hours of that day. Is that a good example? I of course realize that it is just a bunch of more test clauses. And like you I don't know the increased significance of calling trading with no money "walk forward analysis". Can someone turn on the light?

Cheers
 
Last edited:

Tyger

Junior member
20 1
Well I thought this is clear, but let me try to explain it.

The underlying problem with the markets is that the data we process changes all the time.
So a relationship that existed in the past, eg "When the price dropped 50 pips in 3 minutes, then a counter-movement will happen 60% of the times" (completly made up, just for an example) will no always be the same!

For example a month later it could be "When the price dropped 40 pips in 2 minutes, then a counter-movement will happen 55% of the times".
Tough the underlying inefficiency we want to exploit (counter-movements after price movements) stays the same, the exact parameters to describe it correctly will change.

So we need a way to always "get the best parameters" for the current market (not for the market 5 years ago).

And to always get the best parameters for a strategy, to match the current manifestation of the inefficiency we want to exploit, we need to adapt, which is nothing more than to always re-optimise the parameters we use when time goes by.

This is all my current tool can handle, tough, I am working on methods that will even be capable of replacing the whole trading logic, not only the parameters, based on what is working best in the current market situation.

Hope that makes it clear - if not ask me, thats what the thread is about.

-Darwin

Excellent response. But can I play Devil’s Advocate? (please don’t take offense if I appear confrontational – I don’t mean to be!).

So “adaptive” means “changing parameters to optimise results as time goes by”?

So you come up with a theory that a “relationship” truly existed in the past (and was not just data fitting). You surmise that the relationship continues into the future but that the parameters that describe it change (for whatever reason).

So do you then

a). try and find a new relationship that describes how the original parameters change with time? (eg, using your hypothetical figures, we might find that the market cycles between your two examples every month, or we might find a direct relationship to the S&P ATR (average true range), for instance).

Or

b). are you speculating that “if the optimum parameters for the current market are A, B and C, if we trade using these parameters then we can expect X% return”? In this instance, we would, for example, go back 3 years then walk forward each month, “optimising” as we go, then seeing how well this approach worked for the next month, etc, etc. Then if we get sufficiently good results over, say, the last 3 years, we confidently conclude that we have a viable trading strategy.


But surely a). above would be a “Holy Grail” of trading strategies (thus extremely unlikely), and b). is just data mining/fitting? (ie I could do the same with a series of random coin tosses and it would make money most of the time, but be subject to occasional blow-outs).

PS Great thread. Enjoying it immensely!
 
M

member275544

0 0
What you do there is out-of-sample testing, which means you have 1 dataset to optimise your strategy and then 1 dataset to test your optimisation approach.

This is ok, better than a backtest at least.

But are you aware of the fact that you judge the future performance of your strategy based on only 1 datapoint? (in-sample as "past data", out-of-sample as "future data" => 1 datapoint to judge live trading)

A good Walk Forward Analysis can give you at least ~100-150 datapoints to judge from. The approach stays the same, but the results are more reliable as you view a larger part of the picture :)

So even if you disagree with my post, its a matter of fact that a WalkForwardAnalysis is just the way you do it - but one step further :)

Perhaps this convinces you to give it a try, at least ;)

-Darwin
You misunderstand me, I concur with the whole approach of forward testing, I don't need to be convinced. But when someone says backtesting is useless, its so far off the mark its just not worth discussing.
I have generalised my forward testing approach just for ease of understanding, I may indeed have multiple data points..but the real forward testing is doing it live, on demo. None of the tests will account for slippage, late entries, gaps etc etc.
 

new_trader

Legendary member
6,665 1,488
You misunderstand me, I concur with the whole approach of forward testing, I don't need to be convinced. But when someone says backtesting is useless, its so far off the mark its just not worth discussing.
I have generalised my forward testing approach just for ease of understanding, I may indeed have multiple data points..but the real forward testing is doing it live, on demo. None of the tests will account for slippage, late entries, gaps etc etc.

100% correct and the very reason why live trading results vary widely from automated back testing.

IMO: The only way to learn trading is to practice day and night and weekends using a simulator and historical data in real time. Practice, practice study and practice.
 
  • Like
Reactions: Shakone

Shakone

Senior member
2,458 665
Harris Brumfield:

You need to have some kind of game plan. But things evolve so quickly in the market you have to be able to adapt. Combining some kind of systematic foundation with the ability to change things on the fly probably gives you the best of both worlds.

I would read a lot before the trading day, and even during the day, to find a theme, have a direction and develop a trading bias. No market pattern repeats exactly the same, but you can see a lot of things developing [like they did in] the past if your memory is good enough and you remember things that have happened repeatedly.

But you have to mix things up and adjust for the time. One of the problems I saw is that technical ideas eventually get caught and you have to be able to shift real fast. Approaching things the way I did, I figured if I ever found anything [that was really good], it would be harder for people to discover and copy.
 

Darwin-FX

Junior member
35 0
I am still curious about how a walk-forward is any different to testing on out-of-sample data, and how being adaptive is anything other than having just another layer of test clauses to trigger a trade.

Walk forward analysis gives you 100-150 "in-sample/out-of-sample test-pairs", instead of 1, like in classical out-of-sample testing (more on that in the paper I will release in 1-3 days, where I explain how wfa works).

The thing about adaptivity is that you dont get another layer to trigger a trade, but to re-adjust your normal layer (which consists of an EA and its corresponding parameters) so it always gives you the best results. ;)

-------------------------------------------------------------------------------------------------------------------------------------



And like you I don't know the increased significance of calling trading with no money "walk forward analysis". Can someone turn on the light?

Well, thats what it is about, doing the exact same thing like "trading". BUT with the difference that you can do so with 13 years of data in just a few hours.
So it simulates trading, like you do live, but on the past :) Clear now?

-------------------------------------------------------------------------------------------------------------------------------------



Excellent response. But can I play Devil’s Advocate? (please don’t take offense if I appear confrontational – I don’t mean to be!).

So “adaptive” means “changing parameters to optimise results as time goes by”?

[...]
b). are you speculating that “if the optimum parameters for the current market are A, B and C, if we trade using these parameters then we can expect X% return”? In this instance, we would, for example, go back 3 years then walk forward each month, “optimising” as we go, then seeing how well this approach worked for the next month, etc, etc. Then if we get sufficiently good results over, say, the last 3 years, we confidently conclude that we have a viable trading strategy.

[...]

b). is just data mining/fitting?

b) is correct, and of course it is "just" data mining, as this is what the whole EA trading approach is about :)
But it is mining for data with the goal to have a trading plan that produces parametersets for my EA that have a high chance of beeing successful in the future.
And not data mining with the goal to have some nice looking backtests in the past.
Thats the difference :)

But I dont really get how you want to reproduce it using coin tosses, could you please explain that? ;)

-------------------------------------------------------------------------------------------------------------------------------------



But when someone says backtesting is useless, its so far off the mark its just not worth discussing.
[...]
None of the tests will account for slippage, late entries, gaps etc etc.

Ok, you are right, perhaps my thread title was a bit too provocative, but that was what I intended with it, to start a discussion ;) And you are reading the thread, so perhaps it was the right decision :p

And with the other stuff you mentioned, yes, thats the reason why I prefer H4 or D1 timeframes, as on them the "randomness" of live trading is not so heavily influencing the strategy's performance.
But you are right, real live testing (on demo or not) is absolutly vital for success :)

-------------------------------------------------------------------------------------------------------------------------------------



IMO: The only way to learn trading is to practice day and night and weekends using a simulator and historical data in real time. Practice, practice study and practice.

I agree with the learning part, but for an algo-trader its more about research/discussions/learning than about practice :)



-Darwin
 
Last edited:

Solas0077

Active member
236 14
"A WFA takes, for example, the data from 2000 to optimise your system, then tests it on 2001 (which is, from the point of optimisation, the "future" or "live trading").
Then it walks forward and optimises the system on 2001, test it on 2002. Then optimise on 2002, test on 2003 etc.
Do this until you walked through your whole data and only consider the "live trading" in your evaluation.

You see? With this simple tactic you can tackle the 3 problems I have described above, the lack of adaptability, the need for a evaluated parameter selection process and the uselessness of a "past-performance"-backtest."

Imo this does not tackle the problem because you can always find a system if you look hard enough that will pass WFA and fail in forward trading. Just do a neural network search and you will see what I mean. WFA does not work if new conditions develop in the markets that will make an unprofitable system in WFA turn profitable in forward trading. It is a very conservative approach that basically will reject 99.99% of systems and you will end up trading none. Just my 2 cents.
 
Last edited:

Tyger

Junior member
20 1
Quote:
b) is correct, and of course it is "just" data mining, as this is what the whole EA trading approach is about :)
But it is mining for data with the goal to have a trading plan that produces parametersets for my EA that have a high chance of beeing successful in the future.
And not data mining with the goal to have some nice looking backtests in the past.
Thats the difference :)

But I dont really get how you want to reproduce it using coin tosses, could you please explain that? ;)

OK. Randomness. It was mathematician George Spencer-Brown who showed that in a random sequence of 10 to the power of 1,000,007 zeros and ones, we would expect at least 10 non-overlapping sub-sequences of 1 million consecutive zeros. So if you bought a book of random binary numbers for a research project you'd be rather alarmed if all the pages contained just zeros - yet a random sample of random numbers might produce just that. The first iPods had a true random shuffle feature, but users complained when it played the same songs back-to-back (so Steve Jobs famously made the feature "less random to make it feel more random"). Those random anecdotes demonstrate that randomness is a tricky subject!

See attached spreadsheet. I've put together a simple spreadsheet that randomly generates coin tosses (represented by ones and zeroes). Lets pretend they're stock prices. For each Tail the price goes up by 0.25% and for each Head price goes down by 0.25%. The resultant chart looks like a real stock chart BUT IS TOTALLY RANDOM. We look at the chart, or the numbers, and come up with an idea we want to test - say that if we get 4 heads in a row we then buy/Long. We only have to go down to row 68 to show that this is a very profitable trading strategy. We then "walk forward test" and find that it works over the entire 500 row sample.

Therefore, the million dollar question is, how do we know whether the results of our testing represent real "relationships" or are just random results?
 

Attachments

  • Random Coin Tosses 1.xls
    290 KB · Views: 178

Shakone

Senior member
2,458 665
Quote:
b) is correct, and of course it is "just" data mining, as this is what the whole EA trading approach is about :)
But it is mining for data with the goal to have a trading plan that produces parametersets for my EA that have a high chance of beeing successful in the future.
And not data mining with the goal to have some nice looking backtests in the past.
Thats the difference :)

But I dont really get how you want to reproduce it using coin tosses, could you please explain that? ;)

OK. Randomness. It was mathematician George Spencer-Brown who showed that in a random sequence of 10 to the power of 1,000,007 zeros and ones, we would expect at least 10 non-overlapping sub-sequences of 1 million consecutive zeros. So if you bought a book of random binary numbers for a research project you'd be rather alarmed if all the pages contained just zeros - yet a random sample of random numbers might produce just that. The first iPods had a true random shuffle feature, but users complained when it played the same songs back-to-back (so Steve Jobs famously made the feature "less random to make it feel more random"). Those random anecdotes demonstrate that randomness is a tricky subject!

See attached spreadsheet. I've put together a simple spreadsheet that randomly generates coin tosses (represented by ones and zeroes). Lets pretend they're stock prices. For each Tail the price goes up by 0.25% and for each Head price goes down by 0.25%. The resultant chart looks like a real stock chart BUT IS TOTALLY RANDOM. We look at the chart, or the numbers, and come up with an idea we want to test - say that if we get 4 heads in a row we then buy/Long. We only have to go down to row 68 to show that this is a very profitable trading strategy. We then "walk forward test" and find that it works over the entire 500 row sample.

Therefore, the million dollar question is, how do we know whether the results of our testing represent real "relationships" or are just random results?

Since you're comfortable with maths, you'll know that you can make a null hypothesis and test for statistical significance of your trading results.
 

Tyger

Junior member
20 1
Since you're comfortable with maths, you'll know that you can make a null hypothesis and test for statistical significance of your trading results.

My problem with a Null Hypothesis: Using my coin tossing example I formulate a null hypothesis that “longing after tossing 4 heads” does not generate profits over the ensuing 432 coin tosses. But the test shows that the “longing after 4 heads” theory was profitable, therefore the null hypothesis is falsified. As there is no middle ground possible, my original theory must be true. But it clearly isn’t – it is random coin tosses. In a nutshell, the null hypothesis method can’t cope with randomness.

If you can spot an error in my logic I’d be delighted to be enlightened.
 
 
AdBlock Detected

We get it, advertisements are annoying!

But it's thanks to our sponsors that access to Trade2Win remains free for all. By viewing our ads you help us pay our bills, so please support the site and disable your AdBlocker.

I've Disabled AdBlock