Walk Forward Analysis - the only logical successor to backtesting [DISCUSS]

drmark27

Newbie
6 0
I learned a lot about WFA from Howard Bandy through his books and message boards. However, in studying his words, something critical seems to break down.

What ratio of IS:OOS data should you use? Do you use two years IS, two months OOS? Do you use three years IS, six months OOS?

I asked Bandy this question a number of times and never got a good answer. He basically said "use what works." To me, that means you're curve-fitting precisely the tool you are using (WFA) in an effort to prevent curve-fitting! That is not going to give me the confidence to trade any backtested (with WFA) system live.
 

danielfppps

Junior member
21 0
Walk forward optimization is just a more complicated back-testing route where data-mining bias is harder to determine. Here are a few things worth considering:


  • There is no concrete evidence that a system with successful WFO has a better chance at live trading success than a system optimized traditionally for the whole back-testing period. There are many instances where a system developed with successful WFO fails going forward.
  • There is no such thing as an "out-of-sample" period when using historical data. If you use WFO you will always go back to the drawing board (changing parameters, WFO variables, windows, optimization ranges, etc) until you get something that works in WFO. It is a harder fitting exercise - more degrees of freedom - but if you try for long enough you can always mine/find something that works on a WFO analysis. The OS periods are not "unknown" they are actually known because the WFO testing process is repeated as many times as needed. This repetition process introduces hindsight that is as bad as for any historical testing process (as I say above you simply modify things until they work in the WFO).
  • The data-mining bias is really hard to measure in WFO because the degrees of freedom are in fact increased by the WFO process (adds complexity).

I have developed many systems that have achieved profit without the need for WFO and have traded systems with successful WFO tests that failed in live trading as well. This technique gives no real advantage (larger probability of live trading success) over traditional back-testing optimization where the data-mining bias has been properly determined and accounted for. My advice is simply to use your whole data for back-testing/optimization and use randomly generated data to determine the data-mining bias for your strategy.
 
Last edited:

drmark27

Newbie
6 0
Hi Daniel,

I want to state up front that nothing of my response is meant to pick on you personally. What I say about you goes for anyone else including me. We're just having a theoretical discussion here about trading system development, here.

I'll start out by claiming of course there is no concrete evidence of anything pertaining to real trading/investing performance for a few different reasons. First, many of those who succeed trade on their own or with a big firm and do not advertise their results to the public. Second, many of those who fail walk quietly into the night with their tails between the legs. They do not advertise their results (failure) to the public, either. Third, many of those who claim success are snake oil salesmen (and women) of one sort or another and cannot be trusted anyway.

In other words, don't try and argue against any trading approach because "there is no concrete evidence."

Similarly, don't try and argue against any trading approach based on your supposed personal experience. We haven't seen your brokerage records, bank accounts, tax returns, etc., so we ultimately have no idea what you have really done. You may be a snake oil salesman yourself. These supporting elements are therefore irrelevant.

You say there is no OOS period when using historical data once it has been used. I think there's a common misconception that once data has been used it is corrupted and any further use leads to some sort of bias. I don't believe that is what causes the potential harm. I believe the problem is the failure to assess whether profitable results in backtesting are a matter of luck, curve-fitting, or likely Edge. To determine this, optimize parameters over a range and explore that parameter space. Spike regions are likely fluke, in my opinion, and not suggestive of future success. High plateau regions are what we want to see.

I agree with you that a problem of WFO is constant returning to the drawing board until you find something that works if you are referring to changing up the ratio of IS:OOS time intervals. I wonder, though, if this can be optimized over a range as discussed above and inspected for spike vs. plateau regions as well.


Walk forward optimization is just a more complicated back-testing route where data-mining bias is harder to determine. Here are a few things worth considering:


  • There is no concrete evidence that a system with successful WFO has a better chance at live trading success than a system optimized traditionally for the whole back-testing period. There are many instances where a system developed with successful WFO fails going forward.
  • There is no such thing as an "out-of-sample" period when using historical data. If you use WFO you will always go back to the drawing board (changing parameters, WFO variables, windows, optimization ranges, etc) until you get something that works in WFO. It is a harder fitting exercise - more degrees of freedom - but if you try for long enough you can always mine/find something that works on a WFO analysis. The OS periods are not "unknown" they are actually known because the WFO testing process is repeated as many times as needed. This repetition process introduces hindsight that is as bad as for any historical testing process (as I say above you simply modify things until they work in the WFO).
  • The data-mining bias is really hard to measure in WFO because the degrees of freedom are in fact increased by the WFO process (adds complexity).

I have developed many systems that have achieved profit without the need for WFO and have traded systems with successful WFO tests that failed in live trading as well. This technique gives no real advantage (larger probability of live trading success) over traditional back-testing optimization where the data-mining bias has been properly determined and accounted for. My advice is simply to use your whole data for back-testing/optimization and use randomly generated data to determine the data-mining bias for your strategy.
 

meanreversion

Senior member
3,398 537
I use a 4 year IS period then a 1 year OOS period. I don't recall the precise reason I use that, but I remember researching it at the time and deciding that was a good mix.

Even then, you still need a way to determine whether the system is any good or not. Walk forward analysis definitely poses a number of fresh questions.
 

avano

Junior member
34 1
WFA is just another form of optimization on shorter periods. The system will fail regardless if new conditions arise not covered by WFA.
 

drmark27

Newbie
6 0
I don't know whether or not WFO works because I haven't tried it [yet?]. Better yet, I don't know whether or not WFO works because I haven't tried it on a sufficiently large sample size of systems. I say this just to state my bias here: I have none. I don't know.

I believe WFO is based on the idea that what has happened more recently is a better guide to what will happen in the immediate future. I also think this is obvious.

Optimization is only a bad word if one uses it to pick out the best--thinking what was best in the past will be best in the future.

I believe optimization is necessary to see how the system performs over a range of parameter values. Based on this idea, if you don't optimize then you're much less likely to succeed. It's not about determining what worked in the past. It's about seeing something that worked in the past and having some sort of idea whether that was fluke.

Finally, I don't see why a system will necessarily fail if new conditions arise "not covered by WFO." It seems entirely possible that the "new conditions" may find a match with a particular set of parameter values and if the new condition was a gradual development then a WFO process may well be sufficient to adapt.
 

Darwin-FX

Junior member
35 0
I learned a lot about WFA from Howard Bandy through his books and message boards. However, in studying his words, something critical seems to break down.

What ratio of IS:OOS data should you use? Do you use two years IS, two months OOS? Do you use three years IS, six months OOS?

I asked Bandy this question a number of times and never got a good answer. He basically said "use what works." To me, that means you're curve-fitting precisely the tool you are using (WFA) in an effort to prevent curve-fitting! That is not going to give me the confidence to trade any backtested (with WFA) system live.


He can not tell you because he does not know because WFA is flawed.
Well, I am currently writing a new article which should clarify this point, but at the moment know that WFA tries to judge a system that has a parameterspace in the billions (actually most of them have) by taking like 100 samples. That can and will not produce reliable results!

Also, WFA is flawed because for each IS:OOS datasizes to test you need a new WFA procedure that takes time.

My private algorithm (DATFRA - Darwins Algorithmic Trading Framework :D) is able to solve both issues. First, it can generate about 8 million samples in ~24 hours. Also, it can analyse as many IS:OOS datasizes as one wants, in only one analysis procedure.






---------------------------------------------------------------------------------------------------------------------------------






Walk forward optimization is just a more complicated back-testing route where data-mining bias is harder to determine. Here are a few things worth considering:


  • There is no concrete evidence that a system with successful WFO has a better chance at live trading success than a system optimized traditionally for the whole back-testing period. There are many instances where a system developed with successful WFO fails going forward.
  • There is no such thing as an "out-of-sample" period when using historical data. If you use WFO you will always go back to the drawing board (changing parameters, WFO variables, windows, optimization ranges, etc) until you get something that works in WFO. It is a harder fitting exercise - more degrees of freedom - but if you try for long enough you can always mine/find something that works on a WFO analysis. The OS periods are not "unknown" they are actually known because the WFO testing process is repeated as many times as needed. This repetition process introduces hindsight that is as bad as for any historical testing process (as I say above you simply modify things until they work in the WFO).
  • The data-mining bias is really hard to measure in WFO because the degrees of freedom are in fact increased by the WFO process (adds complexity).

I have developed many systems that have achieved profit without the need for WFO and have traded systems with successful WFO tests that failed in live trading as well. This technique gives no real advantage (larger probability of live trading success) over traditional back-testing optimization where the data-mining bias has been properly determined and accounted for. My advice is simply to use your whole data for back-testing/optimization and use randomly generated data to determine the data-mining bias for your strategy.

You are right because of the reasons I just mentioned.

The thing is, when I first experimented with DATFRA, I noticed a thing:
I did a test, just like WFA (WFA would have analysed ~50 IS->OOS datapoints for this test), but I heavily increased the amount of data.
But how much is enough? Well, 25.000 still was not (for a simple system with 3 parameters and a parameterspace size of 10000) enough, data looked like random noise:




Data only began to show clear trends when I looked at about 500.000 single InSample->OutOfSample tests:



(Tests were done on the Default Moving Average Expert Advisor that comes with every new MT4 Installation on EURUSD H4 for 2005-2013)






---------------------------------------------------------------------------------------------------------------------------------






In other words, don't try and argue against any trading approach because "there is no concrete evidence."

Similarly, don't try and argue against any trading approach based on your supposed personal experience. We haven't seen your brokerage records, bank accounts, tax returns, etc., so we ultimately have no idea what you have really done. You may be a snake oil salesman yourself. These supporting elements are therefore irrelevant.

Right, analysis methods can only be discussed based on theory, because as soon as you try to prove or disprove them by using them with some EAs you
a) well only analyse some EAs with them, you would have to analyse hundreds or thousands to get reliable results
b) if a WFA shows good results and the strategy works, well, that does not mean the WFA has anything to do with it. Perhaps the trader designing the strategy was just very good
c) you would have to forward trade the strategies for many years to get reliable results per test imo






---------------------------------------------------------------------------------------------------------------------------------






e. To determine this, optimize parameters over a range and explore that parameter space. Spike regions are likely fluke, in my opinion, and not suggestive of future success. High plateau regions are what we want to see.

Well, it still depends on how you look at the data. You know, you always have to set them in context. So analyse parameters and fitness in optimisation <-> fitness in forward trading / in the relative future.

Then you might get the real insight. Let me bring up DATFRA as an example again:


As you see, you can analyse the relationship between up to 4 things with it. So it makes it very easy to see what parameters work if one wants to get high forward trading profit.
Disclaimer: I implemented this 3d/4d plotting yesterday and I have the feeling it is still buggy, so the plot shown above might be screwed, but its enough to get the concept.






---------------------------------------------------------------------------------------------------------------------------------






Even then, you still need a way to determine whether the system is any good or not. Walk forward analysis definitely poses a number of fresh questions.

Dont worry, I think I have solved this, also :p
Remember when I said I do not just analyse like 100 IS->OOS Datapoints but 500.000 to a few millions? Well, with all this you can do some things...



First, you can empirically see how well a given parameter should work in live trading:



You can also do this based on Optimisation Profit etc. If a system tends to behave not like in the past, based on millions of measurements, it might be broken.




But there is more you can do with all this data, you can train Artificial Neural Networks in order to forecast profitability of your next live trading period:



And it will give you a Number between 0-100% determining the probability for a successfull live trading period. This is also a new feature and I am currently experimenting with it, but first results were very very good. So if forecast and reality tend to differ too much and too often, a system might be broken.



But DATFRA is new, so there might be more sophisticated methods to detect broken systems, but I know the answer lies in the Data generated by it!






---------------------------------------------------------------------------------------------------------------------------------






Better yet, I don't know whether or not WFO works because I haven't tried it on a sufficiently large sample size of systems.
And I thought nobody would ever mention that, too many ppl just want to see it tested on one or a few systems ;)






---------------------------------------------------------------------------------------------------------------------------------








Optimization is only a bad word if one uses it to pick out the best--thinking what was best in the past will be best in the future.
[...]
Finally, I don't see why a system will necessarily fail if new conditions arise "not covered by WFO." It seems entirely possible that the "new conditions" may find a match with a particular set of parameter values and if the new condition was a gradual development then a WFO process may well be sufficient to adapt.

Well, the second picture in this post actually analysed the correlation showing how well your live trading performance will be when picking, for example, always the best candidate. The thing is, in the few systems I have analysed until now (still busy coding this tool...) one thing is obvious: It is always the best idea to pick the best result from optimisation.
But this only gets clear when you look at hundred thousand cases, so in the traditional WFA it is just pure random luck what system you draw

Your second part is also true I guess. What I have found is that "new conditions" do not just arise, but often when your systems will get less profit in forward trading, you can already see that if you look at your optimisation results. Like with Neural Networks as mentioned above, or just if you see "well the best profit in optimisation was <5000$, in the database most candidtes with such a bad IS-Profit also had bad OOS-Profit".

So it is not about fearing unknown conditions, but about measuring the likelyhood of a system to still make profit



-Darwin

PS: If you are interested in DATFRA, and are a professional trader, please add me on Skype Darwin-FX , I am happy to discuss with you to get new input for my development. But no, you can not buy this yet. Perhaps in the future, perhaps only a few to keep it private, perhaps everyone.. I did not settle yet :) But I am sure if you helped me with cool input in form of discussions, it wont hurt you ;)
 
 
AdBlock Detected

We get it, advertisements are annoying!

But it's thanks to our sponsors that access to Trade2Win remains free for all. By viewing our ads you help us pay our bills, so please support the site and disable your AdBlocker.

I've Disabled AdBlock