Walk Forward Analysis - the only logical successor to backtesting [DISCUSS]

numbertea

Well-known member
257 9
I can understand the will to build a testing bed for trading. Truth is that I built my own as well. I was not able to achieve the analysis I was looking for without doing it myself. Once I built the back testing system it was easily integrated into a completely automated trading system, and that was my overall purpose. Along the way, with total intellectual involvement, I discovered my "edge". Good luck with your job prospects.

Cheers
 

Darwin-FX

Junior member
35 0
I can understand the will to build a testing bed for trading. Truth is that I built my own as well. I was not able to achieve the analysis I was looking for without doing it myself.
Cheers

Well, yes, at the beginning I also wanted to generate a full-automated analysis method, but now it became a framework (not so much the WFanalyzer but the currently developed algo), a tool, that just helps you "doing it yourself", but with capabilities you could never reach with manual analysis, as they are just too time-consuming (imagine doing 100.000 backtests per hand and then work with the results in some kind of excel sheet or something... not very efficient).

But of course, "do it automated"-kind of stuff can always be added to such a framework, but as you said, in the end it is best to do it yourself, and use the algos as tools.
But "do it automated"-features are cool for people still learning.

Once I built the back testing system it was easily integrated into a completely automated trading system, and that was my overall purpose. Along the way, with total intellectual involvement, I discovered my "edge". Good luck with your job prospects.

Yes, that was my overall purpose too, and still is. But I soon noticed that the analysis part is the most important one and would also be great for non-algo-EA-creation, so I decided to focus on that part, and make the EA-building-functions only one feature, not the core feature.

-Darwin
 

trendie

Legendary member
6,879 1,430
Darwin-FX said:
....
I WILL RELEASE A WALK-FORWARD-ANALYZER-TOOL FOR FREE WITHIN THE NEXT FEW DAYS THAT DOES ALL OF THIS 100% AUTOMATED; UNTIL THEN, USE THE TIME TO MAKE YOURSELF FAMILIAR WITH THE CONCEPTS.
....
-Darwin

Is this analyser available yet?
I have tried to make sense of it, but I still can't get my head around this WFA.
I thought seeing an example would clarify in my mind.

So, waiting for the WFA.
 

Darwin-FX

Junior member
35 0
Well, will start releasing tomorrow, perhaps first on 1 forum, so if a bug would be in the program I could fix it before releasing on more forums.

Otherwise, released is released, so with the disclaimer that it is still an alpha 0.1 and there might be bugs in it I might release it here, too, tomorrow afternoon or something like that.

What is the part of WFA you dont fully understand? :)

-Darwin
 
Last edited:

trendie

Legendary member
6,879 1,430
Well, will start releasing tomorrow, perhaps first on 1 forum, so if a bug would be in the program I could fix it before releasing on more forums.

Otherwise, released is released, so with the disclaimer that it is still an alpha 0.1 and there might be bugs in it I might release it here, too, tomorrow afternoon or something like that.

What is the part of WFA you dont fully understand? :)

-Darwin

Step Four: Walk Forward Analysis

9g9s5m6r.png

The above illustration you used serves well for my lack of understanding.

I get the "use X data" and walk forward for Y periods.
You are essentially using the past X data to determine actions going forward.
I get the idea that you are optimising for the past X periods to determine optimum parameters for forward trading.

But I dont understand why this is any different to conventional techniques.
For example, a Moving Average is the optimum parameter for X periods, and as each bar disappears out of the calculation, a new one is added, and the value of the MA changes slightly.

No different to deciding that the past 3 days Highs and Lows determine volatility.
As each bar disappears from the calc, a new is one added, and the value changes.
Ichimoku is probably a better example for this.

The only difference as far as I can see is the recalculating of what the MA might be for X periods.
But, if we take a holistic view of the market and consider it as mostly rangebound (around 70% of time if I am correct) followed by tipping-point sentiment shifts, the speed at which the range breaks and trends until a new range is established, the WFA will, as with most indicators, be too slow to react.

(reminds me too much of traders who are constantly re-assessing the value of their indicators after X losing trades, and end up with the "if only Y indicator was set to Z I would have won Q pips" mentality.)

I suppose WFA depends on what your starting points are.
Years ago, I would be totally with you on WFA, (mostly MA and stochastics) but years of trading have educated me that markets move when they feel like it, in their own time, and an inherent insight (screen-time) is needed to first understand the state of the market before deciding which trading technique to use.
And to have a value in mind which convinces you that the market has shifted from rangebound to trend (breakout), and thus your trading style.

I will, of course, give the WFA my best shot once its released.
 
Last edited:

Darwin-FX

Junior member
35 0
First of all, I do not come from a trading-background, but from a coding-background, and started AI-programming before I even knew what forex is.
So I might have had a different point of view from the beginning :)

This said, the crossvalidation of computer-models ("walk forward analysis" in the widest sense) is a common approach to determine robustness of such models.

So it is not so much about finding ideal parameters (that is just a nice side effect), it is about having as much (more or less independent) "past=>future-relationship-testcases" as possible, because the conventional method of backtesting only gives you 1, which is of course not enough to draw meanignfull conclusions.

If you are a very good trader, the significance of a backtest does not come from the testing-method itself (as it is only 1 datapoint, which is more or less irrelevant and a product of chance), but because you put knowledge into the trading-system.

Now, if you want to trade using algorithms (which are dumb) or are not a highly skilled trader and need to rely on the significance of your testing-results, the conventional techniques are not enough and you need something better, like a WFA.

Hope that answers your question. If not or if I overlooked something, please ask me again :)

-Darwin
 

Purple Brain

Experienced member
1,613 179
Robert Pardo is another 'expert' that has made his name writing quality gobbledegook. WFO (or WFA) is simply back-testing, but using only the data up to the current test point in the historical test data on the historical data that follows it and then acting surprised when one of the 10,000 optimised tests shows a positive correlation. It's a dynamic curve fit. All of these techniques back-testing, data mining, curve fitting, walk forward - only work on historical data.

But it's a relatively new name and able to masquerade as a brand new, different and 'scientific', and there are only a dozen or so commercial WFO/WFA offerings out there at the moment. So good luck Darwin for jumping on the WF band-wagon. You'll likely make more from your work when you go commercial than you would from trading it.
 
  • Like
Reactions: avano

numbertea

Well-known member
257 9
Robert Pardo is another 'expert' that has made his name writing quality gobbledegook. WFO (or WFA) is simply back-testing, but using only the data up to the current test point in the historical test data on the historical data that follows it and then acting surprised when one of the 10,000 optimised tests shows a positive correlation. It's a dynamic curve fit. All of these techniques back-testing, data mining, curve fitting, walk forward - only work on historical data.

But it's a relatively new name and able to masquerade as a brand new, different and 'scientific', and there are only a dozen or so commercial WFO/WFA offerings out there at the moment. So good luck Darwin for jumping on the WF band-wagon. You'll likely make more from your work when you go commercial than you would from trading it.

Exactly.

Cheers
 

avano

Junior member
34 1
A) What do you do with trades at cross-points in WFO?

B) This method breaks down in a big way when markets change (ex. 2007 top to 2008 bust). You may lose more by using parameters determined during strong up trends when markets plunge than when you use parameters that where determined over the whole history.

C) As someone else said WFO is dynamic backtesting and AFAIK there is no rigorous justification for it. To the contrary, when using smaller samples significance of the results is degraded than when using a large sample.

D) Just develop the system in first 50% of the data and do an out-of-sample in the other 50%. If the profit factor is no lower than 50% and max DD is the same or less then you may have something. The rest is for vendors to keep people playing with their platforms hoping they get something. Every system developer should read this article. It also applies to WFO: if you reuse the data many times and you change the back and forward period many times you will finally get your fitted system.
 
Last edited:

Darwin-FX

Junior member
35 0
WFO (or WFA) is simply back-testing, but using only the data up to the current test point in the historical test data on the historical data that follows it and then acting surprised when one of the 10,000 optimised tests shows a positive correlation. It's a dynamic curve fit.

Thats wrong.
First of all, it's not about finding one positive correlation, its about looking at all the samples.

Second, and this is the important part, WFA is a crossvalidation-method.
Crossvalidation is a wide spread method to evaluate computer models, not just in trading, but in all scientific fields that use such models.

You could say it's not the end of the road and I would agree, but saying that it is the same as back-testing, just because it uses the same data, is wrong and everybody that has ever used computer models will agree with that :)

You could also say that systems can still be overfitted or crappy, and I would agree, too, but saying that WFA produces overfitting in it's nature is incorrect. ;)

Don't forget: this is an interdisciplinary method that has proved that it benefits robust modells.
After all it's just a tool, like backtesting, but one that is a lot more powerfull.





All of these techniques back-testing, data mining, curve fitting, walk forward - only work on historical data.

That is also wrong. Don't judge from your own experience or the experience most traders had with EAs.
Just because 99% of all EAs out there are crappy, it does not mean that the underlying concepts can not work.

It's more because ppl do not know how to do it right, as most of them come from a trading-background, just knowing some MQL basics.
That will not work, of course.
But with solid algorithms and state-of-the-art AI techniques one should be able to write working code - don't forget that more and more trading, also in huge banks or hedge funds etc is taken over by algorithms.. So the question is not if it can work, but how it works!

Tough, I do not claim I know how this is done, yet. I just know that it CAN work and that I am on the best way towards a working solution, but I am still learning.

Btw, my private algotrading framework is already ~30.000 lines of code, so please do not compare "real" algorithms with EAs. :p





But it's a relatively new name and able to masquerade as a brand new, different and 'scientific', and there are only a dozen or so commercial WFO/WFA offerings out there at the moment. So good luck Darwin for jumping on the WF band-wagon. You'll likely make more from your work when you go commercial than you would from trading it.

Don't get me wrong, I am not using WFA myself, I have a set of private algorithms that are able to analyse about 2500x as much data per system than a WFA.
For example, this diagram shows 250.000 single optimisation/forward-trading pairs:

6ijtimq2.png


X-achsis: Profit in in-sample.
Y-achsis: Profit in forward trading.

You see the clear trend? That is an in-depth view that no manual trader can EVER get on a trading system, as only algorithms are capable of generating and analysing such an amount of data.

Also, it takes only 1 click to show a parameter (like "moving average period") on the X-achsis, so I could clearly spot the parameters the system works best with.

So, what I want to say is: Tough algorithms are not easy to code (and all the EAs out there have nothing in common with real algotrading), they are still the most powerful tools a trader can work with, as long as they are seen as tools and not as replacement for humans.





A) What do you do with trades at cross-points in WFO?

Sophisticated algorithms will wait until the last trade closes, and then start the new WF-window afterwards. But that requires changes on the mql scripts, so the public WFanalyzer can not do this.




B) This method breaks down in a big way when markets change (ex. 2007 top to 2008 bust). You may lose more by using parameters determined during strong up trends when markets plunge than when you use parameters that where determined over the whole history.

Don't limit your imagination by my beginners write-up ;)
Again, sopthisticated algorithms will implement "emergency stops" to stop trading in such events.
Also, this is even more important, they will not even trade in timespans where the underlying and analysed inefficiency is not really present in the markets.

So you have a trend following system and the current markets are ranging? Then there is no trading!

These mechanisms are quite easy to implement when you look at the whole parameterspace of a system (with ~250.000 single test-cases, such trends get very obvious.)

Let's take the diagram I posted above as example. You see that the EA in question made most live-trading results when the in-sample profits are very high.
So when the profit during optimisation is not "very high", the system would not trade at all.




C) As someone else said WFO is dynamic backtesting and AFAIK there is no rigorous justification for it. To the contrary, when using smaller samples significance of the results is degraded than when using a large sample.

When you have 1 large sample and 100 smaller samples (that show the same amount of data than the large sample), I can not see a reason why the signigicance of the second sample-set should be less than the one of the first - of course only if you look at all of them, not only on some.




D) Just develop the system in first 50% of the data and do an out-of-sample in the other 50%. If the profit factor is no lower than 50% and max DD is the same or less then you may have something. The rest is for vendors to keep people playing with their platforms hoping they get something. Every system developer should read this article. It also applies to WFO: if you reuse the data many times and you change the back and forward period many times you will finally get your fitted system.

The second part is very, very right. Thanks for pointing that out :) Yes, if you do this, then of course you will get overfitted systems in the end.
As I said, WFA is just a tool, and it is not protecting a trader from it's own lack of knowledge ;)

Tough, the first part is questionable. Did you ever make in-depth analysis about that? Why is it 50% of the data, not 45 or 60?
Why look at the profit factor, not the expected payoff? Or would the drawdown/profit be better?
Why "not lower than 50%"? Why not 75% or 30%?

You see? That is one of the problems, relying on inutitive chosen numbers will not work very well.
But algorithms could be used to determine the exact numbers, and thats what algo trading is all about - having very powerful tools and code to evaluate every single aspect of a system based on tons of data.
It is NOT about "starting a script and wait until I get rich".



For example, my above mentioned private evaluation algorithm is 100% parameterless, it has only 3 inputs:
a) The EA file to analyse (of course)
b) The Market/Timeframe (also obvious)
c) The timespan to analys (eg 2000-2012)

Every single other aspect regarding the EA, from optimisation and trading timespans to parameter ranges and everything you can imagine is evaluated. There is no "best guess" included in an solid algortihm.
And if it is, then this best guess is verified based on a proper evaluation before it is used within the system.



-Darwin


EDIT: Regarding the image above. That is just one way to look at the data. You can also average all trades on a per-day basis and get something like an equity curve, if that's what you want:

6jj8np7x.png


So the purpose of WFA and the more sopthisticated concepts is to harvest as much data on a single system as possible - whereas a backtest only gives you 1 point of view.
 
Last edited:

meanreversion

Senior member
3,398 537
Robert Pardo is another 'expert' that has made his name writing quality gobbledegook. WFO (or WFA) is simply back-testing, but using only the data up to the current test point in the historical test data on the historical data that follows it and then acting surprised when one of the 10,000 optimised tests shows a positive correlation. It's a dynamic curve fit. All of these techniques back-testing, data mining, curve fitting, walk forward - only work on historical data.

But it's a relatively new name and able to masquerade as a brand new, different and 'scientific', and there are only a dozen or so commercial WFO/WFA offerings out there at the moment. So good luck Darwin for jumping on the WF band-wagon. You'll likely make more from your work when you go commercial than you would from trading it.

Is your point that any form of back-testing is redundant? If you think that back-testing has some merit, then WF is just an attempt to prevent over-fitting, that's all.
 

Darwin-FX

Junior member
35 0
Hi meanreversion,

I would not say it is _just_ an attempt to prevent overfitting, it also generates a different type of data.

Backtesting generates data about a system's performance, WFA generates data about the correlation of a system's performance in the (know) past and the (unknown) relative future.

-Darwin
 
 
AdBlock Detected

We get it, advertisements are annoying!

But it's thanks to our sponsors that access to Trade2Win remains free for all. By viewing our ads you help us pay our bills, so please support the site and disable your AdBlocker.

I've Disabled AdBlock