Forward Testing.

sask said:
Coming from a data mining background, I do not get what the big deal is with backtesting. I think what is more important is how the strategies you built play in the time after you have built.

For ex: I plan to build a system using machine learning which will tell me to Hold(in my case - no trade to enter), Buy or Sell with fixed risk-reward ratio(1:5). Since I have 3 possible decisions - the system I'm planning to build will be a 3-class classification model. (using R - a link to which someone has posted to this post earlier and PostgreSQL database)

In order for the machine learning algorithm to learn the patterns, I need data (currently in the process of acquiring tick data from Oanda) to train my algorithm.

Say I use data from 01NOV2007-01NOV2012(training period) to learn some patterns/rules. Now, there are 2(actually many more) things to do. I need to understand that my system has learned something novel. This will be accomplished by testing the model performance(in terms of its classification accuracy) on a number of randomly (using bootstrapping) held out records from the training period.

Once satisfied with the performance, the final test(before getting on to the practice account stage) would be to see how it performs on data that my model has not seen before. This is what is called as the out-of-time sample(say data from 02NOV2012-28FEB2013) or simply test data or in terms of trading strategy set-up, forward testing.

This is where I would collect statistics such as Sharpe ratio, draw-down rate, win-loss among a myriad of other things. This is my best estimate of what the future of my trading strategy looks like.

In other words, forward testing is what I will be more concerned with. No point in using a system which uses the same data to learn the patterns and once again the same data to measure its performance.

If these numbers look promising, then start with a practice account and if traded profitably then progress to live account otherwise back to the drawing board.

PS: I was on this forum maybe 3 yrs ago, at that time I was looking into getting into prop trading but I realised I needed a more stable job so ended up joining an Analytics dept. at a bank.
Click to expand...

Nice.
 
sask said:
Coming from a data mining background, I do not get what the big deal is with backtesting. I think what is more important is how the strategies you built play in the time after you have built.

For ex: I plan to build a system using machine learning which will tell me to Hold(in my case - no trade to enter), Buy or Sell with fixed risk-reward ratio(1:5). Since I have 3 possible decisions - the system I'm planning to build will be a 3-class classification model. (using R - a link to which someone has posted to this post earlier and PostgreSQL database)

In order for the machine learning algorithm to learn the patterns, I need data (currently in the process of acquiring tick data from Oanda) to train my algorithm.

Say I use data from 01NOV2007-01NOV2012(training period) to learn some patterns/rules. Now, there are 2(actually many more) things to do. I need to understand that my system has learned something novel. This will be accomplished by testing the model performance(in terms of its classification accuracy) on a number of randomly (using bootstrapping) held out records from the training period.

Once satisfied with the performance, the final test(before getting on to the practice account stage) would be to see how it performs on data that my model has not seen before. This is what is called as the out-of-time sample(say data from 02NOV2012-28FEB2013) or simply test data or in terms of trading strategy set-up, forward testing.

This is where I would collect statistics such as Sharpe ratio, draw-down rate, win-loss among a myriad of other things. This is my best estimate of what the future of my trading strategy looks like.

In other words, forward testing is what I will be more concerned with. No point in using a system which uses the same data to learn the patterns and once again the same data to measure its performance.

If these numbers look promising, then start with a practice account and if traded profitably then progress to live account otherwise back to the drawing board.

PS: I was on this forum maybe 3 yrs ago, at that time I was looking into getting into prop trading but I realised I needed a more stable job so ended up joining an Analytics dept. at a bank.
Click to expand...

The big deal with back testing is quite simply.

Taking a block of time in the past (as close to present as possible) and testing a newly developed strategy. The markets work on numbers and human emotion hence why technical analysis "works" what happened one pattern has a high probability of happening again. If your building a robot you should know this better than anyone else. If back testing didn't work that would mean the markets are 100% variables and 100% unpredictable which we know isn't technically true or else your robot and technical analysis is a complete waste of time.

Furthermore the difference between forward testing and back testing can sometimes be blurred into one. Each trader may see forward testing as a different concept than another. For example I see forward testing as demo account testing, your trading based on data in the future. However other traders sometimes take today's data and check for trade set up on it tomorrow that to them is forward testing.

It's an interesting idea to build a robot to trade, but I feel a robot can't trade data it's "never seen" if it's never seen a pattern it's got no idea of the best possible outcome. It would have to place equal probability on all out comes 33% to bull, bear and sideways. Next time the same setup appears the probability might change slightly 40% bull, 30% bear and so on. In this situation it gives clear support to back testing.

Look forward to hearing any future results.

Steven
 
sask said:
Coming from a data mining background, I do not get what the big deal is with backtesting. I think what is more important is how the strategies you built play in the time after you have built.

For ex: I plan to build a system using machine learning which will tell me to Hold(in my case - no trade to enter), Buy or Sell with fixed risk-reward ratio(1:5). Since I have 3 possible decisions - the system I'm planning to build will be a 3-class classification model. (using R - a link to which someone has posted to this post earlier and PostgreSQL database)

In order for the machine learning algorithm to learn the patterns, I need data (currently in the process of acquiring tick data from Oanda) to train my algorithm.

Say I use data from 01NOV2007-01NOV2012(training period) to learn some patterns/rules. Now, there are 2(actually many more) things to do. I need to understand that my system has learned something novel. This will be accomplished by testing the model performance(in terms of its classification accuracy) on a number of randomly (using bootstrapping) held out records from the training period.

Once satisfied with the performance, the final test(before getting on to the practice account stage) would be to see how it performs on data that my model has not seen before. This is what is called as the out-of-time sample(say data from 02NOV2012-28FEB2013) or simply test data or in terms of trading strategy set-up, forward testing.

This is where I would collect statistics such as Sharpe ratio, draw-down rate, win-loss among a myriad of other things. This is my best estimate of what the future of my trading strategy looks like.

In other words, forward testing is what I will be more concerned with. No point in using a system which uses the same data to learn the patterns and once again the same data to measure its performance.
Click to expand...

Good post.
What kind of data are you after, why don't you buy the data?
Tickdata or Olsendata are good quality, obviously you pay for quality though.
Feed recording is OK, I do that as well, I wouldn't want to wait to accumulate the data though.
 
sask said:
In order for the machine learning algorithm to learn the patterns, I need data (currently in the process of acquiring tick data from Oanda) to train my algorithm.

PS: I was on this forum maybe 3 yrs ago, at that time I was looking into getting into prop trading but I realised I needed a more stable job so ended up joining an Analytics dept. at a bank.
Click to expand...

Just realised as you are pulling data from Oanda, you are obviously after FX data.
Olsendata only supply FX tick data to insti clients (might be able to source it as part of your job?)

Failing that give this a try:
Historical Data Feed :: Dukascopy Bank SA | Swiss Forex Bank | ECN Broker | Managed accounts | Swiss FX trading platform
 
Here is something that I couldn't quite figure out a satisfactory solution to myself:

"How much of any strategy is Edge, Hindsight, and Luck?"

Example said:
Say for instance I am betting on the outcome of a coin flip. I know nothing about the coin, only that it has two sides. I have no empirical data on historical coin flips or knowledge about how the coin was made. I decide to test the strategy HHTHTTHT, and it results in 100% "winners". In this case the strategy is 0% Edge 0% Hindsight 100% Luck.

Now let's say that someone sells me a record of the last 1000 coin tosses, and the results are 57% Heads 43% Tails. I switch my strategy to HHHHHHHH, and it results in 57% Heads 43% Tails. The strategy is devised out of 100% Hindsight (it's curve fitted to the data).

Lastly, let's say that I do more research on the coin and discover that it is weighted to the Heads side. I stick to my HHHHHHHH strategy, but this time the strategy is now made up of 100% Edge.
Click to expand...

Now the example I give is simplified, but the parallels with a mechanical strategy are clear to see. I never found a solution I was happy with, but it continued to plague my mind. I couldn't imagine putting serious capital behind something without figuring this out first.

To complicate matters I was testing this on single stocks, and so I would also use index data as a comparison.

Anyone got any thoughts?
 
Hakuna Matata said:
Here is something that I couldn't quite figure out a satisfactory solution to myself:

"How much of any strategy is Edge, Hindsight, and Luck?"



Now the example I give is simplified, but the parallels with a mechanical strategy are clear to see. I never found a solution I was happy with, but it continued to plague my mind. I couldn't imagine putting serious capital behind something without figuring this out first.

To complicate matters I was testing this on single stocks, and so I would also use index data as a comparison.

Anyone got any thoughts?
Click to expand...

"How much of any strategy is Edge, Hindsight, and Luck?"
Truthfully, its not something I can honestly answer.
Not even sure there is an answer TBH.

I don't see an algo as a definitive, all time answer anyway.
I spose in a nutshell, I do the best I can to avoid the hindsight issue.

I'd even go as far to say that all trading involves Edge, Hindsight, and Luck.
I doubt you can ever truly eradicate luck and hindsight from the equation.
We may kid ourselves that it is possible, or that we have eradicated them.
Personally I think the best you can hope for is to minimise their effects.
Thats as good as its gets, for me at least.
 
Hakuna Matata said:
"How much of any strategy is Edge, Hindsight, and Luck?"
Click to expand...


This is an interesting question, and I'm not sure there's an answer

Approaching this from my perspective I'd probably argue that something like a random entry Could possibly eliminate curve fitted entries based on hindsight, but then luck would a greater effect on the outcome.

It seams to me that if you reduce the effects of one element, you increase the effect on some other element, and vice versa. So if you are genuinely able to increase the effects of edge, luck becomes less important

There's some sort of trade off triangle, and any strategy can sit anywhere inside that triangle, and as strategy developers we get to decide where, and everyone will pick a different point.
 
sask said:
Coming from a data mining background, I do not get what the big deal is with backtesting. I think what is more important is how the strategies you built play in the time after you have built.

For ex: I plan to build a system using machine learning which will tell me to Hold(in my case - no trade to enter), Buy or Sell with fixed risk-reward ratio(1:5). Since I have 3 possible decisions - the system I'm planning to build will be a 3-class classification model. (using R - a link to which someone has posted to this post earlier and PostgreSQL database)

In order for the machine learning algorithm to learn the patterns, I need data (currently in the process of acquiring tick data from Oanda) to train my algorithm.

Say I use data from 01NOV2007-01NOV2012(training period) to learn some patterns/rules. Now, there are 2(actually many more) things to do. I need to understand that my system has learned something novel. This will be accomplished by testing the model performance(in terms of its classification accuracy) on a number of randomly (using bootstrapping) held out records from the training period.

Once satisfied with the performance, the final test(before getting on to the practice account stage) would be to see how it performs on data that my model has not seen before. This is what is called as the out-of-time sample(say data from 02NOV2012-28FEB2013) or simply test data or in terms of trading strategy set-up, forward testing.

This is where I would collect statistics such as Sharpe ratio, draw-down rate, win-loss among a myriad of other things. This is my best estimate of what the future of my trading strategy looks like.

In other words, forward testing is what I will be more concerned with. No point in using a system which uses the same data to learn the patterns and once again the same data to measure its performance.

If these numbers look promising, then start with a practice account and if traded profitably then progress to live account otherwise back to the drawing board.

PS: I was on this forum maybe 3 yrs ago, at that time I was looking into getting into prop trading but I realised I needed a more stable job so ended up joining an Analytics dept. at a bank.
Click to expand...


I personally wouldn't discount any type of testing, but i would have to agree that some types are more useful than others (ambiguous)

A simple average approach probably works best ie. any given market has an average daily % move.

Nice post btw, on an interesting thread.
 
the hare said:
This is an interesting question, and I'm not sure there's an answer

Approaching this from my perspective I'd probably argue that something like a random entry Could possibly eliminate curve fitted entries based on hindsight, but then luck would a greater effect on the outcome.

It seams to me that if you reduce the effects of one element, you increase the effect on some other element, and vice versa. So if you are genuinely able to increase the effects of edge, luck becomes less important

There's some sort of trade off triangle, and any strategy can sit anywhere inside that triangle, and as strategy developers we get to decide where, and everyone will pick a different point.
Click to expand...

Exactly what I mean, there are three "payoff" relationships, and I was trying to devise a way of knowing where on each of the three my strategy was - for instance on the Hindsight vs Edge relationship (going from step 2 to step 3 in the coin toss example), what proportion was HIND and what was EDGE..??

I do think that it is possible to get to the bottom of it, with approaches like factor analysis or principle component analysis, but I couldn't quite figure it out for myself.
 
Liquid validity said:
He didn't outright say back testing is useless (although that may well be his view - arguably perfectly valid as well).
Backtesting has several pitfalls, data mining bias being probably the worst:

Data dredging - Wikipedia, the free encyclopedia
The Bootstrap Test: How significant are your back-testing results? | Au.Tra.Sy blog - Automated trading System
Stupid Data Miner Tricks: How Quants Fool Themselves And The Economic Indicator In Your Pants - Forbes
Click to expand...

You are very correct, back testing has its pitfalls. That's why it should be combined with a solid testing plan. A combination of both back and forward testing and as I said a couple of post previously live trades.

Back testing is not the holy grail and never will be, it's an assessment of a strategy. However combined with both forward an live trades it becomes a very usefull addition.

By all means abandon back testing in favour of forward testing. But better yet create a strategy and launch it straight into the market without any prior knowledge of whether it works.

I perhaps think you caught the wrong signal from what am saying or I may not have projected it well enough. If that's the case I apologise.

Just to reiterate back testing isn't the all mighty guarantee, it's got more holes than a pin cushion. But it's "another" method of checking the profitability of a strategy, combining it with other testing increases it's viability.

Thanks for takin the time to respond and for your input.
Kind regards
Steven
 
Hakuna Matata said:
Exactly what I mean, there are three "payoff" relationships, and I was trying to devise a way of knowing where on each of the three my strategy was - for instance on the Hindsight vs Edge relationship (going from step 2 to step 3 in the coin toss example), what proportion was HIND and what was EDGE..??

I do think that it is possible to get to the bottom of it, with approaches like factor analysis or principle component analysis, but I couldn't quite figure it out for myself.
Click to expand...

TBH, I think the closest you will ever get to a definitive answer is after 5-10 yrs of live trading.
If you aint bust, its safe to say you're in the edge part of the triangle (unless something changes).
Should have a pretty good idea after a few years though.
 
Hakuna Matata said:
Exactly what I mean, there are three "payoff" relationships, and I was trying to devise a way of knowing where on each of the three my strategy was - for instance on the Hindsight vs Edge relationship (going from step 2 to step 3 in the coin toss example), what proportion was HIND and what was EDGE..??

I do think that it is possible to get to the bottom of it, with approaches like factor analysis or principle component analysis, but I couldn't quite figure it out for myself.
Click to expand...


I don't have an edge. The reason i say this is because i have no unique information. All the information i have is freely available to any other party.

Hindsight and luck.

Luck. Probably better described as random chance.

Random. can it be harnessed positively?

Cut losses - run profits.
 
Liquid validity said:
TBH, I think the closest you will ever get to a definitive answer is after 5-10 yrs of live trading.
If you aint bust, its safe to say you're in the edge part of the triangle (unless something changes).
Should have a pretty good idea after a few years though.
Click to expand...

Well, fast forward 5-10 years, how would I know then?

Given enough data in-sample and out-of-sample data, I believe it should be possible to find whereabouts on the triangle any strategy is.
 
Hakuna Matata said:
Well, fast forward 5-10 years, how would I know then?
If you aint bust?

Given enough data in-sample and out-of-sample data, I believe it should be possible to find whereabouts on the triangle any strategy is.
Click to expand...

If you mean ball park area, then yes.
Even then, for me personally, I'm pretty sure mine would have to be canned
if an 80's type bull occurs again.
I'm quite curious about that, obviously I have no way of knowing.
Might be OK, I really don't know.

In terms of an outright mathematical answer to the problem I don't believe its possible.
I focus primarily on randomness, game theory, and drawdown with a nod to mean reversion.
Cumulative profit is a by product for me, not the primary development metric.
 
kimo'sabby said:
I don't have an edge. The reason i say this is because i have no unique information. All the information i have is freely available to any other party.

Hindsight and luck.

Luck. Probably better described as random chance.

Random. can it be harnessed positively?

Cut losses - run profits.
Click to expand...

This comes down to how you define an "edge", something that has been discussed here over and over, and never got anywhere as the usual "positive expectency" arguments come out.

For my purposes, I define an "edge" as any systematic property of the markets that make them "opposite of random". So insider information counts as an edge. But so does, say, the repeating habits of participants that make trends occur (if you believe they do). Or the idea that stocks fall faster than they rise, etc.

If you didn't believe that these properties existed, why would you bother trading at all?
 
Hakuna Matata said:
This comes down to how you define an "edge", something that has been discussed here over and over, and never got anywhere as the usual "positive expectency" arguments come out.

For my purposes, I define an "edge" as any systematic property of the markets that make them "opposite of random". So insider information counts as an edge. But so does, say, the repeating habits of participants that make trends occur (if you believe they do). Or the idea that stocks fall faster than they rise, etc.

If you didn't believe that these properties existed, why would you bother trading at all?
Click to expand...

I agree about basic market characteristics, I think that is where randomness is often misunderstood.
Randomness can be applied to those types pf market mechanics, purely because,
although you know those characteristics exist you will never know exactly
when they will exhibit themselves each and every time.

Sure you can have a very good idea quite often that certain types of behavior
are going to occur.
Not every time though.
Then the other random element is individual trade outcome.
Even with an edge, if you knew which trades would lose, you wouldn't open them.

For the most part, as you've said before, probably comes down to personal market beliefs.
There really is no concrete answer for everyone.

Good thread BTW.
 
kimo'sabby said:
That is a really good point.

Trends?
Click to expand...

You can get "trends" in random or non-random data* because they are only identified after they have occurred. If you don't subscribe to the idea that "edges", as I define them, exist, then you can still trade a "trend". But your results are exclusively down to chance (not that there is anything explicitly wrong with that). You could equally trade a mean reverting strategy with the same consequences.

But if you think that there is an edge in trends (for example that they occur more often than you would expect from random data, or there are events that are a precursor to a trend developing), you might like to know how much your exploitation of it contributed to your P&L, and how much was down to luck.

* this is a bit ambiguous, but for the sake of argument we'll run with it.
 
You must log in or register to reply here.

Similar threads

D
Backtesting vs Forward Testing
Replies
0
Views
996
davetapson
D
Jay Ng
  • Article
Swing & Position Trading Day Trading & Scalping Why Back Testing is Better than Forward Testing
Replies
6
Views
7K
foroom lluzers
F
I
Forward Testing
Replies
5
Views
3K
zentrader22
Z
S
Back/forward testing & Expectancy & Profit Factor
2
Replies
30
Views
6K
tony1979
tony1979
S
Conducting and Evaluating a Forward Test
Replies
4
Views
5K
selfmade
S
Top