Forward Testing.

[FringFX]

that is exactly the goal of procrastination

 

[rawrschach]

Ok so the forum had a huge banner across it telling me I hadn't posted for a while and to make a contribution so although I'm sure it's been asked before, after how much forward testing can you assume a statistically significant result to show an edge?

A time frame of say 1 month, 3 months or 6 months?
An amount of trades like 10, 50 or 100?

Define statistically significant?

Monte Carlo simulation would be more useful. Back/forward testing both useless imo.

edit: I see the bunny already said this.

 

[Steven2805]

I think back testing is important but should not be looked at as expected results.

My general rule is :
Half the profits,
Double the time,
Double to draw down

If it's still in profit then you might be onto something

So this is where Forward testing comes into play.
Whip out the demo account and start trading that sucker.

On a final note, time scales.

I back test depending on the time frame am trading.
1 hour time frame is normally 2~3 months back testing, 1 month forward testing and 1 month real trading.

Compare all results and draw a conclusion on whether the strategy works or not. Also I think back testing isn't a once does it deal, you have to always be back testing

 

[forever_young]

My general rule is :
Half the profits,
Double the time,
Double to draw down

If it's still in profit then you might be onto something

This is strict but fair. You haven't ever use scalping, have you?

Compare all results and draw a conclusion on whether the strategy works or not. Also I think back testing isn't a once does it deal, you have to always be back testing

That is good thing about conclusion and testing always the system. A few people do it usually.

 

[brewski1984]

I'm liking this monte carlo simulation thing. I didn't realise you could do so much with Excel.

 

[SavantK]

Excel makes wonders... Till lately I thought I knew excel. But now I can say there is vast untapped potential...

 

[blackyzero99]

forward test of what? EA? NS? or what?

 

[Hakuna Matata]

Ok so the forum had a huge banner across it telling me I hadn't posted for a while and to make a contribution so although I'm sure it's been asked before, after how much forward testing can you assume a statistically significant result to show an edge?

A time frame of say 1 month, 3 months or 6 months?
An amount of trades like 10, 50 or 100?

When I was experimenting with mechanical strategies, an over-simplified version of what I did goes something like this:

On old data (the data I had backtested my strategy on, say 2006 - 2008), I'd take a sample of the asset returns during the time my strategy would either be long or short, and collect these in a new data set. So I end up with a series of daily returns from when my strategy would have a position on. I would then collect a new sample of data (forward testing, but you can do this on old data, say 2009-20011) and take out a new sample using the same rules - so I end up with two sets of daily returns.

I then made up some descriptive statistics on the former sample, and did confidence tests that the "forward sample" was from the same population as the "backward sample". This turned out to be one of my rules for trading the strategy (which never came to fruition), that a recent sub-sample was still part of the larger population (with x degree of confidence).

As for Monte-Carlo simulations, it's important that the data you generate shares the same properties as the data you sampled. So draw your "random" daily returns from a population with the same distribution statistics as the samples you collected - mean, sd, skew, kurtosis etc. Otherwise you are testing your strategy with data that bears no relevence to the market conditions you have a position for. It can also be interesting to look at descriptive statistics for your "non-sampled" data (when your strategy is flat).

Another test I did was bootstrapping, IMO preferable to Monte-Carlo, whereby you collect your sample data, jumble it all around, and test the strategy again. This is very useful when examining the potential for drawdowns or stop-placement.

 

[Liquid validity]

When I was experimenting with mechanical strategies

Good post, was a bit surprised to see you post about mechanical stuff though.
Never knew you had previous involvement with it until now.

BTW, for Brewski, Ninja has built in monte carlo testing.
Maybe you don't like the platform, I don't know, just thought I'd mention it.

 

[Hakuna Matata]

Good post, was a bit surprised to see you post about mechanical stuff though.
Never knew you had previous involvement with it until now.

When you're looking over long time spans, the "sheep" mentality is ever-present. Bubbles and Crashes are always going to happen. There are several reasons why it is natural for trends to occur. It all boils down to what you believe about markets.

I was testing "breakout" strategies on volatility.

 

[brewski1984]

I have had a few demo accounts with Ninja but never really knew about monte carlo testing until this topic. I'll check it out at some point.

 

[Liquid validity]

I have had a few demo accounts with Ninja but never really knew about monte carlo testing until this topic. I'll check it out at some point.

NinjaTrader Version 7 monte carlo

 

[brewski1984]

NinjaTrader Version 7 monte carlo

Legend

 

[Steven2805]

This is strict but fair. You haven't ever use scalping, have you?



That is good thing about conclusion and testing always the system. A few people do it usually.

Depends, I have a scalping strategy which I intend to backtest when I have some free time. The rules aren't meant to be fair but realistic. Lower expectations while maintaining confidence is a steady balance. In my experance many people backtest by just "looking" at the chart. On the other hand I could tell you my expected drawdown, win/lose, p/l per trade ect because I document and constantly review the strategys performance and equally important my performance.

It's not a perfect science but neither is trading, no strategy is full proof but your better going in knowing it's at the very least worked in the past.

Don't get me wrong I think there will be times when back testing isn't practical. Namely really quick scalping (trading an over extended candle for example). However there is nothing stopping you logging the trade as you would a backtest as the move is developing. It's in theory a backtest done in real time (forward test)

 

[Hakuna Matata]

FYI anyone that wan't to look at backtesting seriously would do well to look at something like R

The R Project for Statistical Computing

it's free and can do some very fancy things if you're computer savvy.

 

[Liquid validity]

FYI anyone that wan't to look at backtesting seriously would do well to look at something like R

The R Project for Statistical Computing

it's free and can do some very fancy things if you're computer savvy.

Looks like the data analysis section of matlab for free

 

[Liquid validity]

When you're looking over long time spans, the "sheep" mentality is ever-present. Bubbles and Crashes are always going to happen. There are several reasons why it is natural for trends to occur. It all boils down to what you believe about markets.

I was testing "breakout" strategies on volatility.

Sorry, missed this reply somehow.

Broadly speaking this briefly sums up my market belief:
http://www.trade2win.com/boards/tra...-trading-strategy-how-easy-5.html#post2007050
Teh arab sums it up well.

The basis of what I do is hidden somewhere in that quote.
Basically though, I spose you would call it mean reversion.
Nothing eye opening really.

 

[Aesop20]

FYI anyone that wan't to look at backtesting seriously would do well to look at something like R

The R Project for Statistical Computing

it's free and can do some very fancy things if you're computer savvy.

This looks pretty coool, checking it out.

 

[sask]

Coming from a data mining background, I do not get what the big deal is with backtesting. I think what is more important is how the strategies you built play in the time after you have built.

For ex: I plan to build a system using machine learning which will tell me to Hold(in my case - no trade to enter), Buy or Sell with fixed risk-reward ratio(1:5). Since I have 3 possible decisions - the system I'm planning to build will be a 3-class classification model. (using R - a link to which someone has posted to this post earlier and PostgreSQL database)

In order for the machine learning algorithm to learn the patterns, I need data (currently in the process of acquiring tick data from Oanda) to train my algorithm.

Say I use data from 01NOV2007-01NOV2012(training period) to learn some patterns/rules. Now, there are 2(actually many more) things to do. I need to understand that my system has learned something novel. This will be accomplished by testing the model performance(in terms of its classification accuracy) on a number of randomly (using bootstrapping) held out records from the training period.

Once satisfied with the performance, the final test(before getting on to the practice account stage) would be to see how it performs on data that my model has not seen before. This is what is called as the out-of-time sample(say data from 02NOV2012-28FEB2013) or simply test data or in terms of trading strategy set-up, forward testing.

This is where I would collect statistics such as Sharpe ratio, draw-down rate, win-loss among a myriad of other things. This is my best estimate of what the future of my trading strategy looks like.

In other words, forward testing is what I will be more concerned with. No point in using a system which uses the same data to learn the patterns and once again the same data to measure its performance.

If these numbers look promising, then start with a practice account and if traded profitably then progress to live account otherwise back to the drawing board.

PS: I was on this forum maybe 3 yrs ago, at that time I was looking into getting into prop trading but I realised I needed a more stable job so ended up joining an Analytics dept. at a bank.

 

[malaguti]

Coming from a data mining background, I do not get what the big deal is with backtesting. I think what is more important is how the strategies you built play in the time after you have built.

.

And back to the drawing board you will possibly collect more data, from a different sample period for example, and do what with it; Back test it perhaps? Thats the big deal ...