Backtesting / Historical testing -- how far?

[ed45626]

How far have you gone for back testing? I have a nice little system -- not going to make me a millionare in two weeks, but solid, that I've back tested.

2003 - 2005: results look good, steady positive flow, manageable drawdowns
1998 - 2002: loss every year

Any thoughts? Did the forex market change in 2003? How far should I worry about?

I'm a little concerned about going forward.

Thanks for any input!

Ed

 

[sccz97]

backtesting is always a useful tool to determine the general characteristics of a particular strategy but of course historic time series are exactly that, historic. The finanical markets evolve and its characteristics change along with the evolution of the industry and climate. A very good risk metric that is used amongst financial insitutions as a means of capital allocation is Value-at-Risk which broadly states the maximum amount of money that can be lost within a given confidence interval over a particular time horizon.
There are 3 different methodologies for VaR including a bog standard variance/covariance approach and more sopihsticated monte carlo apporaches but the one adopted by the larger banks is historical simulation. Similar to backtesting, the calculation of VaR via historical simulation uses the prices, spreads, volatilities and all other risk factors for the last x years to determine a p&l distribution with which the 99th percentile is taken as the VaR. The lookback period is important as the prices etc for 5-10 years ago will not be as relevant as those in the last 2-5 years say. What is more .. if you performed a 2 year simulation varying the start and end date you will end up with different results. So, like VaR calcd via historical simulation, in order for your backtesting to mean anything more just how much you would have made in the past, you need a backtesting engine that can have parameters tweaked so that your outcome more accurately reflects the current environment.
I reliased I've gone on about something that may not appear to have anytihng to do with your problem but if you genuinely want to succeed in trading, you need to do as much researhc and analysis as you can.

 

[Charlton]

Analyse 2002/3 market changes to advise future strategy

How far have you gone for back testing? I have a nice little system -- not going to make me a millionare in two weeks, but solid, that I've back tested.

2003 - 2005: results look good, steady positive flow, manageable drawdowns
1998 - 2002: loss every year

Any thoughts? Did the forex market change in 2003? How far should I worry about?

I'm a little concerned about going forward.

Thanks for any input!

Ed

sccz97 said it all and you should follow his advice.

I don't know specifically what happened to forex in 2003, but I'm sure that others can enlighten us. Doing some searches for this period (google: FOREX 2003) I did find:

China's inter-bank foreign exchange transactions volume hit a record 151.1 billion US dollars in 2003, the Shanghai-based China Foreign Exchange Center announced on Friday. By the end of 2003, there were 344 of forex trade dealers at the China Foreign Exchange Center, 21 more than at the end of the previous year.

I also noticed that a lot of forex brokers were offering new online solutions to traders.


Whateve it was It suggests that the parameters you are using in your system are dependent upon conditions that changed around that time. These conditions may change again in the future, so I think it would be worthwhile investigating that period and tweaking your parameters specifically for that period to see if you can get a better match for that period. This might provide you wih pointers for how you would have to change your system in the future. It may also give you ideas of other factors you may have to monitor going forward.

Charlton

 

[TWI]

What I would suggest is that you identify something in the historical system performance which will signal to you that it is no longer working in the current market environment. This is a major issue for all of us who trade systematically, you never know for sure if the next x trades will be net winners of losers but what you can do is see how the systems performs in the last x trades vs. how it performed over the period where it was consistently profitable. This is why it is, in my opinion, important to run a system or systems across a number of markets and timeframes and to make sure that the correlation between these systems is not high. When you see what you have determined to be an uncharacteristic behaviour in any single system then it is best to discontinue taking the signals but keep following it and wait to see some sign that it has not fallen off a cliff before throwing any more money at it.
One of the best ways I have found to obtain this "stop" parameter is to take the results during the historically acceptable period and do a monte carlo simulation on these results so randomly mixing up the trade order many many times. After running this you should have a good appreciation of your worst case draw during times that the system is considered working and hence some criteria for failure.
An easier method that I know some traders use is to technically trade the equity curve of your system. That means stopping to trade when the uptrend is taken out and only starting to get back in on new highs, or something along those lines.
I have been trading systematically for a few years and I find that the systems do continually evolve. The simplest systems I use have produced the most consistent returns although the volatility of those returns has been and was expected to be larger than that of the more involved stuff. One of the most valuable things I learned is the benefit of combining systems that operate on different time frames in similar markets. I actually consider an FX system I run on weekly, daily and hourly data across dollar majors as one system rather than three as the shorter time frames hedge or increase the risk of the longer.
One thing is for sure, unless you start trading something, you do not get anywhere and although some systems can show positive results over all available data many of them, when you drill down to certain short time periods have nasty runs. In many respects you have to strike while the iron is hot but always know your out and always be working on the next holy grail.

 

[josephforex]

I believe that there will never be a system that will work. You can bet your last penny some broker/software guru or the guy/gal ready this is always looking for an edge to get your money. Some suggest going both ways on a pair and look for the retracement. Try doing a study on these periods and you will have a better edge than any other. Some will notice seasonal directions/trends and some will change. USD/CAD is a great example, how many thought that after 1.1600 it would retrace back to 1.2200-1.2500+ ....never happen. Big boys hog tied the option market for those elliot wave projections. You can always back test and back test. There are always going to be surprises no matter what news, trendline, channel happen on the charts. Best bet is to find a time when you can count on a small rally and get out. IE EUR/USD after 5 min. into the market sometimes you see a small spike or direction, and again on the 3:30 and 4:30 am est for other small spikes. I don't know about systems , but more and more gurus are making more selling these ideas/software than they do trading (just an guess with no back test to back it up)

 

[sccz97]

One of the best ways I have found to obtain this "stop" parameter is to take the results during the historically acceptable period and do a monte carlo simulation on these results so randomly mixing up the trade order many many times. After running this you should have a good appreciation of your worst case draw during times that the system is considered working and hence some criteria for failure.

whilst there are many benefits to using monte carlo there are more traits of a mc-driven simulation that the average person is not going to realise or understand well enough to make the simulation results meaningful. Where do you find time series to calculate your input parameters to the simulation? Are you going to include a jump diffusion model to simulate gaps that occur over announcements. Market returns are not normally distributed and getting an MC engine to take a non-gaussian random number generator is not the simplest of matters. Further .. how will you factor in correlation? What will you do if your correlation matrix cannot be decomposed via normal cholesky? There are so many things factors that need to be taken into account that placig too much faith in the results will inevitably lead you down the wrong path

 

[josephforex]

****** claps ***********

 

[sccz97]

****** claps ***********

if you're gonna make sarcastic comments then it's prob best you refrain from posting at all. THe problem with ppl here is that they're all too close-minded looking for that holy grail be it wrt to trading or backtesting. I'm gonna regret saying this but you probably don't make money trading at all and the thought of you providign advice to ppl is laughable at best

 

[TWI]

Sccz97,
I understand that there are always ways to further drill down into testing criteria when trying to assess how robust a system is, or indeed, when coding the system in the first place. I cannot directly answer your questions because the way I utilise mc has worked for what I have developed. I have found that it is not necessary to use anything to complex to account for big figure days. Indeed over time and many tests these days net out to become an irrelevance.
My input parameters are found by basic stability testing. Price and Vol are just variables in the systems calculation and so it is important that values ascribed to user defined constants remain stable across as large a range of values as possible . I found that it is necessary to make calculations as dynamic as possible but user defined values are nearly always necessary and this is where the stability testing is essential of course. I do not have a use for mc until a system which is workable in test and blind set. Once it has demonstrated a suitable positive expectancy I run it through the mc sim to mix up the trades and see how it pans out over many sims. 1000's of sims over a large number of trades is as good a test set as I have required. For correlation I use some commercial software which has been proven reasonable when I have compared real-time runs with historical test sets. What I produce works, it is how I make my living but I am always keen to hear anything I do not yet know regarding this field. Please expand on what you wrote earlier.