The Tenets of a Successful, Backtested, Mechanical Trading System

ybfjax

Junior member
31 0
What about the performance evaluation? Which parameters matter? Which ones are junk?
I'm referring to win rate, avg trade return, profit factor, etc...

Just more distractions. None of those really matter. The primary performance evaluation is seeing if you get a net account increase over many repetitions or tradecycles. Then repeat these tradecycles as much as possible.
 

jaella

Newbie
9 0
I don't seem to be allowed to start new threads; this thread seems relevant though so I will post here.

Contrary to what my post count may indicate, I am a longtime lurker of this and other trading forums. I am not looking to sell any wares, I am not looking to have an esoteric discussion about "trading psychology", and I am not here to profess the wonders of "price action" trading.

If it helps my credibility, I am familiar with posters from some time ago such as Bramble, EliteJets, Trader_Dante etc. This is not me vouching for any of them and nor do I have anything to do with such posters, I am just trying to show that I am not a blow-in.

I am someone who has learnt how to trade discretionally, and am ready to move into the realm of mechanical trading. I would like to seek some advice, if anyone would be willing to share. Trolls need not apply (but will anyway).

With reference to this thread, my question comes down to what constitutes a successfully backtested mechanical system? I suppose what I would like to know is "How much is enough" with regards to testing? I realise that there is no definitive answer to this, so let me tell you some of what I have done and maybe you can add to my thoughts.

I have developed a mechanical system for equities and futures. It work across multiple equities (not all), multiple timeframes (incl. intraday), and multiple exchanges. My success across these different realms leads me to believe that I have a bonifide edge. My results are not marginal.

The backtesting has been completed mechanically, but not in your typical manner. I know the many limitations of EasyLanguage and NinjaTrader, and prefer to have the confidence that can only come through manual backtesting. Yes, testing in this manner takes time.

Yes I have forward tested over several months. Due to the nature of my backtesting, the results from my forward testing essentially match the results from backtesting.

Yes I have completed Monte Carlo testing of my results (although I question the usefulness of the results, due to actual trade performance being unique).

The system has not been optimised in any way and I do not intend to optimise for fear of curve fitting. If this means that I underperform in some manner, then so be it. Frankly, the results are more than suitable as they are.

So back to my question, how much is enough?

The system generates ~2.7 trades per day. This means that on any one instrument (equity/future), I will generate 800-900 trades per year. Obviously more is better for statistical purposes, but there are obstacles to testing, particularly with regard to data quality.

Yes, quality data exists, but I have found it to generally be prohibitvely expensive at this stage. Hence, any data obtained through other means usually needs to be quality controlled for missing/erroneous data. This is quite the exercise if you are using say, 5 minute charts.

At what point do I say, "be done with it, time to go live?".

You are always at the mercy of the market, no matter how much testing you have done.
 

ybfjax

Junior member
31 0
.......

At what point do I say, "be done with it, time to go live?".

You are always at the mercy of the market, no matter how much testing you have done.

When you are confident in the forward tested results performing as expecting in typical market conditions.

There is 1/2 of 1/2 of systems implementation that we have control over and 1/2 that we have no control over. Focus on the 1/2 that you have control over; mainly these are entry, position sizing, and exits.
 

BeginnerJoe

Senior member
3,329 350
No amount of testing, forwards, backwards, sideways will be enough because when you want to take someone's money for real they will fight you every part of the way. But you already know that since you submit to the market's mercy. The market ain't dumb like your simulation.
 

jaella

Newbie
9 0
By manual backtesting, I mean going through the historical data by hand, as opposed to programmatically backtesting. I am forward testing with each new day that passes and am into my second month of forward testing. Just getting a little antsy, because I know that I am onto something.

ybfjax, that was what I was thinking. Perhaps I will hold out until the new year.
 

Shakone

Senior member
2,458 665
Going through by hand is useful, but is also dangerous, as we have hidden biases, and look for what we want to see, and ignore some things we don't want to see. So it's easy to get misleading results from this.
 

jaella

Newbie
9 0
Going through by hand is useful, but is also dangerous, as we have hidden biases, and look for what we want to see, and ignore some things we don't want to see. So it's easy to get misleading results from this.

Yes, I understand what is involved. I have removed the possibility of bias in my backtesting. Excel is used to an extent. I was trying to make the point that my results are not due to some programming quirk. I find testing in this manner to be very transparent, hence why it performs similar when forward tested.
 

jaella

Newbie
9 0
Is there any reason why you think simulating trades manually is better than letting a computer do it?

Aside from the mentioned problems of unrealibility, I imagine going through years of charts and simulating trades is also extremely boring :(. Any significant tests, such as walk forward analysis, can not be done manually anyway. You can do only simple backtests with fixed parameters and they do not tell much about the performance in real trading.

When backtesting programmatically you have the annoyance of intrabar granularity and how the program deals with this problem. A solution can usually be coded, but it complicates even the simplest of backtests. This is not generally a problem if your entry/exit is onbarclose, but if you are using a threshold that can occur within a certain time period then you will have issues.

The parameters that I use are dynamic and adjust with the market, allowing for simple testing.

As for being boring, I can handle the monotony because I know what this is potentially leading to.
 
L

Liquid validity

0 0
When backtesting programmatically you have the annoyance of intrabar granularity and how the program deals with this problem. A solution can usually be coded, but it complicates even the simplest of backtests. This is not generally a problem if your entry/exit is onbarclose, but if you are using a threshold that can occur within a certain time period then you will have issues.

I agree, Intrabar granularity is one of the biggest stumbling blocks anyone will face in
backtesting / automation, if using time series data.

Many discretionary rule based methods (which otherwise work) fall apart at the seams for this very reason
when backtested / fully automated.

I don't know about other platforms, but Ninja can for instance, reference
daily charts (or any other tf for that matter) whilst executing on a lower time
series, or even tick level.
 
L

Liquid validity

0 0
By manual backtesting, I mean going through the historical data by hand, as opposed to programmatically backtesting. I am forward testing with each new day that passes and am into my second month of forward testing. Just getting a little antsy, because I know that I am onto something.

ybfjax, that was what I was thinking. Perhaps I will hold out until the new year.

Personally, 2 months wouldn't be enough for me.
I would forwards test until:

1/ First major drawdown reached.
2/ 300+ forward test trades.
3/ 12 months forwards test duration

Once all 3 criteria fulfilled, forwards testing ends.
Not hard and fast rules, just my own preference.
 

wheresthejelly

Newbie
8 2
Just wanted to update everyone here and thank some of you specifically for things I've learned while reading this thread since I made the original post.

I started trading live my completely automated mechanical system in late October 2012. I am now up 12% since then with a 5% drawdown so far. These numbers are nice, but here is what really made me happy: my live performance closely tracks my simulated performance over that same time period. :)

Thanks bbmac for getting the replies started in a positive light.

Thanks Spec-K Trading for exploring other angles of mechanical systems. Too many points in your huge post to reply to everything right now. I pretty much agree with all the basic ideas behind your post. My system trades many different markets with lots of independent trades every day. My overrides are in place in case of a black swan event. My system backtests about 20,000 trades between 1990 and 2013 with just over +1% trade expectancy. If I open my pre-filtering a bit, I can get 60,000 trades with about +0.5% trade expectancy.

Thanks doh for pointing out forward testing. My system is very adaptable to changing market conditions and doesn't use many fixed parameters. Forward-testing did not prove useful for me. Optimization over the long time frame of 1990 to 2012 proved VERY useful.
 

Spec-K Trading

Junior member
14 1
Congrats on your live trading success Jelly. I've also had great success now for 10 months running, trading several hundred thousand continuously (198 trades completed to date). +22% currently, and only one losing month (Feb 2013), with new equity highs again this month (Mar 2013). And yes, overall I'm running at about 1.1% average per trade return. My system keeps all the money active all the time, spread across 10 trades. The faster I can reenter on a profit exit (which on average occur several times a week, 2-5 typically), the faster returns grow, so I frequently am reentering same day as exit.

I'm more than a little aware the market has been sweet for the most part for long-only systems, and I'm sure to have a significant drawdown when the current market cracks, but that's just part of the game, I intend to trade through it (because that's how I found I maximize overall profits for the long term, vs. back off strategies that I found always reduce long term profits with my system).

As for the many questions about when enough testing is enough, there is no right answer. What I'd suggest is when you feel confident based on all the testing, start putting your money to work, ideally incrementally (scale up). That's what I did, starting with 1/2 size trades, and scaling up over the first two months as I got comfortable with serious $'s in the market all the time, etc.
 

numbertea

Well-known member
257 9
Complete automation of a profitable trading system, also staying profitable, is a dream that I have been cooking for the past 5 years. Backtesting and forward testing is of course the only way to test. I have not found a commercially available demo that is completely realistic so I had to build one into my own system so that I could test it. The importance of granularity on intrabar trades wasn't rocket science to solve, level 2 book data and trade que tracking was the only way that I could finally reproduce actual market conditions. I use Lightspeed API and with a fast Internet connection I've been able to have results of testing meet results of real world trading. Now if I could only quit my job and spend more time programming I could quit my job. Do I have to really keep paying my bills?

Cheers all
 

graydrake

Junior member
39 2
I find a lot of posts in this forum are asking questions rather than providing solutions. So, I'm going to provide what I believe to be the major important tenets of a mechanical trading system. I see a lot of confusion out there about what makes a mechanical trading system successful. This post should clarify what a system designer should aim for and what he/she should believe about trading.

Major tenets and beliefs:
1. Human traders or discretionary traders lose trades because of fear and greed.
2. Human traders or discretionary traders cannot maintain the necessary discipline to trade consistently forever.
3. Market movements of all kinds (price, volume, volatility) are 99.99% determined by human factors rather than mechanical systems.
4. Every profitable trade in history will be repeated in the future.

These are beliefs, which I feel are very close to reality because I have tried discretionary trading, and I also design mechanical systems. I have been lurking on T2W for about a year, and I've designed two mechanical trading systems. Full disclosure: I have not made any real money with these systems yet, but I feel that I will in the next few months once I move away from paper trading. I am not fully qualified to tell you what works in a real-money system. However, I am qualified to tell this forum how to design a system that backtests very successfully.

Just starting out last year with backtesting software, I could barely get an upward sloping equity curve in simulation. Now, I've got an upward sloping equity curve with very low drawdown over a span of 20 years. Some simple stats I can give you are my backtested cummulative annual return (53%), and the maximum system drawdown (28%). Both of these stats are over the time period from January 1, 1992 to June 29, 2012. I've learned so much in how to design a successful system in backtesting software over the past year. So, I am qualified to tell this forum how to design a successful system in backtesting software.

Key Success Factors in designing a successful mechanical trading system in backtesting software:
1. Performance goals
2. Accurate and comprehensive data
3. Positive expectation
4. Risk management
5. Time frame
6. Optimization

If you can understand and do well at each of these areas, you will design a profitable mechanical trading system in simulation. Here's more elaboration on each of these key success factors:

1. Performance goals: Decide for yourself BEFORE backtesting anything what drawdown you can handle. Imagine you had a working, real-money system. Then imagine that next week your system lost 25% of your money. Would you break down and cry? Would you stop trading out of fear? If you feel a cringe at losing that much money, try 20% or 15%. If you're okay with 25%, what about 30% or 35%? Another stat you should define is time in a drawdown. Let's say you recently had a good month of trading, and your account value is the highest it has every been. Feeling good! Then, let's say your system loses 10% next week and doesn't recover for 6 months. Could you handle that? If you trade for clients, could your clients handle that? What about 5 months? The last stat to define is your desired cumulative annual return. Would you be happy with 25% per year? If so, what about 20%? Keep going down until you find a number that sounds unacceptable to you. These three statistics are your performance goals. Design your system to achieve them. Change your system if you feel it doesn't meet these goals. Be serious and realistic about this! Don't be silly and choose a 5% drawdown, 2-week time in drawdown, and a 300% cumulative annual return.

2. Accurate and comprehensive data: Your data source should provide data for all the stocks you want to trade. Also, it must provide data for all de-listed stocks you would have traded. Make sure you go through some stocks to make sure that stock splits were handled correctly. Some backtesting software and data sources do not account for dividends. It is up to you to decide whether that invalidates your backtest results or not. A system that has an average trade length of a few days should be fine with not accounting for dividends. A system that holds positions for weeks or months should somehow account for dividends. Remember that short positions pay out dividends rather than receive them.

3. Positive expectation: It is very very important that your trading rules have a positive expected value. Mathematically, this means (win rate)*(win amount percentage) - (loss rate)*(loss amount percentage) > 0. For example, let's say you design a fictitious system that buys on a moving average crossover and sells 4 days later. Your system wins 58% of the time and loses 42% of the time. Your winning trades make 3.5% on average and your losing trades lose 3.7% on average. Your expectation is 0.58*0.035 - 0.42*0.037 = 0.00476, which is greater than zero. This means that you can expect to make 0.476% on average on every trade you enter in the future (Tenet 4 above). It is very difficult to design a profitable trading system with a negative expectation.

4. Risk Management: Risk is the potential that a particular trade you enter will results in a loss. Every trade you enter has a chance of losing. Some trades have a bigger potential loss than others. You can find out some factors that produce bigger winners and smaller losers. Perhaps stocks with a very high average true range win big with your system? If so, bet more on those positions and less on others. Also, consider how big your stop losses and profit targets are if you use them. There are many position sizing methods to use here, but the main ones are "fixed risk" and "fixed fractional" position sizing. The Kelly formula will give you a decent starting point for position sizing. Google that.

5. Time frame: Just about every random mechanical system ever created is very successful over a short time frame like a month or a year. However, you have to know when to start trading that system and when to stop. A short-only, trend-following system would have been great during the time frame of September 2008 to March 2009! How do you know when to stop trading the system? It's very hard to predict turning points in the market like March 9, 2009. It's much simpler if you design a system that performs well or at least okay during up, down, volatile, and quiet markets. It's okay if you can get your system to make hordes of money in some types of markets and just limit losses other types of markets. The key statistic is your expectation over a long period of time (like 10-20 years). Hopefully, it is positive. If you have positive expectation, you can adjust your risk management to achieve your performance goals.

6. Optimization: Run your backtest over your selected time frame with a range of different parameters. For example, going back to our moving average crossover system, test over different periods for the moving average. You can test a range of values for the periods of any indicator, the stop loss amount, the fixed risk amount, the number of days to stay in a position, etc. Any parameter that is numeric can be optimized. In your optimization, you want to make sure that your system makes money over a wide range of values for each of these parameters so that your simulation results are robust. Once you confirm that, choose the parameter values that produce the best performance (e.g. highest annual return, lowest drawdown, highest win rate, etc.).

I hope this helps some people and stirs some discussion. Thanks for reading!



Is there anyone out there that has successfully used a truely mechanical system?

If so,
What was your annualized return?
Over how long a calendar time?
How many trades completed?
Win/loss ratio?
What market?

I have successfully back tested several. Unfortunately they work in todays back test because the shape of the index (or ticker) price/time movement is compliant with success in the specifics of the mechanical system. Tomorrow's shape will be unique and almost certainly will require an alternate mechanical system to deliver similar results as in the past.

Comments?

Drake
 

robocop

Well-known member
495 1
Here my contribution of how I screen the results for a backtest without having years of intraday tick data available.

The first thing I look for is a nice and steady equity curve, like this system on the DAX with 5 min bars.

StrategyTester.gif

And has good figures.

3-5-2013 21-35-18.png

Then I test it for robustness by running an optimization report with a different set of combinations. If almost all combinations have a profit factor above 1 then I can look further to see which combinations give the best result.

I also look at the best combinations and the worse combinations to see similarities. It is possible that the system doesn't show good results at the start. Playing with the timeframe and atr filter can help a lot to optimize the results. The atr filter helps to filter out any noise.

OptimizationReport.gif

The robustness can be tested in many ways. If it works on 5 min bars, it should work on 15 min bars, but you will need more money to trade the system because your stoploss is more far away.

In the chart below I only have changed the period from 5 min to 15 min bars. Everything else equal. Notice that is still profitable (profit factor of 1.43).

TesterGraph.gif

Now I can run optimization for 15 min bars to optimize it further.
TesterGraph15min.gif

The most optimized combination gives the following equity curve with a profit factor of 2.27.
TesterGraph15minOptimized.gif

This is with only an 300 euro account. What if I had a 3000 euro account. Does the system change character? It doesn't matter so much. I now have a profit factor of 2.05 and it still shows the same system characteristics.

TesterGraph15min3000.gif

There are lots of other tests that you can do, but this is the way I screen symbols.

If you have any questions :?: or suggestions :idea:, please let me know.
 
 
AdBlock Detected

We get it, advertisements are annoying!

But it's thanks to our sponsors that access to Trade2Win remains free for all. By viewing our ads you help us pay our bills, so please support the site and disable your AdBlocker.

I've Disabled AdBlock