Calling all System Developers...

[Chorlton]

Hello All,

For all those who have developed & traded their own systems, I would welcome your thoughts & comments on the following system which I am currently developing.

It is a Weekly LONG only system, trading stocks on the FTSE250.

The profit results are not that great compared to others I have looked at but given that the system is easily to manage, I would be happy with the return, especially as it can be traded with minimum time & effort.

The system has initially been backtested using price data from 1998 to 2004 (inclusive). This is my "in-sample" data. Following any feedback on this thread, I will look to test the system on "out-sample" data from 2004 to present.

The results are as follows:


Monte Carlo Report

Trade Database Filename: v3,81 FTSE250.csv

Simulation Summary
Simulation Date:
28/01/2008

Simulation Time:
3:48:01 PM

Simulation Duration:
85.89 seconds


Trade Parameters
Initial Capital:
$50,000.00

Portfolio Limit:
100.00%

Maximum number of open positions:
100

Position Size Model:
Fixed Percent Risk

Percentage of capital risked per trade:
1.00%

Position size limit:
10.00%

Portfolio Heat:
100.00%

Pyramid profits:
Yes

Transaction cost (Trade Entry):
$15.00

Transaction cost (Trade Exit):
$15.00

Margin Requirement:
100.00%

Magnify Position Size(& Risk) according to Margin Req:
No

Margin Requirement Daily Interest Rate (Long Trades):
0.0000%

Margin Requirement Yearly Interest Rate (Long Trades):
0.0000%

Margin Requirement Daily Interest Rate (Short Trades):
0.0000%

Margin Requirement Yearly Interest Rate (Short Trades):
0.0000%


Trade Preferences
Trading Instrument:
Stocks

Break Even Trades:
Process separately

Trade Position Type:
Process all trades

Entry Order Type:
Default Order

Exit Order Type:
Default Order

Minimum Trade Size:
$0.00

Accept Partial Trades:
No

Volume Filter:
Ignore Volume Information

Pyramid Trades:
No

Use Level Zero trades only:
Yes


Simulation Stats
Number of trade simulations:
20000

Trades processed per simulation:
189

Maximum Number of Trades Executed:
148

Average Number of Trades Executed:
146

Minimum Number of Trades Executed:
143

Standard Deviation:
0.98

Profit Stats

Maximum Profit:
$58,075.74 (116.15%)

Average Profit:
$54,055.41 (108.11%)

Minimum Profit:
$51,240.66 (102.48%)

Standard Deviation:
$1,296.74 (2.59%)

Probability of Profit:
100.00%

Probability of Loss:
0.00%


Percent Winning Trade Stats
Maximum percentage of winning trades:
53.10%

Average percentage of winning trades:
51.57%

Minimum percentage of winning trades:
50.00%

Standard Deviation:
0.67%

Percent Losing Trade Stats
Maximum percentage of losing trades:
50.00%

Average percentage of losing Trades:
48.43%

Minimum percentage of losing trades:
46.90%

Standard Deviation:
0.67%


Average Relative Dollar Drawdown Stats
Maximum of the Average Relative Dollar Drawdown:
$310.96

Average of the Average Relative Dollar Drawdown:
$289.82

Minimum of the Average Relative Dollar Drawdown:
$270.55

Standard Deviation:
$7.11


Average Relative Percent Drawdown Stats
Maximum of the Average Relative Percent Drawdown:
0.3875%

Average of the Average Relative Percent Drawdown:
0.3603%

Minimum of the Average Relative Percent Drawdown:
0.3315%

Standard Deviation:
0.0102%


Maximum Peak-to-Valley Dollar Drawdown Stats
Maximum Absolute Dollar Drawdown:
$2,544.99

Average Absolute Dollar Drawdown:
$2,457.06

Minimum Absolute Dollar Drawdown:
$2,402.11

Standard Deviation:
$26.10




Maximum Peak-to-Valley Percent Drawdown Stats
Maximum Absolute Percent Drawdown:
2.7241%

Average Absolute Percent Drawdown:
2.7162%

Minimum Absolute Percent Drawdown:
2.6984%

Standard Deviation:
0.0064%


All comments welcome.

Thanks in advance,

Chorlton

 

[Tadragh1]

I'll try to share a few thoughts, first let me recap the info most relevant for me:

The system has initially been backtested using price data from 1998 to 2004 (inclusive).
The results are as follows:

Initial Capital:
$50,000.00
Number of trade simulations:
20000
Trades processed per simulation:
189
Question - what is the difference between trades processed vs. executed?

Maximum Number of Trades Executed:
148
Average Number of Trades Executed:
146
Minimum Number of Trades Executed:
143

Maximum Profit:
$58,075.74 (116.15%)
Average Profit:
$54,055.41 (108.11%)
Minimum Profit:
$51,240.66 (102.48%)

Maximum percentage of winning trades:
53.10%
Average percentage of winning trades:
51.57%
Minimum percentage of winning trades:
50.00%

Maximum Absolute Percent Drawdown:
2.7241%
Average Absolute Percent Drawdown:
2.7162%
Minimum Absolute Percent Drawdown:
2.6984%

For me this system looks very well - the profit is not killing, but is definitely adequate and the system seems very stable. Drawdown is very low, worst Win/Loss ratio 1:1. You might want to include average winning amount and average losing amount for completeness. Also worst single loss and best single win would be nice to know. Relatively low number of trades might be a concern for testing purposes (20 per year). What is your average holding period for a position?

I like it, really. I'd put my money in it if more research would not reveal any flaws.

 

[Chorlton]

I'll try to share a few thoughts, first let me recap the info most relevant for me:



For me this system looks very well - the profit is not killing, but is definitely adequate and the system seems very stable. Drawdown is very low, worst Win/Loss ratio 1:1. You might want to include average winning amount and average losing amount for completeness. Also worst single loss and best single win would be nice to know. Relatively low number of trades might be a concern for testing purposes (20 per year). What is your average holding period for a position?

I like it, really. I'd put my money in it if more research would not reveal any flaws.

Hi Tadragh,

Thanks for the response. I wasn't sure if anyone was going to reply so its much appreciated.

I'm currently at work but when I get back home, I shall post the other trade result stats. I agree with you that the overall profit could be better but as you say the MaxDD is very low IMO. Most long-term systems seem to suggest a MaxDD of around 10% and as minimising risk is paramount to me, keeping this figure below this value is important even if overall Profit is affected.

Given that its a weekly system, the frequency of trades taken is around 2 per month. Although, this may be low, I'm happy with this as it is a lazy system.

Out of interest, are you involved in system development and if so have you ever designed something to trade the UK markets? If so, I'd be interested in the kind of number stats that can be achieved in this current climate.

Regards,

Chorlton

 

[Chorlton]

Hello Tadragh1,

Here are the remaining stats:


Trade Parameters and Preferences

Trade Parameters
Initial Capital: $50,000.00
Portfolio Limit: 100.00%
Maximum number of open positions: 100
Position Size Model: Fixed Percent Risk
Percentage of capital risked per trade: 1.00%
Position size limit: 10.00%
Portfolio Heat: 100.00%
Pyramid profits: Yes
Transaction cost (Trade Entry): $15.00
Transaction cost (Trade Exit): $15.00
Margin Requirement: 100.00%
Magnify Position Size(& Risk) according to Margin Req: No
Margin Requirement Daily Interest Rate (Long Trades): 0.00%
Margin Requirement Yearly Interest Rate (Long Trades): 0.00%
Margin Requirement Daily Interest Rate (Short Trades): 0.00%
Margin Requirement Yearly Interest Rate (Short Trades): 0.00%

Trade Preferences
Trading Instrument: Stocks
Break Even Trades: Process separately
Trade Position Type: Process all trades
Entry Order Type: Default Order
Exit Order Type: Default Order
Minimum Trade Size: $0.00
Accept Partial Trades: No
Volume Filter: Ignore Volume Information
Pyramid Trades: No
Use Level Zero trades only: Yes


Detailed Report

Simulation Summary
Simulation Date: 31/01/2008
Simulation Time: 6:12:36 PM
Simulation Duration: 0.75 seconds

Trade Summary
Earliest Entry Date in the Trade Database: 24/12/1997
Latest Entry Date in the Trade Database: 12/11/2004
Earliest Exit Date in the Trade Database: 6/02/1998
Latest Exit Date in the Trade Database: 24/12/2004

Start Trade Entry Date: 24/12/1997
Stop Trade Entry Date: 12/11/2004
First Entry Date: 24/12/1997
Last Entry Date: 12/11/2004
First Exit Date: 6/02/1998
Last Exit Date: 24/12/2004

Total Trading duration: 2557 days

Profit Summary
Profit Status: PROFITABLE
Starting Capital: $50,000.00
Finishing Capital: $102,314.89
Maximum Equity/(Date): $52,314.89 (24/12/2004)
Minimum Equity/(Date): $308.35 (6/02/1998)
Gross Trade Profit: $75,897.91 (151.80%)
Gross Trade Loss: -$23,583.02 (-47.17%)
Total Net Profit: $52,314.89 (104.63%)
Average Profit per Trade: $358.32
Profit Factor: 3.2183
Profit Index: 68.93%
Total Transaction Cost: $4,380.00
Total Slippage: $0.00
Total Trade Interest: $0.00
Daily Compound Interest Rate: 0.03%
Annualized Compound Interest Rate: 10.76%

Trade Statistics
Trades Processed: 189
Trades Taken: 146
Partial Trades Taken: 0
Trades Rejected: 43
Winning Trades: 75 (51.37%)
Losing Trades: 71 (48.63%)
Breakeven Trades: 0 (0.00%)

Largest Winning Trade/(Date): $7,264.86 (28/05/2004)
Largest Losing Trade/(Date): -$1,357.72 (10/12/2004)
Average Winning Trade: $1,011.97
Average Losing Trade: -$332.16
Average Win/Average Loss: 3.0467

Trade Breakdown Long and Short Trades Long Trades Short Trades
Normal Exit: 146 (100.00%) 146 (100.00%) 0 (0.00%)

Total Trades: 146 (100.00%) 146 (100.00%) 0 (0.00%)

Trade Duration Statistics Winning and Losing Trades Winning Trades Losing Trades
Maximum Trade Duration: 357 (days) 357 (days) 140 (days)
Minimum Trade Duration: 7 (days) 35 (days) 7 (days)
Average Trade Duration: 63.57 (days) 88.56 (days) 37.17 (days)

Consecutive Trade Statistics
Maximum consecutive winning trades: 8
Maximum consecutive losing trades: 5
Average consecutive winning trades: 2.14
Average consecutive losing trades: 2.09

Trade Expectation Statistics
Normalized Expectation per dollar risked: $0.93
Maximum Reward/Risk ratio: 15.09
Minimum Reward/Risk ratio: -4.38
Average Positive Reward/Risk ratio: $2.43
Average Negative Reward/Risk ratio: -$0.65

Relative Drawdown
Maximum Dollar Drawdown/(Date): $2,335.00 (10/12/2004)
Maximum Percentage Drawdown/(Date): 2.3080% (10/12/2004)

Absolute (Peak-to-Valley) Dollar Drawdown
Maximum Dollar Drawdown: $2,423.71 (2.3950%)
Capital Peak/(Date): $101,181.70 (28/05/2004)
Capital Valley/(Date): $98,757.99 (24/12/2004)

Absolute (Peak-to-Valley) Percent Drawdown
Maximum Percentage Drawdown: 2.7180% ($2,163.32)
Capital Peak/(Date): $79,587.31 (31/03/2000)
Capital Valley/(Date): $77,423.99 (26/04/2002)



Trade Statistics Long and Short Trades Long Trades Short Trades
Trades Taken: 146 146 0
Total Net Profit: $52,314.89 $52,314.89 N/A
Average Trade Profit: $358.32 $358.32 N/A
Maximum Trade Profit: $7,264.86 $7,264.86 N/A
Minimum Trade Profit: -$1,357.72 -$1,357.72 N/A
Break Even Trades: 0 0 0
Winning Trades: 75 75 0
Losing Trades: 71 71 0
Profitable Trades: 51.37% 51.37% N/A
Losing Trades: 48.63% 48.63% N/A
Average Winning Trade Profit: $1,011.97 $1,011.97 N/A
Average Losing Trade Profit: -$332.16 -$332.16 N/A


Looking through them, one thing which stands out is that the largest winning trade accounts for over 10% of the overall profit, which is not acceptable IMO.

I think I shall run the stats again removing the top 3 trades and see what kind of results are achieved.

More worrying, the Annualized Compound Interest Rate is a very disappointing at 10.76% which in this climate is definately not acceptable.

However, I do think the system has some merits so will try to build on these and will post further results in due course.....


As usual I'd welcome any other views / comments......

Regards,

Chorlton

 

[intradaybill]

Hello Tadragh1,

Here are the remaining stats:

Chorlton

Long only FTSE bull market monte carlo fitted system go bust in bear market yo.

Bill

 

[Chorlton]

Long only FTSE bull market monte carlo fitted system go bust in bear market yo.

Bill

The purpose of this thread was to try to start some "constructive" discussion about the steps involved in System Development and I thought by posting a simple system might be a good way to achieve this.

In answer to your very "constructive" comment thou:

Long Only: I've focused on this as a starting point only. Intention will be to look at Short trades as well given the current climate conditions.

FTSE Bull Market: I would argue that the time frame chosen for backtesting encompasses periods of Bull / Bear & Sideways action

monte carlo fitted system: No "parameter" optimisation has yet been done. Looking at various Position Size models has been explored though. I have also made it clear in my post that the only data backtested to date was "in-sample"

 

[intradaybill]

Long Only: I've focused on this as a starting point only. Intention will be to look at Short trades as well given the current climate conditions.

FTSE Bull Market: I would argue that the time frame chosen for backtesting encompasses periods of Bull / Bear & Sideways action

monte carlo fitted system: No "parameter" optimisation has yet been done. Looking at various Position Size models has been explored though. I have also made it clear in my post that the only data backtested to date was "in-sample"

How many winning trades you got yo in bear market 2000 to 2003? Wha tha distributio in time?

I betsya all your losing trades are between 2000 and 2003.

Your position size is fitting for best trades to make more money. A penny I would not put in a long only system when bear market is ahead. Just friendly advice.

Bill

 

[Chorlton]

How many winning trades you got yo in bear market 2000 to 2003? Wha tha distributio in time?

I betsya all your losing trades are between 2000 and 2003.

Your position size is fitting for best trades to make more money. A penny I would not put in a long only system when bear market is ahead. Just friendly advice.

Bill

Hi Bill,

From memory the only losing year was 2002 and this was a very small amount but I will check the results again to confirm this.

I have a filter in place that stops the system taking new trades when the market experiences a decline. It could be argued that this would result in the results following the index (to some extent) but I feel it offers a good level of protection.

I would agree that a long-only system is not the best solution given the current & potential market conditions ahead. However, I do intend to develop the short side of the system but just wanted to keep it simple to begin with.

Out of interest, can I ask how you trade? Is it in a discretionary or Mechanical manner? If its the latter, have you developed anything for the UK markets, and if so, can I ask what were the average stats?

Just interested....

Kind Regards,

 

[Chorlton]

How many winning trades you got yo in bear market 2000 to 2003? Wha tha distributio in time?

I betsya all your losing trades are between 2000 and 2003.

Your position size is fitting for best trades to make more money. A penny I would not put in a long only system when bear market is ahead. Just friendly advice.

Bill

Hi Bill,

From memory the only losing year was 2002 and this was a very small amount but I will check the results again to confirm this.

I have a filter in place that stops the system taking new trades when the market experiences a decline. It could be argued that this would result in the results following the index (to some extent) but I feel it offers a good level of protection.

I would agree that a long-only system is not the best solution given the current & potential market conditions ahead. However, I do intend to develop the short side of the system but just wanted to keep it simple to begin with.

Out of interest, can I ask how you trade? Is it in a discretionary or Mechanical manner? If its the latter, have you developed anything for the UK markets, and if so, can I ask what were the average stats?

Just interested....

Kind Regards,

 

[Tadragh1]

Chorlton, thanks for the additional info. No, I don't have anything for the UK market.
For me, the next phase after the statistics looking good would be what intradaybill suggested in a way - look at the parameters of the system in each year and see if they are significantly different. Then measure the probability of getting the results in a specific year by using the overall statistics you have. Actually, with 140 trades you can look at each trade and see what it looks like.
About the methodology - I assume you back tested the system once and then you ran a Monte Carlo simulation basing on the parameters from the backtest, right?
I guess if the system is easy to "invert" to get short signals in the same manner, it would be good to have these for completeness.

Have fun!

 

[Chorlton]

look at the parameters of the system in each year and see if they are significantly different. Then measure the probability of getting the results in a specific year by using the overall statistics you have.

Hello Tadragh1,

Can you expand on this as I'm not exactly sure what I would need to do. Thanks....

the methodology - I assume you back tested the system once and then you ran a Monte Carlo simulation basing on the parameters from the backtest, right?

I ran the backtest and looked at the Equity Curve which looked relatively smooth. Then I ran the Monte Carlo sim to see what the stats were. The system's entry conditions are based on P&V as opposed to indicators, which IMO can be more easily curve-fitted by playing around with their individual parameters

I guess if the system is easy to "invert" to get short signals in the same manner, it would be good to have these for completeness.

Their is a possibility to create "short" signals but unfortunately I don't believe that it would simply be a case of "inverting" the buy conditions.

Thanks for your feedback....


Chorlton

 

[Tadragh1]

Chorlton, I'll get back to you to explain what I meant there. Just need some time to get you a clear explanation.
Actually, maybe it would be good to make a T2W "What to look for in backtesting results" FAQ. I am sure that people around here have many good ideas. :idea:

 

[Chorlton]

Chorlton, I'll get back to you to explain what I meant there. Just need some time to get you a clear explanation.
Actually, maybe it would be good to make a T2W "What to look for in backtesting results" FAQ. I am sure that people around here have many good ideas. :idea:

That would be a great idea IMO.... Just post the link to that thread here so I can find it.....

Best Regards....

 

[mshepstone]

Chorlton,

Which Market data are you using to run your test results?

 

[Chorlton]

Chorlton,

Which Market data are you using to run your test results?

I'm testing the system on the FTSE250. Is this what you're asking?

 

[Chorlton]

Chorlton, I'll get back to you to explain what I meant there. Just need some time to get you a clear explanation.
Actually, maybe it would be good to make a T2W "What to look for in backtesting results" FAQ. I am sure that people around here have many good ideas. :idea:

Hi Tagragh1,

I'm currently looking at the links you kindly posted on the new thread but I wonder whether you could expand on your previous comments made on here. see below:

Thanks in advance.....

look at the parameters of the system in each year and see if they are significantly different. Then measure the probability of getting the results in a specific year by using the overall statistics you have.

 

[Tadragh1]

Sure Chorlton, here goes.
The purpose of this stability testing is to see, if the system works as well in the end of the testing period as in the beginning, and if it works in different market conditions (which is rather obvious). You could check it visually comparing the number of losing and winning trades in each year or for example the development of the mean result throughout time.
A good statistical way to do it would be the Kolmogorov-Smirnov Test You could divide the results of your system into two subsamples and see, what is the likelihood, that they come from the same distribution. As you do not have very many trades, I guess just cutting your results sample in two would be the way to go.
There are a few more ideas in the MQL4 webpages, which I posted in the FAQ thread.
Of course, if the system seems unstable over time, it does not necessarily mean it is useless. It is just another way to pinpoint its strength and weaknesses.

 

[Chorlton]

Sure Chorlton, here goes.
The purpose of this stability testing is to see, if the system works as well in the end of the testing period as in the beginning, and if it works in different market conditions (which is rather obvious). You could check it visually comparing the number of losing and winning trades in each year or for example the development of the mean result throughout time.
A good statistical way to do it would be the Kolmogorov-Smirnov Test You could divide the results of your system into two subsamples and see, what is the likelihood, that they come from the same distribution. As you do not have very many trades, I guess just cutting your results sample in two would be the way to go.
There are a few more ideas in the MQL4 webpages, which I posted in the FAQ thread.
Of course, if the system seems unstable over time, it does not necessarily mean it is useless. It is just another way to pinpoint its strength and weaknesses.

Thanks for the reply.

You definately seem to be knowledgable & helpful on this topic, so your advice is greatfully received.

Out of interest, my system so far has only been tested using my "In-Sample" data and has yet to be exposed to the "out-sample" which is the price data from start 2005 to Present Day.

Therefore, at which point should I test my system on this range of price data? My understanding is that I would be looking at refining the system using the current data and then just before "going live" with it, exposing it to the "out-sample" to ensure the system is robust.

Any advice welcome,

Thanks,
Chorlton

 

[Tadragh1]

Hi Chorlton!

The usual disclaimer - I am just a guy out of nowhere with quite a lot of academic knowledge and very sceptical about his own systems But I hope you find something useful in my writing.
About the procedure of testing - I would do the same as you, tweak the system in-sample, then test it out-of-sample. It is a good idea to carefully compare in-sample and out-of-sample results, as they would give you idea about curve-fitting in your system, and also some additional stuff on when it works and when it does not.

 

[Chorlton]

Hi Chorlton!

The usual disclaimer - I am just a guy out of nowhere with quite a lot of academic knowledge and very sceptical about his own systems But I hope you find something useful in my writing.
About the procedure of testing - I would do the same as you, tweak the system in-sample, then test it out-of-sample. It is a good idea to carefully compare in-sample and out-of-sample results, as they would give you idea about curve-fitting in your system, and also some additional stuff on when it works and when it does not.

Agreed!!!

I have kept the last 3 years of data aside which I will use as my out-sample. So as not to expose the system to all 3 years in one go, one idea is to expose the system to one year at a time. Then if I have to make further changes, I will still have the remaining 2 years to backtest on.....