How to Evaluate Trading System Performance?

[MattMLC]

I went through systematic FX fund disclosure doc. It went live quite recently - in Feb 2013. As on June 2013,

Style: High Frequency Trading Algorithm
Instrument: EUR/USD
Profit Factor: 6.48
Hit (Success) Ratio: 94.34%
Expectation: +13.48%
Target Annual Returns: 18-22%
Max DD: 0.30%
Total Trades Closed: 106
Total Days Completed: 129 (incl. Sat & Sun)
No loosing month or week
Fund Manager has 'skin in the game' i.e. has personal funds invested
Management Fee: 0%
Profit Participation: 50% with high-watermark and claw-back provisions
No min. lock-up periods
No side-pockets
No redemption/subscription loads
No gate provision or withdrawal restriction
Fund Manager is less than 30 years old


Questions to pros who are skilled in performance evaluation:
What other metrics do I need to make a thorough and diligent evaluation?
What would you say in response to this 'live' performance statement?
I don't mind the higher than industry benchmark incentive fee in return for low vol/steady returns.

 

[fatowl]

I went through systematic FX fund disclosure doc. It went live quite recently - in Feb 2013. As on June 2013.

I'm confused. Did you start this fund or are you thinking of investing in it?

Just glancing at the stats you gave throw off red flags. TThere is no indication of whether or not this fund will be long-term profitable. A 94% success ratio is great, but only over about 100 trades and only over the past four months?? This data is not statistically significant.

Since the fund is still very young, you should look at the simulated performance over the past 5-10 years. There are bazillions of metrics to choose from, but you basically want to find out this info:

1. how profitable is it? Could use cumulative annual return, average annual return, or net return
2. how risky is it? Could use maximum drawdown, risk-adjusted return, exposure %, or ulcer index
3. how consistent are the returns? Could use K-factor or a table of monthly returns

 

[NVP]

with the best of respect come back in 5 years dude ..........I need a track record of investing in the system and you as a sustainable business .....not simulation or promises

no offence intended - just good business principles

N

 

[Brumby]

I went through systematic FX fund disclosure doc. It went live quite recently - in Feb 2013. As on June 2013,

Style: High Frequency Trading Algorithm
Instrument: EUR/USD
Profit Factor: 6.48
Hit (Success) Ratio: 94.34%
Expectation: +13.48%
Target Annual Returns: 18-22%
Max DD: 0.30%
Total Trades Closed: 106
Total Days Completed: 129 (incl. Sat & Sun)
No loosing month or week
Fund Manager has 'skin in the game' i.e. has personal funds invested
Management Fee: 0%
Profit Participation: 50% with high-watermark and claw-back provisions
No min. lock-up periods
No side-pockets
No redemption/subscription loads
No gate provision or withdrawal restriction
Fund Manager is less than 30 years old


Questions to pros who are skilled in performance evaluation:
What other metrics do I need to make a thorough and diligent evaluation?
What would you say in response to this 'live' performance statement?
I don't mind the higher than industry benchmark incentive fee in return for low vol/steady returns.

The maths don't add up. A targeted annual return of only 18-22 % with a hit rate of 94 % and profit factor > 6. Something terribly negative is missing like reversion to mean i.e. current statistics are overly positive.

 

[MattMLC]

I'm confused. Did you start this fund or are you thinking of investing in it?

Just glancing at the stats you gave throw off red flags. TThere is no indication of whether or not this fund will be long-term profitable. A 94% success ratio is great, but only over about 100 trades and only over the past four months?? This data is not statistically significant.

Since the fund is still very young, you should look at the simulated performance over the past 5-10 years. There are bazillions of metrics to choose from, but you basically want to find out this info:

1. how profitable is it? Could use cumulative annual return, average annual return, or net return
2. how risky is it? Could use maximum drawdown, risk-adjusted return, exposure %, or ulcer index
3. how consistent are the returns? Could use K-factor or a table of monthly returns

Thanks for your response fatowl!

0. I am actually looking to invest in this fund along with some friends. How many min. trades and min. months/years does it take to be statistically significant? The backtested data (over a period of 2-3 years prior to going live) shows similar results.

1. Would you be able to provide the formulae for each of the profitability, riskiness and consistency metrics that you have mentioned above? It'd be very helpful.

 

[MattMLC]

with the best of respect come back in 5 years dude ..........I need a track record of investing in the system and you as a sustainable business .....not simulation or promises

no offence intended - just good business principles

N

1. What you're saying makes sense. And I'm the investor actually.

2. How would you go about evaluating a new fund. Surely, there are many startup funds like PE, VC and hedge funds that raise capital without a long track record. How can I separate the wheat from the chaff?

 

[MattMLC]

The maths don't add up. A targeted annual return of only 18-22 % with a hit rate of 94 % and profit factor > 6. Something terribly negative is missing like reversion to mean i.e. current statistics are overly positive.

Brumby,

I checked with the FM. The figure (annual return) currently stands at 25.42%. The FM prefers to "understate" annual returns rather than "overstate".

 

[MattMLC]

I'm confused. Did you start this fund or are you thinking of investing in it?

Just glancing at the stats you gave throw off red flags. TThere is no indication of whether or not this fund will be long-term profitable. A 94% success ratio is great, but only over about 100 trades and only over the past four months?? This data is not statistically significant.

Since the fund is still very young, you should look at the simulated performance over the past 5-10 years. There are bazillions of metrics to choose from, but you basically want to find out this info:

1. how profitable is it? Could use cumulative annual return, average annual return, or net return
2. how risky is it? Could use maximum drawdown, risk-adjusted return, exposure %, or ulcer index
3. how consistent are the returns? Could use K-factor or a table of monthly returns

Fatowl,

I asked the FM about the strategy: without revealing specifics, he said he trades like an institutional order-flow stop-hunter.

My concern is this - is it a valid strategy on a long-term basis? How do I classify this (for performance benchmarking against peers using similar strategies)? Does it fall under momentum or mean-reversion?

 

[Shakone]

Brumby,

I checked with the FM. The figure (annual return) currently stands at 25.42%. The FM prefers to "understate" annual returns rather than "overstate".

But he's quite happy with overstating win rate and profit factor? Alarm bells...

 

[Brumby]

Brumby,

I checked with the FM. The figure (annual return) currently stands at 25.42%. The FM prefers to "understate" annual returns rather than "overstate".

Something still doesn't add up and is negatively impacting the performance but is not being disclosed in the fund performance (in my opinion). What we are attempting to do is to look at some of the statistics to establish the quality of the returns.

As a comparison, I have an account with the following statistics which is showing a higher return but poorer statistics in comparison :

Period : 5 months
No of trades : 62
Hit rate : 61 %
Profit factor : 1.75
Drawdown : 13.75 %
Return to-date : 39.9%

This fund which has a hit rate of 94 % and a profit factor of > 6 is only getting 25 % return. Something negative is not being told.

A high hit rate typically suggest problem with how losses are being managed but because the profit factor is high, I am having difficulty reading into the statistics.

 

[MattMLC]

Something still doesn't add up and is negatively impacting the performance but is not being disclosed in the fund performance (in my opinion). What we are attempting to do is to look at some of the statistics to establish the quality of the returns.

As a comparison, I have an account with the following statistics which is showing a higher return but poorer statistics in comparison :

Period : 5 months
No of trades : 62
Hit rate : 61 %
Profit factor : 1.75
Drawdown : 13.75 %
Return to-date : 39.9%

This fund which has a hit rate of 94 % and a profit factor of > 6 is only getting 25 % return. Something negative is not being told.

A high hit rate typically suggest problem with how losses are being managed but because the profit factor is high, I am having difficulty reading into the statistics.

1. This is how the FM calculates:

Profit Factor: Gross Profit/Gross Loss
Hit Rate: Winning Trades/(Winning+Losing Trades)

2. Can the lower returns be attributed to his "Profit Ratio" being < 1?

Profit Ratio = Average Profit per Winning Trade / Average Loss per Losing Trade
Profit Ratio = 0.39 which is < 1

Or in other words, his average loss is 2.6 times his average win. :-0

However, with his higher hit rate, ultra-short term holding period (HFT) and hi-prob setups, he manages to have positive expectation and low DD.

Is there any other forensic tool to dig out skeletons (if any)? :cheesy:

 

[MattMLC]

But he's quite happy with overstating win rate and profit factor? Alarm bells...

Had the figures crunched on excel. No overstatement found.

 

[Brumby]

1. This is how the FM calculates:

Profit Factor: Gross Profit/Gross Loss
Hit Rate: Winning Trades/(Winning+Losing Trades)

2. Can the lower returns be attributed to his "Profit Ratio" being < 1?

Profit Ratio = Average Profit per Winning Trade / Average Loss per Losing Trade
Profit Ratio = 0.39 which is < 1

Or in other words, his average loss is 2.6 times his average win. :-0

However, with his higher hit rate, ultra-short term holding period (HFT) and hi-prob setups, he manages to have positive expectation and low DD.

Is there any other forensic tool to dig out skeletons (if any)? :cheesy:

Typically there is an inverse relationship between profit factor and hit rate but in this case it is the reverse, which is highly unusual. I can only conclude that it is either cooked or we have a trading genius. The statistics on hit rate and profit factor if true would put it at the very top (IMHO) of the trading community.

The only way I can reconcile the low returns to these 2 trade statistics is that only a very small portion of the funds are actually traded and the rest is being parked with low returns. I suggest that you ask what percentage of the funds are actively being traded.

 

[NVP]

1. What you're saying makes sense. And I'm the investor actually.

2. How would you go about evaluating a new fund. Surely, there are many startup funds like PE, VC and hedge funds that raise capital without a long track record. How can I separate the wheat from the chaff?

I wouldnt ............its that simple ......I would take a look at the traders involved mainly and see whos running the ship....

If I put my money with people its those I trust to do what I cant sometimes

why not invest in yourself ?.........you will never regret it

N

 

[NVP]

without spinning this off into another thread I am surprised why people continue to use fairly standard principles to still value performance in such isolation

1) work out the returns you want and the mix of risks in your portfolio
2) then go back and get more realistic ..takes about 4 times usually
3) then seek strategies that provide these returns BASED ON THE MARKET CONDITIONS

the market conditions are the key ..........if you get your trading to a decent enough level you will then transcend the problems associated with system failure and begin to appreciate what the market is doing and where it is delivering opportunities and also when it is not..........you can then either stay out or Jump in when it moves accordingly ..........

the Holy grail is having a mix of systems that can produce decent consistent returns regardless of the market conditions .....damn tough if the markets are choppy

so if you are reviewing systems ....take a look at the performance in choppy markets ...........thats the acid test ..........

N