How to Evaluate Trading System Performance?

[MattMLC]

I went through systematic FX fund disclosure doc. It went live quite recently - in Feb 2013. As on June 2013,

Style: High Frequency Trading Algorithm
Instrument: EUR/USD
Profit Factor: 6.48
Hit (Success) Ratio: 94.34%
Expectation: +13.48%
Target Annual Returns: 18-22%
Max DD: 0.30%
Total Trades Closed: 106
Total Days Completed: 129 (incl. Sat & Sun)
No loosing month or week
Fund Manager has 'skin in the game' i.e. has personal funds invested
Management Fee: 0%
Profit Participation: 50% with high-watermark and claw-back provisions
No min. lock-up periods
No side-pockets
No redemption/subscription loads
No gate provision or withdrawal restriction
Fund Manager is less than 30 years old


Questions to pros who are skilled in performance evaluation:
What other metrics do I need to make a thorough and diligent evaluation?
What would you say in response to this 'live' performance statement?
I don't mind the higher than industry benchmark incentive fee in return for low vol/steady returns.

 

[fatowl]

I went through systematic FX fund disclosure doc. It went live quite recently - in Feb 2013. As on June 2013.

I'm confused. Did you start this fund or are you thinking of investing in it?

Just glancing at the stats you gave throw off red flags. TThere is no indication of whether or not this fund will be long-term profitable. A 94% success ratio is great, but only over about 100 trades and only over the past four months?? This data is not statistically significant.

Since the fund is still very young, you should look at the simulated performance over the past 5-10 years. There are bazillions of metrics to choose from, but you basically want to find out this info:

1. how profitable is it? Could use cumulative annual return, average annual return, or net return
2. how risky is it? Could use maximum drawdown, risk-adjusted return, exposure %, or ulcer index
3. how consistent are the returns? Could use K-factor or a table of monthly returns

 

[NVP]

with the best of respect come back in 5 years dude ..........I need a track record of investing in the system and you as a sustainable business .....not simulation or promises

no offence intended - just good business principles

N

 

[Brumby]

I went through systematic FX fund disclosure doc. It went live quite recently - in Feb 2013. As on June 2013,

Style: High Frequency Trading Algorithm
Instrument: EUR/USD
Profit Factor: 6.48
Hit (Success) Ratio: 94.34%
Expectation: +13.48%
Target Annual Returns: 18-22%
Max DD: 0.30%
Total Trades Closed: 106
Total Days Completed: 129 (incl. Sat & Sun)
No loosing month or week
Fund Manager has 'skin in the game' i.e. has personal funds invested
Management Fee: 0%
Profit Participation: 50% with high-watermark and claw-back provisions
No min. lock-up periods
No side-pockets
No redemption/subscription loads
No gate provision or withdrawal restriction
Fund Manager is less than 30 years old


Questions to pros who are skilled in performance evaluation:
What other metrics do I need to make a thorough and diligent evaluation?
What would you say in response to this 'live' performance statement?
I don't mind the higher than industry benchmark incentive fee in return for low vol/steady returns.

The maths don't add up. A targeted annual return of only 18-22 % with a hit rate of 94 % and profit factor > 6. Something terribly negative is missing like reversion to mean i.e. current statistics are overly positive.

 

[MattMLC]

I'm confused. Did you start this fund or are you thinking of investing in it?

Just glancing at the stats you gave throw off red flags. TThere is no indication of whether or not this fund will be long-term profitable. A 94% success ratio is great, but only over about 100 trades and only over the past four months?? This data is not statistically significant.

Since the fund is still very young, you should look at the simulated performance over the past 5-10 years. There are bazillions of metrics to choose from, but you basically want to find out this info:

1. how profitable is it? Could use cumulative annual return, average annual return, or net return
2. how risky is it? Could use maximum drawdown, risk-adjusted return, exposure %, or ulcer index
3. how consistent are the returns? Could use K-factor or a table of monthly returns

Thanks for your response fatowl!

0. I am actually looking to invest in this fund along with some friends. How many min. trades and min. months/years does it take to be statistically significant? The backtested data (over a period of 2-3 years prior to going live) shows similar results.

1. Would you be able to provide the formulae for each of the profitability, riskiness and consistency metrics that you have mentioned above? It'd be very helpful.

 

[MattMLC]

with the best of respect come back in 5 years dude ..........I need a track record of investing in the system and you as a sustainable business .....not simulation or promises

no offence intended - just good business principles

N

1. What you're saying makes sense. And I'm the investor actually.

2. How would you go about evaluating a new fund. Surely, there are many startup funds like PE, VC and hedge funds that raise capital without a long track record. How can I separate the wheat from the chaff?

 

[MattMLC]

The maths don't add up. A targeted annual return of only 18-22 % with a hit rate of 94 % and profit factor > 6. Something terribly negative is missing like reversion to mean i.e. current statistics are overly positive.

Brumby,

I checked with the FM. The figure (annual return) currently stands at 25.42%. The FM prefers to "understate" annual returns rather than "overstate".

 

[MattMLC]

I'm confused. Did you start this fund or are you thinking of investing in it?

Just glancing at the stats you gave throw off red flags. TThere is no indication of whether or not this fund will be long-term profitable. A 94% success ratio is great, but only over about 100 trades and only over the past four months?? This data is not statistically significant.

Since the fund is still very young, you should look at the simulated performance over the past 5-10 years. There are bazillions of metrics to choose from, but you basically want to find out this info:

1. how profitable is it? Could use cumulative annual return, average annual return, or net return
2. how risky is it? Could use maximum drawdown, risk-adjusted return, exposure %, or ulcer index
3. how consistent are the returns? Could use K-factor or a table of monthly returns

Fatowl,

I asked the FM about the strategy: without revealing specifics, he said he trades like an institutional order-flow stop-hunter.

My concern is this - is it a valid strategy on a long-term basis? How do I classify this (for performance benchmarking against peers using similar strategies)? Does it fall under momentum or mean-reversion?

 

[Shakone]

Brumby,

I checked with the FM. The figure (annual return) currently stands at 25.42%. The FM prefers to "understate" annual returns rather than "overstate".

But he's quite happy with overstating win rate and profit factor? Alarm bells...

 

[Brumby]

Brumby,

I checked with the FM. The figure (annual return) currently stands at 25.42%. The FM prefers to "understate" annual returns rather than "overstate".

Something still doesn't add up and is negatively impacting the performance but is not being disclosed in the fund performance (in my opinion). What we are attempting to do is to look at some of the statistics to establish the quality of the returns.

As a comparison, I have an account with the following statistics which is showing a higher return but poorer statistics in comparison :

Period : 5 months
No of trades : 62
Hit rate : 61 %
Profit factor : 1.75
Drawdown : 13.75 %
Return to-date : 39.9%

This fund which has a hit rate of 94 % and a profit factor of > 6 is only getting 25 % return. Something negative is not being told.

A high hit rate typically suggest problem with how losses are being managed but because the profit factor is high, I am having difficulty reading into the statistics.

 

[MattMLC]

Something still doesn't add up and is negatively impacting the performance but is not being disclosed in the fund performance (in my opinion). What we are attempting to do is to look at some of the statistics to establish the quality of the returns.

As a comparison, I have an account with the following statistics which is showing a higher return but poorer statistics in comparison :

Period : 5 months
No of trades : 62
Hit rate : 61 %
Profit factor : 1.75
Drawdown : 13.75 %
Return to-date : 39.9%

This fund which has a hit rate of 94 % and a profit factor of > 6 is only getting 25 % return. Something negative is not being told.

A high hit rate typically suggest problem with how losses are being managed but because the profit factor is high, I am having difficulty reading into the statistics.

1. This is how the FM calculates:

Profit Factor: Gross Profit/Gross Loss
Hit Rate: Winning Trades/(Winning+Losing Trades)

2. Can the lower returns be attributed to his "Profit Ratio" being < 1?

Profit Ratio = Average Profit per Winning Trade / Average Loss per Losing Trade
Profit Ratio = 0.39 which is < 1

Or in other words, his average loss is 2.6 times his average win. :-0

However, with his higher hit rate, ultra-short term holding period (HFT) and hi-prob setups, he manages to have positive expectation and low DD.

Is there any other forensic tool to dig out skeletons (if any)? :cheesy:

 

[MattMLC]

But he's quite happy with overstating win rate and profit factor? Alarm bells...

Had the figures crunched on excel. No overstatement found.

 

[Brumby]

1. This is how the FM calculates:

Profit Factor: Gross Profit/Gross Loss
Hit Rate: Winning Trades/(Winning+Losing Trades)

2. Can the lower returns be attributed to his "Profit Ratio" being < 1?

Profit Ratio = Average Profit per Winning Trade / Average Loss per Losing Trade
Profit Ratio = 0.39 which is < 1

Or in other words, his average loss is 2.6 times his average win. :-0

However, with his higher hit rate, ultra-short term holding period (HFT) and hi-prob setups, he manages to have positive expectation and low DD.

Is there any other forensic tool to dig out skeletons (if any)? :cheesy:

Typically there is an inverse relationship between profit factor and hit rate but in this case it is the reverse, which is highly unusual. I can only conclude that it is either cooked or we have a trading genius. The statistics on hit rate and profit factor if true would put it at the very top (IMHO) of the trading community.

The only way I can reconcile the low returns to these 2 trade statistics is that only a very small portion of the funds are actually traded and the rest is being parked with low returns. I suggest that you ask what percentage of the funds are actively being traded.

 

[NVP]

1. What you're saying makes sense. And I'm the investor actually.

2. How would you go about evaluating a new fund. Surely, there are many startup funds like PE, VC and hedge funds that raise capital without a long track record. How can I separate the wheat from the chaff?

I wouldnt ............its that simple ......I would take a look at the traders involved mainly and see whos running the ship....

If I put my money with people its those I trust to do what I cant sometimes

why not invest in yourself ?.........you will never regret it

N

 

[NVP]

without spinning this off into another thread I am surprised why people continue to use fairly standard principles to still value performance in such isolation

1) work out the returns you want and the mix of risks in your portfolio
2) then go back and get more realistic ..takes about 4 times usually
3) then seek strategies that provide these returns BASED ON THE MARKET CONDITIONS

the market conditions are the key ..........if you get your trading to a decent enough level you will then transcend the problems associated with system failure and begin to appreciate what the market is doing and where it is delivering opportunities and also when it is not..........you can then either stay out or Jump in when it moves accordingly ..........

the Holy grail is having a mix of systems that can produce decent consistent returns regardless of the market conditions .....damn tough if the markets are choppy

so if you are reviewing systems ....take a look at the performance in choppy markets ...........thats the acid test ..........

N

 

[MattMLC]

Typically there is an inverse relationship between profit factor and hit rate but in this case it is the reverse, which is highly unusual. I can only conclude that it is either cooked or we have a trading genius. The statistics on hit rate and profit factor if true would put it at the very top (IMHO) of the trading community.

The only way I can reconcile the low returns to these 2 trade statistics is that only a very small portion of the funds are actually traded and the rest is being parked with low returns. I suggest that you ask what percentage of the funds are actively being traded.

1. I checked the trade by trade statement and cross-checked with the metrics. I don't think it is cooked. Plus I re-calculated the metrics by hand and by calculator. I too was expecting an inverse relationship.

2. How do you calculate exposure % for a fund that uses leverage? FM risks about 0.30% of the NAV on every trade? His max dd is 0.28%. Is that the exposure % or is it something else?

 

[Brumby]

2. How do you calculate exposure % for a fund that uses leverage? FM risks about 0.30% of the NAV on every trade? His max dd is 0.28%. Is that the exposure % or is it something else?

You can't determine % of funds deployed for trading. It is a policy decision of the fund which only they can disclose.

A position size risk of 0.3 % of NAV doesn't necessarily tell the whole story because they might place an arbitrary cap on number of maximum positions open at any one time - so in effect doesn't fully deploy all their funds for trading.

It is common that funds only set aside certain % to trade because they have to cater for redemption and limit risk in the event of unexpected events outside of their trade model.

 

[MattMLC]

without spinning this off into another thread I am surprised why people continue to use fairly standard principles to still value performance in such isolation

1) work out the returns you want and the mix of risks in your portfolio
2) then go back and get more realistic ..takes about 4 times usually
3) then seek strategies that provide these returns BASED ON THE MARKET CONDITIONS

the market conditions are the key ..........if you get your trading to a decent enough level you will then transcend the problems associated with system failure and begin to appreciate what the market is doing and where it is delivering opportunities and also when it is not..........you can then either stay out or Jump in when it moves accordingly ..........

the Holy grail is having a mix of systems that can produce decent consistent returns regardless of the market conditions .....damn tough if the markets are choppy

so if you are reviewing systems ....take a look at the performance in choppy markets ...........thats the acid test ..........

N

Agree that it is best to have a "system basket" which will smooth out returns of individual components. e.g. intermediate term trend-following coupled with short-term trend-fading.

In choppy markets, the price is usually whipsawing between extreme overbought/oversold levels. Systems based on Support & Resistance / Supply & Demand will do well whereas trendfollowing systems will be bleeding red.

I am also a trader. However like the A/B Split test for direct marketing/online marketing, this FM does way better than I can, so I'd rather allocate the capital to him rather than myself until and unless I can better his metrics net of fees on a risk adjusted basis.

Think of it like this: Say, I am a systematic trader and I do not know how to code. I come to understand that to fully realise the potential of mechanical systems and algo trading, I need to learn to code and learn the tech side of trading. After countless hours of poring over reference manuals, tutorials and classes, I manage to come up half-decent code. The learning curve is steep and my output might not be efficient. Rather, if I outsource the coding to a bunch of highly competent hacks, I can get better output more efficiently. Since I understand a thing or two about coding, I don't outsource 'blindly'.

I have noticed this thing in an algo trading firm. There is a portfolio manager, a quant modeler/researcher (statistician) and a programmer. Th PM decides the asset allocation, strategy allocation, etc. From time to time, he bounces some trade ideas or strategy to the quant and asks the quant to analyse it statistically incl. backtesting, forward testing, optimisation. Once the quant has something that has potential, he ask the coder to program the strategy. Once the coding and testing is done, PM on the advice of quant and coder will take the call on whether to run the algorithm live.

The coder is an expert programmer and he understands different trading strategies. He need not be a top trader. But he needs to understand markets, instruments and trading strategies. The quant is a highly qualified mathematician. He'd also have a decent coding knowledge which will help him communicate effectively with the coder. The PM is an excellent trader who has kept abreast of tech trends like algo trading and HFT and knows a bit about quant analysis and a bit of coding. He is not an expert quant nor an expert coder. But his knowledge helps him communicate his ideas effectively with the coder as well as the quant.

NVP, if your live results over a period of time betters this guy's, on the basis of the different performance metrics, I'd rather allocate the funds to you rather than this guy. That is the most efficient use of my capital, time and resources. That is the very heart of capitalism!

 

[MattMLC]

You can't determine % of funds deployed for trading. It is a policy decision of the fund which only they can disclose.

A position size risk of 0.3 % of NAV doesn't necessarily tell the whole story because they might place an arbitrary cap on number of maximum positions open at any one time - so in effect doesn't fully deploy all their funds for trading.

It is common that funds only set aside certain % to trade because they have to cater for redemption and limit risk in the event of unexpected events outside of their trade model.

This guy doesn't hold any positions overnight. I noticed that he most of the trades are opened and closed between late Asian session to London close. Only on a few occasions has he closed his positions in the next Asian session. He is 100% out of the market over the weekends. Redemption liquidity is not an issue here I guess.

 

[fatowl]

Fatowl,

I asked the FM about the strategy: without revealing specifics, he said he trades like an institutional order-flow stop-hunter.

My concern is this - is it a valid strategy on a long-term basis? How do I classify this (for performance benchmarking against peers using similar strategies)? Does it fall under momentum or mean-reversion?

I don't exactly know what order-flow stop-hunting is, but I imagine it is very much like what market makers do. These strategies tend to be mean-reverting. Mean-reverting systems typically have very smooth equity curves until (boom) you get a massive drawdown. The good news is that usually the massive drawdown is short-lived. It's very scary though.

There are tons of existing forex trading funds out there. The statistics you gave us from the fund in your OP are very nebulous. I suggest you find at least 5 more funds and evaluate each objectively. Then, make a decision on which to invest in.

The best statistic you can find is a graph of the equity curve itself. That will tell you how the fund behaves in certain market conditions. Of course, it's not the perfect statistic (doesn't exist), and there are no guarantees of future returns. However, the equity curve tells you more info than cumulative annual return, hit rate, and profit factor ever will.