Depends if the system has been fitted to the data or if it has been devised based on a logical premise then simply, without adjustment or optimisation, applied to the data to find the results for the past.
Former case - system is of no value and is what the snake oil vendors sell.
Latter case show me the tropical island. Backtest results can be very informative over matters such as drawdown, losing/winning runs, RR ratios etc all of which are helpful in money management.
Former case - system is of no value and is what the snake oil vendors sell.
Latter case show me the tropical island. Backtest results can be very informative over matters such as drawdown, losing/winning runs, RR ratios etc all of which are helpful in money management.
Back testing?
I rate that a minus 10 and consider it to be of no value whatsoever. What happened in the past bears Little resemblance to what will happen in the future.
The volume/characteristics/buyers/sellers/news all change, and all affect the price.
The only value historic wise are previous sup and res points.
Easy to see, and no testing needed.
Of course I may be missing the point entirely.
I rate that a minus 10 and consider it to be of no value whatsoever. What happened in the past bears Little resemblance to what will happen in the future.
The volume/characteristics/buyers/sellers/news all change, and all affect the price.
The only value historic wise are previous sup and res points.
Easy to see, and no testing needed.
Of course I may be missing the point entirely.
cassiopeia
Active member
- Messages
- 155
- Likes
- 4
I agree with Levll, backtesting can be very useful, but only if used correctly and with caution.
Over-optimisation during backtesting flatters the performance of the system and results obtained in this way should be viewed with caution. The same applies to selectivity. If the backtest suggests that one particular share performs well out of a hundred tested it may be that the strategy just happened to fit the characteristics of that share over the time period.
To improve validity it is advisable to
1) keep the strategy simple (use no more than a few variables),
2) look for consistently good performance across a large range of instruments
3) perform sensitivity checks, that is change the variables slightly and see if the profits respond gradually or erratically, the latter suggests the system is over-fitted
4) look at results over discrete time periods, if these are consistent then this adds weight to the validity of the method
5) Don't optimise using all the data, use part of the data to optimise then apply the optimised values to the remaining data to determine the true performance
6) Test the method using forward data in real time before committing money to it. (4, 5 and 6 are in fact very similar, but 6 allows you to test the practicalities)
7) As Levll implies, strategies which are based on sound deductive arguments add weight to the validity of the inductive method of backtesting
These are just the technical factors, you can underestimate the psychological pressure of using any strategy.
Most traders back-test in some form. After all their own performance/ experience is a crude back-test (if they bother to analyse it). You have to start somewhere and it is a bit stupid to commit money to the market without any evidence the strategy works. Back-testing provides some evidence if conducted in the right way, but it is not everything.
Over-optimisation during backtesting flatters the performance of the system and results obtained in this way should be viewed with caution. The same applies to selectivity. If the backtest suggests that one particular share performs well out of a hundred tested it may be that the strategy just happened to fit the characteristics of that share over the time period.
To improve validity it is advisable to
1) keep the strategy simple (use no more than a few variables),
2) look for consistently good performance across a large range of instruments
3) perform sensitivity checks, that is change the variables slightly and see if the profits respond gradually or erratically, the latter suggests the system is over-fitted
4) look at results over discrete time periods, if these are consistent then this adds weight to the validity of the method
5) Don't optimise using all the data, use part of the data to optimise then apply the optimised values to the remaining data to determine the true performance
6) Test the method using forward data in real time before committing money to it. (4, 5 and 6 are in fact very similar, but 6 allows you to test the practicalities)
7) As Levll implies, strategies which are based on sound deductive arguments add weight to the validity of the inductive method of backtesting
These are just the technical factors, you can underestimate the psychological pressure of using any strategy.
Most traders back-test in some form. After all their own performance/ experience is a crude back-test (if they bother to analyse it). You have to start somewhere and it is a bit stupid to commit money to the market without any evidence the strategy works. Back-testing provides some evidence if conducted in the right way, but it is not everything.
I find the best use of backtesting is to prove that my "big idea" is not so very "big" after all and that it is not the "holy grail" to unparalleled riches, and that I can ditch it before I've wasted time and money on trading it with real beer vouchers.
I like Rogers idea - dont think of the market in terms of money or points, but beer vouchers! All hail the ale!
On a more serious note - I found the reverse to Roger. When I began trading from a chart, I read all the books and started backtesting, but found even the mechanical systems would NEVER perform as well on live data.
All I did was wast a load of weeks/months of my life while I should have been learning to trade with intuition.
On a more serious note - I found the reverse to Roger. When I began trading from a chart, I read all the books and started backtesting, but found even the mechanical systems would NEVER perform as well on live data.
All I did was wast a load of weeks/months of my life while I should have been learning to trade with intuition.
...as for drawdowns, (I know I'll get shot down for this, but...) I don't really have them due to my trade management. I either win or scratch. I lose about 5% of the time. That's another reason why I have no need to backtest. If the trade threatens entry, I exit.
I guess what I'm trying to say here, is what is the point of back testing, as you just don't know how will you will behave when it comes to pulling the trigger? Also, in theory your set up may produce handsome returns, but you may not have the capital free, there may be too much risk to verify the trade, you may not feel happy with the risk profile as although it falls into your profile, you feel weak as you have just taken a hiding on that last trades (because you haven't learned what cutting your losses means yet) you may be away from your desk..... Human issues aren't/can't be taken into account in the clinical science lab.
I guess what I'm trying to say here, is what is the point of back testing, as you just don't know how will you will behave when it comes to pulling the trigger? Also, in theory your set up may produce handsome returns, but you may not have the capital free, there may be too much risk to verify the trade, you may not feel happy with the risk profile as although it falls into your profile, you feel weak as you have just taken a hiding on that last trades (because you haven't learned what cutting your losses means yet) you may be away from your desk..... Human issues aren't/can't be taken into account in the clinical science lab.
Last edited:
cassiopeia
Active member
- Messages
- 155
- Likes
- 4
I find the best use of backtesting is to prove that my "big idea" is not so very "big" after all and that it is not the "holy grail" to unparalleled riches, and that I can ditch it before I've wasted time and money on trading it with real beer vouchers.
Sounds like the doctrine of falsification. This is a very powerful concept and distinguishes good science from the hocus pocus of most other activities. It is very difficult to put into practice though we always want things to work and see the patterns we want to see and ignore others. Disproving systems has saved me a fortune!
Sounds like the doctrine of falsification. This is a very powerful concept and distinguishes good science from the hocus pocus of most other activities. It is very difficult to put into practice though we always want things to work and see the patterns we want to see and ignore others. Disproving systems has saved me a fortune!
BBB said:
Back testing depends on how you are trading. I use very mechanical 'always in' point and figure systems. Sometimes they are storming, other times not. Having some sort of handle on how they performed in certain scenarios is quite important - if I see a problem I can alter the balance of trending versus pointage reversal trades etc. and have a good justification for doing so.
I also have a good handle on what sort of drawdowns are typical within certain products.
Where I would agree with you is when a 'trading' decision has to be made in a non mechanical fashion, in these cases only experience and ability - with a smidgin of luck - count, back testing and "oh, I would have got filled at....." rarely are worth much more than a cursory look.
Grey1 said:
.. and for my next sweeping generalisation..........
The only thing that misleads traders are other traders.
Back testing is OK if you have a system that is absolutely price driven and you have genuine data.
If you have some sort of system that requires 'interpretation' - then back testing is completely worthless - much the same as paper trading - other than for headline " is this worth pursuing or not" decisions.
This also shows that indicators are a complete and utter waste of time.
Grey1
Senior member
- Messages
- 2,190
- Likes
- 204
TBS,
Backtesting only approximate future performance to the extend that the system is not over curve fitted .. it is Simple to optimise for maximise profit , minimise draw down over few market if that makes a system developer feel better but there is absolutely no evidence to suggest the optimised parameters are likely to be more responsive to fractal event than un-optimised set of non-stationary data /. As I said this is IMHO..
Backtesting only approximate future performance to the extend that the system is not over curve fitted .. it is Simple to optimise for maximise profit , minimise draw down over few market if that makes a system developer feel better but there is absolutely no evidence to suggest the optimised parameters are likely to be more responsive to fractal event than un-optimised set of non-stationary data /. As I said this is IMHO..
Similar threads
- Article
