A few brief thoughts...
1) It can show whether a system would have been viable in the past and hence MAY continue to be so in the future.
It can show your maximum potential loss and maximum consecutive number of losing trades. Would the "max adverse excursion" as Investor/RT euphemistically puts it have wiped out your account or at least rendered you unable to trade due to lack of funds?
Optimisation - you can fine tune stop loss levels, trailing stops, indicator parameters etc. to see which settings would have given the most consistently good results.
2) Curve fitting (over optimisation): it's no use if a system gives a 400% return over one year because you have tweaked it within an inch of its life to fit that particular data series. Look for consistency over several different periods with as much data as possible. e.g. I've tested an hourly bar system over 1,2,3,4,5,6 years (&random chunks in the middle) and would consider that just about enough. Not accounting for commission and slippage. Trying to add more and more rules to a simple system to account for every eventuality - simple is usually best.
Can you handle trading the system in real time without breaking the rules? e.g. if it regularly takes 10 losers in a row, or has a low percentage of winning trades, or makes most of its annual return out of a handful of trades etc.
3) I think so but many would disagree.
4) Not necessarily. Some traders swear by mechanical systems, while others thrive on discretionary ones involving little or no mechnical decisions. It is generally accepted that a mechanical system will make less money than a skilled discretionary trader, but a poor discretionary trader may often lose a lot more than either! One's choice of style comes down to personality, commitment, discipline, experience, knowlewdge time available etc.
5) The data for the instrument you plan to trade. So it's no use backtesting INDU (the Dow index) when you can only trade Dow futures, for instance. Common instruments would include index, commodity and currency futures or individual stocks. If you require lots of data be prepared to pay for it (or ask nicely here!).
I'm sure others can add to this. Our resident experts are probably JonnyT, Kimmrunner and Grey1?
More stuff here -
http://www.moneycareers.com/bonanza.htm
