Backtesting Stocks - Thoughts & Questions

[meegwell]

Hi,

I've been in a heavy research mode on trading systems and recently moved into backtesting. I've been pondering something since this move and its now time to ask others I guess.

Within the realm of stocks, what is the norm, if there is one, for what type of stocks you choose to test on? The diversity among stocks is enormous. I've concluded the following at least, which may or may not be right and I welcome comments:

1.) I would choose a 'type' of stock that has a large enough population to give me enough opportunites.

2.) This type of stock would have characteristics desirable to the system, for example if my system was based on fowlowing trends a characteristic of the stock type would be that it spends more time trending relative to other stocks.

3.) I would assume that the type of stock I test on would be the same basket of stocks I would trade - I dont think I really need to state this but too late, I already typed it.

So lets assume the above is valid. I choose some basic screens like stocks that are within x percent their 52 wk high, positive earning, liquidity of x, price range between x and y, beta of x, mid cap, and tends to trend often.

So now I'm getting more toward the meat of my questions. Lets say the above screen gives me a basket of 500 stocks. Obviously I won't be trading 500 stocks. I run my system back test and there is a big distribution of performance for the stocks...that is, the system works well on some, ok on others, poorly on others. Overall the sytem makes money, has good metrics so assume that I'm not looking to scrap it.

Is it good or bad to cherry pick which stocks the system works well on? I can't help but to think this is a lot like curve fitting or over optimizing? If I then as a next step chose the top 50 performers from this back test and used them to trade as opportunities based on my system arrived...is this a valid method or am I fooling myself somehow (outside of the normal fooling myself stuff that goes into backtesting - just that specific to trimming down that basket of stock to those that performed well in the backtest)?

Part of me says that if the systtem is all about x (say, price action), then the basket of stocks should have price action that fits the system and testing the system on "all possible stocks" makes little sense.

Any suggestions of reading material that dives into this discussion would be appreciated. I've read a lot, but I havent found anyone discuss this specific issue much.

If you feel so inclined, please offer up some successful methods, or tell me that I'm demented.

Thanks,

Meegwell

 

[meegwell]

I'll reply to my own topic here This is what I'm going to do. For my testing I'm running three different systems that are substantially different from a signal standpoint besides that fact that they are all going to attempt to capture profit. duh. They share the same stop loss and position size strategy which basically is adaptable to different stock metrics anyway.

I will run these against 12 baskets of stocks each to start. The twelve baskets are derived from 3 size screens x 4 price metrics.

First I will determine if there are any clear winners with regard to system x stock basket y after variable tweaking. I may dig into separating the entry and exits to see if there is an optimal combo.

At a minimum I will dig into the stocks that work vs the stocks that don't to see if there is anything that could explain performance other than randomness. I will step through time periods to furthur confirm their performance.

At the end of the day I hope to have a simple sytem that works well on stock type y.

 

[DashRiprock]

hi

i dont trade stocks or a trend following stategy so you can ignore me and tell be to get off your blog if you wnat

but what i will say is i would also look for non-math things in the stocks in the basket and that are really good in the backtest and as well really bad in the backtests.

so things like stocks that are part of an index might do better or worse than stocks that arent in an index

or all of your good atocks might be in one sector and all of your bad stocks might be in an other one.

just my 2p

good luck with your trading

 

[meegwell]

Hey thanks DashRip - excellent thoughts. See my goal here is some sort of bifurcation if warranted - I do believe it should be done based on the diversity within the stock universe vs. that within commodity futures. I fight with the idea of robustness, and at what point would be over-bifurcation. I am going by my own assumption that what I am building will not be best suited to trade on a highly liquid commodity futures market as well as a liquid small cap low price stock and a blue chip mega cap stock and a common exchange contract....for example. This comparison is just an example, but my point being that understanding they all have price in common i can't believe that the same system is best for all. That being said, I feel the same for major differences within universe of stocks only.

You mention some excellent points to consider - particularly the index one since that place a big role in the size of the herd trading it and goes precisely along with what IM thinking here.

Thank You,

Meegwell

 

[palm]

Hi,

I also ponder your question. I think if you like the backtest results, you should attempt to trade them all! I did this with my latest system (short-term breakout in stocks). So i would try to capture every signals and doing so by divided my total cash into small size. This, i reasoned, would mimic the back-test result as close as possible. If your system gives 500 signals per day, and you have no good way of filtering them, it might be better to randomly pick some of them. Yes, it might sound absurd to randomly choose stock from your list. But it can be a good way to preserve the statistical property of your system while make the execution possible.

Hope i answer your question.
Palm

 

[meegwell]

I've actually come a long way since my posts. I've created baskets based on price oriented filters - which would include direct price filters such as a range of trading price as well as secondary merics like items that effect the volatility of price (or direct measurement there of). The baskets are sitting tight for now as I develop some systems

I have an outline for thre foundations, each will have a few varieties. The first foundation is a volatility based system that starts with a BBand based signal, although not your basic tag, breakthrough, or RSI combo. Im about finished with the first entry point signal...I've been developing it by testing against just a random 5,10,15 day exit - no stops. This helps give me an indication of whether Im doing better than random entry - ie I was shooting for 55%+ success on entry, with success defined as a winning trade regardless of profit. From here I will incorporate a stop loss strategy and then an exit or profit taking signal....doing similar type of testing for each. I'll combo test my best performers using typical profit v. risk v. drawdown measuremnts....Then I'll move on to the seconf foundation

fun stuff...

 

[numbertea]

I paper trade stocks and when looking for trades I run my system on all the stocks in the nasdq 500. I backtest on all the stocks there to find my trading system. I will verify my system paper trading for the next 6 months before using it. By being able to use any of 500 stocks it affords me many hits that will make me money later. I have built my system for trading reversals so far. The trades I will make will not last longer than 8 days according to my backtesting so far.

 

[adrianjm]

As soon as you make a decision to reduce your inputs through some type of filtering method, you have already started your trading system, and need to think about the impacts that decision makes.

If you have not decided to include extinct tickers (which is difficult to do in it's own right), then your already excluding potential would be losers from your strategy.

Many of my older strategies depended on me choosing x number of equities prior to running my automated strategy. This had a much larger impact on out-of-sample testing than I desired. I picked the best looking stuff over the last x periods, and invariably these performed far worse in future than just a random choice.

With that in mind, I would set things like min volume etc to ensure liquidity, but other than that, I'd just pick a random selection. My systems never did any better by being more selective.

 

[numbertea]

I agree with adrianjm, backtesting and setting up strategy vs results of any price movement source will be much better than picking stocks or markets and then just trading those. Price movement patterns are found similarly in all markets, just at different times. Don't fool yourself into thinking that for future trades one market will be more predictable than another because currently one seems more predictable now, it will change. It always does.

 

[amnonco]

simple.
just trade all of them (that pass the liquidity test).
if it works than statistically you have something.