Best Thread Firewalker's Journey: A path of discovery in search for enlightenment

[firewalker99]

question about backtesting

firewalker

That back testing will then answer questions about expectancy, stop loss, "target" etc.

jon

Hi barjon, this one is not directed at you in particular, it's a question about backtesting. At what point does your backtesting becomes curve-fitting or optimizing instead of truly testing a strategy. Suppose I found a setup that holds a 50% win ratio. Included in my setup is a fixed stop of 1 point. If I'd use the setup with a 3 point stop the setup would have a 60% win ratio, of course with greater risk. Or if I only had a target of 3 points instead of 5, my ratio would increase by 5%, and so forth,...

For the moment I'm defining parameters on sample data for about two months. That's not a lot, I know, but I was wondering if going back to a couple of years data is very useful as markets change over time (but what time?). The next phase is comparing my setup with another two months of unseen data so that every part of the market cycle is included. Is there a reason to assume seasonal or cycliclal components that would render data analysis of a couple of months more or less useful than other months? I've calculated that I have to test about three to fourhundred "elements" to have results that are statistically significant up to 5% error. [elements] could be a [signals] or [days]. One possible answer could be that I had to use as much data as possible in order to be confident of the results of the backtesting. However, I'd like not to relate my personal "feeling" of confidence in this matter and only rely on the numbers.

In essence, what would you define as curve-fitting, where is the line between optimizing your strategy or just looking for the optimal elements that are part of the best possible setup? I've checked out the thread "Tradestation (or any other type of...) backtesting, optimising, curve-fitting" but there hasn't been much activity going on there for the last half year.

 

[dbphoenix]

Hi barjon, this one is not directed at you in particular, it's a question about backtesting. At what point does your backtesting becomes curve-fitting or optimizing instead of truly testing a strategy. Suppose I found a setup that holds a 50% win ratio. Included in my setup is a fixed stop of 1 point. If I'd use the setup with a 3 point stop the setup would have a 60% win ratio, of course with greater risk. Or if I only had a target of 3 points instead of 5, my ratio would increase by 5%, and so forth,...

For the moment I'm defining parameters on sample data for about two months. That's not a lot, I know, but I was wondering if going back to a couple of years data is very useful as markets change over time (but what time?). The next phase is comparing my setup with another two months of unseen data so that every part of the market cycle is included. Is there a reason to assume seasonal or cycliclal components that would render data analysis of a couple of months more or less useful than other months? I've calculated that I have to test about three to fourhundred "elements" to have results that are statistically significant up to 5% error. [elements] could be a [signals] or [days]. One possible answer could be that I had to use as much data as possible in order to be confident of the results of the backtesting. However, I'd like not to relate my personal "feeling" of confidence in this matter and only rely on the numbers.

In essence, what would you define as curve-fitting, where is the line between optimizing your strategy or just looking for the optimal elements that are part of the best possible setup? I've checked out the thread "Tradestation (or any other type of...) backtesting, optimising, curve-fitting" but there hasn't been much activity going on there for the last half year.

Define "backtesting".

 

[barjon]

firewalker

imo the main problem in backtesting is the impact of price shocks. By their nature, these are wholly unexpected events which cause exceptionally volatile price moves - terrorist activity, for example. Or, at a lower level, a sudden and unexpected heavy hike in interest rates. Obviously, such extreme volatilty has a major impact on the system's results. Fiddling with the system's parameters to capture such moves would clearly be a pretence and an example of "illegitimate" curve fitting.

However, I would say that where you are examining possible variables in the system's parameters - such as the stop loss level you example - it is "legitimate" optimisation. Bearing in mind, of course, that the system should be logically based on a careful observation of the market.

As far as how much is concerned, I suppose the classic answer is as much as you can making sure to cover all phases (up, down, sideways) of the market. I did about three years, I think, but I must admit that most of the fine tuning came via the paper trading phase and by a continuing review of every 100 trades. The advantage to that is that I fine-tune in relation to current market conditions which is also a disadvantage when those conditions change. Probably not a route I should be taking .

cheers

jon

 

[firewalker99]

Define "backtesting".

What follows is my definition of backtesting. Not one of the definition available from the net, else this would be only a copy&paste exercise.

Backtesting is testing a setup (a strategy if you wish to call it that) on historic charts in order to see if that what you have devised would have been profitable or if the signals/patterns you're building your strategy on, also appear in earlier data. Backtesting can consist of a looking and reading from left-to-right of old(er) charts, or replaying a trading day as if it were in realtime. The purpose of backtesting can be multi-layered. It can help you in refining your setup and identifying possible loopholes. The more backtesting one does, the more data he has based his setup on, and the more confident he will be in the results it produced. Furthermore, backtesting should help in answering questions like: "where should I place my stop, what should my target be, what is the system's expectancy, how frequent does my setup appear, what is the average P/L, W/L,..."

I know you wrote in #9 of Trading Journals that the purpose of backtesting is to define the setup. And I don't mean to criticize but I think in order to start "testing", you need to have at least something else you'd just be looking for a needle in a haystack.

 

[dbphoenix]

So if to you the purpose of backtesting is to test your setup, how are you going to define the setup you're testing?

 

[firewalker99]

So if to you the purpose of backtesting is to test your setup, how are you going to define the setup you're testing?

Backtesting can consist of testing a setup, or refining it. So refining in a way is also defining. But you have a point. Before you have a setup to test, you of course need to look at historic data (also, next to charts in real time), so in that context my definition isn't entirely correct.

 

[dbphoenix]

No, you don't have to look at historic data in order to have a setup to test. Whatever you test is derived from your hypothesis. You can develop a hypothesis without looking at old charts at all.

First, you're mixing in computerized backtesting with whatever it is you believe about backtesting in general, particularly the purpose of it. None of what I'm asked you to do has to do with curve-fitting.

Second, your continued faith in numbers is not justified by your trading experiences. If you want to believe that the answers lie in formulas (other than the simple calculations you can do in your head), then you just haven't been through enough disappointments.

What, if anything, have you learned from the exercise you have been working on?

 

[firewalker99]

No, you don't have to look at historic data in order to have a setup to test. Whatever you test is derived from your hypothesis. You can develop a hypothesis without looking at old charts at all.

No, but if I would only look at realtime charts during the day, than formulating a hypothesis would take me much longer. By looking back at historic data I'd have more than one chart available for building something upon.

First, you're mixing in computerized backtesting with whatever it is you believe about backtesting in general, particularly the purpose of it. None of what I'm asked you to do has to do with curve-fitting.

You're right, my question about curve fitting wasn't in relation with what you asked me to do. I acknowledge that your definition of backtesting differs from mine and perhaps I should use another word, but I don't want to engage in a semantic discussion. I asked that particular question basically I was wondering at what point am I changing my setup to fit the chart's data and stretching if just in order to confirm my hypothesis.

Second, your continued faith in numbers is not justified by your trading experiences. If you want to believe that the answers lie in formulas (other than the simple calculations you can do in your head), then you just haven't been through enough disappointments.

I don't think I mentioned any formulas in that particular post. I use numbers to illustrate things. Instead of "a lot of backtesting", I like to have something objective. Don't know why you're so opposed to that. As for disappointments, I think a lot of people would have quit already.

What, if anything, have you learned from the exercise you have been working on?

That a simple strategy can be profitable.

 

[dbphoenix]

No, but if I would only look at realtime charts during the day, than formulating a hypothesis would take me much longer. By looking back at historic data I'd have more than one chart available for building something upon.

True. But, so far, you have no defined setup, at least anything that can be tested. Therefore, whatever you're doing seems to have taken "much longer".

You're right, my question about curve fitting wasn't in relation with what you asked me to do. I acknowledge that your definition of backtesting differs from mine and perhaps I should use another word, but I don't want to engage in a semantic discussion. I asked that particular question basically I was wondering at what point am I changing my setup to fit the chart's data and stretching if just in order to confirm my hypothesis.

This isn't a semantic discussion. If you're engaging in two forms of backtesting simultaneously and aren't sure what you expect from either, then both of us are wasting our time.

As for the hypothesis, you have not yet formulated one.

I don't think I mentioned any formulas in that particular post. I use numbers to illustrate things. Instead of "a lot of backtesting", I like to have something objective. Don't know why you're so opposed to that. As for disappointments, I think a lot of people would have quit already.

Levels of confidence (or significance) require formulas. And I'm not opposed to "having something objective". But numbers that are obtained from faulty premises and sloppy methodology have only the appearance of objectivity.

That a simple strategy can be profitable.

And how would you state this in the form of a hypothesis?

 

[firewalker99]

True. But, so far, you have no defined setup, at least anything that can be tested. Therefore, whatever you're doing seems to have taken "much longer".

This isn't a semantic discussion. If you're engaging in two forms of backtesting simultaneously and aren't sure what you expect from either, then both of us are wasting our time.

As for the hypothesis, you have not yet formulated one.

And how would you state this in the form of a hypothesis?

I think you just changed the last sentence while I was typing. You asked me if I knew what I was looking for, then I'd have to admit no. I thought this was the first part of the exercise... I'm looking at S/R levels and as far as I know (you didn't state the purpose of the exercise) you're trying to show me what elements I should be evaluating in order to look for a profitable strategy for dealing with breakouts. So what I've done, is been observing what price does after a breakout, and comparing this with entries with a predefined stop and target.

Two types of backtesting, is that the one being where you define a setup, the other where you test it? As for a setup, in the beginning of my journal, I posted 3 types of things I was looking for, I know you said that if you were to trade that, you'd be banging your head against the wall too... Although I put a lot of time and work into developing that, we then focused on the first one and left a lot out of it. I'm not sure what it is you want me to do next. I'm willing to continue on that exercise or another one or just continue with my journal as before, it's not because I didn't put it online, I gave up on it.

My idea of what a hypothesis would look like: If a breakthrough S or R happens with price moving away further then 3 ticks, price will continue to move in that direction.

 

[dbphoenix]

So you don't know what you're looking for, you haven't defined your setup, you don't know what to expect from whatever kind of backtesting you're doing, and you're not clear about what a hypothesis is?

 

[dbphoenix]

But, more than this, dbphoenix is showing himself to be an incredible tutor.

Thanks for saying so, but if I were, we wouldn't be stuck in this loop.

I'm still trying to figure out where you are and what you know, FW, and I'm not having much luck. As soon as I begin to think we're on the same page, I discover you're in an entirely different library.

So, we'll start over.

You say you have a setup. What is it?

 

[passive]

I think you just changed the last sentence while I was typing. You asked me if I knew what I was looking for, then I'd have to admit no. I thought this was the first part of the exercise... I'm looking at S/R levels and as far as I know (you didn't state the purpose of the exercise) you're trying to show me what elements I should be evaluating in order to look for a profitable strategy for dealing with breakouts. So what I've done, is been observing what price does after a breakout, and comparing this with entries with a predefined stop and target.
...

I find it interesting to contrast this with the your first post on record here:

I'm trying very hard to understand the relationships that move the market. Based on essentials like support and resistance I try to determine a direction of the trend. Price is the main element I focus on, using volume to support my hypotheses. Therefore I believe I've read and studied enough material (mainly the posts of dbhoenix on ET and your excellent PDF files and noted charts).

Unfortunately it doesn't really seem to work out, I've been trying at this for a couple of months now and each time I think I got/see something and implement it after succesfull backtesting, it just doesn't seem to work quite as often as I wish it to be.

Can you give me any feedback? Lamont_C from ET said you'd be more than willing to give me support or suggestions to help me out. I'm trying to look at this from other perspectives but don't know what I'm missing or doing wrong here.

Attached you find a first chart where I noted my entry point.

http://www.trade2win.com/boards/attachment.php?attachmentid=21494

I'm curious if you have any comments on your progress since then.

 

[firewalker99]

I'm still trying to figure out where you are and what you know, FW, and I'm not having much luck. As soon as I begin to think we're on the same page, I discover you're in an entirely different library.

I can't deny having the same feeling. I'm trying to answer everything you throw at me, and I think I've been quite honest throughout this whole process, posting everything you asked and participating in every excercise. Not because I felt obliged to do so, but because I wanted. But I'm a bit at a loss as why you suddenly declare this... Yesterday and the days before I was working on the S/R exercise, chart by chart. So why not continue with whatever you had in mind there? You didn't quite state the purpose or the target of that particular exercise, but I was willing to follow wherever. I didn't mean to intervene by that post about backtesting/curve-fitting which was a reply to something barjon posted.

 

[firewalker99]

I find it interesting to contrast this with the your first post on record here:

I'm curious if you have any comments on your progress since then.

Hi passive, glad you ask. Not such a bad idea. Let me summarize what's changed. I've taken volume of the charts, and also refrained from drawing trendlines. As db recommended, I looked for S/R lines on intermediate term and draw them on the chart before the day opened, including the S/R lines of the previous day. Then throughout the day I observed price move and where necessary, added short term S/R lines/zones.

Then the starting point for the daily journalizing was to establish breakouts and WRB and look for retracements to that line. You can find that in post #78 where I give my essentials for taking an entry and #95. So what would be different now: the chart is in B/W, it's a 5min chart instead of 2min, there's no volume on it. S/R are drawn better, I would not take any countertrend trades. Considering all this, based on the "setup" (or whatever you like to call what I made up), I would probably take a long entry at 10.10 and would not have gone long where it's annotated on the chart. I suppose if you look at it that way, it's an improvement because I'm not talking so many bad trades, but on the other hand I'm not talking any good ones either...

 

[firewalker99]

I also commend Firewalker for his efforts. He will achieve success sooner rather than later.

But, more than this, dbphoenix is showing himself to be an incredible tutor.
He is not giving anything away on a plate, but making FW understand from within himself the issues he has to address, and getting FW to make the intellectual steps to the understanding.

Good work, FW. And DBP.

Thank you trendie. I hope you're right. I do think I've made one step, that's accepting the fact that trading means hard work and success isn't going to come rolling into my hands. But the next step, i.e. knowing to choose the right building blocks to build the house, seems more elusive than ever. Next, I think there's something contrarian about what people say about trading. On the one hand, "it's not like anything else", but then you hear "trading is just like any other business, hard work". Nevermind what other people think of course, I know there's no easy way. Yes, dbphoenix is a tough taskmaster, but my mental reserves are getting pretty much worn out.

 

[firewalker99]

So you don't know what you're looking for, you haven't defined your setup, you don't know what to expect from whatever kind of backtesting you're doing, and you're not clear about what a hypothesis is?

No. I you believe that, than I would understand why you think I am not doing the work. But I have defined all those things for myself. Perhaps they are not what you would define, but if I recall correctly you told me to find out for myself. I'll give some of my definitions in the next post. But I don't think it's fair to say that I have no clue what backtesting is, only based on the fact that I consider it a means of testing a setup on historic data while you consider it more to be a means of defining a setup. Same goes for when you ask me what W/L is, and I give you a definition which is more or less the same of what I read on Traderpedia or any other website, but you reply that it's wrong.
In both cases, why is any of the two wrong, it's just a different point of view not? But we must be clear about what we are doing and what we say we are doing and you could be right that I'm not doing what I'm saying/thinking or viceversa...

You have to understand that this means is essence I have to forget of all the things I studied and learned. I'm willing to do that if I have something else to hold on to.

 

[firewalker99]

firewalker
imo the main problem in backtesting is the impact of price shocks. By their nature, these are wholly unexpected events which cause exceptionally volatile price moves - terrorist activity, for example. Or, at a lower level, a sudden and unexpected heavy hike in interest rates. Obviously, such extreme volatilty has a major impact on the system's results. Fiddling with the system's parameters to capture such moves would clearly be a pretence and an example of "illegitimate" curve fitting.

Yes, agreed, that's what would be more curve fitting than testing imho too. But trying to fit the system to those "extremities" wouldn't seem easy.

As far as how much is concerned, I suppose the classic answer is as much as you can making sure to cover all phases (up, down, sideways) of the market. I did about three years, I think, but I must admit that most of the fine tuning came via the paper trading phase and by a continuing review of every 100 trades. The advantage to that is that I fine-tune in relation to current market conditions which is also a disadvantage when those conditions change.

Hmm, three years is a lot. Don't you think market conditions have already changed in that time? Perhaps your setup doesn't appear so frequently and you needed to go back that far in order to have a substiantially amount of trades?

 

[barjon]

Yes, agreed, that's what would be more curve fitting than testing imho too. But trying to fit the system to those "extremities" wouldn't seem easy

But not the lesser ones maybe. I trade equities and do not trade immediately prior and across results since there are often "shocks". If I was selling my method it'd be easy to tweak the parameters to have included the best of those results moves.

Hmm, three years is a lot. Don't you think market conditions have already changed in that time? Perhaps your setup doesn't appear so frequently and you needed to go back that far in order to have a substiantially amount of trades

Three years is not so long (I'm using eod remember), about 500 trades I guess.

cheers

jon

 

[firewalker99]

You say you have a setup. What is it?

Don't shoot me here. But... my "setup" was defined as following. See posts #95 and following.

If a break through a previous resistance level should occur on a wide range body (closing above middle, this does not necessarily imply one bar try to "blend" bars) then wait for a retracement to touch the resistance line. If it waffles around or slightly above the line for a minimum of three bars, than take an entry. Also check for support to the left of the weekly/monthly chart if the marubozu upbar didn't create an air pocket without support.
In case of a support line -> similar situation but turn everything around

In the next step, I further defined what was too vague:
1) What is "near"? A point? Two? 5%? 10%? Are you shorting all your contracts at once?
Near = exactly to that point
Entering a trade is immediately with two contracts, exiting is done following fixed targets based on ATR as described.

2) Again, retrace how close? Short more contracts? How many?
Retracement to that same level, so exactly as it reaches 5690. For the moment I'm not adding to a position nor averaging down. So if I were to be in a trade that would be one going in the right direction and I'd leave it alone.

3) Ditto re "retrace" and "short". "Wider" than what? How many is "a lot"?
Wider = spread at least double than the down bars before. A lot is at least 3.

I know you'd probably say this doesn't qualify for a setup, but it seems we left of there somewhere at post #107.

So if you want me first to refine this "setup", that's fine by me. If you like me to continue working on this checking it with historic data (that's what I've been doing over the last couple of weeks) or papertrading it (which is what I was doing initially), that's fine too.