Back Test / Forward Test vs REALITY

[ozzymandius]

Hello. Hoping some people with Forex trading experience (esp. auto trading) can chime in here.

I have yet to trade, but have been trying out different mechanical Forex systems in backtester software (AmiBroker) then forward-testing those systems on out-of-sample data. I have a database of 6 years worth of historical, 1-minute Forex data for the major pairs.

But here's the rub - in the backtester, it makes a HUMONGUS difference if the sellprice/coverprice is set to either Close or High or Low. Profitable systems can turn into huge losers just by changing what the exit price is on the same bar.

I know we can set SellPrice and CoverPrice from within any trading software like AmiBroker, and even set minimum profits levels and what not with coding. But what about the actual Sell and Cover Prices? Is there any way to set minimum fullfillment Prices (for Forex, not stock trading)? And eventually feed those prices into the broker's interface in an automated fashion?

For example let's say I buy EURUSD at 1.5305 and my trading code has a profit target at 1.5310 (5 pips). When a sell signal is triggered by my code and I send in a sell order, the price of the next bar might be fluctuating anywhere from 1.5300 to 1.5320 (or wider). I want to ensure that the actual sell price is at minimum 1.5310. In real life trading, is there any way to specify that? Or will the broker just secure a price, at market, at the instant they receive my sell order?

And finally, given the realities of trading, what, if any, is a realistic backtester SellPrice and CoverPrice? To be conservative should SellPrice always be set to Low and CoverPrice always be set to High? Or is Close a good compromise for both that mimics an "average fill price" that one is likely to see in the market? In other words, how can we simulate the most realistic fill prices?

Please excuse my newbie ignorance if my questions are a bit off base!

Any feedback much appreciated.

 

[FXSCALPER2]

Hello. Hoping some people with Forex trading experience (esp. auto trading) can chime in here.

I have yet to trade, but have been trying out different mechanical Forex systems in backtester software (AmiBroker) then forward-testing those systems on out-of-sample data. I have a database of 6 years worth of historical, 1-minute Forex data for the major pairs.

But here's the rub - in the backtester, it makes a HUMONGUS difference if the sellprice/coverprice is set to either Close or High or Low. Profitable systems can turn into huge losers just by changing what the exit price is on the same bar.

I know we can set SellPrice and CoverPrice from within any trading software like AmiBroker, and even set minimum profits levels and what not with coding. But what about the actual Sell and Cover Prices? Is there any way to set minimum fullfillment Prices (for Forex, not stock trading)? And eventually feed those prices into the broker's interface in an automated fashion?

For example let's say I buy EURUSD at 1.5305 and my trading code has a profit target at 1.5310 (5 pips). When a sell signal is triggered by my code and I send in a sell order, the price of the next bar might be fluctuating anywhere from 1.5300 to 1.5320 (or wider). I want to ensure that the actual sell price is at minimum 1.5310. In real life trading, is there any way to specify that? Or will the broker just secure a price, at market, at the instant they receive my sell order?

And finally, given the realities of trading, what, if any, is a realistic backtester SellPrice and CoverPrice? To be conservative should SellPrice always be set to Low and CoverPrice always be set to High? Or is Close a good compromise for both that mimics an "average fill price" that one is likely to see in the market? In other words, how can we simulate the most realistic fill prices?

Please excuse my newbie ignorance if my questions are a bit off base!

Any feedback much appreciated.

It helps to think about what backtests do. It is highly unlikely you will find something that wins more than chance would allow (50% of the time) with more than a 1:1 win to lose ratio. So backtesting is not an entry and exit edge discovery process but a money and trade management discovery process.

 

[ozzymandius]

It helps to think about what backtests do. It is highly unlikely you will find something that wins more than chance would allow (50% of the time) with more than a 1:1 win to lose ratio. So backtesting is not an entry and exit edge discovery process but a money and trade management discovery process.

Thanks. I understand as much. But I still sense that certain kinds of backtests are more realistic than others. An indication of potential profitability, at the very least, is what I am looking for. As it stands, most of my systems do win more than 50%. But, as mentioned, the results can vary widely depending on how prices are fullfilled in the real world.

Just looking for a little guidance on real world trading with respect to fullfillment prices, before I get too carried away with my backtests.

 

[Chorlton]

Hello. Hoping some people with Forex trading experience (esp. auto trading) can chime in here.

I have yet to trade, but have been trying out different mechanical Forex systems in backtester software (AmiBroker) then forward-testing those systems on out-of-sample data. I have a database of 6 years worth of historical, 1-minute Forex data for the major pairs.

But here's the rub - in the backtester, it makes a HUMONGUS difference if the sellprice/coverprice is set to either Close or High or Low. Profitable systems can turn into huge losers just by changing what the exit price is on the same bar.

I know we can set SellPrice and CoverPrice from within any trading software like AmiBroker, and even set minimum profits levels and what not with coding. But what about the actual Sell and Cover Prices? Is there any way to set minimum fullfillment Prices (for Forex, not stock trading)? And eventually feed those prices into the broker's interface in an automated fashion?

For example let's say I buy EURUSD at 1.5305 and my trading code has a profit target at 1.5310 (5 pips). When a sell signal is triggered by my code and I send in a sell order, the price of the next bar might be fluctuating anywhere from 1.5300 to 1.5320 (or wider). I want to ensure that the actual sell price is at minimum 1.5310. In real life trading, is there any way to specify that? Or will the broker just secure a price, at market, at the instant they receive my sell order?

And finally, given the realities of trading, what, if any, is a realistic backtester SellPrice and CoverPrice? To be conservative should SellPrice always be set to Low and CoverPrice always be set to High? Or is Close a good compromise for both that mimics an "average fill price" that one is likely to see in the market? In other words, how can we simulate the most realistic fill prices?

Please excuse my newbie ignorance if my questions are a bit off base!

Any feedback much appreciated.

I only trade Equities and not Forex so won't be able to help in all areas. However, regarding what price you should use for Exits (High, Low or Close) what would happen in reality? If the bottom line is that the price could be anywhere between the High and Low, then have you considered using a RANDOM value between these two levels?

Just a thought,

Chorlton

 

[ozzymandius]

I only trade Equities and not Forex so won't be able to help in all areas. However, regarding what price you should use for Exits (High, Low or Close) what would happen in reality? If the bottom line is that the price could be anywhere between the High and Low, then have you considered using a RANDOM value between these two levels?

Just a thought,

Chorlton

Hi Chorlton. Yes, I've considered using a random value, or a value increased/reduced by a set percentage. But before going there, I was hoping to get an indication of price action by any active Forex traders.

Hoping someone has some specifics.

If not, I would likely use a random number in between the High and Low.

 

[Chorlton]

Hi Chorlton. Yes, I've considered using a random value, or a value increased/reduced by a set percentage. But before going there, I was hoping to get an indication of price action by any active Forex traders.

Hoping someone has some specifics.

If not, I would likely use a random number in between the High and Low.

No Worries... I totally understand why you would prefer "specifics".... btw I'm experiencing a similar problem to you but in my case the issue is trying to define the amount of slippage to include in my models.....

 

[new_trader]

Hello. Hoping some people with Forex trading experience (esp. auto trading) can chime in here.

I have yet to trade, but have been trying out different mechanical Forex systems in backtester software (AmiBroker) then forward-testing those systems on out-of-sample data. I have a database of 6 years worth of historical, 1-minute Forex data for the major pairs.

But here's the rub - in the backtester, it makes a HUMONGUS difference if the sellprice/coverprice is set to either Close or High or Low. Profitable systems can turn into huge losers just by changing what the exit price is on the same bar.

I know we can set SellPrice and CoverPrice from within any trading software like AmiBroker, and even set minimum profits levels and what not with coding. But what about the actual Sell and Cover Prices? Is there any way to set minimum fullfillment Prices (for Forex, not stock trading)? And eventually feed those prices into the broker's interface in an automated fashion?

For example let's say I buy EURUSD at 1.5305 and my trading code has a profit target at 1.5310 (5 pips). When a sell signal is triggered by my code and I send in a sell order, the price of the next bar might be fluctuating anywhere from 1.5300 to 1.5320 (or wider). I want to ensure that the actual sell price is at minimum 1.5310. In real life trading, is there any way to specify that? Or will the broker just secure a price, at market, at the instant they receive my sell order?

And finally, given the realities of trading, what, if any, is a realistic backtester SellPrice and CoverPrice? To be conservative should SellPrice always be set to Low and CoverPrice always be set to High? Or is Close a good compromise for both that mimics an "average fill price" that one is likely to see in the market? In other words, how can we simulate the most realistic fill prices?

Please excuse my newbie ignorance if my questions are a bit off base!

Any feedback much appreciated.

I assume you have time stamps with your 1 minute data so that you know which came first, the high or the low?

I largely agree with fxscalper2 in regard to backtests. I don't trade FX, but normally a target is set by placing a limit order. If you want to trade with market orders then you should use bid and ask prices as the fill price, but most historical data just provides the last price. In this case, if you have a [email protected] then you should (imo) make your fill price 1.5305 + spread. The same thing should be done with your exits. If you trade with limit orders then you should set your test parameter so that you only get a fill if the last price is greater than you limit price for a sell order and less than your limit price for a buy order.

 

[ozzymandius]

I assume you have time stamps with your 1 minute data so that you know which came first, the high or the low?

I largely agree with fxscalper2 in regard to backtests. I don't trade FX, but normally a target is set by placing a limit order. If you want to trade with market orders then you should use bid and ask prices as the fill price, but most historical data just provides the last price. In this case, if you have a [email protected] then you should (imo) make your fill price 1.5305 + spread. The same thing should be done with your exits. If you trade with limit orders then you should set your test parameter so that you only get a fill if the last price is greater than you limit price for a sell order and less than your limit price for a buy order.

Thanks. I suppose adding a 2 pip spread to the open of the next bar might be a reasonable compromise that might mimics real results. I'm going to open a $1,000 test account with a broker soon and do some tests with real trades. Guess that's the only way to be sure. But right now I think 2 pip slippage sounds about right. We shall see!

 

[£10kLoser]

Surely the fact that relatively small changes make the difference between profit and loss means that you don't have enough edge and that results are based more on luck?

Back testing is of limited value. You create a system that fits the curve and very little more. In the future, the curve changes. This often means that the further back you test, the less valid are the results.

I back tested my life for 50 years using daily, hourly and 1 minute time frames and during all that time I never died. I can therefore say with great statistical certainty that I am extremely unlikely to die in the future...

 

[new_trader]

Thanks. I suppose adding a 2 pip spread to the open of the next bar might be a reasonable compromise that might mimics real results. I'm going to open a $1,000 test account with a broker soon and do some tests with real trades. Guess that's the only way to be sure. But right now I think 2 pip slippage sounds about right. We shall see!

No, you misunderstood. The current market price of an instrument is made up of 2 components, the current BID and the current ASK price. The LAST price is the price that the last trade took place, it is history, the market price may or may not have changed. Historical data doesn't normally tell you if the Last trade took place @bid or @ask and this can make a difference when testing with historical data. The small difference might average out or they might add up to a significant difference.