Vectorized vs Event Driven Backtesting

[kojinakata]

Hi everyone,

What are the differences between vectorized and event-driven backtesting?
Which type do you use?
Is the difference important for non-HFT strategies like swing trading?
Which one is more realistic?

Any help is appreciated, thank you and I wish everyone profitable trades.

 

[NetTecture]

One is vectorized, one is event driven Obviously?

I am not sure there is a question of realism here - it is quote directly about technological approaches only. Not everything has a clear better/worse.

Realism is not about what fundamental programming approach you take, but how good you program (saying as someone just rewriting his exchange simulator into I think version 6 no to handle some issues I have with timing).

 

[kojinakata]

One is vectorized, one is event driven Obviously?

I am not sure there is a question of realism here - it is quote directly about technological approaches only. Not everything has a clear better/worse.

Realism is not about what fundamental programming approach you take, but how good you program (saying as someone just rewriting his exchange simulator into I think version 6 no to handle some issues I have with timing).

Thank you for your answer.

The following quotation is from quantsart blog:
"We've spent the last couple of months on QuantStart backtesting various trading strategies utilising Python and pandas (http://pandas.pydata.org/). The vectorised nature of pandas ensures that certain operations on large datasets are extremely rapid. However the forms of vectorised backtester that we have studied to date suffer from some drawbacks in the way that trade execution is simulated. In this series of articles we are going to discuss a more realistic approach to historical strategy simulation by constructing an event-driven backtesting environment using Python."
http://www.quantstart.com/articles/Event-Driven-Backtesting-with-Python-Part-I

He suggests that a event driven backtesting is more realistic due to some drawbacks in trade execution simulation. I don't how big a difference there is and even that drawback is significant. If someone could elaborate on this, I would appreciate it. Also I think Python can be used to simulate demo live trading (paper trading), but I am not sure.

The reason I was asking the differences between them was that I do not know R, MATLAB or Python. I wanted to start learning the most realistic one.

So what you are saying is, if I do the coding with slippage, commissions and other cost included, the realism will be the same in R or MATLAB or Python. Or did I misunderstand you?

 

[NetTecture]

Not at all. He suggests that the vector based library he is using has this problem. Whether you use a vector or a event based approach, let me tell you most software out is one thing: utter ridiculous crap. NinjaTrader, for example. has obvious well documented issues that mean you can not trust a single backtest at all-. and it is semi event based.

THer are serious advantages in the event approach, in that you will trade in this way, so an event driven backtest means same code base for backtest and optimization. But that "is it". Most discrepancies really go down towards inferior software.

 

[kojinakata]

Not at all. He suggests that the vector based library he is using has this problem. Whether you use a vector or a event based approach, let me tell you most software out is one thing: utter ridiculous crap. NinjaTrader, for example. has obvious well documented issues that mean you can not trust a single backtest at all-. and it is semi event based.

THer are serious advantages in the event approach, in that you will trade in this way, so an event driven backtest means same code base for backtest and optimization. But that "is it". Most discrepancies really go down towards inferior software.

That is a very pessimistic answer . So what is my best bet if I want to do backtests, walk forward analysis of trading systems with money and risk management, incorporated slippage and commission, portfolio of diverse assets and other elements that will make it as best as possible? Can you give me a direction which I should pursuit?

 

[NetTecture]

It is not pessimistic - it is realistic. head over to http://www.trade-robots.com/ and in the blog look for NinjaTrader. You will find a plenthora of documented idiocies in there (that is my blog, btw.) - which you can or can not work around. The worst is NT randomly dropping whole days from backtesting. Randomly - which means you can not repeat a backtest and get a similar result, without precautions.

Your options? No idea. Decided to go with my own - a VERY expensive route, in terms of development time, and I am just now reworking the simulator again to handle certain edge cases.

You will have to live with what is on the market, but you have to know the limitations, otherwise - they kill you.

Walk forward: do not do it in a backtest, same with portfolio simulations.

Make strategies independent per instrument, load all optimizations into a database and then use the database for walk forward and portfolio optimization / Risk management. And expect the data to be large (my own db now is allocated 3tb, all on SSD, and an upgrade to 6TB is considered...) But that means that if you rework for example walk forward and / or risk management, you do not have to redo all the backtests and optimizations.

You main problem wil lbe the processing power needed. Our own inhouse software id using multiple (I think last count is 23) computers for backtests / optimizations. THere are very few packtes on the (retail) market that can handle HPC (High Performance Computing) setups that distribute the work among multiple computers. It is quite a requirement, though, once you get larger (like for multi instrument work).

 

[malaguti]

reading NetTectures reply its pretty clear you'd want to stay away from NT. To be honest its quite renowned in terms of its capabilities. portfolio level tetsing is missing which in my opinion is a must
I'm not sure I'd agree that you would have strategies per instrument. That involves an element of curve fitting. But if it works clearly don't knock it.
Personally I'd only look at those software that is not unique in its programmability..why choose python that uses its own language. If that goes belly up, you're back to square one. Similarly matlab and R
that leaves you with those that are closest to C, C++, and C# and java
Amibroker is open source, then you have rightedge. Those are two that should fit your needs, however they would both be event driven
Sounds to me like you want to do algos, in which case both I've mentioned is not your thing

 

[NetTecture]

If you do not isolate strategies for backtesting and optimization you are an idiot - or love burning money.

Assume you trade 200 instruments. THat means either you do 200 separate optimiaztions (which you can do on multiple computers) and then dump the data in a database and take it from there (walk forward, risk management).

Or you make one hugh optimization - and when you add anothe r instrument (going to 201) you make the monster optimizatoin again for ALL 201 instruments. Compare that to just running the backtest for the additional instrument and then taking it from there (with the data in the database).

A database also will allow you to walk forward in time - next month you jsut add backtests for the additional month to the database and then do the analysis based on that.

Portfolio optimizations is a computing intensive problem - any approach that cuts down the computing requirements is very welcome. And having separate strategies (strategy instances) per instrument allows you do do a lot more granular processing than having all in one monster optimization that will simply not scale.

 

[malaguti]

If you do not isolate strategies for backtesting and optimization you are an idiot - or love burning money.
.

isolate strategies for backtesting of course, but your comment was that you were isolating a strategy per instrument

"Make strategies independent per instrument" was your comment

that would make potentially 2000 strategies on the Russell 2000 if that was your market for example

 

[NetTecture]

Yes. That would be 2000 strategy instances, running separtely as per the settings of risk management. Obviously an issue with most software that is woefully designed to be non-.manageable for anyone running more than a handfull of strategies

 

[malaguti]

Yes. That would be 2000 strategy instances, running separtely as per the settings of risk management. Obviously an issue with most software that is woefully designed to be non-.manageable for anyone running more than a handfull of strategies

And thats what I would disagree with. I have multiple strategies per market
If the strategy doesn't work for the market I wouldn't trade it on an individual instrument even if it was profitable.
We clearly have differing opinions Net, but it doesn't make another money burning idiots

I would agree about software being woeful, however I am able to run these multiple strategies, which have independent market filters, conditions, sector bias and portfolio risk management across 700 in just minutes, on a simple laptop.
If I were to replicate 700 strategies it would take me weeks to create.
i guess it just depends on the software and what its running and more importantly how its logic in creating the strategy works. The processing time is simply the number crunching through the data. if the data can be cached and not reloaded then that too saves hours

 

[NetTecture]

With all respect, you misunderstand me (on purpose?)

I called it a fool to run everything in one strategy because of a total lack of optimization possibilitiesn regards to - processing time. I am not sure what planed you live on, but on my world I have to pay for electricity. And processing power is not unlimited. So, a strategy trading 500 instruments that needs reoptimization regularly is a lot more expensive than 500 copies of the strategy each trading one instrument. WIth the reasons given. For development, I can save a llot of processing time. Which IS Money.

We also run multiple strategies per market. But it is never "multiple symbols in the same strategy instance". Trading 5 instruments with one strategy? That is 5 copies of the strategy running, independent.

And risk management in a portfolio approach may not be something individual strategies do. For example, I work on a new risk management module now that will limit exposure. You MAY have 2000 strategy instances, but at the same time the risk system may limit the total exposure to 50 positions. Strategy 51 that "wants to go long" will do so on a simulator (thanks to risk management). RIsk management should IMHO never be IN The strategies, of for no other reason than for risk management (code wise). I rather have a central module that gets less often code changed than "a strategy" in general look after my risk exposure.

 

[malaguti]

With all respect, you misunderstand me (on purpose?)

I called it a fool to run everything in one strategy because of a total lack of optimization possibilitiesn regards to - processing time. I am not sure what planed you live on, but on my world I have to pay for electricity. And processing power is not unlimited. So, a strategy trading 500 instruments that needs reoptimization regularly is a lot more expensive than 500 copies of the strategy each trading one instrument. WIth the reasons given. For development, I can save a llot of processing time. Which IS Money.

We also run multiple strategies per market. But it is never "multiple symbols in the same strategy instance". Trading 5 instruments with one strategy? That is 5 copies of the strategy running, independent.

And risk management in a portfolio approach may not be something individual strategies do. For example, I work on a new risk management module now that will limit exposure. You MAY have 2000 strategy instances, but at the same time the risk system may limit the total exposure to 50 positions. Strategy 51 that "wants to go long" will do so on a simulator (thanks to risk management). RIsk management should IMHO never be IN The strategies, of for no other reason than for risk management (code wise). I rather have a central module that gets less often code changed than "a strategy" in general look after my risk exposure.

I give up..you can live in your own little candle lit world running strategies per instrument blind to the fact that there is software out there than can easily be run in minutes without unlimited processing speed potentially saving you of your electricity and your time
if Kojinakata would like to discuss sensible solutions you're free to PM me

 

[systematics]

When you use vectorization - you apply operations to an entire array (or a large section of it) at a time, as opposed to 1 element at a time. Vectorization approach allows much faster calculations (compared to event-driven) using specifically-optimized libraries and tools (such as R, Matlab, etc..).

However, in order to use vectorization - you need a complete set of data / time series, which is obviously not available during real/live trading, where you get new data ticks one at a time. Therefore, for real/live trading you can only use event-driven approach (where an event is arrival of a new data tick or data bar..). Therefore, if you use vectorized code for backtesting - you cannot use the same code in real-live trading, while if you have event-driven code - you can.