Vectorized vs Event Driven Backtesting

This is a discussion on Vectorized vs Event Driven Backtesting within the Techies Corner forums, part of the Trading Career category; Hi everyone, What are the differences between vectorized and event-driven backtesting? Which type do you use? Is the difference important ...

Reply
 
LinkBack Thread Tools Search this Thread
Old Apr 3, 2015, 11:20am   #1
 
9 Posts
Joined Mar 2015
Vectorized vs Event Driven Backtesting

Hi everyone,

What are the differences between vectorized and event-driven backtesting?
Which type do you use?
Is the difference important for non-HFT strategies like swing trading?
Which one is more realistic?

Any help is appreciated, thank you and I wish everyone profitable trades.
kojinakata is offline   Reply With Quote
Old Apr 4, 2015, 9:29am   #2
Joined Mar 2009
One is vectorized, one is event driven Obviously?

I am not sure there is a question of realism here - it is quote directly about technological approaches only. Not everything has a clear better/worse.

Realism is not about what fundamental programming approach you take, but how good you program (saying as someone just rewriting his exchange simulator into I think version 6 no to handle some issues I have with timing).
NetTecture is offline   Reply With Quote
Thanks! The following members like this post: kojinakata
Old Apr 4, 2015, 5:38pm   #3
 
9 Posts
Joined Mar 2015
kojinakata started this thread
Quote:
Originally Posted by NetTecture View Post
One is vectorized, one is event driven Obviously?

I am not sure there is a question of realism here - it is quote directly about technological approaches only. Not everything has a clear better/worse.

Realism is not about what fundamental programming approach you take, but how good you program (saying as someone just rewriting his exchange simulator into I think version 6 no to handle some issues I have with timing).
Thank you for your answer.

The following quotation is from quantsart blog:
"We've spent the last couple of months on QuantStart backtesting various trading strategies utilising Python and pandas (http://pandas.pydata.org/). The vectorised nature of pandas ensures that certain operations on large datasets are extremely rapid. However the forms of vectorised backtester that we have studied to date suffer from some drawbacks in the way that trade execution is simulated. In this series of articles we are going to discuss a more realistic approach to historical strategy simulation by constructing an event-driven backtesting environment using Python."
http://www.quantstart.com/articles/E...-Python-Part-I

He suggests that a event driven backtesting is more realistic due to some drawbacks in trade execution simulation. I don't how big a difference there is and even that drawback is significant. If someone could elaborate on this, I would appreciate it. Also I think Python can be used to simulate demo live trading (paper trading), but I am not sure.

The reason I was asking the differences between them was that I do not know R, MATLAB or Python. I wanted to start learning the most realistic one.

So what you are saying is, if I do the coding with slippage, commissions and other cost included, the realism will be the same in R or MATLAB or Python. Or did I misunderstand you?
kojinakata is offline   Reply With Quote
Old Apr 5, 2015, 4:23pm   #4
Joined Mar 2009
Not at all. He suggests that the vector based library he is using has this problem. Whether you use a vector or a event based approach, let me tell you most software out is one thing: utter ridiculous crap. NinjaTrader, for example. has obvious well documented issues that mean you can not trust a single backtest at all-. and it is semi event based.

THer are serious advantages in the event approach, in that you will trade in this way, so an event driven backtest means same code base for backtest and optimization. But that "is it". Most discrepancies really go down towards inferior software.
NetTecture is offline   Reply With Quote
Thanks! The following members like this post: kojinakata
Old Apr 5, 2015, 7:07pm   #5
 
9 Posts
Joined Mar 2015
kojinakata started this thread
Quote:
Originally Posted by NetTecture View Post
Not at all. He suggests that the vector based library he is using has this problem. Whether you use a vector or a event based approach, let me tell you most software out is one thing: utter ridiculous crap. NinjaTrader, for example. has obvious well documented issues that mean you can not trust a single backtest at all-. and it is semi event based.

THer are serious advantages in the event approach, in that you will trade in this way, so an event driven backtest means same code base for backtest and optimization. But that "is it". Most discrepancies really go down towards inferior software.
That is a very pessimistic answer . So what is my best bet if I want to do backtests, walk forward analysis of trading systems with money and risk management, incorporated slippage and commission, portfolio of diverse assets and other elements that will make it as best as possible? Can you give me a direction which I should pursuit?
kojinakata is offline   Reply With Quote
Old Apr 5, 2015, 7:17pm   #6
Joined Mar 2009
It is not pessimistic - it is realistic. head over to http://www.trade-robots.com/ and in the blog look for NinjaTrader. You will find a plenthora of documented idiocies in there (that is my blog, btw.) - which you can or can not work around. The worst is NT randomly dropping whole days from backtesting. Randomly - which means you can not repeat a backtest and get a similar result, without precautions.

Your options? No idea. Decided to go with my own - a VERY expensive route, in terms of development time, and I am just now reworking the simulator again to handle certain edge cases.

You will have to live with what is on the market, but you have to know the limitations, otherwise - they kill you.

Walk forward: do not do it in a backtest, same with portfolio simulations.

Make strategies independent per instrument, load all optimizations into a database and then use the database for walk forward and portfolio optimization / Risk management. And expect the data to be large (my own db now is allocated 3tb, all on SSD, and an upgrade to 6TB is considered...) But that means that if you rework for example walk forward and / or risk management, you do not have to redo all the backtests and optimizations.

You main problem wil lbe the processing power needed. Our own inhouse software id using multiple (I think last count is 23) computers for backtests / optimizations. THere are very few packtes on the (retail) market that can handle HPC (High Performance Computing) setups that distribute the work among multiple computers. It is quite a requirement, though, once you get larger (like for multi instrument work).
NetTecture is offline   Reply With Quote
Thanks! The following members like this post: kojinakata
Old Apr 5, 2015, 7:25pm   #7
 
malaguti's Avatar
Joined Nov 2009
reading NetTectures reply its pretty clear you'd want to stay away from NT. To be honest its quite renowned in terms of its capabilities. portfolio level tetsing is missing which in my opinion is a must
I'm not sure I'd agree that you would have strategies per instrument. That involves an element of curve fitting. But if it works clearly don't knock it.
Personally I'd only look at those software that is not unique in its programmability..why choose python that uses its own language. If that goes belly up, you're back to square one. Similarly matlab and R
that leaves you with those that are closest to C, C++, and C# and java
Amibroker is open source, then you have rightedge. Those are two that should fit your needs, however they would both be event driven
Sounds to me like you want to do algos, in which case both I've mentioned is not your thing
malaguti is offline   Reply With Quote
Old Apr 6, 2015, 7:07am   #8
Joined Mar 2009
If you do not isolate strategies for backtesting and optimization you are an idiot - or love burning money.

Assume you trade 200 instruments. THat means either you do 200 separate optimiaztions (which you can do on multiple computers) and then dump the data in a database and take it from there (walk forward, risk management).

Or you make one hugh optimization - and when you add anothe r instrument (going to 201) you make the monster optimizatoin again for ALL 201 instruments. Compare that to just running the backtest for the additional instrument and then taking it from there (with the data in the database).

A database also will allow you to walk forward in time - next month you jsut add backtests for the additional month to the database and then do the analysis based on that.

Portfolio optimizations is a computing intensive problem - any approach that cuts down the computing requirements is very welcome. And having separate strategies (strategy instances) per instrument allows you do do a lot more granular processing than having all in one monster optimization that will simply not scale.
NetTecture is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Similar Threads
Thread Thread Starter Forum Replies Last Post
Is there a list of Technology - Driven Hedgefunds in London? Simon Daniels First Steps 0 Jul 23, 2012 11:22am
price driven strategies Christiaan Forex 7 Aug 26, 2009 2:25pm
Speculator VS Commercial/Hedger driven markets MrMiyagi General Trading Chat 8 Feb 25, 2009 1:54pm
quote/order driven markets???? tonyjai First Steps 17 Mar 23, 2005 12:21am

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)