Automatic Day Trading - Is there anything like that ? - Part B

Previous Part

In part A I brought up some basic ideas about how to measure a trading strategy effectiveness. The basic measurements were the delta between "buying point" and "daily max price" (in order to judge the buying decision), the delta between "Selling point" and "daily max price" ("holding methodology") and the delta between selling all at once versus in several phases ("Selling strategy").

Some readers suggested that these basic definitions are on hand obvious for professionals, and insufficient for beginners. I also received questions regarding trady, the multiple computing network that works for me. I tried to respond to all questions as soon as I got it, if I missed someone I would like to take this opportunity and express my apologizes.

It is very hard to bring up a 3-years research and development sub-conclusions, in one article in a discussion forum. In the near future I plan to locate most information on a web site. At the meantime I'm using the forum platforms in order to share information and receive your feedback.

What's in this part ?

In Part B I will provide more information about TRADY, the system that I've developed for this research. I'm aware that part of my text may seem as obvious but in order to raise a theory it is essential to build a case, I hope it won;t be too boring.

More about trady

As explained in part A trady is a multiple computers servers ("pizza servers") that run a parallel software, in which the main purpose of it is to find, track, buy and sell, the best daily opportunities (currently long stock trading only).

The logical structure consists of three major parts which I call (a) wide picking (b) narrow picking and (c) trading implementation. To understand the overall picture it is important to understand my personal day trading philosophy.

My way of trading

I'm a day trader for several years and engineer by background, education and experience for many years. That combination with a strong mathematical background lead me to develop a trading strategy and a system that implements it (called trady). Basically my fundamentals are: (1) I never hold positions overnight (always go to sleep with zero holdings whether I lost or made profit at that day), (2) I always buy long with a momentum (aligned with the trend, not against it) (3) My portfolio is dynamic and changed every day, every hour and nearly every 10 minutes and (4) In some models I use, budget is a key to control the statistical behavior (this will be explained in details later).

Wide picking

My computers network reads information from NYSE, NASDAQ and AMEX (through an on-line broker tdameritrade) and builds an automatic "preferred list" of about 1500-1800 tickers. The list is not permanents and it is built every 10 minutes until about 12 Noon EST. That list brings up the best tickers to watch in a day, based on numerical parameters and mathematical calculations. I call that list "wide picking" because it provides preliminary list of symbols to watch (clearly a human been is unable to do that watch but computers can). No one guarantees that tickers in this list will remain in list, I found that during a day, a 10 minutes refresh is good enough and about 80% of list remains permanent for the trading day. In the next day the list can be totally different. To complete that part I will specify that although TRADY picks symbols from $1 and above, I personally don't go below $10 (expect for SUNW from time to time if TRADY signals this one) and I never go with tickers above $80 (although GOOG is a very popular ticker on my screens).

Narrow picking

Out of the 1500-1800 tickers in list (which are assumed to be the best for that trading day or at least trading hours), about 180 tickers will get a tight watch by the network computers. It means that these chosen 180 tickers will be sampled by the network every 6 seconds (give or take) and the accumulated samples will be used to calculate and generate trading signals and decisions.
I've been asked several questions at this point regarding the sampling rate and the effectiveness of this technological model, and I feel this is the place to reply. The whole idea is to be able to sample the price and volume charts and make some heavy floating point and int64 calculations in order to identify and verify a trend (for buying) and an exit point (selling). by sampling a symbol 10 times a minute I assume that drastic price/volume changes in a ticker chart are made slower than 5 times a minute, which in 99% of cases is true (for long stock trading, I wouldn't take the same assumption for options trading or currency). Between each and every sample the computer makes the backward calculations which are required to identify and verify the signals. From my experience this a good working point for the system. One asked me about reading every tick, which I wouldn't reject, however, for the purpose of this system, I found that there's no big different between sampling a symbol graph every 6 seconds or triggered by every market tick.

The purpose of narrow picking

Each computer can handle a certain number of symbols at once, therefore the number of computers servers in the rack is the number of symbols my network can track and control. Right now due to resources issues I use 16 computers servers to watch tightly about 160-200 symbols out of the 1500-1800 wide picking list. As explained, each and every symbol - out of the 160-200 watched ones - is sampled every 6 seconds. A sample includes the following parameters: price, change relative to last day close and volume. AV, WMA and EMA are calculated on the fly per each and every sample.

Trading implementation

Each and ticker in the narrow list is handled by two logical sub systems (a) sampling and (b) trading. The sampling sub system reads price, change to close, volume, computes AV,WMA and EMA per sample and stores data in a list which related to that specific ticker (by the way, data is stored forever and used later by TRADY data base for backward analysis and further research), the other sub system (the trading one) runs what I call "a trading program", which is a language I defined that helps me to apply a trading strategy. Each and every ticker can use a different trading program although it makes no sense to provide different trading strategies for randomly-picked symbols (I provide same program for all picked tickers).

Infrastructure

The described infra structure is eventually a parallel computing network that can signal, send buy order, sell order (of course change and cancel as well), and control many tickers very tightly.
During the past years since I started that venture I built several trading models, as most of you may probably know, it takes a lot of time to define a model, to check it, to verify its consistency and to stabilize it. When you pass through these phases the so called "final" model is no longer identical to the initial one. That's why I had to build up a very flexible infrastructure (not that I knew it from the beginnig but at some point about 3 years ago I stopped everything and designed the system from scratch).

Power, Recovery and other technical stuff

I've been asked by readers about power control and recovery. TRADY was designed bottom up, from very basic infrastructure such as encryption and security, through data backup and recovery and power control. I use a chain of two UPS systems to provide a continues power feeding, and most of data is distributed and duplicated over network. The software was written by me over the past two+ years, and right now it is about 600,000 Pascal code lines of EXE and about 200,000 code lines of extension programs and utilities.

Buying signal

Professional traders will acknowledge the "price breakthrough" term. In general a price breakthrough is a point where the demand grows faster than supply and therefore a price goes high. The challenge is to distinguish between temporary strong demand (in which you may buy but lose) and a "real" demand (where you can buy and exit later with some profit). There are many mathematical tests and indicators that were plan to signal a real demand trend. In TRADY the system is "programmed" to identify a demand by a combination of pattern recognition and price/volume acceleration. Over 233 trading days (since the system is so called "stabilized") and more than 35000 tickers which were picked by TRADY due to their daily/hourly behavior, about 88% of triggered symbols generated a price growth for more than 10 minutes since trigger and more than 75% of triggered symbols generated a daily higher price for more than an hour since the trigger time. Ad by the way, I forgot to point out, TRADY will never pick a symbol which its volume is below 100000 shares at 10 AM, 130000 at 10:30 AM, 200000 shares at 11:00 aM and so on (this is also a configurable parameter). I also forgot to point out that all selections and signals were made at real time on my real money and nit a back simulation or regression.

In part C

In the next part I will expand about the buying decision and start "talking" about the "holding methodology"'. I will also introduce more information about TRADY project.

Your comments and feedback is important and welcomed. Please feel free to comment either through he forum or my direct email

TradySupport @ Trady1.Com

The software was written by me over the past two+ years, and right now it is about 600,000 Pascal code lines of EXE and about 200,000 code lines of extension programs and utilities.

Click to expand...

Bloatware!
Have you ever worked as a programmer for Microsoft?

Pascal code lines? Pascal was meant as a training language for beginners, not really suited to use in the wide world.

Have you looked at more modern languages?

Is this all leading to you wanting to sell a system?

If so, shouldn't you declare yourself a Vendor ?

Sounds interesting, though a little light on detail. If you are just looking for feedback, i think you need to go into more detail about your buy signals, risk management etc. etc.

Cheers.

jmreeve said:

Bloatware!
Have you ever worked as a programmer for Microsoft?

Click to expand...

The purpose was not to be sound like a bloatware but to give the idea that I would love to discuss any professional question and detailed comment on either the technological side \, mathematical side or trading side. If it sound like being a bloatware I'm sorry, that's not me.

Trady1

*JDR* said:

Is this all leading to you wanting to sell a system?

If so, shouldn't you declare yourself a Vendor ?

Sounds interesting, though a little light on detail. If you are just looking for feedback, i think you need to go into more detail about your buy signals, risk management etc. etc.

Cheers.

Click to expand...

JDR good morning

No, I'm not a vendor, I'm a personal person, and neither the technology nor the system or any part it - smaller or larger - are for sell, lease etc. I'm an ex-hi tech startup founder, and at the time I lived in New York, during the bubble days I was exposed to the challenge and beauty of day trading and the rest is history. I will expand later what's behind the so called "exposure" of what I'm doing.

Trady1

As explained, each and every symbol - out of the 160-200 watched ones - is sampled every 6 seconds. A sample includes the following parameters: price, change relative to last day close and volume. AV, WMA and EMA are calculated on the fly per each and every sample.

Click to expand...

You really don't need 20 computers to calculate these values for 200 stocks every 6 seconds. In fact you don't need more than one to do it 'real time' ie every tick. And then the PC would be barely raising a sweat.

daytradingUK said:

Pascal code lines? Pascal was meant as a training language for beginners, not really suited to use in the wide world.

Have you looked at more modern languages?

Click to expand...

dcraig1 hi

I'm in the field of software and multicomputing for many years, I started with the microprocessors 8080 and 8085 and ELF-1802 (years before IBM PC XT...) and I programmed and later on managed projects in many software environments and languages. As per nowadays technology I'm not familiar with any mathematical problem or numbers crunching challenge that can be solved by C (all of its versions) better than PASCAL (especially that Borland DELPHI Pascal is a very rich paschal, not like the original one). If you take a look on the ExpertsExchange web site you'll be surprised to see how many professionals are using DELPHI (I'm not claiming that Visual C++ or Borland C++ cannot do that, I just think that at some point and some fields the difference between languages becomes unimportant).
If you find - and I'm brining it up as honest as it can be - that some math problems may receive better handling by a different language, please bring it up, it is something I would love to discuss with you.

Trady 1

dcraig1 said:

You really don't need 20 computers to calculate these values for 200 stocks every 6 seconds. In fact you don't need more than one to do it 'real time' ie every tick. And then the PC would be barely raising a sweat.

Click to expand...

dcraig1 hi

Clearly the computer does more than just computing WMA or EMA (which of course can be calculated from the previous sample and there is no need to go all the way back to the first sample...) however, the pattern recognition, the requires more tough calculations, and the auto picking of symbols as I described in part B takes lot of sweat. Also, I guess there is a "cost" in terms of CPU power paid for the flexibility of the system which is able to run multiple models and strategies. Eventually I assume that I could go back and re design it I would do some stuff better...

Anyhow, I'm not bothered by the # of computers but y the efficiency of my algorithms.

Thanks

Trady1

trady1 said:

dcraig1 hi

I'm in the field of software and multicomputing for many years, I started with the microprocessors 8080 and 8085 and ELF-1802 (years before IBM PC XT...) and I programmed and later on managed projects in many software environments and languages. As per nowadays technology I'm not familiar with any mathematical problem or numbers crunching challenge that can be solved by C (all of its versions) better than PASCAL (especially that Borland DELPHI Pascal is a very rich paschal, not like the original one). If you take a look on the ExpertsExchange web site you'll be surprised to see how many professionals are using DELPHI (I'm not claiming that Visual C++ or Borland C++ cannot do that, I just think that at some point and some fields the difference between languages becomes unimportant).
If you find - and I'm brining it up as honest as it can be - that some math problems may receive better handling by a different language, please bring it up, it is something I would love to discuss with you.

Trady 1

Click to expand...

It not necessarily a matter of language, it's a matter of algorithms and system design above all else. In particular when calculating derived time series like oscillators, MAs and so on, only the last bar needs to be recalculated each tick and that generally involves only a small lookback. Indeed in many cases, if you are prepared to carry some state between ticks, then at the expense of a little complexity you can make it even quicker.

I'm not floating my own boat here, but I have written a real time market scanner in Java that can scan for arbitary boolean combinations of relations on all the ordinary TA indicators, MA's, VWAP (and some others). Each scan condition is evaluated every tick. For a test I used tick data recorded from IB for 50 fairly high volume stocks replayed through the same code path as realtime data. Even at 200 times 'real time' rate, the computational load on a very ordinary Athlon 2800 XP Socket A PC was still well under 20%. The real issue performance wise is not the mathematical calculations but high speed updating of the alarm table GUI (a JTable). All calculations use doubles for consistancy and to minimise the risk of rounding errors biting me in some unforseen way.

I simply cannot see the necessity large amounts of computing power to do this sort of thing.

dcraig1 said:

It not necessarily a matter of language, it's a matter of algorithms and system design above all else. In particular when calculating derived time series like oscillators, MAs and so on, only the last bar needs to be recalculated each tick and that generally involves only a small lookback. Indeed in many cases, if you are prepared to carry some state between ticks, then at the expense of a little complexity you can make it even quicker.

I'm not floating my own boat here, but I have written a real time market scanner in Java that can scan for arbitary boolean combinations of relations on all the ordinary TA indicators, MA's, VWAP (and some others). Each scan condition is evaluated every tick. For a test I used tick data recorded from IB for 50 fairly high volume stocks replayed through the same code path as realtime data. Even at 200 times 'real time' rate, the computational load on a very ordinary Athlon 2800 XP Socket A PC was still well under 20%. The real issue performance wise is not the mathematical calculations but high speed updating of the alarm table GUI (a JTable). All calculations use doubles for consistancy and to minimise the risk of rounding errors biting me in some unforseen way.

I simply cannot see the necessity large amounts of computing power to do this sort of thing.

Click to expand...

Hi
You may be right, but I found that when the computer goes out to the internet in order to send a buy order to the broker (automatically) the line becomes "freezy" and I lose samples, so I thought that splitting the opportunities between different computers will allow multiple orders to be sent at the time other symbols are sampled. Anyhow, you may be right and another API could provide better performance or larger scope of symbols.

Cheers

Trady1

trady1 said:

Hi
You may be right, but I found that when the computer goes out to the internet in order to send a buy order to the broker (automatically) the line becomes "freezy" and I lose samples, so I thought that splitting the opportunities between different computers will allow multiple orders to be sent at the time other symbols are sampled. Anyhow, you may be right and another API could provide better performance or larger scope of symbols.

Cheers

Trady1

Click to expand...

You need to use a proper programming language and use multi-threading.

jmreeve said:

You need to use a proper programming language and use multi-threading.

Click to expand...

Thanks :idea:

jmreeve said:

You need to use a proper programming language and use multi-threading.

Click to expand...

... and an event driven programming model. Sampling is not best way to go about these things.

jmreeve is bang on as is dcraig, I have used Java for home projects like the one described which gives a simple html output of candle patterns, MAs, EMAs and a bit of other analysis - the hard part is really the display.

20 computers is a very wasteful system - C is faster than Java but Java is nicer to program and uses OO techniques (am Programmer myself).

With Java you would be lucky to stress a basic PC with 200 stocks, now the entire UK market is a different matter!
Also need to watch how much backdata you are storing - it will begin to eat memory as garbage collection can be a pain at times.

dcraig have you tried using lots of JLabels and SwingWorker instead of JTable so you don't end up repainting and resorting too many times?

daytradingUK said:

jmreeve is bang on as is dcraig, I have used Java for home projects like the one described which gives a simple html output of candle patterns, MAs, EMAs and a bit of other analysis - the hard part is really the display.

20 computers is a very wasteful system - C is faster than Java but Java is nicer to program and uses OO techniques (am Programmer myself).

With Java you would be lucky to stress a basic PC with 200 stocks, now the entire UK market is a different matter!
Also need to watch how much backdata you are storing - it will begin to eat memory as garbage collection can be a pain at times.

dcraig have you tried using lots of JLabels and SwingWorker instead of JTable so you don't end up repainting and resorting too many times?

Click to expand...

Actually, I think you could handle the entire UK market on a decent PC with lotsa memory by some carefull design starting with very efficient time series class and event mechanism, but that is neither here nor there.

Your right about memory utilisation. I come from more of a Unix C/C++ background and my Java has been learned the hard way by myself. Garbage collection issues are definately important and the doco is atypically for Java, not that good. A very good tool for monitoring is JConsole (with JDK 1.5 and fully supported in 1.6). Calling System.gc() once in a while performs a full mark/sweep/compact gc - otherwise the heap can grow to close to the max heap size before compacting (I think ! - the doco is not clear on this).

JTable - there are ways and means of speeding them up. There is a decent article about JTables and "christmas tree apps" that can be found by googling. In general to become truely proficient or expert in Swing (which I most certainly am not) takes a long, long, long time but it is remarkably flexible.

I don't use SwingWorker, but I do ensure that all GUI stuff goes on the AWT event processing queue and is handled in the AWT thread as recommended (or mandated) by Sun. Wherever possible I use InvokeLater. For charting, I use a kind of producer/consumer multithreaded arragement to decouple chart rendering from the event source(s) and a lookahead arrangement on the producer queue to discard events that are no longer relevent.

One major optimisation you should use is to make sure you only update the row using the tableEvents otherwise it will resort, paint and generally mess with every row.
You are probably already doing this but I thought I should mention it.

I am a Java / Swing / J2EE Dev and used to do that for HP (no the sauce) and have to say it looks harder than it is. Try getting the Deitel & Deitel books as they are mint!

If you really don't like messing with Swing then just grab Netbeans or Eclipse and have a play with the GUI designer.
Same goes for you trady1.

If you need any minor coding help I will be happy to oblige (not 50,000 lines of code with the tag why ain't this working?)

Sorry to post again so soon but can I ask what you are actually looking for to initiate trades trady1, what TA indicators, etc... you are looking for as just having 500 indicators may be information overload - most will be showing the same info in a different way.

As for is there anything like this well there is a market period in the LSE which informs people to turn off their Automated Input Facilities so I think that answers your question. I saw this when stocks went majorly south.

Good Morning DayTradingUK

Sorry for the late reply, I will post a detailed answer right after the Jewish holidays (within about a week)

Sincerely

trady1 said:

Good Morning DayTradingUK

Sorry for the late reply, I will post a detailed answer right after the Jewish holidays (within about a week)

Sincerely

Click to expand...

Still waiting or have you found your answers?