Deadline June

This is a discussion on Deadline June within the Trading Journals forums, part of the Reception category; So far though so good. The killer bug that they introduced in NT7-release candidate 1 which their support team was ...

Reply
 
LinkBack Thread Tools Search this Thread
Old Dec 7, 2010, 11:41am   #591
 
Adamus's Avatar
Joined Mar 2008
NinjaTrader 7 General Release - no more beta, or perhaps I shouldn't be so optimistic

Adamus started this thread So far though so good.

The killer bug that they introduced in NT7-release candidate 1 which their support team was completely unable to reproduce has mysteriously vanished in the final release version 1.

The one thing that NinjaTrader are good at is making their users feel uncertain about the stability of their product. Give it a couple of weeks and if it's still gone I'll be satisfied.

In a show of total over-optimism I also checked the NinjaTrader - Interactive Brokers connectivity bug that neither NinjaTrader nor Interactive Brokers will admit is their problem and it's still there.

On Friday I was reading up about Amitrader and Neoticker with a view to trialling one of them this week because NinjaTrader had become unusable due to the killer bug just coming back again and again - normally it would only appear once or twice a day. So just as NinjaTrader reached the point of me uninstalling it, they released the new version and the bug was gone. Sunday evening restart was a dream, no crashes, no errors, no reloads required. Yesterday was fine too. So apart from the connectivity bug, NinjaTrader is running smoothly. At this rate I might get something done this week instead of sitting here stroking NinjaTrader and mopping its brow.
__________________
What matters most is how well you walk through the fire.
Adamus is offline   Reply With Quote
Old Dec 7, 2010, 12:04pm   #592
 
Adamus's Avatar
Joined Mar 2008
Forex data

Adamus started this thread I just discovered that I had tick data from back to 1998 for AUD/JPY, CHF/JPY and GBP/CHF which I hadn't loaded from DTC.

It takes about 4 hours to load each instrument so now it's done (and now I have a "stable" version of NinjaTrader) I'm going to do a back-up of it all.

I should also be able to load up data from a Russian site forexite.com via Forex Tester.com who claim that it's the best free data available because it's just from one source and not integrated with the ticks from multiple other providers. Sounds good in principle but my guess is that it's actually not as representative of the whole forex market as you might want. A couple of backtests on it will show whether it's any good. The attraction is that they have data back to 2001 for EUR/CAD and other crosses and also if it proves more similar to IB data than DTC, then I'm onto something useful.
__________________
What matters most is how well you walk through the fire.
Adamus is offline   Reply With Quote
Old Dec 9, 2010, 2:52pm   #593
 
Adamus's Avatar
Joined Mar 2008
The show goes on.

Adamus started this thread I see Travis is inspecting an EOD trading system on Collective2 called TurningPoints, which is what my main system is called. I wonder what my system would be like tested on EOD data on all those traded markets. I can't test it straight away because I haven't checked that the old continuous contracts that I did back in Feb last are still on my hard drive, let alone actually useable.

I'll put that down on the list of things to do.

The Turning Points system is working moderately OK at the moment. It made a monthly equity high on Tuesday at £37K in the aftermath of Friday's US unemployment data.

That day got so stressful so quickly with the NinjaTrader bug killing my ability to trade. About an hour before non-farm payrolls was due, I had just got all my systems back online but I had missed the start of the hourly bar when Turning Points makes its entries, so I was checking through the logs to see what trades I had just missed and the system claimed contrary to what it actually had done that this time it was long since the night before.

Like Sorceror's Apprentice that I am, I started messing around trying to get in to the EUR/USD market to take the last few pips profit left on the table below the limit profit target. Of course the first time I tried, the market crept down instead and hit my chicken stop. When I saw that at about 15 mins before the data was due to come out, it had gone back up again to where I'd entered and the profit target was still sitting there, that the system had automatically entered thinking it was long.

So I tried again, only this time NinjaTrader died on me, I restarted everything and now the system decided that it had hit its target and was flat. And I was long and the profit target sell order was still in place just above the market. By a stroke of luck I thought I could make a few pips more because the market was stepping upwards nicely on the 5 min chart, and then non-farm payrolls came out and the next thing I knew I exited at 60 ticks profit.

Thankfully because I was caned for 40 pips on USD/JPY and got whipsawed for 40 and 40 on CHF/JPY.

Shame I hadn't gone short CAD/CHF which fell 200 pips that hour.

What annoyed me more though was the next stunt that NinjaTrader pulled and I couldn't work out why it did it but that was the old version so I've decided not to worry about it - basically NinjaTrader crashed again and on the restart after missing another hourly bar ticking over, the system decided that yes, that was actually a turning point for several markets. So I had theoretical entries at the 15:00 bar Open in six markets and it was now half an hour later and they were all retracing like bungee jumpers. I decided to give it two hours to retrace before I put anything on and saved myself a few hundred dollars, although a couple of markets never looked back.

So that was Friday - too much going on to bother working out how much it cost me and where my equity in my account would be now if I had a stable trading platform and the ability to stop myself playing under the guise of getting the systems running.

I figure there's something about the Turning Points reaction to big bars that can be modified profitably, but I haven't worked out what. Just putting in a mechanism to make it hold off entering a position after a massive move sounds crass - it might be profitable. Definitely backtestable. But I think there is something eluding me about the whole thing. Maybe it would work better on tick-based bars rather than hourly bars - although I don't trust IB's throttling and compression algorithm on their data feed to let me use that reliable, even if I could test it.
__________________
What matters most is how well you walk through the fire.
Adamus is offline   Reply With Quote
Old Dec 9, 2010, 3:27pm   #594
 
Adamus's Avatar
Joined Mar 2008
Backups

Adamus started this thread Finally took my head out of the sand and did a backup of my historical data to a removable harddrive over USB. Took long enough despite being 23 Gig. It ran for 72 hours. Either the USB port I am using is one of the slow obsolete ones on the machine or I should use a different method to copy it over - maybe zipping it first will help.

For some reason NinjaTrader decided to go with some kind of hybrid file-based database. The database directory contains over 2 million files.

That's mostly down to tick data from DTC no doubt.

Anyway, backup done for this month. I should be doing this on the last weekend of every month - that's the intention anyhow despite this being the first I've done since getting the PC back in May.
__________________
What matters most is how well you walk through the fire.
Adamus is offline   Reply With Quote
Old Dec 9, 2010, 3:57pm   #595
 
Adamus's Avatar
Joined Mar 2008
Re: NinjaTrader 7 General Release - no more beta, or perhaps I shouldn't be so optimi

Adamus started this thread
Quote:
Originally Posted by Adamus View Post
So far though so good.

The killer bug that they introduced in NT7-release candidate 1 which their support team was completely unable to reproduce has mysteriously vanished in the final release version 1.
The bug is back.

This is bad. It's not as bad as Friday when I couldn't get things running properly, but it's bad as in a big disappointment that the last 3 days without the bug were just a fluke.

I think that NinjaTrader 7 is just a can of worms.

I'm now going to do 3 things:

(1) re-open the forum thread at NinjaTrader which they closed and marked as "old beta thread" - if they don't look like they're going to find the bug, I'll tell them to get the vaseline out, to make it less painful for them when I give them back their license
(2) evaluate Amibroker and
(3) evaluate Neoticker.
__________________
What matters most is how well you walk through the fire.
Adamus is offline   Reply With Quote
Old Dec 10, 2010, 2:11pm   #596
 
Adamus's Avatar
Joined Mar 2008
Backtesting against last price, running live against bid/ask

Adamus started this thread The spread on the GBP/CHF caught me out yesterday after stopping and starting NinjaTrader.

I was short GBP/CHF and I had my profit target at 1.5475. Earlier, it had dropped to that level and the bid traded at 75, but the ask never did although it was only 3 pips and the market rose again.

On re-initialization, NinjaTrader fed the historical data through the trading system and the low for the bar was 1.5475 so it decided I'd hit my target and closed the position.

The market retraced to 1.5515 by the time I stopped and started NinjaTrader, so I had to close the position. Otherwise I'd still be short.

I think this happens about once every couple of weeks, or about 1 every 100 trades. I think I just have to ignore it because I'm always going to be testing against last and not bid/ask.
__________________
What matters most is how well you walk through the fire.
Adamus is offline   Reply With Quote
Old Dec 10, 2010, 2:21pm   #597
Joined Jan 2009
Re: Backups

Quote:
Originally Posted by Adamus View Post
Finally took my head out of the sand and did a backup of my historical data to a removable harddrive over USB. Took long enough despite being 23 Gig. It ran for 72 hours. Either the USB port I am using is one of the slow obsolete ones on the machine or I should use a different method to copy it over - maybe zipping it first will help.
What you could consider (I don't know if you already do this) is having a separate PC on which you run your robot. You can then access this PC either directly or remotely from another machine, using TeamViewer (which is free). It's then easy to copy files between the two machines, so that's one element of backup taken care of.
meanreversion is offline   Reply With Quote
Old Dec 10, 2010, 3:24pm   #598
 
Adamus's Avatar
Joined Mar 2008
Exclamation Re: Networking unknown unknowns

Adamus started this thread
Quote:
Originally Posted by Adamus View Post
.....
The latest theory from a networking mailing list I'm on is that BT is throttling my broadband connection.

So if this doesn't work, the next step is to try the workstation from someone else's LAN. If that works, then it depends who their ISP is, although hopefully whatever I'll be closer to an answer.

....

Probably less work than switching ISP.
I took my computer around to a friend's house and plugged in to their ISP - O2 - and ran up IB and NinjaTrader. No networking issues. No "connection broken panic" from NinjaTrader or IB.

I took the computer back home and ran it all up again and tried it again - "connection broken panic".

So, what does that mean?

First of all, NnjaTrader said the error must be with IB's connectivity because it works for everyone else and they even reviewed the code in their IB connection module and decided it couldn't be a fault with NinjaTrader.

Secondly, InteractiveBrokers asked me to check whether I get the same errors when I load up the charts in IB. I don't. They only appear when I load up NinjaTrader charts. IB then saw that there are a huge number of IB errors saying "Invalid date format" in the logs. So IB concluded that NinjaTrader is making bad data requests and this is causing the IB server to disconnect. That sounded implausible, I mean, why disconnect? Aren't error messages enough? This is their support statement on the issue:

Quote:
Base on the log file, it appear that Ninja Trader was submitting invalid historical data request periodically. It is feasible that connection between TWS and NT immediately due to countless invalid request from Ninja Trader.
For further investigation, I strongly recommend that you connect TWS and Ninja Trader, however do not request any information in NT yet. Increasingly, request one contract at a time until receiving the disconnection message again. It is feasible that the core issue is only related to one of the existing contracts.
These guys are Swiss, hence the bad English.

So, now that I have discovered that NinjaTrader will connect without problem to IB on O2, it invalidates what IB are saying.

However it still doesn't mean IB is the problem.

What I have tested so far means shows that the problem is that IB is unable to make a robust 24*7 connection via my BT broadband. Sometimes it can, sometimes it can't.

It's not my LAN or my software or my hardware because I've swopped it all out and tried it on different machines with different modems and different configurations. Actually went and bought a new modem for the purpose.

The best theory then is that BT Broadband are doing 'profile shaping' or bandwidth throttling on my connection which accounts for the periodic nature of the fault. So my traffic is passing through one of the BT servers where they are intentionally dropping packets to reduce my bandwidth. In normal circumstances, the Internet Protocol error correction process should make this invisible to the user, but in this case there is something else wrong on my network pathway across the internet to IB's server farm that is killing the error correction process.

But why just me? Why doesn't anyone else get this problem? There must be thousands of people using IB and NinjaTrader, but I'm the only one reporting it.

I have work-around for this bug, but I also have another bug with NinjaTrader with the work-around where the loading of data hangs. I think this may be the same network problem.

So I need a different ISP.

I should be able to prove this theory by checking where on the network pathway BT Broadband is dropping my packets, and where the error correction packets are going missing.
__________________
What matters most is how well you walk through the fire.
Adamus is offline   Reply With Quote
Old Dec 13, 2010, 3:55pm   #599
 
Adamus's Avatar
Joined Mar 2008
Networking known unknowns

Adamus started this thread Now asked BT direct if they are throttling my connection and they said no.

I checked my usage and it's about 15Gigs a month which doesn't put me on their radar as far as punitive throttling is concerned. Apparently P2P and other file sharing stuff goes way beyond that sort of usage.

But this doesn't let BT off the hook yet. BT could be throttling my packets (poor little ****ers) just as part of the routine management of the broadband down my street.

Apart from approaching IB Support again and licking their boots to get some kind of attention from them, I can try to identify the hosts on my network pathway to their server which is failing when I get these historical data server disconnects. That means spending 10 mins with ping trying to find the point of failure at 8 o'clock every evening.

Even then I have no idea whether it would be under BT's control, and if it were, whether they'd do anything about it.
__________________
What matters most is how well you walk through the fire.
Adamus is offline   Reply With Quote
Old Dec 13, 2010, 4:07pm   #600
 
Adamus's Avatar
Joined Mar 2008
Where's that system gone!?!?

Adamus started this thread I'm having trouble getting the Auld Three Duckees system to work.

After way too long polishing turds, I thought I'd go back to basics and resurrect the original 3Ducks that I traded for a month in September.

I have a slight problem though. I can't find it. It looks like I have edited the original system script beyond recognition and the back-up I thought I had is actually not the original either.

So I have a directory full of excel spreadsheets of results but no system.

Logically I should be able to re-invent it but this is no way to work. I thought I had a completely foolproof back-up method storing the data on two machines and removable media - but as someone said, when designing something completely fool-proof, never underestimate the ingenuity of complete fools.

I must be close to the original system though because my current incarnation is hopeless on short trades. Or was it long trades that it was hopeless at? ****!

Just bought Oxygene by Jean-Michel Jarre. What a classic. Click the image to open in full size.
__________________
What matters most is how well you walk through the fire.

Last edited by Adamus; Dec 13, 2010 at 8:48pm.
Adamus is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Similar Threads
Thread Thread Starter Forum Replies Last Post
FX Trading June 4 - June 8 trendie Forex 216 Jun 8, 2007 7:02pm
Brent Crude Trading - June 4th - June 8th ceydababy Commodities & Money Markets 10 Jun 7, 2007 7:00pm
june palladium dentist007 Commodities & Money Markets 2 Jul 7, 2005 11:16am
June T bond (ZB M4) bgold Commodities & Money Markets 1 Mar 9, 2004 10:07pm

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)