I use .SPX 1-minute data. It is part of my "Test" dataset. The first time the "Test" dataset was updated, .SPX showed something like 870000 bars. Everything worked fine. I added 3 more symbols to the data set and everything was ok. Then, later when I added 1 more symbol, and updated the dataset, .SPX showed approximately 200000 bars. Since I was using all of the data to generate results, the results changed.

I assumed that maybe there was some type of data limit, so I killed off the added symbol, but .SPX still showed about 200000 bars. So, I killed off all of the data and started over (I checked the "Delete data for Symbols that are not contained in DataSets" option under the "Update Data" tab that is on the "Data Manager" dialog box). The .SPX data loaded with 870000 bars again.

Is there some data limit, or am I doing something wrong, or is there an issue with updating this data?

Thanks.

1. What were the other 3 symbols that you added?
2. What kind of data update did you run that time: Update All Data or Update DataSet?
3. Is the issue reproducible - if you add the same or different symbols now?

You're using the Fidelity provider, are you not? This problem of corruption in static data files occurs infrequently, but it's not uncommon. The reason is unknown since whenever we're looking for the problem to occur, it doesn't happen. [Very] intermittent problems like this one are a programmer's nightmare.

The best you can do is to remain vigilant, periodically checking the data; especially if running a serious test. The "Bad History Data Check [Rev. A]" script in the Utility folder can help, especially for detecting gaps in data. (If you don't have [Rev. A], be sure to perform a Strategies download from the Strategy Explorer.

Tip:
The easiest way to refresh a single symbol is to right click its chart (in the native time frame) and select "Reload Chart History". Be patient, especially for intraday data, and an "OK" dialog will appear when the data has been refreshed.

Eugene, Cone, thanks for the quick response.

1. The data source is Fidelity.

2. At the time, I thought there might be some data limit, so I'm not 100% sure what the other 3 symbols were (probably UVXY, SVXY, VXX). However, the last symbol that I loaded (that seemed to cause the problem) was VIX. Also, this was during trading hours, so when I entered the VIX symbol and called for an update, it required only 10 or 20 bars to update .SPX. That's why I noticed that it said .SPX only had 2000000 bars after the update. I'll try to repeat the problem.

3. I just tried to download the "Community Components Library" (for the Bad Data History Check) and it said that I didn't have permission to download the Extension. What do I need to do to get permission?

4. It was Update Dataset.

Thanks for the pointers. I hope they might help reproduce the issue.

QUOTE:
3. I just tried to download the "Community Components Library" (for the Bad Data History Check) and it said that I didn't have permission to download the Extension. What do I need to do to get permission?

Since you're not a MS123 LLC customer, we'd like to check your WLP entitlement. Attaching a full-size screenshot of WLP 6.4 (main workspace from title to status bar) logged in to Fidelity and displaying an actual intraday streaming chart (available when the U.S. market is open) with WLP's status bar visible will do the job. Preferably saved in PNG format (MS Paint that comes with any Windows will do it). Please attach the screenshot file when replying.

Eugene,

I'm not an MS123 customer. I downloaded WLP (64 bit) through Fidelity from:

https://www.fidelity.com/trading/advanced-trading-tools/strategy-testing/overview

I'm currently testing WLP to see if I want to use it. Is there a difference in being an MS123 customer versus a Fidelity client?

QUOTE:
Is there a difference in being an MS123 customer versus a Fidelity client?

Yes, it's outlined here: What's the difference between Wealth-Lab Developer and Wealth-Lab Pro Version 6?

QUOTE:
I'm currently testing

Then please disregard the screenshot. On our position regarding extension downloads by trial users, please see this FAQ: I can't download an Extension: the site says Wealth-Lab extensions are available only to Wealth-Lab Version 6 customers.

bill_080 wrote:

QUOTE:
I think I have zeroed-in on why occasional unwanted DataSet changes happen. It seems to happen because of internet connection errors while Streaming.

My data source is Fidelity (1 minute).

When I run a strategy with Streaming data (1 minute data) during market hours, if my internet connection goes down for more than a minute, an error message comes up. Regardless of clicking/not-clicking the OK button on the message box, there's a chance that the historical data for symbols that are being streamed will be changed. How do I know the data changed? Because the calculations under the "Performance" tab of my strategy will have changed once I get things running again.

My code commonly monitors more than one symbol using the following:

Bars FOOBars = GetExternalSymbol("FOO", true);

The more symbols it monitors, the more likely the unwanted changes will happen if my internet connection drops.

How common is this problem? At least once a week. I have to delete the bad DataSet, then "Update DataSet", close/restart WL, then "Create a new DataSet", and "Update DataSet".

I realize that my crappy internet connection is an issue (wired, then via WiFi), but at the moment it's the "least bad" of my options.

Anyway, I hope this helps to chase down the problem.

QUOTE:
an error message comes up.

What is the exact error message text?

QUOTE:
How do I know the data changed?

But how exactly has the data changed? This is important.

QUOTE:
I have to delete the bad DataSet, then "Update DataSet", close/restart WL, then "Create a new DataSet", and "Update DataSet".

Restarting WL is absolutely not necessary, and how can you update DataSet that's just been deleted?

Okay, what actually might be going on here is doing "Update All Data" (not DataSet) with "Delete data not contained in DataSets" option enabled. This deletes the symbol's data, enabling to refresh it (or them) on next update. Still, there are much easier ways to do it:

1. If only a handful of symbols are affected, right click on the instrument's chart and select "Reload chart history".
2. For bulk refresh, use the Data Tool extension's feature called Wipe the entire DataSet data.

Thanks for combining this into the old thread. I looked for it, but couldn't find it.

QUOTE:
What is the exact error message text?

I'll recreate the problem and post the exact message in a day or two (I need to get the error during market hours).

QUOTE:
But how exactly has the data changed? This is important.

Likewise, in a day or two, I'll also provide a data dump of "right" and "wrong" data.

QUOTE:
Restarting WL is absolutely not necessary

Actually, it has been on several occasions. If I simply click OK on the error message and my internet connection comes back up, the chart on my screen may or may not be updating with the Streaming data. I'll double check this to be sure, but more than once I have killed the strategy that was driving the screen and reopened it (without closing WL) and the Streaming data was not updating the screen. I had to close WL for everything to work normally.

QUOTE:
and how can you update DataSet that's just been deleted?

If I simply delete a corrupt Fidelity DataSet, the corrupt data still exists on my machine. For example, if I delete DataSet "Boo" that contains AAPL (and no other DataSet contains AAPL), I can still see it under:

C:\Users\Windows User Name\AppData\Roaming\Fidelity Investments\WealthLabPro\1.0.0.0\Data\FidelityStaticProvider\1 minute\A\AAPL.WL

If I create a new Fidelity DataSet called "Boo1" that contains AAPL, when I "Update DataSet", it will tell me:

Updating DataSet Boo1 ...
Symbols already up to date: AAPL
Update completed (0.xxx secs)

In other words, it still sees the corrupt AAPL data, so it won't update AAPL with fresh data.

So, to actually kill the corrupt data for the symbol AAPL (that is in only one DataSet), I have to delete that DataSet, then click the "Update Data" tab in the Data Manager and make sure the "Delete data for Symbols that are not contained in DataSets" box is checked. I also make sure that the "Fidelity Investments" selection is checked in the scroll box, and then I click "Update all data for selected Providers now" bar.

When I look on my machine, AAPL.WL is now gone. I can now upload fresh data.

The reason that I go through all of this trouble is to make sure that the corrupt data is gone in all files. The only way to be sure is for me to actually see the .WL files die. I'll play around with the method outlined above to see if it saves time.

QUOTE:
If I simply delete a corrupt Fidelity DataSet, the corrupt data still exists on my machine.

Right. I already figured that all out as can be seen in my reply.

QUOTE:
I had to close WL for everything to work normally.

Next time, you might try closing Strategy Monitor + streaming charts/quotes switching streaming providers to e.g. Yahoo and back to Fidelity in Wealth-Lab's Preferences > Streaming.

QUOTE:
Likewise, in a day or two, I'll also provide a data dump of "right" and "wrong" data.

Looking forward to it. Thanks.

Today, the problem showed up again. It was a little different than some previous episodes in that the "Net Profit" under the "Performance" tab was higher than it should have been, rather than lower. That won't help solve the problem, but I didn't want to leave anything out.

The problem happened sometime today (08/16/2013), between 12:00 and 1:00PM EDT (market time). I was at lunch so I don't know the exact time. The message box was up when I returned to my desk.

QUOTE:
What is the exact error message text?

The error message was:

"Unable to connect to the remote server Unable to get Data on Demand for symbol: XIV"

I clicked OK on the message box, but the WL screen locked up (the chart went black, and I think I know why this time), so I killed WL and restarted.

QUOTE:
But how exactly has the data changed? This is important.

During this run, the main symbol was XIV. Other symbols were accessed using:

Bars FOOBars = GetExternalSymbol("FOO", true);

These other symbols were accessed for plotting, testing, or just dead code (that will be killed when I'm satisfied that it's not needed anymore). The other symbols are:

.SPX
.VIX
SPY
VXX
VXZ

I have the "bad" .WL files (for all 6 of the symbols) ending at today's market close. I added the suffix "_Bad" to each of these files to keep them from being overwritten with fresh data. I then loaded fresh versions of each file. Below are the filesizes:

%2ESPX.WL 42,699 KB
%2ESPX_Bad.WL 42,699 KB

%2EVIX.WL 42,095 KB
%2EVIX_Bad.WL 36,131 KB

SPY.WL 42,755 KB
SPY_Bad.WL 2 KB

VXX.WL 20,045 KB
VXX_Bad.WL 20,045 KB

VXZ.WL 10,815 KB
VXZ_Bad.WL 10,815 KB

XIV.WL 10,525 KB
XIV_Bad.WL 10,525 KB

Obviously, the .VIX and SPY files have problems. I don't know if the other files have problems.

When I backup, then swap/rename the .VIX_Bad file, I get the same "Net Profit" value that I had when the problem happened earlier today, so that potentially explains that problem. However, the .VIX symbol is not plotted on the WL screen, so it probably had nothing to do with the WL screen lockup. The only symbols plotted on the WL screen are XIV and SPY.

When I backup, then swap/rename the SPY_Bad file (that only has 2 KB of data), the WL screen locks up just like it did earlier today (again, SPY is a plotted variable). So, that potentially explains that problem.

My questions now are:
1. Do you want me to e-mail all, or some, of the above files?
2. If so, do you have a filesize limit on incoming e-mail? My e-mail is at Yahoo, I can't remember their filesize limit.
3. Instead of e-mail, do you want me to attach the files to this thread using "Add Attachment(s)"?

Thank you for the detailed report. No need to attach the files: clearly they're corrupt by virtue of a rewrite/append operation (not missing a bar or two as I thought initially).

The way I see things, you have the "Automatically update data for symbols on-demand" option enabled and when you experience a network connectivity problem, Wealth-Lab fairly tells that it's "Unable to connect" to finish an on demand data request. As a workaround, try disabling that option in the Data Manager for now to see if it has a positive effect on stability. This implies that instead of relying on background updates, you will have the external symbols' data in a separate DataSet and keeping it up to date.

Possible corruption of external symbol's data during failed on demand data update request is something that the Fidelity team should try to reproduce. However, I wasn't able to reproduce the problem by suddenly dropping the connection in the midst of a data download request.

QUOTE:
you have the "Automatically update data for symbols on-demand" option enabled

That's correct.

QUOTE:
As a workaround, try disabling that option in the Data Manager for now to see if it has a positive effect on stability. This implies that instead of relying on background updates, you will have the external symbols' data in a separate DataSet and keeping it up to date.

Because of the ongoing problems, I currently have only one Fidelity DataSet. It contains all six of the symbols listed in my previous post. So, do I move the 5 external symbols to a new DataSet and keep XIV as the only symbol in the existing DataSet?

As my 1-minute script runs, doesn't "Automatically update data for symbols on-demand" keep all accessed symbols up-to-date, regardless of where they are located? If I disable that option, how do I keep the external symbols up-to-date (again, this is a 1-minute script and the external data is also 1-minute data)?

QUOTE:
Possible corruption of external symbol's data during failed on demand data update request is something that the Fidelity team should try to reproduce. However, I wasn't able to reproduce the problem by suddenly dropping the connection in the midst of a data download request.

Do I need to contact Fidelity and let them know about this?

A while back, I tried to trigger this error by unplugging or turning off my router while WL was streaming during market hours. It didn't work, but I didn't spend much time on it either. My guess was that that type of hard failure was different than some type of soft failure from AT&T. In other words, to reproduce a soft failure, I would have to turn the router off at the exact time that the 1-minute data was in the middle of coming through. The odds of me doing that at just the right time were really low, so I gave up.

QUOTE:
Do I need to contact Fidelity and let them know about this?

According to Robert (see his post #3 above), the problem seems to be already known at their end but unfortunately, intermittent. It won't hurt to call them, I think, and explain how the problem appears when the connection quality is poor.

I don't know how different may be the Fidelity provider (as I said, I couldn't reproduce your issue with other provider(s) available to me), but writing files to the disk should not happen until the historical data is downloaded completely (which is prominent in the case of SPY.WL 42,755 KB vs. SPY_Bad.WL 2 KB).

QUOTE:
If I disable that option, how do I keep the external symbols up-to-date (again, this is a 1-minute script and the external data is also 1-minute data)?

Now that you told that the symbols are kept in one DataSet, they are not really external and there's no need in splitting it. From a logical standpoint, putting the strategy in the Strategy Monitor in Portfolio Backtest mode should update the symbols before executing the strategy, and keep on trying to update if update failed.

QUOTE:
I realize that my crappy internet connection is an issue (wired, then via WiFi), but at the moment it's the "least bad" of my options.

Only my personal opinion, but troubleshooting and fixing the problem -- be it replacing a slightly damaged Ethernet wire or switching to another ISP -- appears to be the shortcut.

Eugene,

Thanks for the help.

QUOTE:
Only my personal opinion, but troubleshooting and fixing the problem -- be it replacing a slightly damaged Ethernet wire or switching to another ISP -- appears to be the shortcut.

For future reference, I'm using AT&T U-Verse as my ISP (DSL, copper over standard phone lines). My router/gateway is a standard 2WIRE router. The main U-Verse switchbox that feeds my router (located down the street) services approximately 200 houses in my neighborhood. When someone cancels their existing U-Verse service, or starts their new U-Verse service, AT&T makes software changes at their office that affect the switchbox, possibly causing some type of reset. I can see the disconnect when certain lights on my router stop blinking. My browser and other software ride through the outage without major problems.

Anyway, I'll call Fidelity in a day or two (maybe I can make the error reproducible) and let them know what's going on.



Edit: ============================

One other thing that I should add is, this problem seems to be related to four things:

1. Streaming
2. 1-minute data (the more often the update, the higher the odds of an error)
3. Grabbing external data using Bars FOOBars = GetExternalSymbol("FOO", true);
4. An interruption in the internet connection

I'm not sure, but I think the problem showed up more often when I monitored more symbols using the above GetExternalSymbol() call. So, to increase the odds of someone else duplicating the error, maybe it's worthwhile to monitor ten, twenty, or more symbols using GetExternalSymbol() and then unplugging the internet connection. I'll probably try that tomorrow during market hours.

QUOTE:
I can see the disconnect when certain lights on my router stop blinking. My browser and other software ride through the outage without major problems.

This sounds like a usual periodic ADSL reconnect (drop old session and connect again every 12-24 hours).

QUOTE:
One other thing that I should add is, this problem seems to be related to four things:

I did not verify it while Streaming, and the market was closed. Will try looking at it next week.

I tried reproducing this error today again, now with Streaming and On Demand Update enabled. I removed a few thousands bars with the help of the Data Tool (to request more data) and dropped the connection when a new bar should be added to the chart (and the Strategy re-executed itself, making a GetExternalSymbol call). Unfortunately (or fortunately), it worked flawlessly - however, I was not using the Fidelity provider.

As it's always easier to fix the last mile than a million code line program, let me suggest you add a backup ISP connection and get a Dual-WAN router with automatic failover. Anyone who is serious about high frequency trading should have a toy like this.

I ran a few tests today. I didn't reproduce "the error", but I did get different performance, depending on when the Stream cycle was interrupted/reconnected. After a certain sequence, Streaming stopped working. That's a "new error", so I'll put that on the back burner until I've beat the crap out of "the error". More trials tomorrow.