How to stop WLP caching daily scale data
Author: kazuna
Creation Date: 1/29/2013 3:20 PM
profile picture

kazuna

#1
Do you have any idea why WLP caches daily scale data while it is neither enabled for on-demand update nor in the datasets?

Since I started using GetAllDataForSymbol, I noticed one of my machine got slightly different results in the optimizer.
After looking at the bar data, I found it is due to the daily scale data being cached under Data\FidelityStaticProvider\Daily folder.
The other machines don't have the daily scale data, so I assume daily external symbols are created from intraday scale data.
In fact, if I delete the cached daily scale data, I got the same result.
In order to get consistent results across all the machines, I would like to stop WLP caching daily scale data.
So I disabled the on-demand data update which was enabled on that machine and the Data Provider Update Option is always set to "Update only Symbols that are contained in DataSets".
However, daily scale data is still cached on that machine when I run the strategy in the optimizer.
profile picture

kazuna

#2
Additional information.

If I do "Update all data for selected Providers now" in Data Manager, daily scale data is updated despite it is not in the DataSets. The daily data update is not reported in the Data Update Log.

If I enable "Delete data for Symbols that are not contained in DataSets", the daily scale data is deleted when I do "Update all data for selected Providers now" and it is reported in the Data Update Log.

Once the daily scale data is deleted, however, while Optimizer is running, do "Update all data for selected Providers now" then the daily scale data is cached again.
profile picture

kazuna

#3
Just confirmed that this issue is not specific to that machine but all other machines as well.
The scheduled data update caches the daily scale data if at the same time the optimizer running a strategy accesses the daily scale data (via GetAllDataForSymbol) even if the daily scale for the symbol is not contained in the DataSets.
profile picture

kazuna

#4
So the answer would be just to use the daily scale data in the datasets?

I preferred daily bars converting from intraday bars instead of relying on two separate datasets (daily and intraday).
Because I found some symbols to have different results using between daily scale data and daily bars converted from intraday scale data.
profile picture

Eugene

#5
While I really don`t see here any issue, GetAllDataForSymbol (BarScale.Daily) would probably grab the data in the specified bar scale (Daily) from Fidelity`s servers if it`s not present on your disk. And since the method is internal and unsupported, I believe there may be a hook to download the data bypassing the on demand update option.
profile picture

kazuna

#6
Not a problem but rather little concerned due to the facts that:

1) There are difference between bars from daily data and bars converted from intraday data into daily scale.
2) I prefer referring as fewer as possible datasets while executing strategy in live trading.
profile picture

Eugene

#7
1) Sure there are. GetAllDataForSymbol is apparently not a compressor.
2) How about using the other overloaded method then i.e. w/o the DataSet name?
profile picture

kazuna

#8
1) Is it also applicable to GetExternalSymbol / SetScaleDaily combo?
2) Could you elaborate it a little more?
profile picture

Eugene

#9
2. I`m talking about this:
CODE:
Please log in to see this code.
profile picture

kazuna

#10
2) I have been using the one without the DataSet name and having the problem.
profile picture

Eugene

#11
To stress this once again, the method is internal and therefore unsupported. It works just how it works. It does not do any intraday to daily compression.
profile picture

Cone

#12
If you've selected an intraday DataSet for the backtest and don't specify another folder to access the Daily data in GetExternalSymbol (or whatever method), then the scaled [Daily] bars are converted from the current intraday DataSet.

For sure there will be differences between Daily and Intraday-Converted-To-Daily. It's mainly due (I think) to settlement pricing applied as corrections to Daily data.
profile picture

kazuna

#13
QUOTE:
If you've selected an intraday DataSet for the backtest and don't specify another folder to access the Daily data in GetExternalSymbol (or whatever method), then the scaled [Daily] bars are converted from the current intraday DataSet.

Yes, that's exactly what I expected all the time.
However, the issue is the automatic schedule update (also happen updated manually) while running optimizer caches the Daily data even if it is not contained in DataSet. The next time whatever method accesses the external Daily data, the Daily bars come from the cached Daily data. All of the sudden, the optimizer result changes from that point.

QUOTE:
For sure there will be differences between Daily and Intraday-Converted-To-Daily. It's mainly due (I think) to settlement pricing applied as corrections to Daily data.

It sounds like Daily data is more precise than Intraday-Converted-To-Daily data?
Because I'm interested in the volatility actually happened in the day, I still prefer Intraday-Converted-To-Daily data.
This website uses cookies to improve your experience. We'll assume you're ok with that, but you can opt-out if you wish (Read more).