Hi,

I've read this in the wiki:

How do I use eSignal (IQFeed etc.) with Wealth-Lab Developer 5.1?

Since Wealth-Lab Developer 5.1 uses a different architecture, APIs have changed, rendering all the existing providers incompatible. Hold on, the Wealth-Lab team is working to deliver the missing data providers (i.e. TC2007, ESignal, Bloomberg, IQFeed, QuoteTracker).

So my question is, if this hasn't been fixed yet(?). When could one expect it to be fixed?

(edit) I'm especially interrested in IQFeed.

Regards

QUOTE:
(edit) I'm especially interrested in IQFeed.

Luckily, IQFeed is to see the light first - we hope it can happen as soon as this month.

Will this and other non-Fidelity data feeds be available for Pro, or only Developer?

If non-Fidelity data feeds will be available for Pro, will they allow streaming quotes to work in the strategy manager in place of the existing slow Fidelity snapshot quotes?

They'll work for both Pro and Developer and almost certainly (just depends on the provider) won't have the current limitations of the Fidelity snapshots, which I sure hope will change someday.

It is not clear to me why third-party streaming data feeds will be provided to Strategy Manager, but not Fidelity's streaming data.

Or are these third-party feeds to Strategy Manager going to be snapshots?

Are there any plans to provide facilities to allow WL users to develop interfaces to data feeds not supported by Fidelity, for use in Strategy Manager?

The Strategy Monitor doesn't work with Streaming data, only static data. Other providers have different methods for updating static data. If the provider's API allows for it, there's no reason why you can't retrieve a few thousand snapshots in a few seconds.

QUOTE:
Are there any plans to provide facilities to allow WL users to develop interfaces to data feeds not supported by Fidelity, for use in Strategy Manager?

Sure, it's already "allowed". See all those dlls in the installation folder? Nearly everything in WL5 is connected by public interfaces - all data adapters, performance visualizers, indicators, strategy packs, commission structures, etc. - even the ones for Fidelity.

While the interfaces do exist, Fidelity removed their documentation. It would be really nice if Fidelity would re-release the development guide (also known as the API documentation). It's been offline for close to a year! I know MS123 does not directly control it, but can't you use the red bat-phone to call Dion and ask him to get it re-published?

http://www2.wealth-lab.com/WL5Wiki/apiDevelopmentGuide.ashx

From the WL Wiki:

Is there any API for the development of a data adapter?

Yes, Wealth-Lab Pro/Developer 5.x contains a number of APIs to build your own streaming and static data providers as well as many other extensions (see Wealth-Lab Version 5 (.NET) Development Guide).

However, at the moment these documents are not available. Please stay tuned for the news.

Cone, you said:

QUOTE:
If the provider's API allows for it, there's no reason why you can't retrieve a few thousand snapshots in a few seconds.

Yes, but although theoretically a provider can easily offer fast snapshot quotes, and such quotes would be much less demanding of local bandwidth and processor resources, I don't know of any services that actually provide this this. Do you? Does anyone?

Because the focus among data providers is streaming quotes, as far as I know, the only way to get fast 1 minute snapshot quotes is to receive streaming data, build bars from it, and write the bars to disk every minute. It is my impression that snapshots from any provider will tend to be slow, like those from Fidelity. Does anyone have any more concrete information on this?

Among streaming providers, there are differences. IQFeed is reported to have clean data, and is well priced, but the IQFeed client software's design is CPU-inefficient; the interface on the application side needs to be carefully written so as not to worsen the problem. ESignal supports lots of symbols but costs a lot more than IQFeed. IQFeed vendor DTN offers a product called NxCore that they say is extremely efficient in its use of bandwidth and CPU resources, but it is very expensive. Some brokerage datafeeds offer APIs.

If WL were to be leading instead of lagging in the area of data, it would offer the following:
1) WL Pro would offer lightning-fast snapshot quotes from Fidelity.
2) Strategy Manager would work with either static or streaming data, and in WL Pro would work with Fidelity streaming data.
3) A well-documented API to import real-time streaming and static data into WL would be provided, with well-written sample code (e.g., for Quotetracker).
4) Well-written interfaces would be provided to Quotetracker, IQFeed and Esignal, TC2007 (static), and any other providers popular with customers (e.g., Tenfore in Europe).

And:
5) Fidelity would provide automated trading in WL Pro to those who want it, not only to those who have done 500 recent trades (their current policy).

There are a lot of commercial and open-source products trying to offer portfolio-level programmable EOD and intraday backtesting and automated execution to small traders. Some are newly developed, some are undergoing major enhancements, and few if any are mature as of yet. WL5 is the only one I have heard of that is limited to static data for intraday execution.

If WL5 would get competitive by offering 1-5 above, and incorporate the wonderful feature set of WL4 that makes WL4 such an appealing and useful program, WL5 would be a superior product and attract new brokerage customers to Fidelity, and increase commission revenue from existing customers.

Oh yes:
6) Offer 64 bit versions of WL Pro and Developer to overcome memory limitations in backtesting.

QUOTE:
fast snapshot quotes

What's the difference between "fast" snapshot quotes and snapshot quotes? (About 8 years ago, I built a RealTick client solution that fetched about 5000 OHLC/V bars for different symbols in about 15 seconds.)

Note that we're not looking for "quotes" per se - we need the entire OHLC/V bar. While streaming would work fine, its a tremendous waste of resources and bandwidth at both the server and client because the strategy only works on the end of interval data. That's why it makes sense to use static/snapshot data.

Fidelity can make this work much much better, but apparently there aren't enough of you guys calling them to tell them that you need it. So, resources are directed at other features.

QUOTE:
the only way to get fast 1 minute snapshot quotes is to receive streaming data, build bars from it,write the bars to disk every minute.

We already know that this isn't the "only" way. But, if you want to do it that way, be my guest! (You'll find that writing huge data files to disk every minute just to save 1 bar will be a futile effort. It's no problem for a few symbols, but try it with 100.)

1 - Still don't know the difference between "lightning-fast" snapshot quotes and snapshot quotes.
2 - A streaming interface for the SM isn't going to happen, but WL5 doesn't limit how a Provider accesses data. That's the great thing about indirection - WL just asks for the data but doesn't say how it must be retrieved.
3 - APIs are already well-documented, they're just not available to the public yet. Call and complain, I have.
4 - We at MS123 are going to eventually provide each of those, but not Tenfore.
5 - Can't help you there, and writing it here won't help either. Call and discuss it.
6 - Developer 5.3 will fully support 64-bit, but Fidelity has to get rid of their legacy component for Pro to take advantage of it.

I was using the term "fast snapshot quotes" to refer to something like what you provided using RealTick, as distinct from Fidelity's "snapshot quotes" which can easily take 45 seconds to retrieve 100 one minute OHLC/V bars. With that kind of performance, if you want to trade 3000 symbols, even hourly bars would be too short.

I agree with you that it is a tremendous waste to use streaming data when OHLC/V bars will suffice, which they almost always will. The problem is that data providers are not necessarily oriented in this direction. Case in point: Fidelity. I notice that their streaming data in a quote window seems to keep pace with lots of symbols a lot better than their "snapshot quotes," although it's chewing up zillions of times more of my bandwidth and CPU.

I was referring to processsing streaming data into one minute bars, and writing the one minute bars to disk, which should be quick even on thousands of symbols. Writing tick data to disk is hard to do quickly enough and requires a specialized database, is not necessary, and I was not suggesting it.

When you guys write your interfaces, if getting static data from a given provider proves horribly slow as it is with Fidelity, which may or may not be true for any given provider, you may want to consider a user-configurable option specific to the interface for it to get its data either through a) requests to the provider for static data, or b) through streaming of the provider's tick data, and creation by your interface of OHLC/V bars.

QUOTE:
I was referring to processsing streaming data into one minute bars, and writing the one minute bars to disk,

I know, it won't work when the file sizes are large - take a look at the size of 1-minute WL file (over 21MB).

QUOTE:
When you guys write your interfaces

Requesting 10 bars of data at a time on one thread like the current Fidelity solution is awful when you're processing more than 30 symbols, I agree. I'm certain no other providers impose a limitation like that.

I just called Fidelity. The person I spoke with said that they plan to provide a fix for the slow static data with the 5.3 patch, and that this is planned for release within the month.

API documentation and 64 bit support; he said those requests will be put in to the developers.

500 trade rule; he said they don't plan to change that. The 500 trades can be done anywhere, not just at Fidelity.

Re: 1 minute bars. I am talking about 5 pieces of data -- OHLCV -- per symbol per minute, which at 64-bit double precision is 40 bytes per minute per symbol, or 20,000 bytes for 500 symbols. This amount of data is of course the same whether derived from streaming data or transmitted per se from the provider. I don't see how writing this amount of data to disk every minute is a problem, but if it is, I suppose it can be kept in memory only. I inferred that if SM wants static data, it will want it on disk.

It's only a problem if you want to save the bar each minute to a 21MB historic data file per symbol. But it's true, the SM doesn't care if you save the data or not.

QUOTE:
I just called Fidelity. The person I spoke with said that they plan to provide a fix for the slow static data with the 5.3 patch, and that this is planned for release within the month.

Was a "fix for slow static data" included in the 5.3 patch?

QUOTE:
Requesting 10 bars of data at a time on one thread like the current Fidelity solution is awful when you're processing more than 30 symbols, I agree. I'm certain no other providers impose a limitation like that.

If there has been improvement in how Fidelity provides static data to Wealth-Lab.NET, how does it differ from what's described above?

QUOTE:
Was a "fix for slow static data" included in the 5.3 patch?

No, and there's no "fix" for that, though in the future Fidelity has other data options.

For now, the 10-symbols at a time method is the fastest method used by the Fidelity Static Provider. It works sufficiently well for WatchLists of 30 to 50 symbols or so on a 1-minute interval.

For stop/limit scripts you can add symbols almost proportionally for longer intervals, but I wouldn't recommend too many more symbols than 50 (even at longer intervals) for market order scripts. The problem there is that too many round trips will delay getting a signal out at the beginning of the bar. Of course in some cases that could actually result in a better price, but customers only seem to want to call in about the "bad" slippage.

Thanks for that information. I'll contact Fidelity to ask about data options to be offered in the future.

When support for other Data Providers (eSignal or IQFeed) becomes available, will that include capability to deliver both realtime static bars and streaming data?

In the meantime, can Wealth-Lab.NET access realtime static one minute bars or streaming ticks through a Yahoo Premium account? If so and in the case of one minute bars, what limitations might apply.

QUOTE:
In the meantime, can Wealth-Lab.NET access realtime static one minute bars or streaming ticks through a Yahoo Premium account?

Yahoo! does not provide intraday historical data. The provider supports streaming daily data (free account - delayed, Premium account - real-time).

QUOTE:
I'll contact Fidelity to ask about data options

You'll probably confuse anyone you talk to on the phone by saying that. The "option" is that their supplier of data could change, and in concert the method of accessing data will change too.

QUOTE:
will that include capability to deliver both realtime static bars and streaming data?

Of course. It's possible, though, that a Static Provider solution be available before Streaming.

QUOTE:
static one minute bars or streaming ticks through a Yahoo Premium account

I could be wrong, but the Y! Streaming adapter only powers the Quotes tool.
Eugene or Aleksey, can you clear this up?

QUOTE:
Y! Streaming adapter only powers the Quotes tool.

Yes it can power streaming Quotes as well, but since Y! isn't a intraday data provider the difference in a streaming Strategy or Chart window would be delayed or real-time daily bars.

Cone and Eugene: Thanks for your help.

QUOTE:
For now, the 10-symbols at a time method is the fastest method used by the Fidelity Static Provider. It works sufficiently well for WatchLists of 30 to 50 symbols or so on a 1-minute interval.

From the Version 5.3 User Guide -

Due to round-trip request delays for Fidelity Static data, consider limiting the number of symbols to approximately 50 per strategy for lower intervals such as 1 and 2-minutes.

Can I infer from the "per strategy" reference that more than 30 to 50 symbols could be monitored in a given interval if (i) they were being monitored by different strategies, (ii) they were in different datasets, or are all requests for data sent to single queue for processing by the Fidelity Static Provider?

Each strategy in the Strategy Monitor runs on its own thread; in other words, it's on a per-Strategy basis.

I have looked at several threads which ask specifically about importing EOD Data files but haven't found an answer to any of the questions.

Mine is simple, I have the fields in the order that the Ascii DataSet type understands and have told it to ignore first line (header row), but can't get a single row imported.

The error says input string was not in the correct format and points to the symbol field line 1. Field: 0001.HK Field Name: Symbol Field type: Custom

I found it interesting that I had to create a custom field called ticker, but I moved past that.

Is this because there is a period in the Symbol field and the Ascii import expects only periods in numbers as decimal point?

The raw data looks like this:
Symbol,Date,Open,High,Low,Close,Volume
0001.HK,01 Jan 1997,68.75,68.75,68.75,68.75,0
0002.HK,01 Jan 1997,27.234,27.234,27.234,27.234,0

*Note on earlier discussions re: data tool - volume is zero!

Thanks,

Hank

QUOTE:
I found it interesting that I had to create a custom field called ticker, but I moved past that.

What you need to here is to add a Security Name column. Custom fields can only contain double values. For more details see the User Guide.

QUOTE:
Is this because there is a period in the Symbol field and the Ascii import expects only periods in numbers as decimal point?

In a comma-delimited file, you simply couldn't mix numbers containing decimal point with decimal field separators. Neither you nor computer could decipher such file unless the numbers were wrapped in double quotes. This is just a side note (not applies to your data).

QUOTE:
The raw data looks like this:
Symbol,Date,Open,High,Low,Close,Volume
0001.HK,01 Jan 1997,68.75,68.75,68.75,68.75,0
0002.HK,01 Jan 1997,27.234,27.234,27.234,27.234,0

Here is the correct way of importing this data:

Thanks Eugene, that works.

However, as EOD Data provides the data in a file per security per year rather than a file per symbol, I created a giant unusable mess!

The import creates a dataset with individual members as a date collection incoproating all prices for all securities in that day.

This is by design: one file = one security. Stuff like popular Indian "BhavCopy" format that keeps daily updates of all stocks in one file, or 'file per security per year' have never been supported.

Speaking about possible workarounds, you could:

1. Put the data in a relational database and pick it up with the Database provider (Wealth-Lab Wiki | Bulk import of data in CSV files to an SQL Server database .

2. Or write a macro in scripting language of your choice for 'glueing' related files by instrument together (in a new folder) before creating the DataSet.

If you like to continue discussing this particular question, please consider starting a new thread. This one is outdated, too general, and too loosely related to your ASCII problem. Thanks in advance.