GetExternalSymbol and 1 minute streaming data

algtrader

#1

8/20/2014 5:00 AM

I have a strategy that runs in 1 minute streaming mode, which looks at 2 to 3 other symbols.
I get this done as follows:

CODE:
Please log in to see this code.

Question #1
My big issue is that I have to call GetExternalSymbol() each time the system calls
Execute(), while in reality, I only need the latest Open/High/Low/Close/Volume values
for each symbol. Is there a way to get the latest values (only) for the current iteration of Execute()?

Question #2
When this strategy runs in streaming mode,. the graph of the main symbol is drawn and dynamically updated. Looking at the graph, I do see that within the 1-minute interval, the tail end of the graph is flashing what appears to be either last bid/ask values or last trade values.
How can I get access to this data?

Eugene

#2

8/20/2014 5:49 AM

1 - You could try to instantiate a StaticDataProvider and impose a limit on the number of bars returned by RequestData. I haven't tested it and making no promises that it would work in Strategies (although chances are higher with the Fidelity data provider.)

Why is that a big issue? HDD being the bottleneck when accessing large data files (as opposed to SSD)? Or more likely, unable to assign enough RAM to VirtualBox by virtue of unsupported and outdated 32-bit XP OS limited to 3GB RAM?

2 - Sorry, you can't. Only a data provider module has access to its own bid/ask values.

Cone

#3

8/20/2014 10:59 AM

Back to Q #1:
Unfortunately you have to go to great lengths to access external intraday data in a streaming strategy. Usually what you get from a secondary symbol in a streaming window is one bar old since the request is static. There are [at least] three work-arounds:

1. Making sure that "Update Data on demand" is enabled, sleep the thread for 10 seconds or more before calling GetExternalSymbol() to give the data provider time to update the static bar. Add this statement before the call to GetExternalSymbol():

CODE:
Please log in to see this code.

2. Use the Strategy Monitor. Fidelity "streams" bars for both the primary and secondary symbols almost simultaneously. Other data providers issue requests for static data, which is essentially updated at the same time.

3. The most reliable solution, imho, is to use this template. First, run your secondary symbol in its own streaming window with this strategy that saves its Bars object in global memory.

CODE:
Please log in to see this code.

Then, use this template for your strategy, which very quickly retrieves the updated secondary bars from global memory.

CODE:
Please log in to see this code.

Size:

Color:

Back to Q #1: 
Unfortunately you have to go to great lengths to access external intraday data in a streaming strategy.  Usually what you get from a secondary symbol in a streaming window is one bar old since the request is static.  There are [at least] three work-arounds:

1. Making sure that "Update Data on demand" is enabled, sleep the thread for 10 seconds or more before calling GetExternalSymbol() to give the data provider time to update the static bar.  Add this statement before the call to GetExternalSymbol():

[code]System.Threading.Thread.Sleep(10000); [/code]

2. Use the Strategy Monitor.  Fidelity "streams" bars for both the primary and secondary symbols almost simultaneously.  Other data providers issue requests for static data, which is essentially updated at the same time.

3. The most reliable solution, imho, is to use this template.  First, run your secondary symbol in its own streaming window with this strategy that saves its Bars object in global memory.

[code]			string s2Bars = Bars.Symbol + "(Bars)";	// global memory key
			SetGlobal(s2Bars, Bars);				// save the Bars object global mem
[/code]

Then, use this template for your strategy, which very quickly retrieves the updated secondary bars from global memory.

[code]using System;
using System.Collections.Generic;
using System.Text;
using System.Drawing;
using WealthLab;
using WealthLab.Indicators;

namespace WealthLab.Strategies
{
	public class PairsTemplate : WealthScript
	{
		protected override void Execute()
		{			
			string stock1 = Bars.Symbol;		// the 'clicked' symbol
			string stock2 = "CSCO";				// pair symbol
			
			Bars xBars = GetBarsFromStreaming(stock2);			
			/* Now use xBars, xBars.Close, etc. for any indicators required for the secondary symbol  */
			
			// Strategy logic goes here...		
			for (int bar = 0; bar < Bars.Count; bar++)
			{
				if (IsLastPositionActive)
				{
					Position p = LastPosition;
				}
				else
				{
					
				}
			}
			
			// For illustrative purposes...
			DrawLabel(PricePane, "Last bar " + stock2 + ": " + xBars.Date[xBars.Count - 1].ToString("yyyyMMdd HHmm"));
			ChartPane xPane = CreatePane(100, true, true);
			PlotSymbol(xPane, xBars, Color.Green, Color.Gray);
		}
		
		/* Add this method to your WealthScript class */
		private Bars GetBarsFromStreaming(string pairSymbol)
		{
			string s2Bars = pairSymbol + "(Bars)";	// global memory key
			
			DateTime lastDate = Date[Bars.Count - 1];
			Bars resBars;						// retrieved from global memory in loop below
			DateTime s2date;
			
			int watchdog = 0;
			do
			{
				System.Threading.Thread.Sleep(100);		// 100 msec delay
				resBars = (Bars)GetGlobal(s2Bars);		// these bars are unsynched
				s2date = resBars.Date[resBars.Count - 1];
				
				if (++watchdog > 50) 			// 5 secs max
					break;
			} while (s2date != lastDate);	
						
			return Synchronize(resBars);		// synch the xBars
		}		
	}
}[/code]

algtrader

#4

8/21/2014 8:12 AM

@Eugene

Re: Your response to: Q1: I've just formally registered for the WL Wiki, so I'll look at the referenced StaticDataProvider example when I get access.

As far as why it's an issue, it's mostly because of all that you mention, and 1 more: speed. At this point, everything can work with doing 2 GetExternalSymbol() calls at each time Execute() is called, but I feel that I can have more headroom if I don't have to fish out so much data that I don't need.

Additional trick that I'll try:
- Do a bar.SaveToFile() preparatory run to save the "large-window" intraday 1m tick data for all symbols,
- Run the strategy using:
- a small time window (1day), but:
- with startup calls to LoadFromFile() / AppendWitCorrections() to reload the saved data.

Re: XP.... The problems are actually not due to WinXP; all have to do with the fact that I have a 6year old PC with 3Gb RAM and Intel E2200@2.20GHz (ie. no on-chip vm support). In the near future, I'll get a new PC with 8Gb+ of RAM, wipe it clean of the Win(Doze)8 install, replace with 64bit Xubuntu, and run any Win8 or 32bitWinXP under VirtualBox. I expect it to be a lot zippier because of the larger RAM that I can allocate, and the fact that the current multi-core Intel CPUs have native vm support. Unless I can strip down Win8 or Win7 to the same level that I did with XP (ie. not XP pro), and install only the needed .NET frameworks, nothing beats using XP as a bare bones .NET platform (...except for Linux running .NET apps under mono.)

Re: Your response to: Q2:
That's too bad that I don't have access to the current bid/ask/last trade values.

Thanks very much for your reply.

Size:

Color:

@Eugene

[b][u] Re: Your response to: Q1:[/u][/b]   I've just formally registered for the WL Wiki, so I'll look at the referenced [u]StaticDataProvider[/u] example when I get access.

As far as why it's an issue, it's mostly because of all that you mention, and 1 more: speed.  At this point, everything can work with doing 2 [b]GetExternalSymbol()[/b]  calls at each time Execute() is called, but I feel that I can have more headroom if I don't have to fish out so much data that I don't need.

Additional trick that I'll try: 
- Do a [b]bar.SaveToFile()[/b] preparatory run to save the "large-window" intraday 1m tick data for all symbols,
- Run the strategy using:
     -  a small time window (1day), but:
     -  with startup calls to [b] LoadFromFile()[/b] / [b]AppendWitCorrections()[/b] to reload the saved data.

Re: XP.... The problems are actually not due to WinXP; all have to do with the fact that I have a 6year old PC with 3Gb RAM and Intel E2200@2.20GHz (ie. [b][u]no[/u][/b] on-chip vm support).  In the near future, I'll get a new PC with 8Gb+ of RAM, wipe it clean of the Win(Doze)8 install,  replace with 64bit Xubuntu, and run any Win8 or 32bitWinXP under VirtualBox.  I expect it to be a lot zippier because of the larger RAM that I can allocate, and the fact that the current multi-core Intel CPUs have native vm support.   Unless I can strip down Win8 or Win7 to the same level that I did with XP (ie. not XP pro), and install only the needed .NET frameworks,  nothing beats using XP as a bare bones .NET platform  (...except for Linux running .NET apps under mono.)

[b][u] Re: Your response to: Q2:[/u][/b]
That's too bad that I don't have access to the current bid/ask/last trade values.

Thanks very much for your reply.

algtrader

#5

8/21/2014 8:37 AM

@Cone....

Re: Your reply #1
I'll try the suggestion on enabling "Update Data on demand" and calling System.Threading.Thread.Sleep(10000) before the call to GetExternalSymbol() to make sure that the data for the external symbols have been properly posted.

Re: Your reply #2
I'll look into Strategy Monitor, which I've never used.

Re: Your reply #3
So, if I have 3 external symbols, I'll have to run an instance of the 2-line "Streamer" strategy that you gave, each running on a different "clicked" symbol? Is the snippet example below OK? Do I have to do any plotting or calls to Synchronize() ? It would be great if these instances of the "Streamer" strategy just did nothing else and just minimize in the background. when started.

CODE:
Please log in to see this code.

Thanks for the GetBarsFromStreaming(string sym) routine. This is crucial to the whole scheme (esp the sleep callss, which could have only been arrived at by past experience)!

algtrader

#6

8/21/2014 8:44 AM

@Eugene and @Cone -

Thanks very much for the responses. You guys are great!! I hope that Fidelity recognizes this fully. I've been from time to time reading your posts over the years (originally, Eugene's) , and you guys make this site really happen.

Thanks again.

/AL

Cone

#7

8/21/2014 9:52 AM

QUOTE:
So, if I have 3 external symbols, I'll have to run an instance of the 2-line "Streamer" strategy that you gave, each running on a different "clicked" symbol? Is the snippet example below OK?

Right. That´s all it takes.

QUOTE:
Do I have to do any plotting or calls to Synchronize() ?

No, that's taken care of in the GetBarsFromStreaming() method.

QUOTE:
It would be great if these instances of the "Streamer" strategy just did nothing else and just minimize in the background. when started

Answer 1: Since you have Fidelity data, you could run this strategy in the Strategy Monitor on all the secondary symbols. The "Streaming Bars" however, do come with a small delay, perhaps more than 5 seconds, in which case you'd have to increase the watchdog to > 100 in the GetBarsFromStreaming() method.

Answer 2: That said, the most-responsive solution is to use all-streaming Strategy Windows. Updates will occur within milliseconds of the end of the bar, which means your trading strategy will execute faster.

When using streaming windows, you should create a trading Workspace that minimizes the use of Performance Visualizers. That way there is no delay filling the Trades list, calculating Buy & Hold Equity, etc. If you do that, then those secondary symbol windows will actually be doing nothing but putting their Bars into global memory. And, there's certainly no problem if you minimize those windows.