It seems that my WLP6 has a intermittence problems to execute orders. After placing orders gerenrated by S. Monitor or Quotes or even Trade Ticket, the statuse of the orders remain Submitted and never change to Active, and consequently I don't get them in my fidelity account. I have both 32 and 64bit versions installed on different machines and both have the same problem. Again, it is intermittence and somtimes it works without a problem.

Thank you for reporting. You're not alone, and we're feeding this info back to Fidelity.

Okay, thanks Eugene, it works for me now. Whatever you did, it "unstuck" the topic for me.

I have had the same problem with orders failing to reach the Fidelity server. After shutting down WL6 and restarting, the order status would appear as “Unknown”. It appears that this happens after I have run the Genetic Optimizer. I hope this helps.

I got the same problem the day before yesterday and submitted a ticket and working with Cone.

One thing I noticed looking at the log file wlp.txt was an error code "bz" logged at around the time when the submitted order didn't get through.

QUOTE:
2010-09-01 xx:yy:zz,nnn [Runner0] ERROR bz [(null)] - Unhandled re-direct.

Thanks for the report Ray60. For sure this is a problem we've got a lot of eyeballs looking at, but one thing I'm pretty confident in saying is that using the GA Optimizer won't affect order entry.

Eugene,

Do you have any update on this from Fidelity?

My rough understanding is that the problem is in authentication's periodic "login refresh" that's not keeping you logged in to the trading server. There could be a patch release within a few weeks.

To work-around the problem, you can manually log out and then back in periodically (every 90 min or so) throughout the day, or just before you plan to be placing orders. Also, if you already have orders in the stuck Submitted state, logging out and in should put the order through.

This might explain another problem: I've noticed that sometimes WLP6 freezes during the "Log in" and "Log out" process.

Hmmm, I just noticed that myself today.. and it might be a separate problem altogether. We'll need to mark it for investigation.

I have the same issue as above. I manually execute an order, but it stuck in 'submitted' state.
Here are the steps:
- Strategy Monitor generates 'buy' order.
- I go to 'Orders' window and right click "Place selected order'.
- The order never get executed.
- This happens only sometimes.

Any update on this issue?

thanks

Unfortunately, this intermittent issue continues to exist - we've recently had two more reports of it in the Support area. Which version of Wealth-Lab Pro are you using? 6.9.15?

Mohammad, can you find any log in wlp.txt at around the time when the order got stuck?

First day when I upgraded to 6.9.15, all of the sudden in the afternoon, all sell and buy orders stopped working and they all got in stuck submitted state along with the following errors logged in wlp.txt.

QUOTE:
2016-01-25 10:43:04,371 [Runner0] ERROR Fidelity.Components.RequestManager.Request [(null)] - WebException thrown
System.Net.WebException: The remote server returned an error: (500) Internal Server Error.

I have switched back to 6.9.12 since then and I haven't had the problem for few weeks now.

QUOTE:
My rough understanding is that the problem is in authentication's periodic "login refresh" that's not keeping you logged in to the trading server. There could be a patch release within a few weeks.

This is exactly correct.

Fidelity recently switched from http to https connections, and security certificate for "most" https connections is server IP address aware. In other words, the encryption used is dependent on both the IP address of the server and the client. So on re-authentication, the WLP client must connect to exactly the same server IP address to continue the encrypted connection between the two. This prevents a man-in-the-middle attack.

The problem is the load balancer on the server cluster isn't properly configured so the WLP client reconnects to exactly the same server (with the same IP address) so re-authentication is successful. There are several alternatives. One alternative is to have Verisign generate server certificates that would accept an address range of sequential IP addresses for the servers in the cluster, but that's less secure (although it does enable better dynamic load balancing).

So--for the time being anyway--please configure the server cluster's load balancer so the same https server (and server IP address) is associated with the same WLP client for the duration of the WLP login session. That should work for now, although a more flexible cluster configuration might be desirable for the future.

Meanwhile, users will just have to restart a totally new WLP login session periodically by quitting, then restarting Wealth Lab 6.9.15+. :(

superticker, have you had a chance to discuss with Fidelity support on this 6.9.15 issue so they are aware of this?
Because Fidelity plans decommissioning the old servers (http), we will be forced to upgrade to 6.9.15 and start having this problem. It's unacceptable for me. Instead, they should figure it out before switching to new servers (https).

QUOTE:
Meanwhile, users will just have to restart a totally new WLP login session periodically by quitting, then restarting Wealth Lab 6.9.15+. :(

Did you find how periodically you have to restart WLP in order to avoid this issue?

QUOTE:
superticker, have you had a chance to discuss with Fidelity support on this 6.9.15 issue so they are aware of this?

Well, kind of. I reported a "related problem" to Cone three weeks ago, and he passed it on to Fidelity. I have more recently talked to Daniel Achebodt at Fidelity about another "related problem" with the failure of WLP to re-login (and re-authenticate) during a given WLP session, and he's directly in touch with the developers. (I have not specifically mentioned the order "Submitted" problem, but that will go away when the re-login problem is fixed; all these server connectivity Internal-Error-500 problems are related to failed https authentication.)

But this is not a software or developer problem. It's a problem the computer engineer in the server room needs to address, not the developers. I guess I failed to mention that to Daniel. Oops. I see your point; the developers are helpless to fix it because it's not a software issue. Daniel will be calling me the middle of next week. I'll clarify it to him then so he talks to the machine room engineer instead.

QUOTE:
Meanwhile, users will just have to restart a totally new WLP login session periodically by quitting, then restarting Wealth Lab 6.9.15+. :(

QUOTE:
Did you find how periodically you have to restart WLP in order to avoid this issue?

Well, that's a good question.

This all depends on the probably of the load balancer handing off the re-authentication to the original server you first authenticated with. I'm not sure how many servers they have in their cluster. Typically, I've only had to crash my WLP session once a day, but your millage may vary.

Web browsers typically employ cookie session IDs to help the load balancer reconnect to the right server in the https cluster. But the WLP client is not a web browser, so I'm not sure if the WLP client includes a sessionID in its https requests to the server cluster. I guess you could snoop its https packets to see. :)

QUOTE:
"related problem" with the failure of WLP to re-login (and re-authenticate) during a given WLP session

Wow, that sounds even worse problem. Is it a new issue due to the new servers used with new 6.9.15? I have never had a problem like that in the past several years with 6.9.12 and earlier.

QUOTE:
I'll clarify it to him then so he talks to the machine room engineer instead.

Yes, please. I greatly appreciate if you can update this thread with your findings.

QUOTE:
This all depends on the probably of the load balancer handing off the re-authentication to the original server you first authenticated with.

That sounds like the load balance can happen in any minutes.

QUOTE:
I've only had to crash my WLP session once a day, but your millage may vary.

For me, it was 4 hours after I started WLP and start getting this issue.

QUOTE:
I've only had to crash my WLP session once a day, but your millage may vary.

QUOTE:
For me, it was 4 hours after I started WLP and start getting this issue.

That sounds about right for me too. The WLP client shouldn't need to re-authenticate more than every couple hours.

My Internet connection is beamed through the air, so if there's a wet, heavy snow storm, my connectivity is constantly interrupted. Because of the WL re-authentication problem, I just can't trade on those days anymore. At least not using WL. My web browser has no problem with the intermittent connectivity. Fidelity has the load balancers for their https web servers configured right.

QUOTE:
Because of the WL re-authentication problem, I just can't trade on those days anymore.

Why not switching back to 6.9.12 so that it won't use https that causing this new problem?

QUOTE:
Why not switching back to 6.9.12 so that it won't use https that causing this new problem?

That would be an alternative plan. Of course, an eavesdropper could monitor your trades then. And when Fidelity shuts down the http servers, you'll need to switch back.

Ha, ha; I guess I was hoping they would fix it soon. I didn't think--silly thought--it would take that long to fix.

Where can I download WLP 6.9.12.0?

QUOTE:
Where can I download WLP 6.9.12.0?

I don't think you can download old version once new one is released. I keep every single version so I can switch back to any version if necessary.

QUOTE:
Where can I download WLP 6.9.12.0?

You can't. But chances are you still have it: Downgrade from 6.9 to a previous version

Re: https connections

The only change for https in 6.9.12 was for GICS downloads. Fidelity trading and authentication has always used a secure connection.

QUOTE:
Fidelity trading and authentication has always used a secure connection.

Do you know if the authentication activity went through a https connection or did it employ a remote procedure call (RPC) [better known as DCOM or Distributed Common Object Model on the Windows OS platform] through a SSL (Win)socket connection instead?

I'm guessing they originally used a RPC through a SSL connection instead to authenticate, and did all the authentication through a single authentication server so no load balancer was involved. That's the way we use to do it before https existed. The down side of this approach is that the authentication server is a single point of failure; if it goes down all new logins stop, although existing logins may continue.

Using https requires more coding, but that coding is already built into the .NET framework (SessionID management for https), so there's less code for the app developers to write/support. And for servers, the actual SSL encryption (using either approach) is done on the network (ethernet) adapter hardware instead of the processor, so there's little server processor overhead.

For internal enterprise LAN traffic between servers, RPC (or DCOM for Windows) is commonly still employed. But for WAN traffic over the greater Internet, I think http and https is preferred today despite its overhead.

Suggestion:
1) Fidelity should hold off enforcing 6.9.15 upgrade and decommissioning old servers until this issue is figured out.
2) WLP should consider monitoring the connection to the server and notify the user the connection is down (preferably dialog pop-up as it is critical situation).
3 WLP should consider adopting status timeout after submitting the orders and notify the user the order is failed (preferably dialog pop-up as it is critical situation).

QUOTE:
2) WLP should consider monitoring the connection to the server and notify the user the connection is down (preferably dialog pop-up as it is critical situation).

When talking to Daniel at Fidelity, he asked me what the error message was when the WLP client lost connectivity with the authentication server (failed https secure connection). I told him there's no coding error in WLP, but that its code could be improved if it timed out (watchdog timer) and provided an error message. As it stands today, it just returns a cursor wait circle forever, and the WLP process has to be killed by Windows Task Manager to exit.

I think the older versions of WL also occasionally loose server connectivity (perhaps by design), but they were always able to reestablish it again transparent to the user. With the failed https secure connection problem, that's no longer happening. And the user is now aware of it.

Here is part of wlp.txt. Now the problem repro to me all the time.

2016-02-16 08:41:01,042 [Runner1] ERROR Fidelity.Components.RequestManager.Request [(null)] - WebException thrown
System.Net.WebException: The remote server returned an error: (500) Internal Server Error.
at Fidelity.Components.RequestManager.Request.b()
at Fidelity.Components.RequestManager.Request.Run()
2016-02-16 08:41:01,057 [Runner1] ERROR Fidelity.Components.RequestManager.FidelityWebService [(null)] - Exception parsing XML Response: q1:tradeequityconfirm/q1:order/q3:account
System.Xml.XPath.XPathException: Namespace prefix 'q3' is not defined.
at MS.Internal.Xml.XPath.CompiledXpathExpr.UndefinedXsltContext.LookupNamespace(String prefix)
at MS.Internal.Xml.XPath.BaseAxisQuery.SetXsltContext(XsltContext context)
at MS.Internal.Xml.XPath.CompiledXpathExpr.SetContext(IXmlNamespaceResolver nsResolver)
at System.Xml.XmlNode.SelectSingleNode(String xpath, XmlNamespaceManager nsmgr)
at Fidelity.Components.RequestManager.FidelityWebService.TryParseSingleNode(XmlNode theNode, String& rstr, String xpath, XmlNamespaceManager nsm)
2016-02-16 08:41:01,073 [Runner1] ERROR Fidelity.Components.RequestManager.FidelityWebService [(null)] - Exception parsing XML Response
System.Xml.XPath.XPathException: Namespace prefix 'q1' is not defined.
at MS.Internal.Xml.XPath.CompiledXpathExpr.UndefinedXsltContext.LookupNamespace(String prefix)
at MS.Internal.Xml.XPath.BaseAxisQuery.SetXsltContext(XsltContext context)
at MS.Internal.Xml.XPath.CompiledXpathExpr.SetContext(IXmlNamespaceResolver nsResolver)
at System.Xml.XmlNode.SelectNodes(String xpath, XmlNamespaceManager nsmgr)
at Fidelity.Components.RequestManager.SimpleTrade.ParseResponse(XmlDocument xml)
2016-02-16 08:41:01,073 [Runner1] ERROR Fidelity.Components.RequestManager.RequestManager [(null)] - Null Response object.
2016-02-16 08:41:01,073 [29] ERROR Fidelity.Components.RequestManager.FidelityWebService [(null)] - Failed to fire onTrade event: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index
at System.ThrowHelper.ThrowArgumentOutOfRangeException(ExceptionArgument argument, ExceptionResource resource)
at WealthLab.DataProviders.FidelityBrokerProvider.UpdateOrdersWithResponse(List`1 Response)
--- End of inner exception stack trace ---
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
at System.Delegate.DynamicInvokeImpl(Object[] args)
at Fidelity.Components.RequestManager.Trade.DoCallback()
2016-02-16 08:41:37,103 [Runner3] ERROR Fidelity.Components.RequestManager.Request [(null)] - WebException thrown
System.Net.WebException: The remote server returned an error: (500) Internal Server Error.
at Fidelity.Components.RequestManager.Request.b()
at Fidelity.Components.RequestManager.Request.Run()
2016-02-16 08:41:37,103 [Runner3] ERROR Fidelity.Components.RequestManager.FidelityWebService [(null)] - Exception parsing XML Response: q1:tradeequityconfirm/q1:order/q3:account
System.Xml.XPath.XPathException: Namespace prefix 'q3' is not defined.
at MS.Internal.Xml.XPath.CompiledXpathExpr.UndefinedXsltContext.LookupNamespace(String prefix)
at MS.Internal.Xml.XPath.BaseAxisQuery.SetXsltContext(XsltContext context)
at MS.Internal.Xml.XPath.CompiledXpathExpr.SetContext(IXmlNamespaceResolver nsResolver)
at System.Xml.XmlNode.SelectSingleNode(String xpath, XmlNamespaceManager nsmgr)
at Fidelity.Components.RequestManager.FidelityWebService.TryParseSingleNode(XmlNode theNode, String& rstr, String xpath, XmlNamespaceManager nsm)
2016-02-16 08:41:37,103 [Runner3] ERROR Fidelity.Components.RequestManager.FidelityWebService [(null)] - Exception parsing XML Response
System.Xml.XPath.XPathException: Namespace prefix 'q1' is not defined.
at MS.Internal.Xml.XPath.CompiledXpathExpr.UndefinedXsltContext.LookupNamespace(String prefix)
at MS.Internal.Xml.XPath.BaseAxisQuery.SetXsltContext(XsltContext context)
at MS.Internal.Xml.XPath.CompiledXpathExpr.SetContext(IXmlNamespaceResolver nsResolver)
at System.Xml.XmlNode.SelectNodes(String xpath, XmlNamespaceManager nsmgr)
at Fidelity.Components.RequestManager.SimpleTrade.ParseResponse(XmlDocument xml)
2016-02-16 08:41:37,104 [Runner3] ERROR Fidelity.Components.RequestManager.RequestManager [(null)] - Null Response object.
2016-02-16 08:41:37,104 [29] ERROR Fidelity.Components.RequestManager.FidelityWebService [(null)] - Failed to fire onTrade event: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index
at System.ThrowHelper.ThrowArgumentOutOfRangeException(ExceptionArgument argument, ExceptionResource resource)
at WealthLab.DataProviders.FidelityBrokerProvider.UpdateOrdersWithResponse(List`1 Response)
--- End of inner exception stack trace ---
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
at System.Delegate.DynamicInvokeImpl(Object[] args)
at Fidelity.Components.RequestManager.Trade.DoCallback()
2016-02-16 09:09:23,148 [1] ERROR WealthLabPro.ChartForm [(null)] - Collection was modified; enumeration operation may not execute.
2016-02-16 12:55:16,186 [Runner4] ERROR Fidelity.Components.RequestManager.Request [(null)] - WebException thrown
System.Net.WebException: The remote server returned an error: (500) Internal Server Error.
at Fidelity.Components.RequestManager.Request.b()
at Fidelity.Components.RequestManager.Request.Run()
2016-02-16 12:55:16,206 [Runner4] ERROR Fidelity.Components.RequestManager.FidelityWebService [(null)] - Exception parsing XML Response: q1:tradeequityconfirm/q1:order/q3:account
System.Xml.XPath.XPathException: Namespace prefix 'q3' is not defined.
at MS.Internal.Xml.XPath.CompiledXpathExpr.UndefinedXsltContext.LookupNamespace(String prefix)
at MS.Internal.Xml.XPath.BaseAxisQuery.SetXsltContext(XsltContext context)
at MS.Internal.Xml.XPath.CompiledXpathExpr.SetContext(IXmlNamespaceResolver nsResolver)
at System.Xml.XmlNode.SelectSingleNode(String xpath, XmlNamespaceManager nsmgr)
at Fidelity.Components.RequestManager.FidelityWebService.TryParseSingleNode(XmlNode theNode, String& rstr, String xpath, XmlNamespaceManager nsm)
2016-02-16 12:55:16,206 [Runner4] ERROR Fidelity.Components.RequestManager.FidelityWebService [(null)] - Exception parsing XML Response
System.Xml.XPath.XPathException: Namespace prefix 'q1' is not defined.
at MS.Internal.Xml.XPath.CompiledXpathExpr.UndefinedXsltContext.LookupNamespace(String prefix)
at MS.Internal.Xml.XPath.BaseAxisQuery.SetXsltContext(XsltContext context)
at MS.Internal.Xml.XPath.CompiledXpathExpr.SetContext(IXmlNamespaceResolver nsResolver)
at System.Xml.XmlNode.SelectNodes(String xpath, XmlNamespaceManager nsmgr)
at Fidelity.Components.RequestManager.SimpleTrade.ParseResponse(XmlDocument xml)
2016-02-16 12:55:16,206 [Runner4] ERROR Fidelity.Components.RequestManager.RequestManager [(null)] - Null Response object.
2016-02-16 12:55:16,206 [32] ERROR Fidelity.Components.RequestManager.FidelityWebService [(null)] - Failed to fire onTrade event: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index
at System.ThrowHelper.ThrowArgumentOutOfRangeException(ExceptionArgument argument, ExceptionResource resource)
at WealthLab.DataProviders.FidelityBrokerProvider.UpdateOrdersWithResponse(List`1 Response)
--- End of inner exception stack trace ---
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
at System.Delegate.DynamicInvokeImpl(Object[] args)
at Fidelity.Components.RequestManager.Trade.DoCallback()

Using version 6.9.15.0

MohammadRashid, that's exactly the same problem I got with when I upgraded to 6.9.15.0.

QUOTE:
Using version 6.9.15.0

And downgrading to 6.9.12.0 fixes the problem?

I think you should call Fidelity Wealth-Lab Support (1-800-823-0175) and tell them that you are also (you are not alone!) running into the problem with 6.9.15.0. Otherwise they don't know how serious this problem is and they will enforce 6.9.15.0 upgrade soon.

QUOTE:
2016-02-16 12:55:16,186 [Runner4] ERROR Fidelity.Components.RequestManager.Request [(null)] - WebException thrown
System.Net.WebException: The remote server returned an error: (500) Internal Server Error.

This is the error that's relevant to this thread. In an https connection, only a specific server IP address and can decrypt/encrypt a specific client (WLP) IP address' traffic. This is intentional to prevent a man-in-the-middle attack by an eavesdropper or hacker on a secure client/server connection.

Upon snooping the SessionID passed in the client's https header, if load balancer hands off the https request to the wrong web server (with the wrong IP address), that server will just see gibberish because it will be unable to decrypt the secure https payload (meant for a different web server). All the wrong server can do is return an 500 Error (Internal Server Error) because it can't make out the encrypted gibberish. (Yes, it's not the most descriptive error message.)

The machine room engineer needs to be sure the load balancer can "route" all https requests (based on a given https SessionID) to the right server. These SessionIDs are first generated by the Apache web server(s) when the client first logs in with a secure connection so as to identify which server (and its IP address) to use for subsequent secure https requests. The problem is there are many ways to configure how Apache web servers generate the SessionIDs, and the configuration of the load balancer has to jive with the method used.

I've noticed, when trading on multiple accounts, some orders will go through whereas others won't. So if one account gets stuck ordering, try another. (Perhaps each account is authenticated with a different SessionID.) And you can always restart WL with a fresh login (and SessionID) or go back to WLP 6.9.12.0.

If you want to speak to Daniel about the hanging submitted-order problem, he can be reached at (800) 544-7595 (WL tech support).

Just wanted to add that I too am having this issue, and called Fidelity to report it. I would encourage everyone else to do the same.

I called Fidelity. It seems they are not responsive to the problem, but they said they will look into it.

I called Fidelity Support to see if they have any update on this issue but so far the only reliable solution at this moment is to roll back to 6.9.12.0.

I'm still running on 6.9.12.0 for 5 weeks since then and it is working flawlessly without any single order issue seen with 6.9.15.0.
Does anyone get an update on this issue from Fidelity?
Definitely Fidelity hasn't decommissioned the old servers yet that I guess they are working on this issue.

There remains numerous updating problems with WL 6.9.15.0. For example, the fill price on a trailing-stop sell order doesn't get updated; it remains at 0.00 instead. And you can't relogin to WL 6.9.15.0 without restarting the program. They don't know when there will be an update to fix that.

However, they appear to have fixed the major server side issues. So the Submitted-order problem (which fails to become Active) has gone away as far as I can tell. I haven't seen that one for nearly a month on WL 6.9.15.0.

I would stick with WL 6.9.12.0 for now unless you need to use the new options trading features of 8.9.15.0, which I haven't looked at.

QUOTE:
However, they appear to have fixed the major server side issues.

Did you have a chance to confirm with Daniel or Fidelity about the server side issue if they have changed something which may have fixed it?

QUOTE:
Did you ... confirm with ... Fidelity about the server side issue if they have changed something which may have fixed it?

No, but the Submitted-to-Active failure problem was real and reproducible, and now is now gone, so something on the server side was tweaked since the WL client wasn't changed.

When running a server cluster, one constantly tweaks the cluster--that's normal. Typically, those tweaks aren't noticed by end users. And it's likely some of their performance tuning fixed the problem without them even knowing it.

Your next question is, "Is it possible this problem may reappear when the configuration of the cluster is changed?" And the answer is "yes" if the tuning is adjusting the "timing" of operations (say a time-of-life setting of a router table entry for the load balancer) in the servers. Optimal "timing" will always be a function of cluster configuration and load.

One can shift the design to be more "event driven" so it's less time dependent, but that's a discussion for a network engineering forum.

--------

I recently heard from Cone that the next update of the Wealth-Lab client "might" come out late summer. He assured me many of the remaining WL 6.9.15.0 client-server communication problems will be addressed. As I said, I would stay with WL 6.9.12.0 unless you need the new options trading simulation features.

I do not think these problems are fixed at all. The logout and login failure problem still exists as of this morning, so does the stuck submitted order problem. Incidentally, do not use the chart right click "Buy to Cover", that doesn't work properly either. You will find yourself both short and long the stock.

For what it's worth the stuck order problem seems to exist for me in the mornings, not the afternoon.

QUOTE:
One can shift the design to be more "event driven" so it's less time dependent, but that's a discussion for a network engineering forum.

Great idea I second that.

QUOTE:
For what it's worth the stuck order problem seems to exist for me in the mornings, not the afternoon.

Are you seeing the stuck order problem on 6.9.15.0 but not on 6.9.12.0?
I have seen it when I upgraded to 6.9.15.0 but haven't seen it at all after switching back to 6.9.12.0.

The stuck order problem started immediately I upgraded to 6.9.15.0. I did re-install 6.9.12.0 and they went away, but the auto-trade feature does not seem to work OK for me, so had to go back to the latest version. Now I did not have any problems today. I think its an inactivity time out problem, as I was trading quite heavily this morning. Like I said the problem also seems to go away in the afternoon session.

Just to give you an idea as to how much of a pain this problem is. I booted up around 7:00 am yesterday. Set up all me trades for 9:30 am open only to have them all get struck. By the time I rebooted the opportunities were lost. Total disaster!

How long didn't you have the stuck order problem on 6.9.15.0? Only today?
What's the auto-trade issue you got with 6.9.12.0?
I don't use auto-trade feature any more but I'm still curious about it.

Its always been there with 6.9.15.0?

As for autotrading its extremely difficult to get it to work reliably. The biggest problem is the scripts do not always run. This can change minute to minute. In addition, WLP looses sink with your account as it does not always get back the fill information. When this happens it will place orders with the wrong position information and the sell orders will error out. It's a real nightmare quite frankly. We have been working at this for weeks and mostly it has not worked for more than a couple of trades. So few people I think use this feature that its not given much shrift here or by Fidelity.

QUOTE:
WLP looses sink with your account as it does not always get back the fill information.

This is discussed in the Orders chapter, but WLPro trading scripts never has knowledge of the positions in your live accounts. Scripts always run hypothetically.

Generally, this is not a problem because if a script tries to exit a position that you don't own, the order will "error out". You can, however, synchronize to exit the entire position held in an account in Preferences (F12) > Trading. This is useful so as not to leave odd shares in your account due to a change in Position Sizing setting.

Where the out-of-sync condition can be a problem is particularly with limit order scripts. In this case (discussed at length in the User Guide), your script may be running exit logic when a fill did not occur in your live account (because it assumes you will always be filled precisely at the limit price).

Anyway, what was the result with the new and improved script on the "Loses Sync with Portfolio" conversation? (Please respond there.)

QUOTE:
You can, however, synchronize to exit the entire position held in an account in Preferences (F12) > Trading.

Cone:

Perhaps we do not have this set up right. Lets say we buy 1000 shares, which we sell. If no sell info is returned, which happen very often, the next time we buy another 1000 shares, WLP will hypothetically try to sell 2000 shares. This order will error out?

I'm copying your last reply to the other topic because it's off topic here.

Having traded with this issue all week I can say with a high degree of confidence that the stuck order problem is due to WLP timing out due to lack of trade activity, much like the Fidelity website times out.. Is there a way to ask Fertility to switch this off. Its a problem that seems to have occurred with the upgrade to the latest software version. This is clearly a major problem if you are relying on WLP to exit a trade automatically.

Yes, this is a major problem and a deal breaker for me. I'm currently trading with 6.9.12.0 and so far working great for me. If Fidelity cannot fix the stuck submitted issue but enforces upgrading to the newer version, I have to give up trading with WLP. I have been working around many issues on WLP in the past several years but unfortunately I cannot find a solution for this stuck submitted issue yet.

QUOTE:
Having traded with this issue all week I can say with a high degree of confidence that the stuck order problem is due to WLP timing out due to lack of trade activity, much like the Fidelity website times out.

There are a couple approaches to making the association between WL client and a particular web server (in the cluster) "sticky". One method is to create a routing table in the load balancer that would favor passed routes. The catch is setting the routing table Time-of-Life entries properly. Too short and you loose the association between WL client and server, which is fatal for a secure (https) connection (causing the stucked submitted problem). Too long and the load balancer won't balance the load effectively.

QUOTE:
Is there a way to ask Fidelity to switch this off.

Absolutely not for a secure https connection. If you allowed another server to take the original server's place, then you could have a man-in-the-middle attack, which is unacceptable for a secure connection.

1) You could lengthen the Time-of-Life entry setting in the load balancer's routing table--slightly.

2) You could make the load balancer employ SessionIDs generated by the login of the original web server for routing purposes so you wouldn't need the routing table in the first place. This is a time independent solution, which is typically used when users "login" to services since it's more robust. It does require WL to handle https connections correctly, which means the SessionID will need to be passed with each https request just like your web browser handles it. The .NET framework "might" support https connection management; I don't know.

3) There are other options too (like having shared server IP address(es) or server https-security certificates that span IP address ranges). We're getting into a subject that should be discussed in a network engineering forum.

Auto Mouse Clicker software may be helpful. In the "Orders" and "Account Balances and Positions" windows I do automatic clicks on the "Update" buttons every 5 minutes. So far, this seems to have circumvented the stuck order problem which I have experienced many times. I must say that like many of the posters on this topic, I am losing confidence in Fidelity's ability to support auto-trading.

I wonder if anyone could figure out how to automate clicking the "Update" button using AutoIt which is often recommended in this forum.

QUOTE:
Auto Mouse Clicker software may be helpful. In the "Orders" and "Account Balances and Positions" windows I do automatic clicks on the "Update" buttons every 5 minutes.

Yes, that "may" reset the Time-of-Life counter in the load balancer's routing table. It's a workaround to a poorly tuned load balancer.

Honestly, I haven't seen the stuck submitted problem (using WL 6.9.15.0) for over a month and I trade throughout the day, mostly in the morning. But I also have (1) over ten streaming chart windows open and (2) the streaming ticker bar running all during this time. (I have streaming updates turned off for the Account Balances and Positions window; I just hit the update button when I need to update that.)

How much streaming are WL 6.9.15.0 users doing, which are commonly experienced this stuck submitted problem? I'm "guessing" they are doing less than I'm doing.

NOTE: Not all streaming may be helpful in working around the stuck submitted problem. But any streaming that requires a Fidelity login "might" reset the Time-of-Life counter for the particular server that supports other login services such as placing orders. It's that routing table entry that needs to stay "alive" to maintain the secure https logged-in-service connection. Connections to other servers (like Data Manager services, which don't require a login) aren't that important.

It's strange this has been a reoccurring problem for the last six years, and the server managers don't seem to understand it--weird. And it's really not a Wealth-Lab client problem; it's not a problem the developer can fix. Perhaps now that https connections are being used, the problem will get much worse, and the server managers will finally address this problem.

I think the problem is related to orders not being submitted for a period of time rather than mouse clicks. I say this because I can be do code development for a couple of hours only to find that I'm timed out of the orders server.

Perhaps a possible solution to this problem would be to have a strategy submit an "out of the money" limit or stop order every 30 min or so. A "keep alive" strategy as it were. Here's my crack at a possible solution. I've no way of testing it until tomorrow. Perhaps someone here could look it over and make any changes if they see a problem. I'm unsure if the stop order will automatically cancel itself??? I tend only to use market orders in my strategies.

CODE:
Please log in to see this code.

If you just want to place a stop order 2% above the last price, the following script is sufficient -

CODE:
Please log in to see this code.

I'm not sure this theory is correct - I can log in to Fidelity, immediately place an order via the trade ticket, and I get the stuck submitted problem.

QUOTE:
If you just want to place a stop order 2% above the last price, the following script is sufficient -

Cone: The idea of the keep alive strategy is to fire off on order that will be executed every 30 min to stop WLP timing out. Your script does not do that.

QUOTE:
I can log in to Fidelity, immediately place an order via the trade ticket, and I get the stuck submitted problem.

Please call Fidelity Wealth-Lab Support (1-800-823-0175) and report your problem because your symptom sounds even worse.

QUOTE:
Please call Fidelity Wealth-Lab Support (1-800-823-0175) and report your problem because your symptom sounds even worse.

I don't think just reporting the problem will reveal how to fix it, and that's what we need right now. Fidelity already knows about the problem.

What we need is a watchdog timer that reports a useful server-side error message that pegs exactly what's causing the problem so it can be fixed. And reporting "500 Internal Server Error" as the return error isn't cutting it. I'm guessing this error is caused because the load balancer is handing off the encrypted https request to a third-party server rather than the original login server (which can decrypt the secure https packet), but that's a guess. Let's have the WL client report the IP address of the "expected" (original target) server and the actual server that answered to see if they really match. And if they don't, then we know the load balancer is misdirecting secure encrypted https requests.

Just reporting known problems or looking for silly workarounds isn't going to fix anything. We need to troubleshoot the problem so we can prove what's causing it. Right now we are just stabbing in the dark.

And as I said, I have not experienced the stuck submitted problem (on WL 6.9.15.0) for over a month now. Yes, my streaming charts and ticker bar might be keeping the network routing path to the correct server "alive", but that's just a wild guess. We need more documentation (error messages) or it will never get fixed.

@ boreland #55

Sure it does that if you use 30 minute bars.

@Cone

It does work. The only issue is the order stays active for the entire 30min. Is there a way to immediately cancel this order, say on the next bar, programmatically?

I'm going to let this run and report back

@superticker

QUOTE:
looking for silly workarounds isn't going to fix anything

This statement is itself silly. We would all agree that this work around in not a fix, but the problem has been ongoing for a considerable period of time. Allowing WLP to time out and not let through orders is unacceptable as far as I'm concerned. If this does work, then at least I have something to hang my hat on until it does get fixed, but I'm not holding my breath this will ever happen.

I wasn't aware of the 'cancel' requirement. This one will do the Alert/Cancel thing every 30 minutes with a 1-minute bar chart.

CODE:
Please log in to see this code.

@Cone. This strategy works perfectly. No timing out at all this afternoon. The stop order get cancelled on the next minute bar as expected. Many thanks to you.

QUOTE:
The stop order get cancelled on the next minute bar as expected.

How about sell order that will fail immediately due to insufficient fund? If you disable Portfolio Sync, sell orders against symbols which you don't own in your account will fail immediately on Fidelity server. I think sell order works better than buy order.

Sure that'd work too. You'd just have to create a dummy Position to sell. It just depends if you like seeing errors or not ;).
Anyway, 6 of one, half dozen of another.

I used the above script for several days now and it works great. No more time outs,so no more stuck orders.

QUOTE:
No more time outs,so no more stuck orders.

It sounds promising workaround for this stuck submitted issue.
However, I wonder if the same workaround works for kcbars because he got the stuck submitted order immediately after logging into Fidelity.

@kazuna He did not get back to us as to whether he still has this problem. I'm inclined to think it's a Fidelity permissions issue, rather than this problem.

Cone, is Fidelity saying anything about this workaround?

Hi everyone. So it's been several weeks since I posted the idea of this work around. If I remember to enable the strategy, I never see the stuck order problem, which means I can leave my PC for a couple of hours and know that I can immediately fire off an order. I wonder if Fidelity will ever get around to actually fixing the root problem? But there are of coarse other serious issues, like not being able to cover a short order directly off of a chart. Oh well you can not have everything I guess.

@boreland What's the "not being able to cover a short order" issue?

I know WLP doesn't have a way to short HTB (Hard To Borrow) stocks.
But I haven't run into an issue when covering a short position.

My WLP 6.9.12 has been working perfect until this morning but one of my buy order just got stack submitted.
I have never had anything like this with WLP 6.9.12 but only 6.9.15 which I downgraded from due to the issue.
Did Fidelity do anything with the old servers?
Did anyone else ever get stack submitted order with 6.9.12?

Looks like these stuck submitted issues I got with 6.9.12 are different than the issues with 6.9.15.

I have seen "(500) Internal Server Error." error log all the time when I got the stuck submitted issue with 6.9.15.
I can find nothing in the log at around the time when I got the stuck submitted issue with 6.9.12.
It was exactly when FOMC statement was released at 11am on June 15.
Probably the heavy network traffic caused Fidelity server to drop some others thus no error log but the orders merely disappeared in the NET.

By the way, does anyone have any update on this stuck submitted issue with 6.9.15?
As 6.9.12 still works fine, I'm guessing Fidelity hasn't figured out the solution yet.

The stuck submitted problem with Wealth-Lab 6.9.15.0 is still alive and well; however, it happens very rarely for me. See screenshot. I'm guessing this is really a server side problem, although the Wealth-Lab client should be able to escape from any bad server behavior.

1) Is there a way to delete these stuck submitted orders (without closing the entire WL application)?

2) Will WL 6.9.15.0 be updated soon (when?) so it can better handle bad server behavior?

I'm still running on 6.9.12 due to the new stuck submitted problem introduced on 6.9.15.
According to the reports above, placing dummy orders every 30 minutes would mitigate the new stuck submitted problem but I'm not comfortable switching to 6.9.15 just yet. Have you tired that?

However, there are old stuck submitted problems on 6.9.12 and earlier. One problem I have confirmed is when the orders placed back to back in a very short period of time (~ 4 seconds). From my personal experiments, orders placed at least 5 seconds apart won't get the problem.

I have also observed other stuck submitted problem that was when the market moved very wild (e.g. FOMC statement).

QUOTE:
According to the reports above, placing dummy orders every 30 minutes would mitigate the new stuck submitted problem but I'm not comfortable switching to 6.9.15 just yet. Have you tired that?

I'm sure that will work because that effort will prevent the entries in the load balancer's routing table from going stale. Likewise, if you place orders frequently enough in groups, that also prevents those routing table entries from going stale.

Remember, for an https connection, the keys to the data stream encryption lie with a specific web server. Typically, for a logged in session, communications is kept consistent through a SessionID negotiated between client and server. With that SessionID, the load balancer will direct your https requests to the same web server since it's the only one that can decrypt your https packets. But in the WL 6.9.15 design, they aren't using SessionIDs. Therefore, the load balancer uses a routing table instead to try to give you the same web server. But the time-of-life of these routing table entries is limited, so if your entry is dropped, then you're routed to a difference web server, which can't decrypt your secure packets. And you get the web server 500 error because the new web server can't use your packets.

The solution is to use SessionIDs like every other https site does for logged-in, secure connections. They just haven't done that yet.

Honestly, I only run into the problem about once a month or so. I'm guessing the machine room engineer has increased the time-of-life period on the load balancer's routing table. That's a stop gap measure, but it also makes balancing the load more of a problem. Using SessionIDs would be the preferred choice just like your web browser does when it connects via https.

Hello. Just wanted to report that I've been using the most current version of the software for the past month or so without the stuck submitted problem. Until this morning. For whatever reason, through wealth-lab today all orders showed as stuck submitted. In the first instance, I attempted to send 4 buy orders simultaneously. All showed stuck submitted in wealth-lab, and none went through to Fidelity (via ATP). Then I restarted and attempted to place the orders again, this time one at a time. Each showed stuck submitted in wealth-lab, but this time they did all go through to Fidelity individually. This makes things very difficult, especially if you want to trade intraday through Wealth-Lab.. Thanks.

QUOTE:
... I restarted [Wealth-Lab] and attempted to place the orders again, this time one at a time. Each showed stuck submitted in Wealth-Lab, but this time they did all go through to Fidelity individually.

I've been having the same problem. Orders posted to the WL Orders window get stuck at "Submitted," but they are filled successfully when you check their website.

I'm "guessing" someone misconfigured the machine room firewall so conformation packets from the Fidelity WL Orders server back to the WL client get blocked. But your orders are still going through okay. I called Fidelity about the problem, so they know about it.

UPDATE: This particular "stuck submitted" problem (discovered this week) in the above paragraph has been fixed today, 3/2/2017.

@superticker
In Post #35 you mentioned that "the fill price on a trailing-stop sell order doesn't get updated". We're working on that right now and cannot duplicate it. Can you provide a procedure, data, or screenshots that would help us?

QUOTE:
"the fill price on a trailing-stop sell order doesn't get updated".... Can you provide a procedure, data, or screenshots

If you can't reproduce this easily, then we need to talk. I'll open a trouble ticket and send you a screenshot. We can go from there.

Thanks for working on the problem.

Version 6.9.17.0 Just came out. Do you know if it fixes the problem?

Thanks

Per the User Guide, New and Noteworthy > What's Changed...

Orders Tool and Trading
· Covering short orders were incorrectly being submitted as new long positions in Fidelity trading.
· Fills, partial fills, and execution prices were not consistently reflected in the Order status.
· Potential fix for Wealth-Lab Pro "Stuck Submitted" orders implemented in Fidelity backend

Accounts Tool
· Closed accounts were erroneously appearing in the account list.
· Certain types of alternate brokerage accounts (BrokerageFlex, etc.) were not showing up in the account list.

Miscellaneous
· Various login instability issues.
· Improved application security.

I read that as a "low confidence" chance of fixing the Stuck Submitted issue, but there's a chance nevertheless (implemented on the server end) .

Hi. Just wanted to report that the new upgrade, at least for me, does not fix the stuck submitted issue. Thanks.

Thanks for reporting kc, and sorry :(

The new Version didn't fix the problem for me either. Now I am getting new delay 15 problem, which is reported somewhere else.

I'm curious is anyone in the wealth-lab community uses Fidelity and is able to trade intraday and get around the "stuck submitted" issue? I have found that this has made intraday trading with Fidelity virtually impossible (unless you go back several versions, which comes with other issues).

Along the same lines, I don't believe there is another broker that integrates with wealth-lab and would place orders automatically, but if I am wrong, please let me know. I am absolutely ready to change brokers, even though I've been with Fidelity more than twenty years.

Wanted to update this and let you know I talked a great deal with Fidelity today. As I think most know, they are aware of the stuck submitted issue, but I get the impression that it is WAY down the priority list. Reading through the lines, there are simply not enough wealth-lab users for them to spend the resources on the problem. Apparently I'm about the only user who tries to use wealth-lab on an intraday basis, and it simply does not integrate with Fidelity correctly. They said the first time I reported the issue was in 2010, and after 7 years I can't imagine the issue will ever be fixed.

Nutshell: the problem is that we cannot duplicate the problem! It's discouraging, I know.

However, the fact is that Fidelity has actually spent lot of resources chasing the problem over the years, and I even dedicated a machine to it all this week again. Unfortunately all code reviews and attempts to duplicate it continue leading to dead ends. I'm discouraged too, but we're not throwing in the towel.

QUOTE:
Apparently I'm about the only user who tries to use wealth-lab on an intraday basis

So you're the one! :D

Thanks for responding. Like I told them, I'm happy to have someone get on my computer and watch it happen. I can make it happen anytime, because it happens 100% of the time. All I have to do is run a simple buy limit strategy on one symbol, and it stops on submitted. I'm glad to help, just let me know if I can.

I auto trade many times a day without the problem. I use the code suggested on this page to basically put a fake order every 30 minutes. However, you must wait until about 15 minutes before market open to log in.
Here is the code for 5-minutes intervals:

CODE:
Please log in to see this code.

If I don't do that then I get the stuck problem 100% of the times.

I daytrade on WLP everyday and I used to having the "Stuck Submitted" issue almost 100% of the time on 6.9.15.
Since then I have been using 6.9.12 without having the issue except only once when FOMC statement was released.

But at that time I wanted to be prepared for the "upgrade enforcement", I went ahead and implemented the 30 minutes dummy order workaround.
I have been testing it with 6.9.12 for months until last week Fidelity finally enforced the upgrade.
Fortunately, I'm not having any issue with 6.9.17 but I don't know if it is due to the 30 minutes dummy order workaround.

Apparently MohammadRashid confirmed that the issue still exists on 6.9.17 and the dummy order fixes it.

Thanks for the responses. I will try the code above, but I doubt it will help my problem since orders get stuck right out of the box, regardless of whether there has been activity or not. If I'm using strategy monitor, and leave it running after the order gets stuck, when it tries to cancel and place another order at the next bar, I get an error that says "unable to place order when previous is in submitted state." (Or something like that - I'm paraphrasing.) Thanks.

@kcbars
You get Stuck Submitted orders 100% of the time even at the beginning of the session? I'll certainly be interested to know if dummy orders fix it for you.

Yes, correct. I'll let you know.

Yes, as I suspected, the dummy order gets stuck submitted, then when it comes around again, it says "0 Errors: Can't replace an order in submitted status." Thanks.

How about SellAtMarket order (against a symbol you don't own it in your account) instead of BuyAtStop order?
That's what I do in my strategy and I get a response from Fidelity server immediately:

QUOTE:
The position to Sell was not found or not available. Please review the following:

But I know at least it is reaching to the server.

This seems to be the problem.. When 'Stuck Submitted' is observed, the order does not make it to the server - at least I am not aware of a case in which a 'SS' order was accepted and placed on the server side.

kcbars, please create a new "Stuck Submitted Ticket" and stand by. I'll probably take you up on the offer to help. Thank you.

====

For anyone participating here, please post a nearby city and state from where you're logging in as well as a ball-park percentage of Stuck Submitted orders. Maybe the survey will reveal something!

QUOTE:
This seems to be the problem.. When 'Stuck Submitted' is observed, the order does not make it to the server - at least I am not aware of a case in which a 'SS' order was accepted and placed on the server side.

Just to clarify, the order does not make it to the "correct" https server which can successfully decrypt the WL client https packet payload.

Rather, the order is sent to the wrong https server, which cannot decrypt the payload, so it replies with a generic "Internal Error 500" in the WL debug log. That is, it's make it to some https server okay, just not the right one. In other words, it's a load balancer routing problem.

I don't have this stuck-submitted problem typically, but I constantly have a streaming Quotes window with 30 positions going in my Wealth-Lab session, which likely protects me from load balancer routing table time outs. It's an alternative to doing the 30 minute dummy trades. Certain types of streaming WL windows should protect you from the load balancer dropping your entry in its "sticky" routing table (due to lack of recent network usage).

Technical note: There are numerous ways to setup load balancing in a https server cluster. One alternative method is to let the Ethernet switch perform the https-server routing by MAC address instead of the load balancer perform the routing by the IP address. This approach is commonly used when all the https servers share the same IP address. In this case, the problem rides in the Ethernet switch configuration rather than the load balancer config. My point is the specifics of my above comments need to be take with a grain of salt depending on how the server cluster is configured. (But machine room guys already knew that.)

However, kcbars' stuck-submitted problem appears to be different. It's something else altogether. kcbars, do you see the "Internal Error 500" at all in your Wealth-Lab debug log? I'm guessing you don't because your problem is totally different. I'm guessing your WL orders aren't even making it to any https server, not even the wrong one (Error 500). Perhaps someone should review your WL debug logs immediately after you have submit an order to see what's happening.

I don't see any sort of error message in the logs when this happens (unless there are certain logs that I don't know about). So I think you may be correct.

QUOTE:
I don't see any sort of error message in the logs when this happens (unless there are ... logs that I don't know about).

I'm talking about the %APPDATA%\Fidelity Investments\WealthLabPro\1.0.0.0\Logs\wlp.txt debug log file. It's in your local WealthLab directory in AppData.

In a "classic" stuck submitted problem that's created by miss routing the WL client's https service request, you would expect it to be non-deterministic. That's because sometimes the load balancer's routing table hasn't timed out yet (i.e. the route remains sticky as it should), so the balancer routes the packet to the correct https server which can then decrypt and process request successfully. And you would only get the "Internal Error 500" (in the WL debug log) when the packet goes to the wrong https server, which can't decrypt the packet, so it replies with a nondescript "Error 500".

But the stuck submitted error kcbars describes is deterministic because it fails every time. And the fact he's not seeing the "Error 500" means not even the wrong https server is replying. My guess is that his WL https order requests are not reaching any server. It's a different problem.

So there are two separate stuck submitted problems to investigate here: a (1) non-deterministic one with "Error 500", and a (2) deterministic one with no error message. The latter one is lightly a problem with the WL client itself and can probably be revealed in a detailed WL debug log.

The thing about logs is that they only log what is programmed to log. Apart from the http 500 (which I understand to be a generic catch-all error code) we haven't found a smoking gun in logs for this problem. That said, I like believing some permutation of the load-balancer / external-to-Wealth-Lab hypothesis. It's credible because of the intermittent nature of the problem. But a test has to be set up to test the hypothesis, which is difficult when you can't deterministically duplicate the problem.

Also, kcbars told me that he was able to get to some orders through after placing dummy orders. The fact about Stuck Submitted is that some customers experience more than others - and some never at all!

@kcbars:
The amount of logging is controlled by settings in the WealthLabPro.exe.config file in the installation folder. As long as you haven't made changes to that file (not recommended unless instructed), WLPro is set up for "Error" logging. The log file is C:\Users\[kcbars]\AppData\Roaming\Fidelity Investments\WealthLabPro\1.0.0.0\Logs\wlp.txt.

QUOTE:
[Cone said]... I like believing some permutation of the load-balancer / external-to-Wealth-Lab hypothesis. It's credible because of the intermittent nature of the problem. But a test has to be set up to test the hypothesis,...

So get a second router and setup a bypass (using a secret IP address) to the load balancer (or Ethernet switch, whichever routes for the https server cluster), then have people experiencing problems run a beta test version of WL that connects with that secret IP address (bypass) and see if their problems go away.

Alternatively, the machine room engineer can setup/program a digital network analyzer (i.e. packet sniffer) to snoop the packets for an "Internal Error 500" in their headers. But then he has to determine how stale these offending https connections are that are generation these "Error 500"s. Yes, he can compare successfully routed packets against unsuccessfully routed Error-500 ones for a given WL client IP address, but someone has to give him the IP address of their WL client that's having the problem. (We can't use Cone's IP address [for placing orders] because he's not having the problem. And that's a problem ... sort of.)

QUOTE:
The fact about Stuck Submitted is that some customers experience more than others ...

That's because their WL workspace is setup differently. Some WL users have more streaming windows open than others, so their load balancer routing table entries don't expire. I always keeps a Quotes window open with 30 entries, so I don't experience the problem. But if I closed that Quotes window I would probably loose my place in the routing table due to lack of recent usage. The balancer's routing table only caches the most-recently-used (MRU) entries for managing its sticky connections.

QUOTE:
kcbars ... was able to get to some orders through after placing dummy orders.

So is kcbars stuck submitted problem deterministic or non-deterministic? He just said he didn't see any "Error 500"s after experiencing a stuck submitted issue, which suggest to me he has a deterministic problem.

If his problem is deterministic with no Error 500s, then I think his firewall (or his ISPs firewall) is blocking the port used to communicate with the https orders server. He should move his computer to a different facility that's serviced by a different ISP provider (whos firewall isn't blocking his connections). He should also turn off his own firewall, which might be miss configured if there's a WL installer bug or he didn't install WL under the administrator account.

Also, stuck submitted users should be tracking their TCP/IP usage with TCPview from http://systeminternals.com/. Perhaps someone can indicate the IP name or IP address ranges of the WL https-orders server cluster.

The "30 minutes dummy order workaround" seems not perfect.
I've got the stuck submitted issues at the exactly same time on my two machines.
According to the log, it is "Error 500".

QUOTE:
2017-06-07 07:00:31,547 [Runner0] ERROR Fidelity.Components.RequestManager.Request [(null)] - WebException thrown
System.Net.WebException: The remote server returned an error: (500) Internal Server Error.
at Fidelity.Components.RequestManager.Request.b()
at Fidelity.Components.RequestManager.Request.Run()
2017-06-07 07:00:31,610 [Runner0] ERROR Fidelity.Components.RequestManager.FidelityWebService [(null)] - Exception parsing XML Response: q1:tradeequityconfirm/q1:order/q3:account
System.Xml.XPath.XPathException: Namespace prefix 'q3' is not defined.
at MS.Internal.Xml.XPath.CompiledXpathExpr.UndefinedXsltContext.LookupNamespace(String prefix)
at MS.Internal.Xml.XPath.BaseAxisQuery.SetXsltContext(XsltContext context)
at MS.Internal.Xml.XPath.CompiledXpathExpr.SetContext(IXmlNamespaceResolver nsResolver)
at System.Xml.XmlNode.SelectSingleNode(String xpath, XmlNamespaceManager nsmgr)
at Fidelity.Components.RequestManager.FidelityWebService.TryParseSingleNode(XmlNode theNode, String& rstr, String xpath, XmlNamespaceManager nsm)
2017-06-07 07:00:31,610 [Runner0] ERROR Fidelity.Components.RequestManager.FidelityWebService [(null)] - Exception parsing XML Response
System.Xml.XPath.XPathException: Namespace prefix 'q1' is not defined.
at MS.Internal.Xml.XPath.CompiledXpathExpr.UndefinedXsltContext.LookupNamespace(String prefix)
at MS.Internal.Xml.XPath.BaseAxisQuery.SetXsltContext(XsltContext context)
at MS.Internal.Xml.XPath.CompiledXpathExpr.SetContext(IXmlNamespaceResolver nsResolver)
at System.Xml.XmlNode.SelectNodes(String xpath, XmlNamespaceManager nsmgr)
at Fidelity.Components.RequestManager.SimpleTrade.ParseResponse(XmlDocument xml)
2017-06-07 07:00:31,610 [Runner0] ERROR Fidelity.Components.RequestManager.RequestManager [(null)] - Null Response object.
2017-06-07 07:00:31,610 [28] ERROR Fidelity.Components.RequestManager.FidelityWebService [(null)] - Failed to fire onTrade event: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index
at System.ThrowHelper.ThrowArgumentOutOfRangeException(ExceptionArgument argument, ExceptionResource resource)
at WealthLab.DataProviders.FidelityBrokerProvider.UpdateOrdersWithResponse(List`1 Response)
--- End of inner exception stack trace ---
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
at System.Delegate.DynamicInvokeImpl(Object[] args)
at Fidelity.Components.RequestManager.Trade.DoCallback()

Just to add a bit to the last few posts, what I get is error 500 - at the time I just didn't know how to look at the correct log. I get the same issues on two separate computers at two completely different locations (behind different firewalls, ISP's, etc). Yesterday I placed a market order manually through the order bar, it went through. Immediately after I tried to place a limit sell order through the order bar, and it was stuck submitted. I would think it has to have something to do with the way the orders get routed.

QUOTE:
... I placed a market order manually through the order bar, it went through. Immediately after I tried to place a limit sell order through the order bar, and it was stuck submitted.

The words "immediately after" caught my eye. If both orders are being routed through the same load balancer, then this isn't a time-of-life problem with the routing table entries of the load balancer; it's something else.

My "guess" is that the machine room is trying to route orders such that it's not going through the same load balancer all the time, and that's causing all the confusion. The two (or more) load balancers aren't sharing the same routing table, and they need to be to work in dual mode. As a result, the two balancers are routing requests differently to different https servers. That will work with http, but not https.

They really need to get a load balancer consultant out to configure their balancers correctly, or they need to use the same load balancer for routing all the https-type requests. And all this routing-table complexity could be avoided altogether if they simply switched from using sticky connections with their load balancers and used SessionIDs (generated by the https servers) to direct the routing instead so any balancer could route any https request to the right https server just by looking at its SessionID.

The SessionID approach is more complicated to setup (You need to program the https servers to generate them.), but it will balance the load better than using sticky connections. It's the preferred approach when users are "logged in" to the https servers, which is always the case with the Wealth-Lab client.

Hi kcbars,

QUOTE:
Immediately after I tried to place a limit sell order through the order bar, and it was stuck submitted.

That's a difference issue than the stuck submitted issue.

As I mentioned earlier in #73 as:

QUOTE:
when the orders placed back to back in a very short period of time (~ 4 seconds). From my personal experiments, orders placed at least 5 seconds apart won't get the problem.

Based on my experiments, the order has to be placed at least 5 seconds apart. Otherwise, the client side (WLP) not the server side (Fidelity) seems omitting such a order. In this case, there is no "Error 500" recorded in the log because the 2nd order hasn't even been reached at the server.

These two issues "stuck submitted" and "orders placed one after another in a short period of time" are my outstanding two open tickets not resolved for years.

I'm getting a lots of stuck submitted orders with many orders on multiple machines this morning. Anyone else?

It's like every other buy or sell orders are stuck submitted.
The dummy order workaround isn't helping at all.

QUOTE:
... getting a lots of stuck submitted orders with many orders on multiple machines this morning.

The Fidelity servers, including their website, are unusually slow today. This network slowness will exasperate the stuck-submitted problem. I got a stuck-submitted order this morning, and I rarely get these. I'm running WLP 6.9.18.0 on Windows 7.

I didn't know about the network issue going on Fidelity but it explains this problem.

I just even got this unusual dialog popup right after restarting WLP:

QUOTE:
After you view the following Nasdaq Agreement, you will need to restart Wealth-Lab Pro.
Close Nasdaq User Agreement browser window!!!
Please restart Wealth-Lab Pro.

Apparently Fidelity is really having a problem on their side.

The fix of periodically submitting an order only seems to work if you have Auto Trading enabled.

Is there any fix without Auto Trading enabled?

I also noticed that you can't just Log out of Fidelity and log back in. You receive an error if you do so. You need to completely restart WL. If the fidelity component forces you to disconnect after a period of inactivity perhaps you can have WL reflect that it is logged out and the user just needs to login again without the need to restart the application.

QUOTE:
... noticed that you can't just Log out of Fidelity and log back in.... You need to completely restart WL.

Not true anymore. You need to upgrade to WLP 6.9.18.0.
https://www.wealth-lab.com/Forum/Posts/Wealth-Lab-Pro-6-9-18-is-ready-38879

QUOTE:
The fix of periodically submitting an order only seems to work if you have Auto Trading enabled.

Is this true? (I don't use Auto Trading.) If so, I'm wondering what that suggests about the actual nature of this problem?

If you do not have auto-trading privileges the dummy order workaround is of no value (post #60).

But as some of us have found there are some rules to be followed. I log in after 9.00 am. If you log in earlier you may timeout before the market opens. I've not done any tests to see how early I can, in fact, log-in and not have a problem.

The other issue is if you switch auto-trading off, say to enter manual trade parameters, and forget to go back to live to trading. Then I still timeout which really sucks.

I just switch over to 6.9.18.0 I do not know if the problem still persists.

QUOTE:
Is this true? (I don't use Auto Trading.) If so, I'm wondering what that suggests about the actual nature of this problem?

I don't have auto trading enabled and the general recommendation of placing or updating an order every 35 number of minutes does work.


This has been quiet for some time....
Has there been any follow up on this hugely annoying issue?
Has anybody figured what the maximum interval can be for placing dummy orders?

Stuck Submitted problem is still unresolved. With auto-trading enabled, WLP created 8 entry orders near the open and only 7 executed. One order generated a Stuck Submit error, for $750 in lost profit on the trade.

Today is the second occurrence this year. The first was on January 13 for another $247 in lost profit.

Yes, the stuck submitted problem is still present.
It occurs much less frequently than the past but I still see it once or twice in 1000 to 2000 orders.
When the order gets stuck submitted, the following error is written in Logs\wlp.txt all the time.

QUOTE:
2019-02-11 06:31:10,078 [Runner1] ERROR Fidelity.Components.RequestManager.Request [(null)] - WebException thrown
System.Net.WebException: The remote server returned an error: (500) Internal Server Error.

Error 500 is quite generic but I see that not just you but other customers have also noticed it in the WLP log. Something to keep in mind if this problem is approached some day. There's no ETA to resolve it for now.

When the server returns an error (especially the error 500), I guess your code just bails out from monitoring the order status and therefore the status gets stuck submitted.
I wonder why you don't re-submit the order in case if the server returns an error.

QUOTE:
I wonder why you don't re-submit the order ...

That's what I do. When an order gets stuck, I just re-submit it. Because the problem is nondeterministic, chances are it won't occur again with the same stock.

Although the server end is definitely involved, the WL client side can do more to help. If WL implemented a watchdog timer (see post #57) that would recognize the trade order is in the "submitted state" too long, then: [1] automatically breaks the deadlock on the client end, [2] sends debugging information to its error log describing where in its own code it timed out and [3] how it got there (i.e. an execution trace) so there's a smoking gun, that would be a first step in helping the developer understand the nature of this problem. Until then, what exactly causes this problem will remain a mystery to everyone.

And I still think it's a configuration problem with the load balancer, and not a coding problem with the server. But you can read about that in the discussion above. If it's a load balancer problem, to prove it, the WL error log should include the absolute identity of the server it initially sent the https request to and the absolute identity of the server flagging the 500 internal error. If it's a load balancer problem, the two identities won't match, which is forbidden in a secure connection (https). Understand, in a server cluster, some servers "may" share identical IP addresses (which is okay in particular cluster configurations), so you can't use IP addressing to absolutely identify different servers in such a cluster configuration. Rather than using IP addresses, use the server's Ethernet MAC address instead; they will remain unique.

Good news. This issue is partially fixed for stop or limit orders with a trigger price $1,000 or higher:

Wealth-Lab Pro 6.9.20.7 (07/10/2019) Update

And more work is scheduled to get it fixed for good.

I just experienced another Stuck Submit event. WLP cancelled an open limit order at 3:50 ET successfully, but the replacement market order sent at the same time to flatten the position was apparently not received by Fidelity. WLP shows "Submitted" as the status. I had to close the position manually to prevent carrying it overnight.

This problem still needs more work.

We did not say it's fixed (only partially). It's still listed as an open issue here, marked as "To be partially fixed in WLP 6.9.21":

Open Issues

The wait is not too long. Sure there's work going on. Like @Cone said:

QUOTE:
Many bug fixes are coming very soon, as is standard Fidelity authentication (same as ATP's authentication dialog), and a rewrite of WL's Fidelity broker provider is nearly complete. The proof will be in the pudding, but I'm very hopeful that the next build will bring back confidence in order automation.