Critical XenDesktop Broker Bug


Yesterday afternoon we were struck by a bug that left us unable to enumerate applications, effectively resulting in a near 100% system-down condition.  We’ve been working with Citrix on an intermittent inability to log in with little visible error other than ‘All the Citrix XML Services configured for farm Farm1 failed to respond to this XML Service transaction. ‘ in the Storefront log.  When I received a report that “No desktop was available” I immediately duplicated the issue, then contacted the escalation engineer we were working with.  In short order he put together a team on his side to help review the log files of what was going on. The only change(s) that had been made in the environment were that some applications had been copied earlier, but that change had been tested and seemingly worked without error.

Storefront tracing was alread enabled using the commands, and Scout data was submitted

However, despite our best efforts the logins were continuing to fail.  It became apparenty through examining the Delivery controller event logs that this was a new issue, because we were in fact seeing exceptions in the application event log.

 

 

This error returned nothing of use on the web as most references to DALDataStoreException were in different namespaces, like Cds.  After filtering the DC event logs to find the first occurrence of this error my colleague noted that the time was remarkably close to the time that he copied the applications.

I reviewed the “Configuration Logging” section in Studio and it became increasingly clear that link from copying the applications was likely the culprit.

 

Seemingly successful commands

Seemingly successful commands

XenDesktop Broker Error

Errors began appearing immediately

 

So what happened? Was some unsupported method of application copying used?  No, in fact.  The commands were tested on a single application, then when that was successful, repeated for all applications in a delivery group.

 

First,

After launching and testing the app successfully, it was deleted, then the command below was issued.

Reversing this action resolved the errors

 

So what happened?  Did we do something wrong, or unsupported?  Apparently not, according to a post from Citrix staff in the official forums.

 

Yes, the restriction of one delivery group per application is purely cosmetic in Studio’s app publishing wizard.

This was an intentional design, as the multiple association is meant to be an advanced use case. The process for discovering apps from the Start Menu and automatically publishing applications would become much more complex if we supported multiple delivery groups, as the VDA images are likely different between the groups.

The product fully supports publishing apps to multiple delivery groups, but when setting up these associations via PowerShell, it’s up to the administrator to ensure the VDA images are adequately similar so the application can be launched via the same command line in both groups.

Thanks for the feedback on showing the priority–I’ll raise this as an enhancement to the Studio team.

Unfortunately, something went awry with one or more of the applications that was copied in this way.  At this point the precise root cause isn’t known, but if you need to use this functionality, be on the lookout for potential problems!


Leave a Reply

0 thoughts on “Critical XenDesktop Broker Bug

  • Just Bob

    Reminds me of an old web interface issue bug where publishing an app with too long of a name would completely disable the web interface. This was “XenApp 4.6” I believe, and sounds a bit like this bug.

    • Atum Post author

      Thanks for the comment. For what its worth, this specific error seems to be a generic “Database access error”. If the database is down, you’ll see a similar error in the event viewer. In this case, it was some sort of exception on the query results.