This post was written in reference to the following platforms and versions:
Citrix Web Interface 5.4
Citrix NetScaler 9.x
Citrix XenApp 6.5
Microsoft Windows 2008 R2
Step 1: Verify STA / XML servers are responding
Verify the STA URL. The URL should be in the following format: http[s]://[FQDN of STA server]/scripts/ctxsta.dll.
Confirm you can reach the XML service. Start your favorite telnet client. Open a session to the FQDN of your STA server on the port which your XML service is running. You should receive some sort of response, which verifies the XML service is responding.
If steps 1 and 2 are failing, or you are still having issues, check the event log on the STA servers for any issues and check to make sure all your services that are set to automatic are started.
At one customer site, I found an issue with the XML service not responding on one of the servers. I also found other issues on that server, the SID was not randomized causing authentication issues. So I performed a Sysprep on that server to randomize the SID and re-add it to the domain. After the server was back online the XML service was responding correctly. For more detailed information on how to sysprep a server please see section Solution 1: How to generalize a computer SID.
Step 2: Check the Web Interface log on the NetScaler
Start your favorite SSH client.
Open a session to the NetScaler.
Type shell.
Type cd /var/wi/tomcat/logs.
There will be a bunch of files named localhost.[date].log. Find the log file with the most recent date and open it.
Scroll down to the bottom of the file to see the most recent entries and look for any errors or issues.
I found an issue here in one instance with the following error message:
..."Event Log ID: 18001 A communication error occurred while attempting to contact the Access Gateway authentication service at https://[fqdn of Access Gateway]/CitrixAuthService/AuthService.asmx. Check that the authentication service is running. The message reported by the underlying platform was: ; nested exception is: You have not chosen to trust "GTE Cybertrust Global Root", the issuer of the server's security certificate"...
With the above error, the problem was that the Web Interface call back to Access Gateway isn't working because the Java the certificate wasn't being imported to the java trusted store. This was verified when I went to the Web Interface in question on the NetScaler with the Access Gateway attached and the option "Trust SSL Certificates" was selected but when you next through the options you get to the end and you get the following error:
"Import SSL certificate failed. Following command execution failed: ./export_cert.sh"...
So at this point, I needed to manually add the certificate as a trusted certificate in the java store. Please see section Solution 2: How to manually add a trusted certificate in the java store.
Solution 1: How to generalize a computer SID
The steps below will not delete any data on the computer. But, make sure your data is backed up before proceeding.
Remove the computer from the domain.
Navigate to: c:\Windows\System32\sysprep\sysprep.exe.
The System Preparation Tool will open.
Under System Cleanup Action, select Enter System Out-of-Box Experience (OOBE).
Make sure the Generalize checkbox is selected.
Under Shutdown Options, select Reboot.
The server will then reboot and the SID will be generalized.
Go through the normal system setup steps.
Add the server back to the domain.
Solution 2: How to manually add a trusted certificate in the java store
Open your favorite SSH client and open a session to your NetScaler.
Access the shell by typing: shell and press enter.
Backup your existing java store: You can use WinSCP or if you are familiar with a Linux text editor to modify /netscaler/wi/export_cert.sh. Find line STOREPASS=Type in a 6 digit keystore password after STOREPASS= (e.g. STOREPASS=abc123)Save the file.Copy the file: cp /netscaler/wi/export_cert.sh /var/wi/
Modify the rc.netscaler file to make these changes persistent across reboots: You can use WinSCP or if you are familiar with a Linux text editor to modify /netscaler/rc.netscaler. Enter the following line: cp /var/wi/export_cert.sh /netscaler/wi/export_cert.sh.
Add the certificate to your store: Please note, the certificate you need is in the X.509 format, not PEM. So you will need to grab a copy of your certificate in the X.509 format but you do not need to export the private key.Copy the X.509 certificate to your NetScaler in the following directory: /var/wi/java_home/bin. In the SSH shell session, type: cd /var/wi/java_home/bin. Type the following command to import the X.509 certificate: ./keytool -import -trustcacerts -file [certificate file name] -alias [certificate file name] -keystore /var/wi/java_home/lib/security/cacerts. You will be prompted to enter the keystore password which you defined earlier in the export_cert.sh file. When prompted Trust this certificate? [no] type yes.