4

We have two connections (as a .NET/Windows client) to an IBM MQ endpoint (let's say non-Windows).

We use SSL, which is supported by a pair of certificates (client and server) and correctly labelled in the Windows Certificate Store to ensure we pick the right certificate.

Everything in Queue A works fine.

Queue B has recently stopped working. We now get a 2538 error. Nothing has changed.

  • We keep both sides of both queues the same as we can.
  • We have Wire-traced both Queue A and Queue B and can confirm SSL negotiation is incomplete on Queue B because of a "Connection Reset" by the Client
  • All certificates are valid and in date
  • We've tried with hard-trusted Self-signed certs between parties

It feels like there may be a communication error inhibiting successful SSL handshake.

The problem is exacerbated by the WebSphereMqClient NuGet package we use (8.0.0.7 - latest version) not giving us anything other than a 2538 error on calling:

_mqQueueManager = new MQQueueManager((string)queueManager, (Hashtable)connectionProperties);

Connection Properties being (redacted):

<IbmMqConnection server="ipaddress" 
                         portNumber="1234" 
                         channel="SOME.CHANNEL" 
                         queueManager="QUEUE_MANAGER">
    <SecureConnection enabled="True" 
                      cipherSuite="TLS_RSA_WITH_AES_256_CBC_SHA256" 
                      cipherSpec="TLS_RSA_WITH_AES_256_CBC_SHA256" 
                      peerName="CN=trusted-parties-cn"
                      certificateLabel="ibmwebspheremq"
                       />

Is there anything an expert could give us a clue on? Is there a way to get any diagnostics out of the NuGet black box?

Update 1:

The server is running 8.0.0.9.

Update 2:

@JoshMc suggested I turn on logging.

From this, I learn:

  • It couldn't (rightly) find an mqclient.ini file in the binary folder or my personal folder.
  • Errors start during SSL Authentication, specifically System.ComponentModel.Win32Exception (0x80004005): The credentials supplied to the package were not recognized
  • Followed by the host seemingly disappearing: MQTCPConnection.ConnectSocket(string,string,MQLONG) rc=OK 00000174 10:29:28.917526 5748.1 CompCode: 2, Reason: 2538 AMQ9202: Remote host 'x.x.x.x(y)' not available, retry later.
  • Out of which comes the 2538 CompCode: 2, Reason: 2538
  • Negotiation seems to continue, using appropriate TLS version, etc.
  • Then when trying to get the client (with our private key) certificate, we get System.ComponentModel.Win32Exception (0x80004005): The credentials supplied to the package were not recognized

I have double checked that the user does have permission to the Private Key.

Update 3

Issue found.

The issue was highlighted by the text System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.. On investigation it turns out that the permissions on the Private Key had been removed during a certificate renewal. I know the above says the permissions were confirmed to be there, but for a different certificate. I've left it in to retain history/honesty.

Program.X
  • 7,250
  • 12
  • 49
  • 83
  • Does the not working connection pass via a firewall or VPN tunnel? – JoshMc Sep 10 '19 at 14:58
  • Yes, and the same route as the working connection. We've had firewall and networking teams investigate with no conclusion. We have to assume that these are working. – Program.X Sep 10 '19 at 15:17
  • What error shows up in the queue manager error logs. – JoshMc Sep 10 '19 at 15:41
  • I'm told (the server party is a partner so I don't have direct access) it doesn't get past the the MQIPT (Internet Pass-thru for MQ? https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_9.0.0/com.ibm.mq.ipt.doc/ipt0000_.htm ) and the Queue Manager itself doesn't get involved to do any logs. – Program.X Sep 10 '19 at 16:04
  • What do the MQIPT logs show? – JoshMc Sep 10 '19 at 16:27
  • 1
    If you want to have them turn on a trace for a Managed MQ Client fro NuGet you need to follow the [Using the stand-alone IBM MQ .NET client](https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_9.0.0/com.ibm.mq.dev.doc/q123550_.htm) instructions. Basically you add an entry to the `app.config` file that points to a directory where you create a `trace.config` file that tells it turn turn on the trace. This will usually shed more light on what the client TLS interaction is. – JoshMc Sep 10 '19 at 23:22
  • IBM doesn't support MQ .NET client used via the nuget package. IBM officially doesn't provide any nuget package for the .NET Client.Try using the Redistributable client. – subbaraoc Sep 11 '19 at 05:53
  • 1
    Adding `-Djavax.net.debug=ssl` to the JVM command line args it the mqipt startup can provide useful diagnostic messages as well. – JoshMc Sep 11 '19 at 08:51
  • This may help: https://stackoverflow.com/questions/46364718/2538-error-on-mq-for-ssl-channel-connection/46418392#46418392 – JoshMc Sep 11 '19 at 08:57
  • And this: https://stackoverflow.com/questions/42347768/ibm-mq-net-certificatelabel-cipherspec/42348037#42348037 – JoshMc Sep 11 '19 at 08:59
  • Thanks @JoshMc. Very useful. The logging is just what we need. I'll have a look at the other questions when I get in, too, though I think we've already solved that (getting the SSL to successfully negotiate). It _was_ working, now is not. – Program.X Sep 11 '19 at 17:21
  • Interesting also @subbaraoc, do you mean directly working the the amg*.dlls? Surely that is what the NuGet package is doing anyway? Why should I reinvent the wheel? – Program.X Sep 11 '19 at 17:21
  • Also to note if you use the stock `mqipt.ske` to start your MQIPT you can just set the env variable `MQIPT_JVM_OPTIONS="-Djavax.net.debug=ssl"` before you start MQIPT to get it to pick up that option. Logging should go to the `mqipt.stdout.log` in the `logs` directory. – JoshMc Sep 11 '19 at 18:02
  • By guess is this answer (or another on this question) may help: https://stackoverflow.com/questions/7984945/the-credentials-supplied-to-the-package-were-not-recognized-error-when-authent/7990364#7990364 – JoshMc Sep 12 '19 at 11:48
  • Can you post a sanitized MQIPT route entry? – JoshMc Sep 12 '19 at 11:54
  • HI, seems that a procedural flaw has been discovered when renewing SSL certificates. The logging was critical, thanks for that @JOshMc. That led us straight to the problem. If only that had been found earlier (despite numerous searches). – Program.X Sep 12 '19 at 16:55
  • 1
    I was writing my answer up before you posted this last comment and the answer, hope my answer was helpful and you would consider accepting it. I'll vote your question and answer up as well right now as they are good content. – JoshMc Sep 12 '19 at 17:17

2 Answers2

2

The NuGet package is just the stand-alone amqmdnet.dll that IBM lets you relocate from a full MQ client install or download via the redistributable client zip file.

If you want to have them turn on a trace for a stand-alone Managed .NET MQ Client you need to follow the instructions at Developing applications > Developing .NET applications > Writing and deploying IBM MQ .NET programs > Using the stand-alone IBM MQ .NET client.

You start by adding the following to your app.config and point the value to a directory that you can create a file in.

<appSettings>
<add key="MQTRACECONFIGFILEPATH" value="C:\MQTRACECONFIG" />
</appSettings>

You then create a file called trace.config in the above directory with the following contents where the MQTRACEPATH and MQERRORPATHpoint to a directory that the user running your application can write to.

<?xml version="1.0" encoding="utf-8"?>
<traceSettings>
  <MQTRACELEVEL>2</MQTRACELEVEL>
  <MQTRACEPATH>C:\MQTRACEPATH</MQTRACEPATH>
  <MQERRORPATH>C:\MQERRORLOGPATH</MQERRORPATH>
</traceSettings>

Note that you can toggle the MQTRACELEVEL between 2 and 0 to dynamically turn tracing on or off for a running program.

On the MQIPT side of things if you use the stock mqipt.ske to start your MQIPT instance you can set the env variable MQIPT_JVM_OPTIONS="-Djavax.net.debug=ssl" before you start MQIPT, this will cause java to provide TLS debug logging to mqipt.stdout.log in the logs directory.

JoshMc
  • 10,239
  • 2
  • 19
  • 38
1

Issue was resolved by enabling the logging, as suggested by @JoshMc.

The log entry:

000001C9 11:16:51.194536   2072.1      System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
at IBM.WMQ.Nmqi.MQEncryptedSocket.MakeSecuredConnection()
at IBM.WMQ.Nmqi.MQEncryptedSocket..ctor(NmqiEnvironment env, MQTCPConnection conn, Socket socket, MQChannelDefinition mqcd, MQSSLConfigOptions sslConfigOptions)
at IBM.WMQ.MQTCPConnection.ConnectSocket(String localAddr, String connectionName, Int32 options)

... led us straight to the issue, which was that the credentials used to access the private keys in the Windows Certificate Store were invalid. On investigation, they were not present.

Make sure that you include this step as part of your procedures for renewing SSL certificates.

JoshMc
  • 10,239
  • 2
  • 19
  • 38
Program.X
  • 7,250
  • 12
  • 49
  • 83