2

I have a Worklight 6.1 app on iOS with push notifications. It works fine, until something goes wrong on the Worklight server. All push notifications after that fail until the server is restarted.

The Worklight server is Liberty 8.5.5.0 on Linux x86_64 Using JDK : java-1.7.0-ibm-1.7.0.5.0.x86_64 The Worklight database is Derby (this is a test server)

Everything works great when I start the server.

10 minutes after the server starts, I see: 

[2/20/14 19:39:15:319 CST] 0000003e com.notnoop.apns.internal.ApnsFeedbackConnection             W Failed to retreive invalid devices
java.lang.RuntimeException: javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.j: End user tried to act as a CA

But that doesn't seem to break anything. Push notifications still work. 30 minutes after server start, I see:

[2/20/14 19:59:48:657 CST] 00000061 com.ibm.ws.webcontainer.util.ApplicationErrorUtils           E SRVE0777E: Exception thrown by application class 'org.jboss.resteasy.core.SynchronousDispatcher.handleApplicationException:365'
org.jboss.resteasy.spi.UnhandledException: org.springframework.dao.InvalidDataAccessApiUsageException: Multiple DISTINCT aggregates are not supported at this time. {SELECT COUNT(DISTINCT t0.DEVICE), COUNT(DISTINCT t1.ID) FROM NOTIFICATION_DEVICE t0 INNER JOIN NOTIFICATION_USER t1 ON t0.USERSUBSCRIPTIONID = t1.ID WHERE (t0.APPLICATIONID IN (?) AND t0.PLATFORM = ?)} [code=30000, state=42Z02]; nested exception is <openjpa-1.2.2-r422266:898935 nonfatal user error> org.apache.openjpa.persistence.ArgumentException: Multiple DISTINCT aggregates are not supported at this time. {SELECT COUNT(DISTINCT t0.DEVICE), COUNT(DISTINCT t1.ID) FROM NOTIFICATION_DEVICE t0 INNER JOIN NOTIFICATION_USER t1 ON t0.USERSUBSCRIPTIONID = t1.ID WHERE (t0.APPLICATIONID IN (?) AND t0.PLATFORM = ?)} [code=30000, state=42Z02]

And any call to the push adapter after that results in:

[2/21/14 19:06:44:038 CST] 00000090 com.notnoop.apns.internal.ApnsConnectionImpl                 I Exception while waiting for error code
java.net.SocketException: Socket is closed
    at com.ibm.jsse2.qc.j(qc.java:301)
    at com.ibm.jsse2.e.read(e.java:32)
    at java.io.InputStream.read(InputStream.java:102)
    at com.notnoop.apns.internal.ApnsConnectionImpl$1MonitoringThread.run(ApnsConnectionImpl.java:114)

[2/21/14 19:06:44:650 CST] 00000085 com.notnoop.apns.internal.ApnsConnectionImpl                 I Failed to send message Message(Id=2; Token=499D9813FBC377CCDE787E2749CDA914F826EDF39B0830D4AFEEF7A5D71A1802; Payload={"aps":{"alert":{"body":"You have 4 available messages","action-loc-key":null},"sound":"","badge":4},"payload":"{\"alias\":\"myNotificationPush\"}"})... trying again after delay
javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.j: End user tried to act as a CA

And all the retries after that result in the same failure.

There are some google hits on "End user tried to act as a CA" and "APNS" that suggest this is a JDK 1.5 problem (fixed in 1.6), and that it can be fixed by using the IbmPKIX trust manager, but I am certain that I am using JDK 1.7, and the trust manager in java.security is PKIX

David Dhuyveter
  • 1,416
  • 9
  • 17
  • OK. I thought I had the sequence, but with further testing all I can really say is that sometimes the push through APNS works, and sometimes it fails with the above error. – David Dhuyveter Feb 22 '14 at 03:36
  • notnoop is not working well with JRE 7 AFAIK. Use 6 and observe for any changes. – Idan Adar Feb 22 '14 at 05:43
  • switched to java-x86_64-60 (1.6.0). Now APNS works when the server is started, but then some time later stops working without any errors in the Worklight server log. The notifications just never arrive on the device. GCM keeps working with no issues. – David Dhuyveter Feb 22 '14 at 17:24
  • 1
    Please contacnt JayaKarthik Jayabalan (fellow IBMer and push notifications developer in Worklight). Sounds like this requires investigation. – Idan Adar Feb 22 '14 at 17:31
  • @IdanAdar what is the jayaKarthik jayabalan's stackoverflow id? – BalajiG Nov 19 '14 at 15:21

2 Answers2

4

After upgrading to the Worklight 6.0.0.2 fix pack, we've seen the same issue on Websphere running JDK 7. After some research I found this issue reported for IBM JDK 7. There is an issue with the Entrust certificate that is included in the JDK. I tried the workaround with replacing the cacerts file from the JDK 6 and this worked.

 Replace jre\lib\security\cacerts file in JDK 7 with
 jre\lib\security\cacerts file in JDK 6

http://www-01.ibm.com/support/docview.wss?uid=swg1IV43936

Note: To establish a TLS session with APNs, an Entrust Secure CA root certificate must be installed on the provider’s server. If the server is running OS X, this root certificate is already in the keychain. On other systems, the certificate might not be available. You can download this certificate from the Entrust SSL Certificates website.

https://developer.apple.com/library/ios/documentation/NetworkingInternet/Conceptual/RemoteNotificationsPG/Chapters/CommunicatingWIthAPS.html

Sam Nunnally
  • 2,291
  • 2
  • 19
  • 30
  • That shuts up the error messages. Now APNS push works for a little while after starting the server, and then stops working … at some point … without any errors or messages in the worklight log. The only way to get it going again is to restart the Worklight server. So, same behavior, just less info :-( – David Dhuyveter Feb 25 '14 at 16:25
  • Hmmmm. With the javax.net.ssl.SSLHandshakeException we weren't able to send any APNS at all, and the workaround solved the issue. Sounds like this is another separate issue. No exceptions being thrown at all? – Sam Nunnally Feb 25 '14 at 19:34
  • No, the behavior hasn't changed at all. From the start I was able to send APNS messages right after the Worklight server started (don't ask me how it was working, I would expect the SSL problem to be a complete blocker too. But it _was_ working) Then some time later APNS started working. The only difference was that before the JDK fix, I was seeing errors in the log, and now, nothing. – David Dhuyveter Feb 26 '14 at 14:59
  • @SamNunnally what certificate to be used? – BalajiG Nov 19 '14 at 11:13
  • @BalajiG the cacerts file in the JDK 6 can be used to replace the cacerts file in JDK 7. As of now I believe this has been fixed in the latest version of the JDK 7 from IBM – Sam Nunnally Nov 19 '14 at 15:13
  • @SamNunnally I did it(the cacerts file has been replaced) still, i didnt receive the notification in ios and in android notification is receiving. – BalajiG Nov 19 '14 at 15:19
1

Once again … this took long enough to figure out.

The underlying problem was that the Worklight server was hosted on SoftLayer, and evidently the SoftLayer firewall by default monitors socket connections, and if a connection is idle for ~10 minutes, drops it.

So, the worklight server connected to APN and push worked. Then 10 minutes passed without any push notifications, and the firewall dropped the socket between the Worklight server and the APNS server. All subsequent push requests failed silently because Worklight had a dead socket for the APNS service.

We reconfigured the firewall to allow the connection to APNS to sit idle indefinitely and the problem was resolved.

David Dhuyveter
  • 1,416
  • 9
  • 17