1

I updated an application that subscribes on MQTT topics and persists received messages in a MongoDB database.
Is use the paho Java MQTT driver and Spring Data MongoDB.

The application works like this:
When a message arrives the message processing is moved to a separate Thread (using ThreadPoolExecutor). Thus arrived messages are confirmed very fast, which I need to not get into trouble with dropped messages because of 'inflight queue full' errors when the system is under temporary heavy load. What might happen, but which I normally not observe, is, that the ThreadPoolExecutor Queue grows rapidly because it can not be handled fast enough.

After updating the system I observe that - especially after the start, but not only - the ThreadPoolExecutor queue grows. The application always manages to catch up with the processing, but there are always times when the queue grows to several 10 thousand tasks. And this with a continuously same load of about 700 messages per second (because of testing).

What I tested so far is:

Java  8 and App build with Java 8  -> works great
Java 11 and App build with Java 8  -> works great (some Illegal reflective access warnings of course)
Java 11 and App build with Java 11 -> makes some problems

The application update contains dependency updates of course, mainly the following

spring-boot-starter-data-mongodb         1.15.12.RELEASE -> 2.2.4.RELEASE
  spring-data-mongodb                    1.10.11.RELEASE -> 2.2.4.RELEASE
  mongodb-driver                                          3.4.3 -> 3.11.2

org.eclipse.paho.client.mqttv3                              1.2.0 -> 1.2.2

MongoDB Server Version is 3.4.24

I suspect that something in the libraries is causing the performance to degrade. Something like the changed WriteConcern in the MongoDB Java Driver like described here (but this is not my problem because I already use the mongo driver 3.4.3 in my Java 8 build application). I just can't find what it can be.

If you have any idea what could be causing this behavior, please let me know. Right now I'm at a loss.

Jason Aller
  • 3,541
  • 28
  • 38
  • 38
FrVaBe
  • 47,963
  • 16
  • 124
  • 157
  • Just to clarify, the problems you describe do not occur with updated dependencies on Java 8, right? – Nicolai Parlog Feb 16 '20 at 20:09
  • Do you manually specify the GC algorithm to use? In Java 11, the default has changed from Parallel to G1. – Sebastian Feb 16 '20 at 20:25
  • Try to change the GC to ParrallelGC - https://stackoverflow.com/questions/46582523/why-there-is-performance-degradation-after-6-hours-of-java-9-g1-work-without-th – Dmitriy Dumanskiy Feb 17 '20 at 08:09
  • I tried changing the GC but the observation did not change. The problems occure with the updated app, where I did (more or less) updated every dependency to a new (~current) version (because of the major spring boot update). It is a strange think that the Java 8 Application version (with old dependencies) runs fine on Java 11. - With the updated frameworkd and dependencies I observe a growing number of entries in the ThreadPoolExecutor queue. The jobs in there persists the messages in the database. Problem must be somewhere there I guess. – FrVaBe Feb 17 '20 at 12:01

1 Answers1

0

It turned out that the update of the paho java MQTT client from version 1.2.0 to 1.2.1 (or also 1.2.2) caused a significant increase of CPU usage.

The issue #754 was created in the GitHub issue tracker for this.

FrVaBe
  • 47,963
  • 16
  • 124
  • 157