0

This code gradually consumes memory starting at about 130 MB (due to dependencies) and keeps climbing to 800+ MB before I have to kill it and restart it (as the server runs out of memory).

It's running with OpenJDK 11. I have an older version of this code running on a Java 8 server whose memory usage stays stable and never increases. So I'm not sure if it has to do with the new JDK?

I modified the code here quite a bit to ensure it's as simple as possible - yet still has the problem.

Basic gist - is it queries the database every couple seconds for pending invoices. However, there are no pending invoices (log proves this as well), so it never gets into complex code locations and just continues to repeat every couple seconds.

public static void main(String[] args) {
    ...

    final int interval = Constants.INTERVAL;
    QuickBooksInvoices qbInvoices = new QuickBooksInvoices(filename);
    qbInvoices.testConnection();

    log.log(Level.INFO, "Checking invoices with an interval of " + interval + " seconds...");

    while (isRunning == true) {
        qbInvoices.process();

        try {
            Thread.sleep(interval * 1000);
        } catch (InterruptedException e) {

        }
    }
}

public void process() {
    errorBuffer.clear();  // These are array lists
    successBuffer.clear(); // These are array lists

    try (Connection conn = DriverManager.getConnection(dbURI, dbUser, dbPassword)) {
        ArrayList<com.xxx.quickbooks.model.wdg.Invoice> a = getInvoices(conn);
        OAuthToken token = null;

        if (a.size() > 0) {
            // Never gets here - no results
        }

        for (com.xxx.quickbooks.model.wdg.Invoice invoice : a) {
            // Never gets here - no results
        }
    } catch (Exception e) {        
        writeLog(Level.ERROR, ExceptionUtils.getStackTrace(e));
    }

}

private ArrayList<com.xxx.quickbooks.model.wdg.Invoice> getInvoices(Connection conn) {
    ArrayList<com.xxx.quickbooks.model.wdg.Invoice> invoices = new ArrayList<com.xxx.quickbooks.model.wdg.Invoice>();  

    String sql = 
        "select " +
        "id," +
        "type," +
        "status," +
        "business_partner_id," +
        "invoice_number," +
        "total," +
        "nrc," +
        "szrc," +
        "trans_ts," +
        "warehouse_id," +
        "due_date," +
        "ref_number," +
        "payment_type " +
        "FROM dv_invoice " +
        "WHERE exported_ts is NULL AND exported_msg is NULL ; ";

    try (
        PreparedStatement stmt = conn.prepareStatement(sql);
        ResultSet rs = stmt.executeQuery();
    ) {
        while (rs.next()) {
            // Never gets here - no results
        }
    } catch (SQLException e) {
        writeLog(Level.ERROR, ExceptionUtils.getStackTrace(e));
    }
    return invoices;
}
JustAFellowCoder
  • 300
  • 2
  • 11
  • How much memory did you give Java (`-Xmx` option)? – Andreas Mar 28 '19 at 15:37
  • 2
    Profiler (YourKit, JProfiler) or heap dump at various intervals with analysis tool (https://stackoverflow.com/questions/9154785/how-to-find-memory-leaks-using-visualvm). Is it actually throwing an OOME or is the heap just growing to the threshold it's allowed to and then GC'd away? – Not a JD Mar 28 '19 at 15:38
  • At first glance you never close your `ResultSet rs`, also check if there is nothing inside your `testConnection` that could be consuming resources without closing them – nortontgueno Mar 28 '19 at 15:39
  • Memory growth isn't necessarily a problem, it's not being able to reclaim it that becomes an issue. Have you monitored it over a long period of time, through multiple GC cycles? – Mike Mar 28 '19 at 15:39
  • Standard procedure to pinpoint memory leaks is taking a heap dump, and load it into analysis tools such as eclipse MAT, to find out which objects are being retained. Then, it is usually easy to find the offending code. – meriton Mar 28 '19 at 15:40
  • @ngueno `ResultSet rs` is within try-with-resources, so it is being closed. – rkosegi Mar 28 '19 at 15:41
  • It's using the defaults `java -XX:+PrintCommandLineFlags -version -XX:G1ConcRefinementThreads=8 -XX:GCDrainStackTargetSize=64 -XX:InitialHeapSize=513937728 -XX:MaxHeapSize=8223003648 -XX:+PrintCommandLineFlags -XX:ReservedCodeCacheSize=251658240 -XX:+SegmentedCodeCache -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseG1GC openjdk version "11.0.1" 2018-10-16 OpenJDK Runtime Environment (build 11.0.1+13-Ubuntu-3ubuntu118.04ppa1) OpenJDK 64-Bit Server VM (build 11.0.1+13-Ubuntu-3ubuntu118.04ppa1, mixed mode, sharing)` – Brian Cothran Mar 28 '19 at 15:41
  • It gradually consumes up memory over days - I did try to force through System.gc() on each loop (after thread.sleep) - still an issue. So it appears to be persisting through gc cycles. – Brian Cothran Mar 28 '19 at 15:45
  • I tried profiling in in JProfiler on my defvelopment machine - but running it for hours didn't show any increase in memory usage. The server it's on for production is Ubuntu 18, my dev machine is a Windows environment. – Brian Cothran Mar 28 '19 at 15:47
  • @BrianCothran enforce heap dump on production evn, this will enforce GC to run as well, so you will see if you have leaks on not (by the size of dump simply speaking) – Antoniossss Mar 28 '19 at 15:51
  • @ngueno - The testconnection is a one time run - nothing looks off in it when the jar first starts. I'll comment it out and upload and see if it makes a difference. – Brian Cothran Mar 28 '19 at 15:51
  • @NotaJD - it's just growing - I don't let it get to OOME - I get server alerts about the system running low on memory (<512 MB) at which point I kill and restart it as I don't want to run the risk of another service running out of memory. The other older server we have with Java 8 - memory usage stays constant. I did try with the same java options as the default on the other machine (it uses a different gc as well), however, it doesn't make a difference. – Brian Cothran Mar 28 '19 at 15:54
  • @Antoniossss - I'l try that - dealing with this is a new thing. In the past, I've been able to simply find leaks and resolve them. Haven't had to do heap dumps before, so give me a bit to figure that out... – Brian Cothran Mar 28 '19 at 15:58
  • It could be differences in GC strategy between the JVMs you're running. If it's not actually throwing an OOME, then you're not out of memory. If the heap reveals that active usage is constant, then it's just the JVM doing its thing. – Not a JD Mar 28 '19 at 15:58
  • @NotaJD that was a thought I had. I forced it to use the same gc strategy as an older server (along with lower memory constraints) and got the same result. If it's the JVM doing its thing - that's awful as it's a change, it throws off memory alerts as the server runs low on memory. I assume it'll run to OOME at some point as well - but I suppose I can just sit and watch. – Brian Cothran Mar 28 '19 at 16:15
  • @BrianCothran force GC first, worry later. Maybe you got used heap on level of 20MB but JVM is allocating new memory (cuz why now). It that would be the case, -XX options will do the trick. – Antoniossss Mar 28 '19 at 16:49
  • Peter got the answer - forced the gc, but still had issues. The problem it appears was an older Postgres driver running against a newer Postgres version. It had some leak in the database connection. Thanks for the brainstorming! – Brian Cothran Mar 29 '19 at 21:24

1 Answers1

0

The Driver might produce a memory leak. I had this issue on Unify SQL Databases. Update the driver to the latest version.

See this in postgres

Grim
  • 1,938
  • 10
  • 56
  • 123
  • That's a great thought - it's a postgres db. We've used the old driver for years, and the databases on this server is a different version (newest version of Postgres). I've gone ahead and updated the driver and will see if that makes a difference. – Brian Cothran Mar 28 '19 at 16:11
  • 1
    This was it!!! Super awesome - you have no idea how thrilled I am. This one didn't even cross my mind. But I'm guessing the older driver has some error against newer database versions which causes a leak. – Brian Cothran Mar 29 '19 at 21:21
  • @BrianCothran I like to cite a wise master from the `Tao of programming` : **You dont know the bug because the bug is inside the code you do not know** – Grim Mar 29 '19 at 23:11