-1

My java program runs forever (intentionally); main has a while(true) that does lots of work in reading JSON data from servers and doing lots of internal processing into an embedded h2 database.

there are 2 static functions I have defined outside of main to do the majority of the work: getUrl (perfoms HTTP GET with input parameters) and postUrl (performs HTTP POST with parameters). Both simply request HTTP data from localhost server, and return to the caller a JsonNode object (from jackson json class) that the guts of the while(true) process data from, into h2.

After a few days the java process is mysteriously "Killed"; research shows OOM is most likely culprit. Ive tried using System.gc() at very end of the while(true) but that doesnt help. Here is postUrl (getUrl is nearly identical, postUrl has 2 args input to it, getUrl only needs 1)

  static JsonNode postUrl(String requestType, String postData) throws Exception {
  //use HTTP POST to API and return JSON data
    String output="";
    try {
      URL obj = new URL(post + requestType);
      HttpURLConnection con = (HttpURLConnection) obj.openConnection();
      con.setRequestMethod("POST");
      String urlParameters = urlParmPrefix + requestType + postData;
      con.setDoOutput(true);
      DataOutputStream wr = new DataOutputStream(con.getOutputStream());
      wr.writeBytes(urlParameters);
      wr.flush();
      wr.close();
      BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
      String inputLine;
      StringBuilder response = new StringBuilder();
      while ((inputLine = in.readLine()) != null) {
        response.append(inputLine);
      }
      in.close();
      output = response.toString();
      } catch (Exception postUrlFUNCTION) {
        System.out.println("Function postUrl failed for: " + postData);
        postUrlFUNCTION.printStackTrace(System.out);
      }
    if (arg.equals("debug")) System.out.println(output);
      ObjectMapper mapper = new ObjectMapper();
      return (JsonNode)mapper.readValue(output, JsonNode.class);
  }

These 2 functions are called, literally hundreds of times per second. I can see in HTOP program the RES field growing every minute or so. Its not at all my intention to carry any memory/objects into each interation of that while(true), id like to toss it all, but the System.gc() doesnt seem to help.

is this part of whats causing my issue, or perhaps something else in the while?

Optical Carrier
  • 368
  • 1
  • 4
  • 14
  • 1
    What is the outcome of your profiling session? profiling will be the **only way** to get a meaningful answer! – Timothy Truckle Jul 10 '17 at 15:56
  • Aside: `ObjectMapper` instances are very expensive to construct. Make a static one and reuse it for all `readValue` calls. – Matt Ball Jul 10 '17 at 16:05
  • ok yes thanks for the advice on ObjectMapper. Ive moved it out of those functions and declared it as " public static ObjectMapper mapper = new ObjectMapper();" in the class outside of main and adjusted my 2 functions to use it w.o creating new one. ive also done the same thing to many variables in many of my while/for loops inside that main while(true) loop (but declared them all in main, not in the class). the leak seems to have slowed a great deal. Ive got MAT installed, will start trying to go through that. thanks all for advice here. – Optical Carrier Jul 10 '17 at 20:19

2 Answers2

0

Java will use whatever memory available to it for it's garbage collection:

What are the Xms and Xmx parameters when starting JVMs?

As you described you have a loop that endlessly does something and it does it often, which means you are allocating objects all the time like for example that ObjectMapper and many others.

There is no need to call System.gc() since the garbage collector typically runs always now in the background and is tuned to collect at times that decrease chances of your program being slowed down when collection does happen.

Kristopher Ives
  • 5,838
  • 7
  • 42
  • 67
  • ive not set and Xmx/Xms, though I am now running it with 100M for both values, waiting for it to OOM crash. HTOP shows it starts with about 87K for RES. Its now at 113M but still hasnt crashed for some reason. whenever it does croak on me, Ill load the hprof file (also started the process with (-XX:+HeapDumpOnOutOfMemoryError) into MAT to see whats up. also, i took out that System.gc(). thanks all – Optical Carrier Jul 10 '17 at 20:56
0

Short answer NO. You function looks fine.

Long answer you have to learn to use memory profiling tool lice MAT.

You can configure JVM to save memory dump when it crashes because of OOM. When you get one load it into MAT and analyze count and size of objects. It also can show you chain of references which lead to leaking object.

PS: System.gc() never solution for such problem. it is not recommended to call it.

talex
  • 17,973
  • 3
  • 29
  • 66
  • ok - so i set up my program to run with 20M as Xms and Xmx. it eventually OOM'ed and I opened up my hprof file in MAT. Any way I tell it to present me memory offenders, it gives MVStore, at nearly 80% of the pie chart that MAT gives me. org.h2.mvstore.MVStore. I never use that class directly. I only use PrepareStatemtent, resultset, Statement, and Query. and Im always closing them ater Im done, i went through and checked. H2 bug? – Optical Carrier Jul 12 '17 at 16:11