1

I have multiple threads running a natural language processing application.

Occasionally one of my threads dies with an OutOfMemoryException.

I have diagnosed the problem with a reasonable degree of confidence. As per explanation here, garbage collection is taking up all cpu time.

As per my logs:

[INFO] 2016-07-30 13:43:53,442:

Total Memory: 7165968384 (6834.0 MiB)

Max Memory: 7635730432 (7282.0 MiB)

Free Memory: 2296592136 (2190.2009353637695 MiB)

FileioThread currentFile currentFileProcessTime: 4392

Text Analysis Thread currentFile currentFileProcessTime: 244443

PersistenceThread currentFile currentFileProcessTime: 1588

[INFO] 2016-07-30 13:43:53,442: ikoda.jobanalysis.JobAnalysisThread.run(JobAnalysisThread.java:400) JobAnalysisThread run Going to sleep for 30000 millis.

[INFO] 2016-07-30 13:45:40,717: Total Memory: 6028787712 (5749.5 MiB)

Max Memory: 7635730432 (7282.0 MiB)

Free Memory: 5045040128 (4811.3251953125 MiB)

fileioThread currentFile currentFileProcessTime: 73502

Ta Thread currentFile currentFileProcessTime: 351718

persistenceThread currentFile currentFileProcessTime: 38

Key Observations: There seems to be sufficient memory in the JVM. The Text Analysis Thread has spent over 5 minutes analyzing one file (suggesting that either the file is very long or there are some very long sentences in the file). Most likely the garbagecollection is running overtime cleaning up multiple tokens per word in each sentence of each paragraph of a text file.

Current Behaviour A random thread blows up and dies with the outOfmemoryException. (Not necessarily the TextAnalysis Thread). A monitor thread ensures the dead thread really is dead, pauses for 30 seconds then creates a new instance.

Everything continues hunky dory.

Third Party Known Issue An important note here is that the text analysis thread utilizes third party Stanford NLP packages. These packages do create a lot of objects (nx objects per word) and so gc can be extreme at times, but will recover.

Question Creating replacement thread instances appears benign. Also, the program gathers big data, data loss during thread reinantiation is minor and harmless. Putting the program on a faster server will never fully solve the problem. However, pretending OutOfMemoryErrors didn't happen goes against my instincts and training. Can someone suggest some strategies for dealing with this.

Community
  • 1
  • 1
Jake
  • 4,322
  • 6
  • 39
  • 83
  • 1
    Shouldn't your total memory + free memory add up to the max memory ? Your logging sounds suspicious – Dici Jul 31 '16 at 01:47
  • 1
    More importantly, I agree with you, such error should never be ignored. Either you have a memory leak, or are doing crazy stuff such as loading entire files in memory or you should increase your max heap size. – Dici Jul 31 '16 at 01:51
  • As noted, 3rd party package does do crazy stuff. It will create n tokens (java objects) for each word in a text file. ONly when I am done with the entire file can I be sure that all these objects are collected. It's major gc, but permissible given the purposes (and amazing work NLP can do). – Jake Jul 31 '16 at 02:38
  • btw, could be wrong, but I think freemem is how much of total mem is currently available. Maxmem is maximum that can be allocated to JVM. – Jake Jul 31 '16 at 02:40
  • 1
    And do you have to process the file all at once ? Cannot you split it into reasonable chunks of work ? Think about what would happen if your file was twice bigger, then 10 times bigger, then 100 times bigger. If you find that it wouldn't scale, then you should do something about it sooner rather than later. – Dici Jul 31 '16 at 02:44
  • 1
    I already do, actually. But I'm thinking of being a bit more proactive in dumping out each sentence as soon as its been analyzed. Hey thanks for the thoughts. It helps to get input. – Jake Jul 31 '16 at 05:12
  • Yep, if your library won't penalize you for calling it more often you should definitely process the files line by line or by a fixed number of lines at a time – Dici Jul 31 '16 at 05:50
  • @Dici, sorry, one last q. I've been away from programming for 10 years. Back in the day, setting a field level variable to null was considered bad practice as GC would not necessarily pick up the original referenced object. Is this still the case? – Jake Jul 31 '16 at 06:18
  • 1
    It will clear it... eventually. I don't see why it could be a bad practice since keeping a reference to it is even worse GC-wise. At least if you clear the reference correctly you know that it will be GCed at some point – Dici Jul 31 '16 at 11:45
  • The linked question proposed many solutions to tackle the issue. I think you have accumulated old gen with long lived objects. What is the configuration of your machine? post your JVm parameters – Ravindra babu Aug 03 '16 at 03:02

0 Answers0