0

I'm using an ETL software Pentaho (only mildly relevant) -- it's built on Java. I'm getting heap space errors after 80 loops of a process every time --- I can increase the memory allocated to Java -xmx2048 or what have you, but that just "delays" the error and is a band-aid fix.

I have VisualVM and MemoryAnalyzer but am trying to figure out where to find the root causes/ objects/ threads that are causing extraneous objects to persist. I can see in VisualVM that the heap space memory gradually increases over time. I'm not sure what exactly I should be doing to find the cause though. Analyzing the heap dump upon error?

user45867
  • 887
  • 2
  • 17
  • 30
  • http://eclipsesource.com/blogs/2013/01/21/10-tips-for-using-the-eclipse-memory-analyzer/ may be helpful – JP Moresmau May 21 '15 at 15:14
  • I would look into what you're doing in the process that may need memory optimization. I've run PDI for days on end and it never ran out of memory, and that's without increasing xmx. – Brian.D.Myers May 21 '15 at 17:08
  • I'm not certain what would be 'persisting' in the looped job. A REST call is made to fetch a JSON document -- this is fed direct to JSON input -- this is parsed into an insert/ update (x12 copies) execute SQL script. The SQL connections are definitely closing afterwards. .... One thing I do know with Pentaho is that when you 'loop' a job entry within a job .... the job entry log doesn't reset ... it keeps growing and growing since the parent/ root job started. However I'm not certain if it's "the log" that's causing the leak here. That would happen in all long jobs. – user45867 May 22 '15 at 19:16
  • I have increased the memory xmx to 2g, and that seems fine. Oddly, I've had other similar jobs without as much memory overhead (but still a number of loops) --- I honestly couldn't say. There are no javascript steps in use, and the environmental variables number about 12 --- these would not be created during each loop, I don't believe --- an environmental variable exists one time ... I couldn't tell you. The 2gb seems to work ... there is a gradual increase in memory heap use over time ... but somehow it rests, stops increasing rapidly once it hits 1.5g ... who knows whats happenign. – user45867 May 22 '15 at 19:18

2 Answers2

2

There are several tools that deal with this problem: try :

  1. JProfiler
  2. Yourkit

Jprofiler is actually a great tool that tracks down memory leaks in your application. You can fiter by classes or at the package level.

arnabkaycee
  • 1,634
  • 13
  • 26
1

If command-line tools don't frighten you, try aprof: https://github.com/devexperts/aprof

It is very helpful when you have to profile in the production environment by some reason (e.g. you cannot reproduce an issue anywhere else).

Arseny Mayorov
  • 98
  • 7