4

I use BufferedReader and FileReader on very big files (~100g).

Here is the code I'm using:

BufferedReader reader = new BufferedReader(new FileReader("file path"));
BufferedWriter writer = new BufferedWriter(new FileWriter("output file"));
String line;
while ((line = reader.readLine()) != null) {
    // check if i need this line, and if i need it, i print it
    writer.write(line);
    writer.newLine();
}
writer.close();
reader.close();

When I run this on my files in the beginning it uses a low amount of memory but slowly the used memory grows (can easily use over than 50GB of RAM).

Why it's like that? And, can I fix it somehow?

stefanobaghino
  • 11,253
  • 4
  • 35
  • 63
ldoroni
  • 629
  • 1
  • 7
  • 18
  • 2
    What are you doing with the lines? – Jon Skeet Sep 13 '16 at 07:24
  • @JonSkeet i update my question. i take each line and split it by tabs (my files are tab-delimited), and check if i need this line, i print it with BufferedWriter to a output file. – ldoroni Sep 13 '16 at 07:29
  • So you're not *saving* it anywhere? With the code you've presented, I'd expect the garbage collector to handle it with no problems... unless you have huge *individual lines* in your files. – Jon Skeet Sep 13 '16 at 07:30
  • @JonSkeet im not saving the rows anywhere, i immediatly print the needed rows to another file – ldoroni Sep 13 '16 at 07:32
  • 2
    And you're not saving any *part* of it? It would help if you could provide a [mcve] here so we could try to reproduce it. (As well as a rough idea of what's in your files, so we can generate something similar.) – Jon Skeet Sep 13 '16 at 07:37
  • 2
    50g of RAM? How much heap do you have configured there? And what exactly do you mean by "used memory"? – Marko Topolnik Sep 13 '16 at 08:15
  • Take a heap dump and analyze it :) – Steffen Harbich Sep 13 '16 at 08:24
  • Maybe your files just don't use line breaks? – Mark Sep 13 '16 at 08:56
  • This code does not leak memory. Clearly the problem is in the part of the code you haven't posted. – user207421 Sep 13 '16 at 09:40
  • 1
    Don't edit answers/solution into your question, instead create an answer post. Then don't write "solved" into the title, accept the answer, which solved your issue, instead. And also explain how you used the `-Xmx` parameter. Maybe the passed value was set incorrectly. – Tom Sep 13 '16 at 12:53
  • Hi Idorini, have a look at this question for reading a large file. Add write logic to solve your problem. http://stackoverflow.com/questions/18508836/java-outofmemoryerror-in-reading-a-large-text-file/31724120#31724120 – Ravindra babu Sep 13 '16 at 15:49

1 Answers1

0

I see that when i run java program on linux from command line with -Xmx parameter, for some reason it make my java program to need to alot of RAM.

From when i remove -Xmx parameter, the java program use low memory.

This was my old command:

java -Xmx350g -jar MyJar.jar

This is my new command:

java -jar MyJar.jar
ldoroni
  • 629
  • 1
  • 7
  • 18
  • I didn't knew it was even possible to allocate 350g to the JVM ... Also, maybe the garbage collector only starts after a certain % of the space is taken ?.. – Asoub Sep 13 '16 at 13:21
  • 1
    @Asoub I run it on strong server, so when you have 512g of RAM, it shouldnt be a problem to allocate 350g. And i think you may right about the garbage collector, but i dont really know. – ldoroni Sep 13 '16 at 13:35