0

I had an outofmemory error when launching the batch on the server: below are the details of the error as well as the piece of code written in java, can you help me please , the error is triggered at this line:

rowString = new String(barray, 0, endOfLinePos, charSet).split("\n");

knowing that my file contains 461500 lines and the number of bytes used is 230989152.

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3332)
at java.lang.StringCoding.safeTrim(StringCoding.java:89)
at java.lang.StringCoding.decode(StringCoding.java:230)
at java.lang.String.<init>(String.java:463)
at com.test(Maclasse.java:215)
at com.test(Maclasse.java:181)
at com.test(Maclasse.java:144)
at com.test(Maclasse.java:119)



   try ( FileChannel inputChannel = new FileInputStream("/opt/input.txt").getChannel();)
 {
    MappedByteBuffer buffer = inputChannel.map( FileChannel.MapMode.READ_ONLY, 0L, inputChannel.size());
    buffer.position(0);
    long availableBytes = inputChannel.size();
    int endOfLinePos = (int)availableBytes;
    int get = Math.min(buffer.remaining(), endOfLinePos);
    byte[] barray = new byte[endOfLinePos];
    buffer.get(barray, 0, get);
    Charset charSet = Charset.forName("UTF-8");
    String[] rowString = new String[buffer.remaining()];
   
    rowString = new String(barray, 0, endOfLinePos, charSet).split("\n");   
    Arrays.asList(rowString).stream().map(line -> {
        try ...
Wajih Bouzid
  • 1
  • 1
    This `new String[buffer.remaining()]` doesn't make sense and probably already causes the problem. You don't want an array that holds each byte as an item. The array size is the number of lines, and that final array is created by the `split("\n")` already. So no need to create one in advance. – Andreas Dolk Sep 12 '22 at 13:06
  • 1
    @Andreas Dolk , thanks for your response but it's the same, the same error persists, I removed the line new String[buffer.remaining()]; and replace it directly with: String[] rowString = new String(barray, 0, endOfLinePos, charSet).split("\n"); unfortunately the same error appears on this line – Wajih Bouzid Sep 12 '22 at 13:22
  • Think about what you are doing. You are loading a file in memory that's very large by your own estimation. Then, you load in memory a very large array of strings, and then a *new* `String` object to hold the contents of the byte array. I think Wajid's recommendation should work. Also, allocating more memory, although might work, is not the way to solve these problems (just in case someone suggests it or you were thinking about it). – hfontanez Sep 12 '22 at 13:28
  • You could just use `Files.lines(Path.of("/opt/input.txt"), charSet)` which returns a `Stream` where you can then call `.map(line -> {` etc. on – Lino Sep 12 '22 at 13:40
  • thank you @Lino , thant's true but i need to change Stream wich is actually used with FileChannel because FileChannel and BufferedinputStream are recommended for large files – Wajih Bouzid Sep 12 '22 at 13:59
  • @WajihBouzid as far as I can tell, `Files.lines` uses `FileChannel` internally (with a falback to `BufferedReader`), so you could still use it – Lino Sep 12 '22 at 14:01

0 Answers0