1

I am using this standard piece of code but it gives out of memory. Tried google guava Bytestream.toArray and even IOUtils.toByteArray but no luck. Purpose is to get the length of the input stream for large binaries for further processing of the input stream. Also there is a constraint of using file system.

byte[] data = new byte[4096];
while ((nRead = is.read(data, 0, data.length)) != -1) {
    buffer.write(data, 0, nRead);
}
Gopal
  • 378
  • 2
  • 12
  • 2
    If you intend to keep a lot of data in memory (which is usually not a good idea), then you need a lot of memory. Either give more memory to the JVM, or come up with a smarter design. – Kayaman Aug 07 '17 at 19:11
  • I am looking for that smarter design :-). One option could have been to save the inputstream on a file system but unfortunately I cannot go with this option. I have a constraint where I need to calculate the lenght of the inputstream and then perform certain action on that inputstream. – Gopal Aug 07 '17 at 19:19
  • If your only goal is to find the total size, then why are you keeping the whole thing in memory? Are you keeping the data you write in buffer for later use? – litelite Aug 07 '17 at 19:20
  • You're going to have to explain better what you're trying to achieve. – Kayaman Aug 07 '17 at 19:20
  • @litelite - Based upon the lenght I need to perform certain operation on the inputstream. Calculating just the size could have been easier. – Gopal Aug 07 '17 at 19:22
  • @Kayaman - There is a large binary that I am getting as inputstream. I want to find the size of that inputstream and perform some action on the inputstream or lets say send that inputstream to another api along with its size. – Gopal Aug 07 '17 at 19:24
  • 1
    If you can't read the stream twice (as is usually the case), then you're just going to have to find some way of storing the bytes. Get more memory, it's cheap. – Kayaman Aug 07 '17 at 19:31
  • Is this a file you are reading? – Rabbit Guy Aug 07 '17 at 19:32
  • @Kayaman - I wanted to check if there is something else than just increasing the memory. – Gopal Aug 07 '17 at 19:35
  • @rabbitguy - This is an inputstream coming in a multipart request. If it would have been a file on a file system somewhere it could have been achieved easily and my constraint is I cannot involve a file system here. – Gopal Aug 07 '17 at 19:37

1 Answers1

1

If reading the file is not an option and it comes from multipart request you could try using content-length header.

Another option is to enforce the client to supply file size for the file so you do not need to load it in memory. Note that this makes sense only if your API client is trusted.

Danylo Zatorsky
  • 5,856
  • 2
  • 25
  • 49
  • content-disposition header is also an option as noted in another stack overflow post https://stackoverflow.com/a/11317380/4374481 – Palamino Aug 07 '17 at 19:46