How to convert List to the csv byte array safely?

Question

Initially I had the following code:

Attempt 1

try (var output = new ByteArrayOutputStream();
     var printer = new CSVPrinter(new OutputStreamWriter(output), CSVFormat.DEFAULT)) {
   printer.printRecord(EMAIL);
   for (MyBean mb : items) {
     printer.printRecord(mb.getEmail());
   }
   externalHttpCall(output.toByteArray());
}

Here I found out that sometimes the byte array is not written fully.

I understand that it is because of the fact that stream is not flushed during externalHttpCall invocations.

To fix it I wrote the following:

Attempt 2

try (var output = new ByteArrayOutputStream();
     var printer = new CSVPrinter(new OutputStreamWriter(output), CSVFormat.DEFAULT)) {
  printer.printRecord(EMAIL);
  for (MyBean mb : items) {
    printer.printRecord(mb.getEmail());
  }
  printer.flush();
  log.info("Printer was flushed");

  externalHttpCall(output.toByteArray());
}

It solved the problem, but here I was lost in a thought that it is really bad idea to close stream only after externalHttpCall. So I came up with the following solution:

Attempt 3

externalHttpCall(convertToByteArray(items);

public byte[] convertToByteArray(List<MyBean> items){
  try (var output = new ByteArrayOutputStream();
       var printer = new CSVPrinter(new OutputStreamWriter(output), CSVFormat.DEFAULT)) {
    printer.printRecord(EMAIL);
    for (MyBean mb : items) {
      printer.printRecord(mb.getEmail());
    }
    return output.toByteArray();    
  }
}

I expected that flush will happen before stream close. But based on my experiments it doesn't work. Looks like it happens because of flush happens before stream close but after toByteArray invocation.

How could I fix it?

I think you missed flush() somewhere inside externalHttpCall. I expect that array prepared by your code is OK — Anton Straka, Nov 27 '19 at 21:18
@Anton Straka I don't think that making a flush is responsibility of *externalHttpCall** — gstackoverflow, Nov 28 '19 at 07:20
Any reason why you don't call flush before toByteArray in convertToByteArray? — dpr, Dec 05 '19 at 13:53
What is this supposed to mean? You didn’t see code that calls flush, so you think it should work without? — dpr, Dec 05 '19 at 21:58

dpr · Accepted Answer · 2019-12-09T08:58:22.097

Given the three code snippets in the question I'd assume that this should work:

externalHttpCall(convertToByteArray(items);

public byte[] convertToByteArray(List<MyBean> items){
  try (var output = new ByteArrayOutputStream();
       var printer = new CSVPrinter(new OutputStreamWriter(output), CSVFormat.DEFAULT)) {
    printer.printRecord(EMAIL);
    for (MyBean mb : items) {
      printer.printRecord(mb.getEmail());
    }
    printer.flush()
    log.info("Printer was flushed");

    return output.toByteArray();
  }
}

Depending on the CSVFormat the CSVPrinter is flushed automatically on close (CSVFormat.DEFAULT will not be flushed automatically...). You can use CSVFormat's builder like pattern to make the format flush on close with CSVFormat.DEFAULT.withAutoFlush(true) (thanks to @PetrBodnár for this hint). This will however probably make no difference in the above example.

If you translate the try-with-resource to the actual call order you will get something like this:

var output = new ByteArrayOutputStream();
var printer = new CSVPrinter(new OutputStreamWriter(output), CSVFormat.DEFAULT)
printer.printRecord(EMAIL);
...
var result = output.toByteArray();
printer.close();  // might call flush
output.close();
return result;

As the close operations will be called in the finally-block, they will take place after creation of the byte array. If flush is needed, you will need to do it prior to calling toByteArray.

To make the answer complete, it could be added that changing `CSVFormat.DEFAULT` to `CSVFormat.DEFAULT.withAutoFlush(true)` will take care of the flush on `close()`. Still, this won't help by itself and explicit flush is necessary, unless `output.toByteArray()` is taken out of the `try` block. — Petr Bodnár, Dec 08 '19 at 19:48
@PetrBodnár, thanks for this hint. I added this information to the answer. — dpr, Dec 09 '19 at 08:58

score 0 · Answer 2 · answered Dec 06 '19 at 10:13

0

The following is a correct usage:

var output = new ByteArrayOutputStream();
try (var printer = new CSVPrinter(
            new OutputStreamWriter(output, StandardCharsets.UTF_8), CSVFormat.DEFAULT)) {
    printer.printRecord(EMAIL);
    for (MyBean mb : items) {
        printer.printRecord(mb.getEmail());
    }
}
// Everything flushed and closed.
externalHttpCall(output.toByteArray());

This error behavior might stem from something else.

For instance the externalHttpCall not flushing . Or writing the bytes as text (using a Writer i.o. OutputStream), and expecting UTF-8, whose multi-byte sequences are brittle, maybe raising an exception. Or setting the HTTP header Content-Length wrong, as String.length().

An other cause: items containing a null or getEmail throwing an exception that is not detected.

Available is also:

String s = output.toString(StandardCharsets.UTF_8);

answered Dec 06 '19 at 10:13

Joop Eggen

107,315
7
83
138

Calling `toByteArray` on a closed `ByteArrayOutputStream` works? – dpr Dec 06 '19 at 10:56
Ok, it does. As `ByteArrayOutputStream.close()` is a noop. – dpr Dec 06 '19 at 11:31
1

@dpr actually I always feel slightly guilty as I often do not use close, and would have expected a close call _before_ `toByteArray()`. But yeah, a noop, sparing us many bugs. – Joop Eggen Dec 06 '19 at 11:39
@JoopEggen, the 1st code you give is probably correct - you took the `output.toByteArray()` out of the `try` block, which should cause the needed close -> flush. I would recommend to somewhat revise the rest of your answer in order to avoid downvotes though (e. g. take externalHttpCall - it just uses the resulting byte array). See the answer from @dpr, it explains the problem in a better way... – Petr Bodnár Dec 08 '19 at 19:57
**This error behavior might stem from something else.** Which error behaviour you mean? – gstackoverflow Dec 09 '19 at 09:43
After Petr's comment I am not sure anymore, so for now I would debug `output.toByteArray()`, writing it to file, maybe the length already suffices, a unittest maybe. But as soon as the unlucky closing is resolved (as in my code), the error can only be in other code parts dealing with this data. Not much to go wrong. – Joop Eggen Dec 09 '19 at 10:01
I guess the error behavior really means that the byte array doesn't always contain the full csv output. Anyway, @gstackoverflow, do you still consider your question unanswered even by the 1st answer? If so, then why? – Petr Bodnár Dec 09 '19 at 18:36
**For instance the externalHttpCall not flushing** What does it mean? Should it flush? Byte array? – gstackoverflow Dec 11 '19 at 22:17
Sorry, I am too uninformed about your code; it seems to lead to no solution. – Joop Eggen Dec 12 '19 at 09:43

How to convert List to the csv byte array safely?

Attempt 1

Attempt 2

Attempt 3

2 Answers2