How can I write data to csv in chunks via PrintWriter in Java

Question

I have some data that I am processing and then saving it in stringbuilder to be finally written to csv.

Currently, I am writing the entire stringBuilder at once using println obtained as a result of processing some idData array. But I want to write in batches so maybe let's say if the array data for ids has length = 200, I want to write 50 id's data at a time.

How can I do that?

Existing code of writing the entire string builder result

class A {
 private StringBuilder sb;

 public void appendBody(String[] idData) {
   for(String id: idData) {
     Data data = getData(id);
     processData(data);
     
   }
 writeToCsv();
 } 

 public void processData() {

  .. processing that involves populating private property string builder
  // adding some sample data here
  this.sb.append("Name,Phone,Email\n");
  this.sb.append("Glen,1234,glen@abcd.com\n");
  this.sb.append("John,5678,john@efgh.com");

  // end populating string builder
 }

 public String writeToCsv() throws DataNotFoundException {

   String filename = "outputFileNamePath";  // sample filename, modified to be pasted here
   PrintWriter writer;
     try {
         writer = new PrintWriter(new File(filename));
     } catch (FileNotFoundException e) {
         writeLog("ERROR", e.getMessage);
         throw new Exception("Error while creating output file");
     }
     writer.println(this.sb.toString());
     writer.close();
     return filename;
  }
}

Some suggestions and hints will be helpful.

P.S: Please don't comment on the log message or exception being rethrown or the name of the variables or any nitty gritties not related to my problem since they have been modified to add it here and pardon me for any mistakes in advance!

` writeToCsv();` is not `public String writeToCsv(FeedFileName destFile)` - also you have `writeToCsv()` in a loop and you seem to be overwriting the entire file each time (the `StringBuilder` is only being appended to. Write real code that shows what the real problem is. — Scary Wombat, Jul 02 '21 at 00:09
@ScaryWombat Sorry, it's supposed to be out of the loop. I edited it. Thanks for pointing it out. — Atihska, Jul 02 '21 at 00:57
If you use a FileWriter object you can open the file in append mode. — Scary Wombat, Jul 02 '21 at 01:44

score 1 · Answer 1 · answered Jul 02 '21 at 10:16

Here is the solution to your problem.

Basic idea is to control call to writeToCsv() (this will be equal to desired chunk size) and always open file in append mode

Here, I am used mod operation to control a call to writeToCsv(). You can add any logic that will match your chuck size requirement. Text will be push to file after 5 iterations.

Also, to open file in append mode use

FileWriter fileWriter = new FileWriter(filename, true); // Set true for append mode
writer = new PrintWriter(fileWriter);

Here is the full program.

public class WriteToFileInChunk {
    private static StringBuilder sb = new StringBuilder();

    public static void main(String[] args) throws IOException {
        WriteToFileInChunk writeToFileInChunk = new WriteToFileInChunk();
        writeToFileInChunk.writeInchuncks();
    }

    private void writeInchuncks() throws IOException {
        String[] idData = { "1,", "2,", "3,", "4,", "5,", "6,", "7,", "8,", "9,", "10,", "11,", "12,", "13,", "14,",
                "15,", "16,", "17,", "18,", "19,", "20" };

        for (int i = 0; i < idData.length; i++) {
            StringBuilder data = new StringBuilder(idData[i]);
            sb.append(processData(data));

            if (i % 5 == 0) {
                writeToCsv(sb);
                sb = new StringBuilder();
            }
        }
    }

    private StringBuilder processData(StringBuilder stringBuilder) {
        return stringBuilder.append(System.currentTimeMillis() + ", Name, Phone,Email\n");
    }

    private String writeToCsv(StringBuilder sb) throws IOException {

        String filename = "outputFileNamePath"; // sample filename, modified to be pasted here
        PrintWriter writer;
        try {

            FileWriter fileWriter = new FileWriter(filename, true); // Set true for append mode
            writer = new PrintWriter(fileWriter);
            writer.println(sb.toString());
            writer.close();

        } catch (FileNotFoundException e) {
            System.out.printf("ERROR : %s", e.getMessage());
            new Exception("Error while creating output file", e);
        }
        return filename;
    }
}

zhh · Answer 2 · 2021-07-02T02:32:04.063

-1

You need to split idData into batches of arrays. Then writeToCsv after processing each batch. (Note you have to clear sb)

public void appendBody(String[] idData) {
    for (String[] batch : split(idData, BATCH_SIZE)) {
        sb = new StringBuilder();
        for (String id : batch) {
            Data data = getData(id);
            processData(data);
        }
        writeToCsv();
    }
}

You can use the below code to split an array:

static List<String[]> split(String[] idData, int batchSize) {
    List<String[]> list = new ArrayList<>();
    for (int i = 0; i < idData.length; i += batchSize) {
        int realBatchSize = Math.min(batchSize, idData.length - i);
        String[] batch = new String[realBatchSize];
        for (int j = 0; j < realBatchSize; j++)
            batch[j] = idData[j + i];
        list.add(batch);
    }
    return list;
}

Remember to use APPEND mode when you open a file.

edited Jul 02 '21 at 02:32

answered Jul 02 '21 at 00:56

zhh

2,346
1
11
22

`PrintWriter ` does not have append – Scary Wombat Jul 02 '21 at 01:44
I mean the [file mode](https://stackoverflow.com/questions/8210616/printwriter-append-method-not-appending). – zhh Jul 02 '21 at 02:00
As OP was not even using a `FileOutputStream` without the link, the comment was meaningless. I suggest that you add it to your answer. – Scary Wombat Jul 02 '21 at 02:30

How can I write data to csv in chunks via PrintWriter in Java

2 Answers2