How can I efficiently use threads in this case?

Question

I have to write in a file all possible combinations resulting from the lottery using threads. Example:

1. 0 0 0 0 0 0 (first combination)
2. 0 0 0 0 0 1 (Second combination)
3. 0 0 0 0 0 2
   . . .
   Last. 38 38 38 38 38 38 (Last combination)

In my main class i just use one thread because i dont know how i can use more threads for write in the file faster.

For generates the numbers i uses 6 loops one inside the other this way:

import java.io.BufferedWriter;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.OutputStreamWriter;
import java.util.Scanner;

public class Proceso extends Thread {
    File file = new File("lottery.txt");
    public Proceso(String msg) {
        super(msg);
    }

    public void run() {
        try {
            BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(file, true)));
            StringBuffer linea = new StringBuffer();
            for (int i = 0; i < 15;i++) {
                for (int j = 0; j < 39; j++) {
                    for (int j2 =0; j2 < 39; j2++) {
                        for (int k = 0; k < 39; k++) {
                            for (int k2 = 0 ; k2 < 39; k2++) {
                                for (int l = 0; l < 39; l++) {
                                    linea.append(i + " " +j + " " +j2 + " " +k + " " +k2 + " " +l + "\n");
                                    bw.write(linea.toString());

                                    }
                            }
                        }
                    }
                }
            }
            bw.close();
        } catch (IOException e) {

            e.printStackTrace();
        }

    }


}

Also the file (in middle of the execution) rise a size of 8gb and i have to stop it because my pc collapse.

Edit: if i cant do that, at least i can write with two different threads in the same file at almost the same time? If i can do that how i would do it?

Answer 1

You start by stopping. Seriously: step back for a second.

First of all, as tkausl is correctly commenting, using more than one thread probably will not help when your bottleneck is IO performance. To the contrary - having multiple threads writing into the same file might not all be helpful. Besides, it is hard to get that right!

So, if at all, you would have to look into distributing this task onto several systems. And we then look into your code; it becomes obvious that you can't do that with such code.

That is my initial point: your code is extremely low-level. There is almost no abstraction in there that would separate the different activities within this task from each other. Just loop within loop within loop. That is code that probably "does" its job (or maybe would, if the "expected" output would be smaller); but it doesn't contain any of the helpful abstractions that you would need to "adapt" it to your actual needs.

Long story short: if you really want to try the multi-threaded thing; then start by turning your code into a stream based solution. And then, when that works, you can study parallel streams; and use those to run your code "in parallel".

And just for the record: there is a bug in your code.

linea.append(i + " " +j + " " +j2 + " " +k + " " +k2 + " " +l + "\n");
bw.write(linea.toString())

This appends a new line to your buffer; and then writes the whole buffer. But: the lines stay in your buffer!

So, maybe in this case, the "easy" answer is: clear the buffer after you wrote its content! Like: linea.delete(0, linea.length()). And btw: use StringBuilder; not StringBuffer. You could simply create a new builder for each line (instead of using/clearing the same builder all the time).

Most likely you get your exception because your StringBuffer keeps growing and growing; until you run out of memory.

Final note: stepping back is still the thing to do here. If I am not mistaken, the file you intend to create will have 38*38*38*38*38*38 lines, that makes up 3.010.936.384, so roughly 3 billion entries. Assume that each line contains 10, 15 bytes. So you end up with a file around 30 GB or so. But why? This file will contain a simple permutation of those values. And with a bit of math ... alone the line number could be used to compute the string that would be find in that line! Because you can map any value between 1 and 3.010.936.384 to the corresponding "permutation".

So, longer story short: whatever program is working with that 30 GB file - it doesn't need to. You know how many permutations exist; and you can easily compute (aka predict) the permutations. Any program that needs those permutations can do that! There is no point in writing 30 GB of data to the file system if you can compute the data so easily!

Thus, in essence: forget about writing this stuff to a file. You don't have you. That just costs you IO time writing and reading in later on. And you know, "IO" is still the most expensive thing to do.