Threads and file writing

Question

I have a java program which uses 20 threads. Every one of them write their results in a file called output.txt.

I always get a different number of lines in output.txt.

Can it be a problem with the synchronization of threads? Is there a way to handle this?

Answer 1

can it be a problem of synchronization of threads?

Yes.

There's a way to handle this?

Yes, ensure that writes are serialized by synchronizing on a relevant mutex. Or alternately, have only one thread that actually outputs to the file, and have all of the other threads simply queue text to be written to a queue that the one writing thread draws from. (That way the 20 main threads don't block on I/O.)

Re the mutex: For instance, if they're all using the same FileWriter instance (or whatever), which I'll refer to as fw, then they could use it as a mutex:

synchronized (fw) {
    fw.write(...);
}

If they're each using their own FileWriter or whatever, find something else they all share to be the mutex.

But again, having a thread doing the I/O on behalf of the others is probably also a good way to go.

Answer 2

I'd suggest you to organize it this way: One thread-consumer will consume all data and write it to the file. All worker threads will produce data to the consumer thread in synchronous way. Or with multiple threads file writing you can use some mutex or locks implementations.

Answer 3

If you want any semblance of performance and ease of management, go with the producer-consumer queue and just one file-writer, as suggested by Alex and others. Letting all the threads at the file with a mutex is just messy - every disk delay is transferred directly into your main app functionality, (with added contention). This is especially unfunny with slow network drives that tend to go away without warning.

Answer 4

If you can hold your file as a FileOutputStream you can lock it like this:

FileOutputStream file = ...
....
// Thread safe version.
void write(byte[] bytes) {
  try {
    boolean written = false;
    do {
      try {
        // Lock it!
        FileLock lock = file.getChannel().lock();
        try {
          // Write the bytes.
          file.write(bytes);
          written = true;
        } finally {
          // Release the lock.
          lock.release();
        }
      } catch ( OverlappingFileLockException ofle ) {
        try {
          // Wait a bit
          Thread.sleep(0);
        } catch (InterruptedException ex) {
          throw new InterruptedIOException ("Interrupted waiting for a file lock.");
        }
      }
    } while (!written);
  } catch (IOException ex) {
    log.warn("Failed to lock " + fileName, ex);
  }
}

Answer 5

Well, without any implementation detail, it is hard to know, but as my test case shows, I always get 220 lines of output, i.e., constant number of lines, with FileWriter. Notice that no synchronized is used here.

import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
/**
 * Working example of synchonous, competitive writing to the same file.
 * @author WesternGun
 *
 */
public class ThreadCompete implements Runnable {
    private FileWriter writer;
    private int status;
    private int counter;
    private boolean stop;
    private String name;


    public ThreadCompete(String name) {
        this.name = name;
        status = 0;
        stop = false;
        // just open the file without appending, to clear content
        try {
            writer = new FileWriter(new File("test.txt"), true);
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }

    }


    public static void main(String[] args) {

        for (int i=0; i<20; i++) {
            new Thread(new ThreadCompete("Thread" + i)).start();
        }
    }

    private int generateRandom(int range) {
        return (int) (Math.random() * range);
    }

    @Override
    public void run() {
        while (!stop) {
            try {
                writer = new FileWriter(new File("test.txt"), true);
                if (status == 0) {
                    writer.write(this.name + ": Begin: " + counter);
                    writer.write(System.lineSeparator());
                    status ++;
                } else if (status == 1) {
                    writer.write(this.name + ": Now we have " + counter + " books!");
                    writer.write(System.lineSeparator());
                    counter++;
                    if (counter > 8) {
                        status = 2;
                    }

                } else if (status == 2) {
                    writer.write(this.name + ": End. " + counter);
                    writer.write(System.lineSeparator());
                    stop = true;
                }
                writer.flush();
                writer.close();
            } catch (IOException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }
        }
    }
}

As I understand (and test), there are two phases in this process:

• all threads in the pool all created and started, ready to grab the file;
• one of them grabs it, and I guess it then internally locks it, prevents other threads to get access, because I never see a line combined of contents that come from two threads. So when a thread is writing, others are waiting until it completes the line, and very likely, releases the file. So, no race condition will happen.
• the quickest of the others grabs the file and begins writing.

Well, it is just like a crowd waiting outside a bathroom, without queuing.....

So, if your implementation is different, show the code and we can help to break it down.

Answer 6

You should use synchronization in this case. Imagine that 2 threads (t1 and t2) open the file at the same time and start writing to it. The changes performed by the first thread are overwrited by the second thread because the second thread is the last to save the changes to the file. When a thread t1 is writing to the file, t2 must wait until t1 finishes it's task before it can open it.