Java multi-thread application that reads a single file

Question

I currently wrote my program to use 32 threads and read 1 file per thread (so 32 .txt files). The multi-threading has nothing to do with CPU speed, but making 32 calls to BING's api per second is a lot faster than making 1. Each of the .txt files containing a list of search queries. I create a thread it reads one line at a time from it's file. Is it possible to create all 32 threads and point them to a single .txt file??

I'm parsing big text files and I typically do it like this: one producer is doing the I/O and putting in a queue then as many consumers as I want fetch jobs from the queue. In your case your unique producer could read the txt file and put one entry per line in the queue. — TacticalCoder, Nov 14 '11 at 18:30

score 14 · Accepted Answer · answered Nov 14 '11 at 18:31

14

Use the Producer-Consumer pattern. Have only one thread that reads file and pushes each line/command into ArrayBlockingQueue (thread safe read and write) using put().

All other 32 threads should read from the same queue object by calling take(). They will block if queue is empty, which is nice.

This solution is better because the disk is inherently single-threaded so you won't get much by reading the file concurrently.

answered Nov 14 '11 at 18:31

Tomasz Nurkiewicz

334,321
69
703
674

Thanks, that makes changing the input a lot easier than having 32 separate files. – chrstahl89 Nov 14 '11 at 19:44

score 0 · Answer 2 · answered Nov 14 '11 at 18:32

0

You can synchronize the code that handles the file. Each time you write to the file you open, write and close it. Implement using a monitor and it should be fine.

Could also use java util logging as it is thread safe. If you implement a handler then logging api should take care of the thread safety issues.

answered Nov 14 '11 at 18:32

Sid Malani

2,078
1
13
13

There's no writing involved in the question and this doesn't have a thing to do with logging. – G_H Nov 14 '11 at 18:34

score 0 · Answer 3 · answered Nov 14 '11 at 18:33

It depends on how these threads are implemented. If each has its own reader or input stream on the file, then concurrent reading probably isn't much of a problem. Unless the JVM implementation on the OS implicitly locks a file when an input stream is opened on it and the lock can't be shared across streams.

But you'd be doing a lot of unnecessary work, to be honest. It'd be much better to encapsulate the access to the file in a separate class and pass instances of that to your threads, then have that class do the necessary concurrency handling.

Java multi-thread application that reads a single file

3 Answers3

Linked