Java parsing txt file multithreading

Question

I have a method to parse txt file and saving datas to database. Method named: parseFileAndSaveToDB(String fileAddress); This file are big (100 000 rows) so inserting datas into DB are very long.

Can anyone help me with do this in multithreading?? I think that each thread can use this method with each files forexample:

parseFileAndSaveToDB("c:/file1");//thread 1
parseFileAndSaveToDB("c:/file2");//thread 2
parseFileAndSaveToDB("c:/file3");//thread 3
parseFileAndSaveToDB("c:/file4");//thread 4

But how to do it with multithreads?? And is it good idea?

We can help you to improve or to rectify your code/logic. We are not here to write code for you. — Ankur Shanbhag, Feb 25 '14 at 11:23
Are they writing in the same table? If yes, multithreading has no use at all. — Smutje, Feb 25 '14 at 11:23
It may have some use if their bottleneck was disk access to the files; by multithreading, they can overcome that, and then the database access would be the new bottleneck — Jorge_B, Feb 25 '14 at 11:26
For most cases you want to use batch inserts rather that multiple threads. — krosenvold, Feb 25 '14 at 11:34
@Smutje this depends on the table itself (of course) and on the RDBMS engine... Some engines _do not_ require table-level locks on insertions :p — fge, Feb 25 '14 at 11:41

score 2 · Answer 1 · answered Feb 25 '14 at 11:32

Please take a look at this link here which explains how to use ExecutorService in java. It can be something like:

ExecutorService executorService = Executors.newFixedThreadPool(10);

MyDataLoader myDataLoader1 = new MyDataLoader("c:/file1");
MyDataLoader myDataLoader2 = new MyDataLoader("c:/file2");
MyDataLoader myDataLoader3 = new MyDataLoader("c:/file3");

executorService.execute(myDataLoader1);
executorService.execute(myDataLoader2);
executorService.execute(myDataLoader3);

executorService.shutdown();

You can write MyDataLoader class like:

public class MyDataLoader implements Runnable {

    String fileName = null;

    public MyDataLoader(String fileName) {
        this.fileName = fileName;
    }

    public void run() {
        //Your logic to parse file and insert the data in DB.
    }
}

Hope this helps.

RMachnik · Accepted Answer · 2014-02-25T19:47:02.500

1

This is quick draft. As the argument of program pass list of file names that should be read and store.

public class ParsingThread implements Runnable{
private String fileName;

public ParsingThread(String fName){
    fileName = fName;
}

@Override
public void run(){
    parseFileAndSaveToDB(fileName);

}
public void parseFileAndSaveToDB(String fileName){
    //your implementation
}



public static void main(String args[]){
    for(String fileName : args){
        Thread runAndStore = new Thread(new ParsingThread(fileName));
        runAndStore.run();
    }

  }
}

edited Feb 25 '14 at 19:47

answered Feb 25 '14 at 11:27

RMachnik

3,598
1
34
51

He don't allow me to use in main method and in for loop this Thread, IDE talking that i have to use ParsingThread but if I use this, then i have to create method start() in ParsingThread method – hetmanJIIIS Feb 25 '14 at 19:40
Now it should work I edited my code sory for mistakes. – RMachnik Feb 25 '14 at 19:47

score 0 · Answer 3 · answered Feb 25 '14 at 11:30

Very basic template:

public static void parseFileAndSaveToDB(String filePath)
{
    new Thread(new Runnable() {

        @Override
        public void run() {
            // # TODO : Open file with path "filePath"
            // # Read file
            // # Write to database
        }
    }).start();
}

public static void main(String args[]) {
    List<String> paths; // # TODO : fill this list with the file paths you have

    for(String filePath : paths) 
    {
        parseFileAndSaveToDB(filePath);
    }
}

But before that, i strongly recommend you read some articals on the consecpt of multithreading in java. You might find the following sources helpful:

Official Tutorial on Concurrency to begin with.
tutorialpoint.com with simple concept and demonstration samples
Vogella to dig deeper

score -1 · Answer 4 · edited May 23 '17 at 11:50

Since I can't add comments yet I'll add an answer instead. I agree with Rafik991's answer and in the implementation you can start by counting the number of lines you have in order to distribute your data equally between your threads. check this link for fast implementations. i'm not sure if that's a good idea since you will have go through the file to get the right line but I think that it's what you are trying to do.

Java parsing txt file multithreading

4 Answers4