15

I am trying to download multiple files that matches a pattern using threads. The pattern could match 1 or 5 or 10 files of diff sizes.

lets say for simplicity sake the actual code that would download the file is in downloadFile() method and fileNames is the list of filenames that match the pattern. How do I do this using threads. Each thread will download only one file. Is it advisable to create a new thread inside the for loop. 

for (String name : fileNames){
    downloadFile(name, toPath);
}
user373201
  • 10,945
  • 34
  • 112
  • 168

5 Answers5

39

You really want to use an ExecutorService instead of individual threads, it's much cleaner, likely more performant and will enable you to change things more easily later on (thread counts, thread names, etc.):

ExecutorService pool = Executors.newFixedThreadPool(10);
for (String name : fileNames) {
    pool.submit(new DownloadTask(name, toPath));
}
pool.shutdown();
pool.awaitTermination(Long.MAX_VALUE, TimeUnit.MILLISECONDS);
// all tasks have now finished (unless an exception is thrown above)

And somewhere else in your class define the actual work horse DownloadTask:

private static class DownloadTask implements Runnable {

    private String name;
    private final String toPath;

    public DownloadTask(String name, String toPath) {
        this.name = name;
        this.toPath = toPath;
    }

    @Override
    public void run() {
        // surround with try-catch if downloadFile() throws something
        downloadFile(name, toPath);
    }
}

The shutdown() method has a very confusing name, because it "will allow previously submitted tasks to execute before terminating". awaitTermination() declares an InterruptedException you need to handle.

Philipp Reichart
  • 20,771
  • 6
  • 58
  • 65
  • This requires the `downloadFile()` method to be either defined inside `DownloadTask` (my recommendation) or declared `static` when outside. – Philipp Reichart Sep 21 '11 at 16:07
  • `@Philipp Reichart` New download tasks will not be accepted in the pool after firing `shutdown` I will avoid this if I were you. – Bitmap Sep 21 '11 at 16:21
  • That's a non-issue as no new tasks are added to the pool after it has been shut down in this case. The first code snippet is supposed to live in a method and creates a new pool on every invocation. – Philipp Reichart Sep 21 '11 at 16:23
  • "The first code snippet is supposed to live in a method and creates a new pool on every invocation" - This is an assumption of design. Let's not speculate, try deploying a code like this on tomcat or jboss and see if the download service instance will still be available to proceed next download call, after the shutdown is invoked - I had a bite from doing the same in the past, but I am not using that as an excuse to mark you down am I. – Bitmap Sep 21 '11 at 16:30
  • @Bitmap It seems like you're a bit cranky because Philipp criticized your solution (and for good reasons at that!). The above code is perfectly fine and demonstrates the usual approach nicely. No need for nitpicking.. – Voo Sep 21 '11 at 16:43
  • Excuse me, if I can't know how much `fileNames` in the first place. How can I do? I mean the user might choose to download a single file in page 1, and later in page 3 or 5. And I want to control the asynchronously download task to be n(maximum n task). – Alston Sep 10 '14 at 14:00
  • Is it necessary that make the class static? According to https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ExecutorService.html , the class in example is not. – lalameat Jan 29 '18 at 09:51
  • @lalameat It's not necessary to make the inner class static, but unless you need a reference to the surrounding class I'd keep it static. Makes it more obvious what data the inner class has access to, so easier to reason about. – Philipp Reichart Jan 29 '18 at 11:47
5

Yes, you certainly could create a new thread inside the for-loop. Something like this:

List<Thread> threads = new ArrayList<Thread>();
for (String name : fileNames) {
  Thread t = new Thread() {
    @Override public void run() { downloadFile(name, toPath); }
  };
  t.start();
  threads.add(t);
}
for (Thread t : threads) {
  t.join();
}
// Now all files are downloaded.

You should also consider using an Executor, for example, in a thread pool created by Executors.newFixedThreadPool(int).

maerics
  • 151,642
  • 46
  • 269
  • 291
  • Using the above code, how can i limit the number of simultaneous downloads, say I just want to download 3 files at a time. Do you have an example of using Executor – user373201 Sep 21 '11 at 16:07
  • @user373201: follow the link I provided to `Executors.newFixedThreadPool(int)` and review @Phillip Reichart's answer. – maerics Sep 21 '11 at 16:26
1

Yes you can create the Threads inline.

for (final String name : fileNames){
    new Thread() {
       public void run() {
           downloadFile(name, toPath);
       }
    }.start();
}
Garrett Hall
  • 29,524
  • 10
  • 61
  • 76
1

Use Executor, Try this.

ExecutorService  exec= Executors.newCachedThreadPool()
for (String name : fileNames){
 exec.submit(new Runnable()
 {
  public void run()
  {
    downloadFile(name, toPath);
  }
 });
}

If you want say three download running concurrently, you can use:

Executors.newFixedThreadPool(3)
Bitmap
  • 12,402
  • 16
  • 64
  • 91
  • Now it blocks while `execute()`ing the `Runnable`, you meant to call `exec.submit()` :) – Philipp Reichart Sep 21 '11 at 16:10
  • @Philipp Reichart do you really know what you're talking about? what blocks while execute? Listen bruv! avoid marking people down when you dont know what you on about. – Bitmap Sep 21 '11 at 16:13
  • Yes, I do, I've actually been bitten by this once. [`Execute.execute()`](http://download.oracle.com/javase/1,5.0/docs/api/java/util/concurrent/Executor.html#execute(java.lang.Runnable)) may "... execute in a new thread, in a pooled thread, or **in the calling thread**, at the discretion of the Executor implementation." You really want to declare the variable as `ExecutorService` and use [`submit()`](http://download.oracle.com/javase/6/docs/api/java/util/concurrent/ExecutorService.html#submit(java.lang.Runnable)). – Philipp Reichart Sep 21 '11 at 16:17
  • So you go about marking people down because you've had issue in the past calling `execute`?! bruv you're just taking a mickey. – Bitmap Sep 21 '11 at 16:26
  • I only vote down when I'm certain the answer is not correct, fix it and I'll happily remove the down vote. The possibility that `execute()` might run the command on the calling thread depending on the `Executor` implementation used, and the fact that there obviously is a way that does not exhibit this problem is a rather compelling rationale to me. – Philipp Reichart Sep 21 '11 at 16:32
  • So what next should I add to the answer since you've indicated you'll only mark me back up if I choose to do it your way. – Bitmap Sep 21 '11 at 16:38
0

All the Above mentioned approach creates Threads but the actual Concurreny is not achieved.

ExecutorService pool = Executors.newFixedThreadPool(5);
final File folder = new File("YOUR_FILES_PATH");
int l = folder.listFiles().length;
System.out.println("Total Files----"+folder.listFiles().length);
long timeStartFuture = Calendar.getInstance().getTimeInMillis();
    pool.execute(new DownloadFile(folder,0,l/2));
    pool.execute(new DownloadFile(folder,(l/2),l));
    pool.shutdown();
    try {
        pool.awaitTermination(Long.MAX_VALUE, TimeUnit.MILLISECONDS);
    } catch (InterruptedException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }

    long timeEndFuture = Calendar.getInstance().getTimeInMillis();
    long timeNeededFuture = timeEndFuture - timeStartFuture;
    System.out.println("Parallel calculated in " + timeNeededFuture + " ms");

The above program is used to achieve concurreny and please modify as per your requirement.

ihappyk
  • 525
  • 1
  • 5
  • 16