How to MultiThread HttpURLConnection in Java

Question

Hi guys, I have a client that would like to check the variations around their website. They have 5 million URLs to check. If I was to send requests/pings synchronously, it would take me 23 days. So I'm looking for a multithreaded solution. I originally started this problem off in Python, but didn't see much improvement/couldn't scale well, so here I am in Java, and if this fails too, I'll try in Go before throwing in the towel.

The issue is I'm not seeing any improvements at all with multithreading. Perhaps I'm implementing it wrong, could anyone please help me?

Edits:

I'll just be making edits here and new comers can look at the history of this post to see how I've progressed through the problem.

This is the socket suggestion, fails when I try to run it in a thread, unsure what I'm doing wrong here too.

Main Class:

package com.company;
import java.io.IOException;
import java.util.ArrayList;
import java.util.concurrent.TimeUnit;

public class Main extends Thread{
public static void main(String[] args) throws IOException {
long startTime = System.nanoTime();
Helpers.get("www.google.com", 80); // works here
String path = "test.txt";
boolean append = true;
for (int x = 0; x < 1; x++) {
ArrayList<String> urls = new ArrayList<String>();
// when x = 0, y = 0 | 10 /\ when x = 1, y = 10 | 20
for (int y= x * 10;y < ((x + 1) * 10); y++){
urls.add(String.format("www.google%d.com/", y)); // doesn't work here
}
Thread thread = new Thread(new Helpers(path, append, urls, 80));
thread.start();
thread.interrupt();
}
long endTime = System.nanoTime();
long duration = TimeUnit.NANOSECONDS.toMillis(endTime - startTime);
System.out.println(duration + " ms");
}
}

Helpers Class:

package com.company;
import java.io.IOException;
import java.net.*;import java.io.FileWriter;
import java.io.PrintWriter;import java.util.ArrayList;
public class Helpers extends Thread{
public Helpers(String path, boolean append, ArrayList<String> urls, int port) throws IOException {
this.run(path, append, urls, port);
}
public void run(String path, boolean append, ArrayList<String> urls, int port) throws IOException {
for (String url : urls) {
String status = Helpers.get(url, port);Helpers.writeToFile(path, append, status);
System.out.println(status);
}
}
public static String get(String url, int port) throws IOException {
try {
Socket conn = new Socket(url, 80);
conn.close();
return url + " | Success";
}catch (UnknownHostException error){return url + " | Failed";
}
}

Answer 1

You are implementing it wrong. You should make Helpers class extend thread or implement runnable. Pass everything you need to this class, for example, url, file pointer etc.

In your main class, create object of Helper and then run it as a thread.

Answer 2

You can try a different approach. Instead of making HTTPConnection for every call, you can try creating a socket connection to the webserver and then make multiple calls (GET/HEAD) to different URLs.

/**
 * hostname of the webserver e.g. www.w3.org
 * @param hostname
 * @param urlList
 * @throws IOException
 */
public static void makingHTTPCall( String hostname , List<String> urlList) throws IOException {

    SSLSocketFactory factory = (SSLSocketFactory)SSLSocketFactory.getDefault();
    SSLSocket socket =
            (SSLSocket)factory.createSocket(hostname, 443);


    BufferedReader in
            = new BufferedReader(
            new InputStreamReader(socket.getInputStream()));
    PrintWriter out
            = new PrintWriter(socket.getOutputStream(), true);

    /**
     *  if required create different url List and pass those list to separate thread for better performance
     */

    urlList.forEach(
            url -> {
                System.out.println("Making call to url /" + url);
                out.println("HEAD "  + url + " HTTP/1.1\r\n");
                out.flush();

                String line = "";

                try {
                    while ((line = in.readLine()) != null) {
                        System.out.println("Response" + line);
                        break;
                    }

                } catch (IOException e) {
                    e.printStackTrace();
                }
            }
    );

    try {
            in.close();
    } catch (IOException e) {
        e.printStackTrace();
    }
    out.close();

}

I have tried with the SSL socket you can change this according to your configuration.