Android get website content / source code

Question

I am into project that needs to get the source code of website. So I got my code from Mark B answer: How to get the html-source of a page from a html link in android?

The problem is that HttpClient, HttpGet, HttpResponse are deprecated now.

So I change it into HttpURLConnection but no luck, it forces close.

    URL url = new URL("http://google.com");
    HttpURLConnection response = (HttpURLConnection) url.openConnection();

    String html = "";
    InputStream in = response.getInputStream();
    BufferedReader reader = new BufferedReader(new InputStreamReader(in));
    StringBuilder str = new StringBuilder();
    String line = null;
    while((line = reader.readLine()) != null){
        str.append(line);
    }
    in.close();
    html = str.toString();
    return html;

What's the problem with this? Here is my stacktrace

java.lang.RuntimeException: Unable to start activity ComponentInfo{com.example.listview_load_data_from_json/com.kaleidosstudio.listview_load_data_from_json.MainActivity}: android.os.NetworkOnMainThreadException
        at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2319)
        at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2370)
        at android.app.ActivityThread.access$800(ActivityThread.java:155)
        at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1243)
        at android.os.Handler.dispatchMessage(Handler.java:102)
        at android.os.Looper.loop(Looper.java:136)
        at android.app.ActivityThread.main(ActivityThread.java:5426)
        at java.lang.reflect.Method.invokeNative(Native Method)
        at java.lang.reflect.Method.invoke(Method.java:515)
        at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1268)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1084)
        at de.robv.android.xposed.XposedBridge.main(XposedBridge.java:126)
        at dalvik.system.NativeStart.main(Native Method)
 Caused by: android.os.NetworkOnMainThreadException
        at android.os.StrictMode$AndroidBlockGuardPolicy.onNetwork(StrictMode.java:1156)
        at java.net.InetAddress.lookupHostByName(InetAddress.java:385)
        at java.net.InetAddress.getAllByNameImpl(InetAddress.java:236)
        at java.net.InetAddress.getAllByName(InetAddress.java:214)
        at com.android.okhttp.internal.Dns$1.getAllByName(Dns.java:28)
        at com.android.okhttp.internal.http.RouteSelector.resetNextInetSocketAddress(RouteSelector.java:216)
        at com.android.okhttp.internal.http.RouteSelector.next(RouteSelector.java:122)
        at com.android.okhttp.internal.http.HttpEngine.connect(HttpEngine.java:292)
        at com.android.okhttp.internal.http.HttpEngine.sendSocketRequest(HttpEngine.java:255)
        at com.android.okhttp.internal.http.HttpEngine.sendRequest(HttpEngine.java:206)
        at com.android.okhttp.internal.http.HttpURLConnectionImpl.execute(HttpURLConnectionImpl.java:345)
        at com.android.okhttp.internal.http.HttpURLConnectionImpl.connect(HttpURLConnectionImpl.java:89)
        at com.kaleidosstudio.listview_load_data_from_json.GetMP3.getJSON(GetMP3.java:49)
        at com.kaleidosstudio.listview_load_data_from_json.GetMP3.GetMP3_9Cloud(GetMP3.java:34)
        at com.kaleidosstudio.listview_load_data_from_json.MainActivity.onCreate(MainActivity.java:52)
        at android.app.Activity.performCreate(Activity.java:5296)
        at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1094)
        at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2283)
        at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2370)
        at android.app.ActivityThread.access$800(ActivityThread.java:155)
        at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1243)
        at android.os.Handler.dispatchMessage(Handler.java:102)
        at android.os.Looper.loop(Looper.java:136)
        at android.app.ActivityThread.main(ActivityThread.java:5426)
        at java.lang.reflect.Method.invokeNative(Native Method)
        at java.lang.reflect.Method.invoke(Method.java:515)
        at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1268)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1084)
        at de.robv.android.xposed.XposedBridge.main(XposedBridge.java:126)
        at dalvik.system.NativeStart.main(Native Method)

Please post your stack trace in the question. Please don't link to it off-site. — Mike M., Mar 26 '16 at 08:21

Panda · Answer 1 · 2016-03-26T08:34:33.307

You are calling these functions in your main thread. Now, Their are two possible methods, to deal with this.

AsyncTask : AsyncTask enables proper and easy use of the UI thread. This class allows to perform background operations and publish results on the UI thread without having to manipulate threads and/or handlers. Here is a reference to that AsyncTask

public class myAsyncTask extends AsyncTask<String, Void, Void> {
public String html;

@Override
private void doInBackground(String... params) {
URL url = new URL(params[0]);
HttpURLConnection response = (HttpURLConnection) url.openConnection();

String html = "";
InputStream in = response.getInputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(in));
StringBuilder str = new StringBuilder();
String line = null;
while((line = reader.readLine()) != null){
    str.append(line);
}
in.close();
html = str.toString();
}
}

and then call it like this

 myAsyncTask task= new myAsyncTask();
 task.execute("http://google.com");

Thread: A Thread is a concurrent unit of execution. It has its own call stack for methods being invoked, their arguments and local variables. Each application has at least one thread running when it is started, the main thread, in the main ThreadGroup. The runtime keeps its own threads in the system thread group.

There are two ways to execute code in a new thread. You can either subclass Thread and overriding its run() method, or construct a new Thread and pass a Runnable to the constructor. In either case, the start() method must be called to actually execute the new Thread.

Each Thread has an integer priority that affect how the thread is scheduled by the OS. A new thread inherits the priority of its parent. A thread's priority can be set using the setPriority(int) method. Here is a quick Reference Thread

Rakshith Ravi · Answer 2 · 2016-03-26T08:44:34.700

You are getting a NetworkOnMainThreadException. What this is saying is that you can't do stuff like sending Http requests on the main UI thread. You'd have to do all that in a background thread.

Here is a simple example of how to do it:

public class WebsiteDataWorker extends AsyncTask<String, Void, Void> {
    public String html;

    @Override
    private void doInBackground(String... params) {
        URL url = params[0];
        HttpURLConnection response = (HttpURLConnection) url.openConnection();

        String html = "";
        InputStream in = response.getInputStream();
        BufferedReader reader = new BufferedReader(new InputStreamReader(in));
        StringBuilder str = new StringBuilder();
        String line = null;
        while((line = reader.readLine()) != null){
            str.append(line);
        }
        in.close();
        html = str.toString();
    }
}

Now you can call your background thread by just saying

WebsiteDataWorker worker = new WebsiteDataWorker();
worker.execute("http://google.com");
String html = worker.html;
//do whatever stuff you want with the html variable

Its obsolete, adding the url in execute method, while calling the background function and then, adding it again in the background method — Panda, Mar 26 '16 at 08:32

Rahul Wadhai · Answer 3 · 2016-03-29T10:45:36.053

private class RegistrationAsyncTask extends AsyncTask<Void, Void, String> {
ProgressDialog dialog;

@Override
protected void onPreExecute() {
    dialog = new ProgressDialog(MainActivity.this);
    dialog.setTitle("Registration");
    dialog.setMessage("Registration in process...");
    dialog.setCancelable(false);
    dialog.show();
}

@Override
protected String doInBackground(Void... params) {
    try {

        response = HttpClientWrapper.getResponseGET("Your url put here");
        Log.e(TAG, "Response: " + response);
    } catch (Exception e) {
        e.printStackTrace();
        this.error = e.getMessage();
    }
    return response;
}

@Override
protected void onPostExecute(String result) {
    Log.e(TAG, "result: " + response);

    if (dialog.isShowing()) {
        dialog.dismiss();
    }
}}

use this async task

Rahul Wadhai · Answer 4 · 2016-03-29T10:44:56.537

public static String getResponseGET(String url) {
String response = "";
HttpURLConnection c = null;

try {
    URL u = new URL(url);
    c = (HttpURLConnection) u.openConnection();
    c.setRequestMethod("GET");
    c.setConnectTimeout(15000);
    c.setReadTimeout(15000);

    c.connect();
    int status = c.getResponseCode();

    switch (status) {
        case 200:
        case 201:
            BufferedReader br = new BufferedReader(new InputStreamReader(c.getInputStream()));
            StringBuilder sb = new StringBuilder();
            String line;
            while ((line = br.readLine()) != null) {
                sb.append(line+"\n");
                response = sb.toString().substring(0, sb.toString().length() - 1);
            }
            br.close();
            return response;
    }

} catch (IOException ex) {
    if (c != null) {
        c.disconnect();
    }
} finally {
    if (c != null) {
        try {
            c.disconnect();
        } catch (Exception ex) {

        }
    }
}
return null;}

use this http call for GET method

It forces close. I'm getting this error: http://pastebin.com/raw/Nt2JpPy9 — tokis, Mar 26 '16 at 08:50
Rahul, please combine your posts into one answer. They look like two separate answers, as evidenced by the OP still getting the same Exception. — Mike M., Mar 26 '16 at 09:15

Android get website content / source code

4 Answers4