How to read BIG JSON Conten from URL

Question

im lookin way to read big json content from url. Im write simple app where i test much funcitons . Most function show me time ~40-45 seconds. Than return content(JSON file realy big) Size of JSON FILE 90kb .. 2600 lines First funciton reading all content but very slow (40-45 seconds)

public  String readJsonFromUrl(String urls)  {
         String content = "";
         URL myLink = null;
         String inputLine=null;
        try {
            myLink = new URL(urls);
         } catch (MalformedURLException e) {
            // TODO Auto-generated catch block
        }
         BufferedReader in = null;
        try {
            in = new BufferedReader(
                         new InputStreamReader(
                         myLink.openStream()));
        } catch (IOException e) {
        }
        try {
            while ((inputLine = in.readLine()) != null)
                 content =content+inputLine;
        } catch (IOException e1) {
            // TODO Auto-generated catch block

        }
         try {
            in.close();
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
       return content;
      }

Function work perfect. But time :(. U guys can just input fucn to code , set url and parse.

Second function

public void readJSONFromUrl(String urls) throws IOException, URISyntaxException
    {
        InputStream is = null;
        String response = "";
        String url=urls;
        DefaultHttpClient httpClient = new DefaultHttpClient();
        HttpEntity httpEntity = null;
        HttpResponse httpResponse = null;
        HttpGet httpGet = new HttpGet(url);
        httpResponse = httpClient.execute(httpGet);
        httpEntity = httpResponse.getEntity();
        response = EntityUtils.toString(httpEntity);
        Log.i("RESPONSE","RESPONSE = "+response);
}

this funciton work but very strange. i get only PART of content and very small part of big content.

Anyone maybe know some thing better or how to fix second function for test .. it's show me how much time need for getting content. Or maybe some one have other fucntion which faste than two this function ? Regards Peter.

don't use `content =content+inputLine;`. Use a StringBuilder instead — Pedro Oliveira, Sep 12 '14 at 07:59
and if i use stringbuilder in first function than function get only part of json file (like 30-50 lines) :(((( dont know why :( — Peter, Sep 12 '14 at 08:06
90kb exceeds LogCat's buffer, which is the actual problem (see my answer). — 0101100101, Sep 12 '14 at 08:19

Pedro Oliveira · Accepted Answer · 2014-09-12T08:17:43.370

3

That concatenation will take too much time appending. Use a StringBuilder instead.

    StringBuilder response = new StringBuilder();

    while ((inputLine = in.readLine()) != null) {
        response.append(inputLine);
    }
    content = response.toString();

UPDATE:

Here is an example of a getJsonObject from a url:

private static JSONObject getJSONObject(String _url) throws Exception {
    if (_url.equals(""))
        throw new Exception("URL can't be empty");

    URL url = new URL(_url);
    HttpURLConnection conn = (HttpURLConnection) url.openConnection();
    conn.setReadTimeout(10000 /* milliseconds */);
    conn.setConnectTimeout(15000 /* milliseconds */);
    conn.setDoInput(true);
    conn.setRequestProperty("User-Agent", "android");
    conn.setRequestProperty("Accept", "application/json");
    conn.addRequestProperty("Content-Type", "application/json");
    BufferedReader in = new BufferedReader(
            new InputStreamReader(conn.getInputStream()));

    if (!url.getHost().equals(conn.getURL().getHost())) {
        conn.disconnect();
        return new JSONObject();
    }
    String inputLine;
    StringBuilder response = new StringBuilder();
    while ((inputLine = in.readLine()) != null) {
        response.append(inputLine);
    }
    in.close();
    conn.disconnect();

    return new JSONObject(response.toString());

}

edited Sep 12 '14 at 08:17

answered Sep 12 '14 at 08:02

Pedro Oliveira

20,442
8
55
82

and if i use stringbuilder in first function than function get only part of json file (like 30-50 lines) :(((( dont know why :( – Peter Sep 12 '14 at 08:07
it's like a want to get string "understand" from url - but it's return only "unders" that's all :( – Peter Sep 12 '14 at 08:07
Could you show us the json? Also if lines is the problem try `response.append(inputLine).append("\n");` – Pedro Oliveira Sep 12 '14 at 08:10
http://www.json-generator.com/api/json/get/bWjUhbiVQi?indent=2 it's auto generated file from site . Big file. And still it's get only part of code. :(40-50 lines) – Peter Sep 12 '14 at 08:41
That's not a "big" json. It's just a jsonarray. Change the return of my method from JSONObject to JSONArray. – Pedro Oliveira Sep 12 '14 at 08:46
StringBuilder containe only 40-50 lines from this content. It's not all content. JSON array i build in other function , but here returned not all content. idk how to fix it – Peter Sep 12 '14 at 08:48
anyway bro, thank u ,stringbuilder better than string to string. – Peter Sep 12 '14 at 08:55
Lines doesn't matter friend if the content is all there. Have you at least tried creating an array from the content? – Pedro Oliveira Sep 12 '14 at 08:56
I just tried this function and its working. I get 88 objects in the json array. Like the site. Just because you get less lines that doesn't matter that you don't have all the objects. And I can tell you also that this operation takes 738ms. Which is less than a second. So as I told you. Not a "big" json. – Pedro Oliveira Sep 12 '14 at 09:02
yep. just result 1 seconds. Its amazing. thats why i thinked about not all content. – Peter Sep 12 '14 at 09:07
Good! Used this example and got my download + string building from 10 seconds to 1! – KasparTr Nov 24 '16 at 13:22

score 0 · Answer 2 · edited May 23 '17 at 12:30

0

It's impossible that the second function only gives you a small part of the content. The problem doesn't lie in the response or in the stream reading, it lies in you trying to display that huge content in LogCat: LogCat has a limit (see Android webservice GET request replies only a part of the XML response and What is the size limit for Logcat and how to change its capacity?; 90kb exceeds LogCat's buffer). That means your variable response does contain the full JSON, but LogCat isn't displaying the full variable. Just parse your JSON and everything should work.

edited May 23 '17 at 12:30

Community

1
1

answered Sep 12 '14 at 08:13

0101100101

5,786
6
31
55

no it's poblem not in logcat. it's strange error or some thing else. cos if not using stringbuilder than content getting all but very slow, if using stringbuilder than it's return only part of code.(content) – Peter Sep 12 '14 at 08:46
@Peter Not talking about StringBuilder, but your second approach using HttpResponse. – 0101100101 Sep 12 '14 at 08:57

How to read BIG JSON Conten from URL

2 Answers2

Linked