how to get full html code from website using jsoub or any other

Question

I'm trying to get html code from website and if the website code is small like this one: (https://abdelftahzowail.github.io/WriteUpsideDown/) i get the full code but if the website code is big like this one: (https://www.pixel4k.com/page/1?s=deadpool) i dont get the full code

i tried Jsoup and HttpURLConnection and dosnt give me the full code

this is the code i have

        Thread thread = new Thread(() -> {
            try  {
                Document doc;
                doc = Jsoup.connect(editText.getText().toString())
                        .header("Accept-Encoding", "gzip, deflate")
                        .userAgent("Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.69 Safari/537.36")
                        .maxBodySize(0)
                        .timeout(0)
                        .get();
                Log.i("IMPORTANT !!!!","doc ( "+editText.getText().toString()+" )\n"+doc);
            } catch (Exception e) {
                Log.i("IMPORTANT !!!!","error : "+e);
            }
        });
        thread.start();

and this is the code i get from this website (https://www.pixel4k.com/page/1?s=deadpool)

    <!doctype html>
<html class="no-js" lang="en-US" prefix="og: http://ogp.me/ns#"> 
 <head> 
  <meta charset="UTF-8"> 
  <title>You searched for deadpool - 4k Wallpapers ,Hd Wallpapers,Desktop Wallpapers, Free Backgrounds Download, Widescreen Wallpapers</title> 
  <link rel="icon" href="https://www.pixel4k.com/wp-content/uploads/2018/09/favicon.ico" type="image/x-icon"> 
  <link rel="apple-touch-icon" href="apple-touch-icon.png"> 
  <meta name="viewport" content="width=device-width, initial-scale=1.0"> 
  <meta name="apple-mobile-web-app-capable" content="yes"> 
  <meta name="apple-mobile-web-app-status-bar-style" content="black"> 
  <link rel="stylesheet" type="text/css" media="all" href="https://www.pixel4k.com/wp-content/themes/pxxx/style.css"> 
  <link rel="pingback" href="https://www.pixel4k.com/xmlrpc.php"> 
  <meta name="google-site-verification" content="xHAo1q6wJG7bz-iw00VylrwaMabFjK_xSyU1jakgwaQ"> 
  <meta name="wot-verification" content="317f71c46e1fb6060ce1"> 
  <script async src="//pagead2.googlesyndication.com/pagead/js/adsbygoogle.js" type="f8f50ad6803275492fa5ce1d-text/javascript"></script> 
  <script type="f8f50ad6803275492fa5ce1d-text/javascript">(adsbygoogle=window.adsbygoogle||[]).push({google_ad_client:"ca-pub-2555268506534283",enable_page_level_ads:true});</script> <!--[if lt IE 9]>
    <script src="https://html5shim.googlecode.com/svn/trunk/html5.js"></script>
    <![endif]--> 
  <meta name="robots" content="noindex,follow"> 
  <link rel="next" href="https://www.pixel4k.com/search/deadpool/page/2"> 
  <meta property="og:locale" content="en_US"> 
  <meta property="og:type" content="object"> 
  <meta property="og:title" content="You searched for deadpool - 4k Wallpapers ,Hd Wallpapers,Desktop Wallpapers, Free Backgrounds Download, Widescreen Wallpapers"> 
  <meta property="og:url" content="https://www.pixel4k.com/search/deadpool"> 
  <meta property="og:site_name" content="4k Wallpapers ,Hd Wallpapers,Desktop Wallpapers, Free Backgrounds Download, Widescreen Wallpapers"> 
  <meta name="twitter:card" content="summary_large_image"> 
  <meta name="twitter:title" content="You searched for deadpool - 4k Wallpapers ,Hd Wallpapers,Desktop Wallpapers, Free Backgrounds Download, Widescreen Wallpapers"> 
  <script type="application/ld+json">{"@context":"https:\/\/schema.org","@type":"Person","url":"https:\/\/www.pixel4k.com\/","sameAs":[],"@id":"#person","name":"Mika"}</script> 
  <link rel="dns-prefetch" href="//ajax.googleapis.com"> 
  <link rel="dns-prefetch" href="//www.pixel4k.com"> 
  <link rel="alternate" type="application/rss+xml" title="4k Wallpapers ,Hd Wallpapers,Desktop Wallpapers, Free Backgrounds Download, Widescreen Wallpapers » Feed" href="https://www.pixel4k.com/feed"> 
  <link rel="alternate" type="application/rss+xml" title="4k Wallpapers ,Hd Wallpapers,Desktop Wallpapers, Free Backgrounds Download, Widescreen Wallpapers » Comments Feed" href="https://www.pixel4k.com/comments/feed"> 
  <link rel="alternate" type="application/rss+xml" title="4k Wallpapers ,Hd Wallpapers,Desktop Wallpapers, Free Backgrounds Download, Widescreen Wallpapers » Search Results for “deadpool” Feed" href="https://www.pixel4k.com/search/deadpool/feed/rss2/"> 
  <style type="text/css">img.wp-smiley,img.emoji{display:inline!important;border:none!important;box-shadow:none!important;height:1em!important;width:1em!important;margin:0 .07em!important;vertical-align:-.1em!important;background:none!important;padding:0!important}</style> 
  <link rel="stylesheet" id="wp-block-library-css" href="https://www.pixel4k.com/wp-includes/css/dist/block-library/style.min.css?ver=5.3.8" type="text/css" media="all"> 
  <style id="rocket-lazyload-inline-css" type="text/css">.rll-youtube-player{position:relative;padding-bottom:56.23%;height:0;overflow:hidden;max-width:100%;background:#000;margin:5px}.rll-youtube-player iframe{position:absolute;top:0;left:0;width:100%;height:100%;z-index:100;background:0 0}.rll-youtube-player img{bottom:0;display:block;left:

but this app (https://play.google.com/store/apps/details?id=com.teejay.trebedit&hl=en&gl=US) get the full code

what shuld i do ?

I suspect the thread die before you get the whole document. Could you put ``Thread.sleep(20 * 1_000 )`` after thread.start and see if it changes the behavior ? — spi, May 31 '21 at 15:40

score 0 · Answer 1 · answered Jun 01 '21 at 17:00

You are getting all the data (your two urls and your code produce complete html), but the android logger does not output everything when you call it.

If you try to write a file instead of a log statement, you most likely will notice that all your data are available.

Cf. What is the size limit for Logcat and how to change its capacity?

score -1 · Answer 2 · answered May 31 '21 at 18:41

I searched the maximum length of a String in Java. According to Takahiko Kawasaki in this question, the maximum length is 65536 characters.

Since the method you used writes the HTML code of a web page in a String, that means your code will work as intended if the web page you are trying to download is less than 65.536 bytes.

I do not know what you need to do after you acquired the HTML code of a web page, so the following suggestion may not be adequate to your necessities, but: have you tried storing the HTML code in a StringBuffer instead of a String?

how to get full html code from website using jsoub or any other

2 Answers2