Can't parse specific site with wget

Question

I'm trying to parse https://app.xdao.app/ site using wget.

I used:

wget https://app.xdao.app/

and

wget -r -p https://app.xdao.app/

Always I got the same result:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <link rel="icon" type="image/svg+xml" href="/assets/img/favicon.svg" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>xDAO App</title>
  <script type="module" crossorigin src="/assets/index.e975bc04.js"></script>
  <link rel="modulepreload" href="/assets/vendor.ab9443f5.js">
  <link rel="stylesheet" href="/assets/index.578c0be9.css">
</head>
  <body>
    <div id="app"></div>
    
  </body>
</html>

Why? I want to get more full htm file, like I got result with yahoo.com for instance. I tried curl and got the same empty htm also.

PS: also when I do save as web page in FF I also get full html.

Here page downloaded with FF:

<!DOCTYPE html>
<html lang="en"><head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
    <meta charset="UTF-8">
    <link rel="icon" type="image/svg+xml" href="https://app.xdao.app/assets/img/favicon.svg">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>xDAO App</title>
  <script type="module" crossorigin="" src="xDAO%20App_files/index.js"></script>
  <link rel="modulepreload" href="https://app.xdao.app/assets/vendor.ab9443f5.js">
  <link rel="stylesheet" href="xDAO%20App_files/index.css">
</head>
  <body>
    <div id="app"> <div class="fixed left-0 right-0 bottom-2 text-center z-50"></div> <div class="flex justify-between p-2 lg:p-4 items-center  w-full"><img src="xDAO%20App_files/logo_full_blue.svg" class="self-start h-10 hidden md:block ml-5 mt-1" alt=""> <img src="xDAO%20App_files/logo_blue.svg" class="h-10 md:hidden" alt=""> <div class="md:flex space-x-4 items-center self-end hidden"> <a class="btn btn-ghost" href="#/create"><svg xmlns="http://www.w3.org/2000/svg" width="24px" height="24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round" class="feather feather-plus-circle "><circle cx="12" cy="12" r="10"></circle><line x1="12" y1="8" x2="12" y2="16"></line><line x1="8" y1="12" x2="16" y2="12"></line></svg> <p>Create DAO</p></a> <button class="btn btn-primary">Connect Wallet</button></div> <div class="flex space-x-2 items-center self-end md:hidden"> <a class="btn-mobile btn-mobile-ghost" href="#/create"><svg xmlns="http://www.w3.org/2000/svg" width="24px" height="24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round" class="feather feather-plus-circle "><circle cx="12" cy="12" r="10"></circle><line x1="12" y1="8" x2="12" y2="16"></line><line x1="8" y1="12" x2="16" y2="12"></line></svg> <p>Create DAO</p></a> <button class="btn-mobile btn-mobile-primary">Connect Wallet</button></div></div> <div class="flex"><aside class="h-screen top-0 sticky"><div class="hidden md:flex md:flex-col md:space-y-2 md:px-5 md:py-12 md:w-64"><a href="#/"><div class="flex flex-col md:flex-row items-center space-y-2 md:space-y-0 md:space-x-4 px-3 py-3 md:py-4 md:px-5 rounded-xl text-grey transition duration-200 hover:bg-hovergrey hover:text-primary svelte-1i2lpd1 chosenbutton"><svg xmlns="http://www.w3.org/2000/svg" width="24px" height="24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round" class="feather feather-home "><path d="M3 9l9-7 9 7v11a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2z"></path><polyline points="9 22 9 12 15 12 15 22"></polyline></svg> <div class="text-captionsm md:text-title font-semibold"><p slot="text">Home</p></div></div></a> <a href="#/ecosystem"><div class="flex flex-col md:flex-row items-center space-y-2 md:space-y-0 md:space-x-4 px-3 py-3 md:py-4 md:px-5 rounded-xl text-grey transition duration-200 hover:bg-hovergrey hover:text-primary svelte-1i2lpd1"><svg xmlns="http://www.w3.org/2000/svg" width="24px" height="24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round" class="feather feather-grid "><rect x="3" y="3" width="7" height="7"></rect><rect x="14" y="3" width="7" height="7"></rect><rect x="14" y="14" width="7" height="7"></rect><rect x="3" y="14" width="7" height="7"></rect></svg> <div class="text-captionsm md:text-title font-semibold"><p slot="text">Ecosystem</p></div></div></a> <a href="#/mydaos"><div class="flex flex-col md:flex-row items-center space-y-2 md:space-y-0 md:space-x-4 px-3 py-3 md:py-4 md:px-5 rounded-xl text-grey transition duration-200 hover:bg-hovergrey hover:text-primary svelte-1i2lpd1"><svg xmlns="http://www.w3.org/2000/svg" width="24px" height="24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round" class="feather feather-list "><line x1="8" y1="6" x2="21" y2="6"></line><line x1="8" y1="12" x2="21" y2="12"></line><line x1="8" y1="18" x2="21" y2="18"></line><line x1="3" y1="6" x2="3.01" y2="6"></line><line x1="3" y1="12" x2="3.01" y2="12"></line><line x1="3" y1="18" x2="3.01" y2="18"></line></svg> <div class="text-captionsm md:text-title font-semibold"><p slot="text">My DAOs</p></div></div></a> <a href="#/create"><div class="flex flex-col md:flex-row items-center space-y-2 md:space-y-0 md:space-x-4 px-3 py-3 md:py-4 md:px-5 rounded-xl text-grey transition duration-200 hover:bg-hovergrey hover:text-primary svelte-1i2lpd1"><svg xmlns="http://www.w3.org/2000/svg" width="24px" height="24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round" class="feather feather-plus-circle "><circle cx="12" cy="12" r="10"></circle><line x1="12" y1="8" x2="12" y2="16"></line><line x1="8" y1="12" x2="16" y2="12"></line></svg> <div class="text-captionsm md:text-title font-semibold"><p slot="text">Create</p></div></div></a></div></aside> <main class="flex-1 py-8 px-4 md:py-12 md:px-16 lg:py-9 lg:px-16 w-full"><div class="card"><p class="text-h4 font-semibold">4 Simple Steps to Start</p> <div class="w-full grid grid-cols-4 gap-8 mt-4"><div class="col-span-4 md:col-span-2 lg:col-span-1"><a href="https://metamask.io/download.html" target="_blank"><img src="xDAO%20App_files/01.svg" alt="" class="w-full"></a></div> <div class="col-span-4 md:col-span-2 lg:col-span-1"><button class="w-full"><img src="xDAO%20App_files/02.svg" alt="" class="w-full"></button></div> <div class="col-span-4 md:col-span-2 lg:col-span-1"><button class="w-full"><img src="xDAO%20App_files/03.svg" alt="" class="w-full"></button></div> <div class="col-span-4 md:col-span-2 lg:col-span-1"><a href="#/create"><img src="xDAO%20App_files/04.svg" alt="" class="w-full"></a></div></div> <a href="https://docs.xdao.app/manual/how-to-start" target="_blank"><button class="btn btn-ghost w-full h-7.5 mt-12"><svg xmlns="http://www.w3.org/2000/svg" width="18px" height="18px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="feather feather-book "><path d="M4 19.5A2.5 2.5 0 0 1 6.5 17H20"></path><path d="M6.5 2H20v20H6.5A2.5 2.5 0 0 1 4 19.5v-15A2.5 2.5 0 0 1 6.5 2z"></path></svg> <p>Guide</p></button></a></div> <div class="card mt-2"><div class="grid grid-cols-2 gap-8"><div class="col-span-2 lg:col-span-1"><p class="text-h6 font-medium">About xDAO Project</p> <p class="text-body text-grey mt-1">xDAO is a DeFi protocol for quick and easy creation of Decentralized
        Autonomous Organizations — DAOs. Combine your crypto assets to manage
        them in more efficient and secure way by using auto-generated smart
        contracts. A clear voting system allows you to make collective decisions
        and be confident in their exact execution.</p></div> <div class="col-span-2 lg:col-span-1 space-y-2"><div class="flex space-x-2 items-center text-primary"><svg xmlns="http://www.w3.org/2000/svg" width="22px" height="22px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="feather feather-shield "><path d="M12 22s8-4 8-10V5l-8-3-8 3v7c0 6 8 10 8 10z"></path></svg> <a href="https://github.com/pessimistic-io/audits/blob/main/xDao%20Security%20Analysis%20by%20Pessimistic.pdf" class="font-medium underline" target="_blank">Audit</a></div> <div class="flex space-x-2 items-center text-primary"><svg xmlns="http://www.w3.org/2000/svg" width="22px" height="22px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="feather feather-code "><polyline points="16 18 22 12 16 6"></polyline><polyline points="8 6 2 12 8 18"></polyline></svg> <a href="https://github.com/xDAO-App/xdaocontracts" target="_blank" class="font-medium underline">Source Code</a></div> <div class="flex space-x-2 items-center text-primary"><svg xmlns="http://www.w3.org/2000/svg" width="22px" height="22px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="feather feather-link "><path d="M10 13a5 5 0 0 0 7.54.54l3-3a5 5 0 0 0-7.07-7.07l-1.72 1.71"></path><path d="M14 11a5 5 0 0 0-7.54-.54l-3 3a5 5 0 0 0 7.07 7.07l1.71-1.71"></path></svg> <a href="https://xdao.app/" class="font-medium underline" target="_blank">Official Website</a></div> <div class="flex space-x-2 items-center text-primary"><svg xmlns="http://www.w3.org/2000/svg" width="22px" height="22px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="feather feather-book "><path d="M4 19.5A2.5 2.5 0 0 1 6.5 17H20"></path><path d="M6.5 2H20v20H6.5A2.5 2.5 0 0 1 4 19.5v-15A2.5 2.5 0 0 1 6.5 2z"></path></svg> <a href="https://docs.xdao.app/" target="_blank" class="font-medium underline">Docs</a></div></div></div></div></main></div> <div class="bg-darkactive fixed bottom-0 w-full md:hidden grid grid-cols-4 gap-2 z-50 p-2"><div class="col-span-1"><a href="#/"><div class="flex flex-col md:flex-row items-center space-y-2 md:space-y-0 md:space-x-4 px-3 py-3 md:py-4 md:px-5 rounded-xl text-grey transition duration-200 hover:bg-hovergrey hover:text-primary svelte-1i2lpd1 chosenbutton"><svg xmlns="http://www.w3.org/2000/svg" width="24px" height="24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round" class="feather feather-home "><path d="M3 9l9-7 9 7v11a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2z"></path><polyline points="9 22 9 12 15 12 15 22"></polyline></svg> <div class="text-captionsm md:text-title font-semibold"><p slot="text">Home</p></div></div></a></div> <div class="col-span-1"><a href="#/ecosystem"><div class="flex flex-col md:flex-row items-center space-y-2 md:space-y-0 md:space-x-4 px-3 py-3 md:py-4 md:px-5 rounded-xl text-grey transition duration-200 hover:bg-hovergrey hover:text-primary svelte-1i2lpd1"><svg xmlns="http://www.w3.org/2000/svg" width="24px" height="24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round" class="feather feather-grid "><rect x="3" y="3" width="7" height="7"></rect><rect x="14" y="3" width="7" height="7"></rect><rect x="14" y="14" width="7" height="7"></rect><rect x="3" y="14" width="7" height="7"></rect></svg> <div class="text-captionsm md:text-title font-semibold"><p slot="text">Ecosystem</p></div></div></a></div> <div class="col-span-1"><a href="#/mydaos"><div class="flex flex-col md:flex-row items-center space-y-2 md:space-y-0 md:space-x-4 px-3 py-3 md:py-4 md:px-5 rounded-xl text-grey transition duration-200 hover:bg-hovergrey hover:text-primary svelte-1i2lpd1"><svg xmlns="http://www.w3.org/2000/svg" width="24px" height="24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round" class="feather feather-list "><line x1="8" y1="6" x2="21" y2="6"></line><line x1="8" y1="12" x2="21" y2="12"></line><line x1="8" y1="18" x2="21" y2="18"></line><line x1="3" y1="6" x2="3.01" y2="6"></line><line x1="3" y1="12" x2="3.01" y2="12"></line><line x1="3" y1="18" x2="3.01" y2="18"></line></svg> <div class="text-captionsm md:text-title font-semibold"><p slot="text">My DAOs</p></div></div></a></div> <div class="col-span-1"><a href="#/create"><div class="flex flex-col md:flex-row items-center space-y-2 md:space-y-0 md:space-x-4 px-3 py-3 md:py-4 md:px-5 rounded-xl text-grey transition duration-200 hover:bg-hovergrey hover:text-primary svelte-1i2lpd1"><svg xmlns="http://www.w3.org/2000/svg" width="24px" height="24px" fill="none" viewBox="0 0 24 24" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round" class="feather feather-plus-circle "><circle cx="12" cy="12" r="10"></circle><line x1="12" y1="8" x2="12" y2="16"></line><line x1="8" y1="12" x2="16" y2="12"></line></svg> <div class="text-captionsm md:text-title font-semibold"><p slot="text">Create</p></div></div></a></div></div> <footer class="p-4 text-center text-grey "><p><a class="hover:underline" href="#/termsandconditions">Terms and Conditions</a>
    |

    <a class="hover:underline" href="#/privacypolicy">Privacy Policy</a> |

    <a class="hover:underline" href="#/amlpolicy">AML Policy</a> <br> <br> <a class="font-medium text-primary hover:underline" target="_blank" href="https://www.tis.bizfile.gov.sg/ngbtisinternet/faces/oracle/webcenter/portalapp/pages/TransactionMain.jspx?authNo=X21255309Q&amp;selectedETransId=prdAuth&amp;uen=202112101H">XGROUP GLOBAL PTE. LTD. Singapore</a> <span class="font-medium">| Powered by Binance Smart Chain</span></p></footer> <div class="h-24 md:hidden"></div></div>
   
</body></html>'

Thanks for any advices

score 1 · Answer 1 · answered May 26 '21 at 13:29

1

This

<script type="module" crossorigin src="/assets/index.e975bc04.js"></script>

is <script> tag, when your browser encounter that it does execute JavaScript instruction in file pointed by src. In this case it apparently fill page with content, thus you might get full html via Firefox. So far I know executing JavaScript is out of scope of wget. You would need tool which is able to execute JavaScript, not only download resource.

answered May 26 '21 at 13:29

Daweo

31,313
3
12
25

Thank you. Please check the file that I downloaded with FF (I added them in my post). As you can see there is nested tag
, but in code downloaded with wget only root one. Why? I don't need in js. All info in index is sufficient. But how can I get it correctly?
– vns955 May 26 '21 at 14:22
@vns955 : Daweo is correct.
is filled with content by the Javascript. I already tested, but you may test it for yourself: deactivate JS in browser and access the page - you'll find the element empty. Related SO answer: [link](https://stackoverflow.com/questions/28505501/get-the-content-text-of-an-url-after-javascript-has-run-with-php). Related resource: [link](https://www.webscrapingapi.com/php-web-scraping/) – Tinel Barb Sep 09 '21 at 05:43

Can't parse specific site with wget

1 Answers1