How to get the contents of a webpage in a shell variable?

Question

In Linux how can I fetch an URL and get its contents in a variable in shell script?

codaddict · Accepted Answer · 2010-09-23T13:06:10.033

230

You can use wget command to download the page and read it into a variable as:

content=$(wget google.com -q -O -)
echo $content

We use the -O option of wget which allows us to specify the name of the file into which wget dumps the page contents. We specify - to get the dump onto standard output and collect that into the variable content. You can add the -q quiet option to turn off's wget output.

You can use the curl command for this aswell as:

content=$(curl -L google.com)
echo $content

We need to use the -L option as the page we are requesting might have moved. In which case we need to get the page from the new location. The -L or --location option helps us with this.

edited Sep 23 '10 at 13:06

answered Sep 18 '10 at 18:46

codaddict

445,704
82
492
529

This is a really neat trick. I invoke a shell script via a php script on a proxy server. When asked, the proxy server turns on expensive servers which shut themselves off after 2 hours. I need the output from wget for standard output to feed back to the Jenkins console record. – Dennis Jul 12 '12 at 01:42
i am yet to get this... can anybody demostrate as to how , for eg. get an img tag in a variable for this link http://www2.watchop.io/manga2/read/one-piece/1/4 ?? – juggernauthk108 Oct 30 '16 at 09:05
1

@juggernaut1996: that should be a separate question. Briefly, you have to download the page, extract the `src` attribute of the correct element, then download *that* page. If you install [tq](https://github.com/plainas/tq), this command should do it: `curl -s http://ww1.watchop.io/manga2/read/one-piece/1/4 | tq -j -a src "#imgholder a img" | xargs wget` – pyrocrasty Feb 06 '17 at 02:11
Wget 1.14 version is not accepting `convert_links = on` with `-O-` option. It is failing with error `-k can be used together with -O only if outputting to a regular file.`. Is it expected? – Prasad Bonthu Jul 17 '18 at 14:40

score 38 · Answer 2 · edited Jan 31 '19 at 18:12

38

There are many ways to get a page from the command line... but it also depends if you want the code source or the page itself:

If you need the code source:

with curl:

curl $url

with wget:

wget -O - $url

but if you want to get what you can see with a browser, lynx can be useful:

lynx -dump $url

I think you can find so many solutions for this little problem, maybe you should read all man pages for those commands. And don't forget to replace $url by your URL :)

Good luck :)

edited Jan 31 '19 at 18:12

Benjamin W.

46,058
19
106
116

answered Sep 18 '10 at 20:43

julianvdb

389
2
2

If I were you, I'd double quote the url's. – Mar 23 '21 at 20:39

score 9 · Answer 3 · answered Sep 18 '10 at 18:47

9

There is the wget command or the curl.

You can now use the file you downloaded with wget. Or you can handle a stream with curl.

Resources :

answered Sep 18 '10 at 18:47

Colin Hebert

91,525
15
160
151

score 3 · Answer 4 · answered Sep 18 '10 at 18:55

3

content=`wget -O - $url`

answered Sep 18 '10 at 18:55

Jim Lewis

43,505
7
82
96

@rjack: (But the article you linked to does make a pretty good case for the $(...) syntax.) – Jim Lewis Sep 18 '10 at 19:33

score 3 · Answer 5 · answered Sep 18 '10 at 19:13

You can use curl or wget to retrieve the raw data, or you can use w3m -dump to have a nice text representation of a web page.

$ foo=$(w3m -dump http://www.example.com/); echo $foo
You have reached this web page by typing "example.com", "example.net","example.org" or "example.edu" into your web browser. These domain names are reserved for use in documentation and are not available for registration. See RFC 2606, Section 3.

score 3 · Answer 6 · answered Sep 18 '10 at 20:50

If you have LWP installed, it provides a binary simply named "GET".

$ GET http://example.com
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<HTML>
<HEAD>
  <META http-equiv="Content-Type" content="text/html; charset=utf-8">
  <TITLE>Example Web Page</TITLE>
</HEAD> 
<body>  
<p>You have reached this web page by typing &quot;example.com&quot;,
&quot;example.net&quot;,&quot;example.org&quot
  or &quot;example.edu&quot; into your web browser.</p>
<p>These domain names are reserved for use in documentation and are not available 
  for registration. See <a href="http://www.rfc-editor.org/rfc/rfc2606.txt">RFC 
  2606</a>, Section 3.</p>
</BODY>
</HTML>

wget -O-, curl, and lynx -source behave similarly.

score 2 · Answer 7 · edited Mar 23 '21 at 20:32

No curl, no wget, no ncat, nothing? Use telnet:

$ content=$(telnet localhost 80)
GET / HTTP/1.1
Host: localhost
Connection: close
 
Connection closed by foreign host.

$ echo $content
HTTP/1.1 200 OK Date: Mon, 22 Mar 2021 12:45:02 GMT Server:
Apache/2.4.46 (Fedora) OpenSSL/1.1.1j Last-Modified: Mon, 31 Dec 2018
15:56:45 GMT ETag: "a4-57e5375ad21bd" Accept-Ranges: bytes
Content-Length: 164 Connection: close Content-Type: text/html;
charset=UTF-8 Success! 192.168.1.1

How to get the contents of a webpage in a shell variable?

7 Answers7

Linked