How do I download a website using python 3?

Question

Just an http get request like file_get_contents in php, or something simple where I enter URL and it get contents to variable.

http://docs.python.org/py3k/library/urllib.request.html – Ignacio Vazquez-Abrams Dec 19 '10 at 21:12 — Ignacio Vazquez-Abrams, Dec 19 '10 at 21:12

score 31 · Accepted Answer · answered Dec 19 '10 at 21:11

31

Use urllib:

from urllib.request import urlopen
html = urlopen("http://www.stackoverflow.com/").read().decode('utf-8')
print(html)

answered Dec 19 '10 at 21:11

alexn

57,867
14
111
145

1

And you can't assume it's UTF-8, really. Either don't decode it, or check the headers. – Lennart Regebro Dec 20 '10 at 02:37
2

This only gets the HTML though. Is there a way to get images and external js files and stuff? Or do we just go through the html and look for the tags, because on some sites that takes a while – rassa45 Jun 28 '15 at 16:32
3

SSLCertVerificationError - resolved by using `import requests` instead – MrMartin Dec 04 '19 at 16:32

score 0 · Answer 2 · answered Sep 28 '20 at 22:15

Your Intention seems to be to get a static version of a Website. This can be achieved using WGET. This tool can in one command retrieve the files for a given URL. Use the -r (recursive) Parameter with care, example:

wget -erobots=off -p -H -k -E -w 1 --random-wait -r -l 1 http://your.site.

Download a working local copy of a webpage

How do I download a website using python 3?

2 Answers2