Questions tagged [urllib]

Python module providing a high-level interface for fetching data across the World Wide Web. Predecessor to urllib2. In Python 3, urllib2 and urllib have been reorganized and merged into urllib.

For Python 2, the urllib module is the predecessor of the urllib2 module, while the latter still uses some functionality of the former.

For Python 3, the urllib package was reorganized. It now has no content of its own. All methods and classes are in several submodules:

Note that doesn't exist in Python 3 anymore.

3960 questions
1119
votes
30 answers

How to download a file over HTTP?

I have a small utility that I use to download an MP3 file from a website on a schedule and then builds/updates a podcast XML file which I've added to iTunes. The text processing that creates/updates the XML file is written in Python. However, I use…
Owen
  • 22,247
  • 13
  • 42
  • 47
1005
votes
11 answers

What are the differences between the urllib, urllib2, urllib3 and requests module?

In Python, what are the differences between the urllib, urllib2, urllib3 and requests modules? Why are there three? They seem to do the same thing...
Paul Biggar
  • 27,579
  • 21
  • 99
  • 152
749
votes
15 answers

How to urlencode a querystring in Python?

I am trying to urlencode this string before I submit. queryString = 'eventName=' + evt.fields["eventName"] + '&' + 'eventDescription=' + evt.fields["eventDescription"];
James
  • 9,694
  • 5
  • 32
  • 38
511
votes
12 answers

UnicodeEncodeError: 'charmap' codec can't encode characters

I'm trying to scrape a website, but it gives me an error. I'm using the following code: import urllib.request from bs4 import BeautifulSoup get = urllib.request.urlopen("https://www.website.com/") html = get.read() soup = BeautifulSoup(html) And…
SstrykerR
  • 7,982
  • 3
  • 12
  • 11
438
votes
47 answers

urllib and "SSL: CERTIFICATE_VERIFY_FAILED" Error

I am getting the following error: Exception in thread Thread-3: Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner self.run() File…
user3724476
  • 4,720
  • 3
  • 15
  • 20
405
votes
6 answers

How can I percent-encode URL parameters in Python?

If I do url = "http://example.com?p=" + urllib.quote(query) It doesn't encode / to %2F (breaks OAuth normalization) It doesn't handle Unicode (it throws an exception) Is there a better library?
Paul Tarjan
  • 48,968
  • 59
  • 172
  • 213
325
votes
7 answers

How to send POST request?

I found this script online: import httplib, urllib params = urllib.urlencode({'number': 12524, 'type': 'issue', 'action': 'show'}) headers = {"Content-type": "application/x-www-form-urlencoded", "Accept": "text/plain"} conn =…
user1113569
  • 3,441
  • 2
  • 14
  • 10
222
votes
20 answers

Downloading a picture via urllib and python

So I'm trying to make a Python script that downloads webcomics and puts them in a folder on my desktop. I've found a few similar programs on here that do something similar, but nothing quite like what I need. The one that I found most similar is…
Mike
  • 2,247
  • 2
  • 14
  • 5
193
votes
13 answers

AttributeError: 'module' object has no attribute 'urlopen'

I'm trying to use Python to download the HTML source code of a website but I'm receiving this error. Traceback (most recent call last): File "C:\Users\Sergio.Tapia\Documents\NetBeansProjects\DICParser\src\WebDownload.py", line 3, in
delete
162
votes
4 answers

How to convert a dictionary to query string in Python?

After using cgi.parse_qs(), how to convert the result (dictionary) back to query string? Looking for something similar to urllib.urlencode().
Sam
  • 4,727
  • 9
  • 23
  • 17
158
votes
10 answers

can we use XPath with BeautifulSoup?

I am using BeautifulSoup to scrape an URL and I had the following code, to find the td tag whose class is 'empformbody': import urllib import urllib2 from BeautifulSoup import BeautifulSoup url = …
Shiva Krishna Bavandla
  • 25,548
  • 75
  • 193
  • 313
151
votes
9 answers

Download Returned Zip file from URL

If I have a URL that, when submitted in a web browser, pops up a dialog box to save a zip file, how would I go about catching and downloading this zip file in Python?
user1229108
  • 1,621
  • 3
  • 13
  • 9
151
votes
5 answers

Python: Importing urllib.quote

I would like to use urllib.quote(). But python (python3) is not finding the module. Suppose, I have this line of code: print(urllib.quote("châteu", safe='')) How do I import urllib.quote? import urllib or import urllib.quote both…
imrek
  • 2,930
  • 3
  • 20
  • 36
143
votes
4 answers

In Python, how do I use urllib to see if a website is 404 or 200?

How to get the code of the headers through urllib?
TIMEX
  • 259,804
  • 351
  • 777
  • 1,080
137
votes
3 answers

'module' has no attribute 'urlencode'

When I try to follow the Python Wiki's example related to URL encoding: >>> import urllib >>> params = urllib.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0}) >>> f = urllib.urlopen("http://www.musi-cal.com/cgi-bin/query", params) >>> print…
Croll
  • 3,631
  • 6
  • 30
  • 63
1
2 3
99 100