Stack Overflow
Stack Overflow

Questions tagged [urlopen]

The urlopen is a method of the urllib library in Python, used to open a particular URL.

The urlopen is a method of the urllib library in Python, used to open a particular URL. As a result, a file-like object is returned that contains information about the URL - headers, response data and other details about the requested URL resource.

369 questions
186
votes
12 answers

Let JSON object accept bytes or let urlopen output strings

With Python 3 I am requesting a json document from a URL. response = urllib.request.urlopen(request) The response object is a file-like object with read and readline methods. Normally a JSON object can be created with a file opened in text…
Peter Smit
  • 27,696
  • 33
  • 111
  • 170
82
votes
10 answers

Python check if website exists

I wanted to check if a certain website exists, this is what I'm doing: user_agent = 'Mozilla/20.0.1 (compatible; MSIE 5.5; Windows NT)' headers = { 'User-Agent':user_agent } link = "http://www.abc.com" req = urllib2.Request(link, headers =…
James Hallen
  • 4,534
  • 4
  • 23
  • 28
65
votes
10 answers

Python 3.5.1 urllib has no attribute request

I have tried import urllib.request or import urllib The path for my urllib is /Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/__init__.py I am wondering where is urlopen, or is my python module pointing to the wrong file?
user1999806
  • 681
  • 1
  • 5
  • 10
55
votes
10 answers

How to fetch a non-ascii url with urlopen?

I need to fetch data from a URL with non-ascii characters but urllib2.urlopen refuses to open the resource and raises: UnicodeEncodeError: 'ascii' codec can't encode character u'\u0131' in position 26: ordinal not in range(128) I know the URL is…
onurmatik
  • 5,105
  • 7
  • 42
  • 67
54
votes
5 answers

Parsing HTTP Response in Python

I want to manipulate the information at THIS url. I can successfully open it and read its contents. But what I really want to do is throw out all the stuff I don't want, and to manipulate the stuff I want to keep. Is there a way to convert the…
Colton Allen
  • 2,940
  • 2
  • 23
  • 33
29
votes
11 answers

How can I speed up fetching pages with urllib2 in python?

I have a script that fetches several web pages and parses the info. (An example can be seen at http://bluedevilbooks.com/search/?DEPT=MATH&CLASS=103&SEC=01 ) I ran cProfile on it, and as I assumed, urlopen takes up a lot of time. Is there a way to…
Parker
  • 8,539
  • 10
  • 69
  • 98
29
votes
6 answers

timeout for urllib2.urlopen() in pre Python 2.6 versions

The urllib2 documentation says that timeout parameter was added in Python 2.6. Unfortunately my code base has been running on Python 2.5 and 2.4 platforms. Is there any alternate way to simulate the timeout? All I want to do is allow the code to…
rubayeet
  • 9,269
  • 8
  • 46
  • 55
27
votes
5 answers

Use "byte-like object" from urlopen.read with JSON?

Just trying to test out very simple Python JSON commands, but I'm having some trouble. urlopen('http://www.similarsitesearch.com/api/similar/ebay.com').read() should…
r123454321
  • 3,323
  • 11
  • 44
  • 63
19
votes
3 answers

python urllib2 urlopen response

python urllib2 urlopen response: > expected: {"token":"mYWmzpunvasAT795niiR"}
Shown
  • 201
  • 1
  • 2
  • 7
17
votes
4 answers

How to use urllib2.urlopen to make POST request without data argument

I am trying to use urllib2.urlopen to perform GET and POST requests via the Facebook Graph API. I noticed from here: https://stackoverflow.com/questions/2690723/facebook-graph-api-and-django that I can perform GET request fairly easily. And from…
airfang
  • 1,238
  • 3
  • 14
  • 22
17
votes
2 answers

ImportError : cannot import name urlopen

I am trying to open up an URL for my project and here is my code: from urllib2 import urlopen page = urlopen("https://docs.python.org/3/howto/urllib2.html") contents = page.read() It's just a simple code for a demo however, when I run the codes, I…
plzhelp
  • 185
  • 1
  • 3
  • 9
17
votes
3 answers

Tell urllib2 to use custom DNS

I'd like to tell urllib2.urlopen (or a custom opener) to use 127.0.0.1 (or ::1) to resolve addresses. I wouldn't change my /etc/resolv.conf, however. One possible solution is to use a tool like dnspython to query addresses and httplib to build a…
Attila O.
  • 15,659
  • 11
  • 54
  • 84
15
votes
3 answers

How to set TCP_NODELAY flag when loading URL with urllib2?

I am using urllib2 for loading web-page, my code is: httpRequest = urllib2.Request("http:/www....com") pageContent = urllib2.urlopen(httpRequest) pageContent.readline() How can I get hold of the socket properties to set TCP_NODELAY? In normal…
Andrey Rubliov
  • 1,359
  • 2
  • 17
  • 24
14
votes
5 answers

Does urllib2.urlopen() cache stuff?

They didn't mention this in python documentation. And recently I'm testing a website simply refreshing the site using urllib2.urlopen() to extract certain content, I notice sometimes when I update the site urllib2.urlopen() seems not get the newly…
Shane
  • 4,875
  • 12
  • 49
  • 87
14
votes
2 answers

with and closing of files in Python

I have read, that file opened like this is closed automatically when leaving the with block: with open("x.txt") as f: data = f.read() do something with data yet when opening from web, I need this: from contextlib import closing from…
nekomimi
  • 1,310
  • 3
  • 10
  • 14
1
2 3
…
24 25