27

So my brother wanted me to write a web crawler in Python (self-taught) and I know C++, Java, and a bit of html. I'm using version 2.7 and reading the python library, but I have a few problems 1. httplib.HTTPConnection and request concept to me is new and I don't understand if it downloads an html script like cookie or an instance. If you do both of those, do you get the source for a website page? And what are some words that I would need to know to modify the page and return the modified page.

Just for background, I need to download a page and replace any img with ones I have

And it would be nice if you guys could tell me your opinion of 2.7 and 3.1

Dan
  • 554
  • 2
  • 10
  • 18

5 Answers5

48

Use Python 2.7, is has more 3rd party libs at the moment. (Edit: see below).

I recommend you using the stdlib module urllib2, it will allow you to comfortably get web resources. Example:

import urllib2

response = urllib2.urlopen("http://google.de")
page_source = response.read()

For parsing the code, have a look at BeautifulSoup.

BTW: what exactly do you want to do:

Just for background, I need to download a page and replace any img with ones I have

Edit: It's 2014 now, most of the important libraries have been ported, and you should definitely use Python 3 if you can. python-requests is a very nice high-level library which is easier to use than urllib2.

leoluk
  • 12,561
  • 6
  • 44
  • 51
11

An Example with python3 and the requests library as mentioned by @leoluk:

pip install requests

Script req.py:

import requests

url='http://localhost'

# in case you need a session
cd = { 'sessionid': '123..'}

r = requests.get(url, cookies=cd)
# or without a session: r = requests.get(url)
r.content

Now,execute it and you will get the html source of localhost!

python3 req.py

Timo
  • 2,922
  • 3
  • 29
  • 28
6

If you are using Python > 3.x you don't need to install any libraries, this is directly built in the python framework. The old urllib2 package has been renamed to urllib:

from urllib import request

response = request.urlopen("https://www.google.com")
# set the correct charset below
page_source = response.read().decode('utf-8')
print(page_source)
Caner
  • 57,267
  • 35
  • 174
  • 180
0

The first thing you need to do is read the HTTP spec which will explain what you can expect to receive over the wire. The data returned inside the content will be the "rendered" web page, not the source. The source could be a JSP, a servlet, a CGI script, in short, just about anything, and you have no access to that. You only get the HTML that the server sent you. In the case of a static HTML page, then yes, you will be seeing the "source". But for anything else you see the generated HTML, not the source.

When you say modify the page and return the modified page what do you mean?

Jim Garrison
  • 85,615
  • 20
  • 155
  • 190
  • for all img files on a certain page, replace with a new one – Dan Aug 20 '10 at 18:59
  • the link you sent me is very big. What are the minimums i should read – Dan Aug 20 '10 at 19:25
  • Google search for information about HTTP. This is the underlying protocol that carries the HTML from the server to your browser. I assume you already understand HTML and have a strategy for parsing it. If not, all the pieces are available but you will have some research and learning to do to put them together. – Jim Garrison Aug 20 '10 at 20:36
0

All the above will fail on an https request behind Cloudflare. You can try this to fetch both http and https html:

import requests
url = 'https://your.link.here'   
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36',}
response = requests.get(url, headers=headers)

if response.status_code == 200:
    print(response.text)
else:
    print(f'Request failed with status code: {response.status_code}')
pebox11
  • 3,377
  • 5
  • 32
  • 57