25

I am using beautiful soup and I am writing a crawler and have the following code in it:

  print soup.originalEncoding
                #self.addtoindex(page, soup)


                links=soup('a')
            for link in links:

                if('href' in dict(link.attrs)):                   
                    link['href'].replace('..', '')
                    url=urljoin(page, link['href'])
                    if url.find("'") != -1:
                        continue
                    url = url.split('?')[0]
                    url = url.split('#')[0]
                    if url[0:4] == 'http':
                        newpages.add(url)
        pages = newpages

The link['href'].replace('..', '') is supposed to fix links that come out as ../contact/orderform.aspx, ../contact/requestconsult.aspx, etc. However, it is not working. Links still have the leading ".." Is there something I am missing?

John Machin
  • 81,303
  • 11
  • 141
  • 189
sdiener
  • 273
  • 1
  • 3
  • 5

3 Answers3

64

string.replace() returns the string with the replaced values. It doesn't modify the original so do something like this:

link['href'] = link['href'].replace("..", "")
mechanical_meat
  • 163,903
  • 24
  • 228
  • 223
joel goldstick
  • 4,393
  • 6
  • 30
  • 46
15

string.replace() returns a copy of the string with characters replaced, as strings in Python are immutable. Try

s = link['href'].replace("..", '')
url=urljoin(page, s)
jan zegan
  • 1,629
  • 1
  • 12
  • 18
11

It is not an inplace replacement. You need to do:

link['href'] = link['href'].replace('..', '')

Example:

a = "abc.."
print a.replace("..","")
'abc'
 print a
'abc..'
a = a.replace("..","")
print a
'abc'
Urjit
  • 1,150
  • 8
  • 15