Python recursive crawling for urls

Question

I have this method that when supplied with a list of links will get the child links and so on and so forth:

def crawlSite(self, linksList):
    finalList = []
    for link in list(linksList):
        if link not in finalList:
            print link            
            finalList.append(link)
            childLinks = self.getAllUniqueLinks(link)
            length = len(childLinks)
            print 'Total links for this page: ' + str(length)

        self.crawlSite(childLinks)
    return finalList

It eventually will repeat itself with the same set of links and I can't seem to figure it out. When I move the self.crawlSite(childLinks) inside of the if statement. I get the first item in the list repeated over and over.

Background on the self.getAllUniqueLinks(link) method get a list of links from a given page. It filters to all click-able links within a given domain. Basically what I am trying to do is get all click-able links from a website. If this isn't the desired approach. Could you recommend a better method that can do the exact same thing. Please also consider that I am fairly new to python and might not understand more complex approaches. So please explain your thought processes. If you don't mind :)

agf · Answer 1 · 2011-07-21T06:58:35.903

You need

finalList.extend(self.crawlSite(childLinks))

not just

self.crawlSite(childLinks)

You need to merge the list returned by the inner crawlSite()s with the list already extant in the outer crawlSite(). Even though they're all called finalList, you have a different list in each scope.

The alternative (and better) solution is to have finalList be an instance variable (or nonlocal variable of some type) instead of just a local variable, so that it's shared by all the scopes of the crawlSite()s:

def __init__(self, *args, **kwargs):
    self.finalList = set()

def crawlSite(self, linksList):
    for link in linksList:
        if link not in self.finalList:
            print link            
            self.finalList.add(link)
            childLinks = self.getAllUniqueLinks(link)
            length = len(childLinks)
            print 'Total links for this page: ' + str(length)
            self.crawlSite(childLinks)

You just need to make sure you self.finalList = [] if you want to start over from scratch with the same instance.

Edit: Fixed the code by putting the recursive call in the if block. Used a set. Also, linksList doesn't need to be a list, just an iterable object, so removed the list() call from the for loop. Set was suggested by @Ray-Toal

score 2 · Answer 2 · edited May 23 '17 at 11:47

2

You are clearing the finalLinks array on each recursive call.

What is needed is a more global set of links that you have already visited. Each recursive call should contribute to this global list, otherwise, if your graph has cycles, you are sure to eventually visit a site twice.

UPDATE: Check out the nice pattern used in DFS on a graph using a python generator. Your finalList can be a parameter, with a default of []. Add to this list in each recursive call. Also, FWIW, consider a set rather than a list --- it is faster.

edited May 23 '17 at 11:47

Community

1
1

answered Jul 21 '11 at 06:21

Ray Toal

86,166
18
182
232

Good alternatives. Using a default parameter is less flexible than an instance variable though, as you can't clear it; you need to create a new instance if you want to start over. – agf Jul 21 '11 at 06:39
@agf, true, but the approach with the parameter is threadsafe. It's a tradeoff. :) – Ray Toal Jul 21 '11 at 06:44

Python recursive crawling for urls

2 Answers2