How to write a crawler?

Question

I have had thoughts of trying to write a simple crawler that might crawl and produce a list of its findings for our NPO's websites and content.

Does anybody have any thoughts on how to do this? Where do you point the crawler to get started? How does it send back its findings and still keep crawling? How does it know what it finds, etc,etc.

Answer 1

You'll be reinventing the wheel, to be sure. But here's the basics:

• A list of unvisited URLs - seed this with one or more starting pages
• A list of visited URLs - so you don't go around in circles
• A set of rules for URLs you're not interested in - so you don't index the whole Internet

Put these in persistent storage, so you can stop and start the crawler without losing state.

Algorithm is:

while(list of unvisited URLs is not empty) {
    take URL from list
    remove it from the unvisited list and add it to the visited list
    fetch content
    record whatever it is you want to about the content
    if content is HTML {
        parse out URLs from links
        foreach URL {
           if it matches your rules
              and it's not already in either the visited or unvisited list
              add it to the unvisited list
        }
    }
}

Answer 2

The complicated part of a crawler is if you want to scale it to a huge number of websites/requests. In this situation you will have to deal with some issues like:

• Impossibility to keep info all in one database.

• Not enough RAM to deal with huge index(s)

• Multithread performance and concurrency

• Crawler traps (infinite loop created by changing urls, calendars, sessions ids...) and duplicated content.

• Crawl from more than one computer

• Malformed HTML codes

• Constant http errors from servers

• Databases without compression, wich make your need for space about 8x bigger.

• Recrawl routines and priorities.

• Use requests with compression (Deflate/gzip) (good for any kind of crawler).

And some important things

• Respect robots.txt

• And a crawler delay on each request to dont suffocate web servers.

Answer 3

Multithreaded Web Crawler

If you want to crawl large sized website then you should write a multi-threaded crawler. connecting,fetching and writing crawled information in files/database - these are the three steps of crawling but if you use a single threaded than your CPU and network utilization will be pour.

A multi threaded web crawler needs two data structures- linksVisited(this should be implemented as a hashmap or trai) and linksToBeVisited(this is a queue).

Web crawler uses BFS to traverse world wide web.

Algorithm of a basic web crawler:-

1. Add one or more seed urls to linksToBeVisited. The method to add a url to linksToBeVisited must be synchronized.
2. Pop an element from linksToBeVisited and add this to linksVisited. This pop method to pop url from linksToBeVisited must be synchronized.
3. Fetch the page from internet.
4. Parse the file and add any till now not visited link found in the page to linksToBeVisited. URL's can be filtered if needed. The user can give a set of rules to filter which url's to be scanned.
5. The necessary information found on the page is saved in database or file.

6. repeat step 2 to 5 until queue is linksToBeVisited empty.

   Here is a code snippet on how to synchronize the threads....

    public void add(String site) {
      synchronized (this) {
      if (!linksVisited.contains(site)) {
        linksToBeVisited.add(site);
        }
      }
    }
   
    public String next() {
       if (linksToBeVisited.size() == 0) {
       return null;
       }
          synchronized (this) {
           // Need to check again if size has changed
          if (linksToBeVisited.size() > 0) {
             String s = linksToBeVisited.get(0);
             linksToBeVisited.remove(0);
             linksVisited.add(s);
             return s;
          }
        return null;
        }
     }
   

Answer 4

Crawlers are simple in concept.

You get a root page via a HTTP GET, parse it to find URLs and put them on a queue unless they've been parsed already (so you need a global record of pages you have already parsed).

You can use the Content-type header to find out what the type of content is, and limit your crawler to only parsing the HTML types.

You can strip out the HTML tags to get the plain text, which you can do text analysis on (to get tags, etc, the meat of the page). You could even do that on the alt/title tags for images if you got that advanced.

And in the background you can have a pool of threads eating URLs from the Queue and doing the same. You want to limit the number of threads of course.

Answer 5

If your NPO's sites are relatively big or complex (having dynamic pages that'll effectively create a 'black hole' like a calendar with a 'next day' link) you'd be better using a real web crawler, like Heritrix.

If the sites total a few number of pages you can get away with just using curl or wget or your own. Just remember if they start to get big or you start making your script more complex to just use a real crawler or at least look at its source to see what are they doing and why.

Some issues (there are more):

• Black holes (as described)
• Retries (what if you get a 500?)
• Redirects
• Flow control (else you can be a burden on the sites)
• robots.txt implementation

Answer 6

Wikipedia has a good article about web crawlers, covering many of the algorithms and considerations.

However, I wouldn't bother writing my own crawler. It's a lot of work, and since you only need a "simple crawler", I'm thinking all you really need is an off-the-shelf crawler. There are a lot of free and open-source crawlers that will likely do everything you need, with very little work on your part.

Answer 7

You could make a list of words and make a thread for each word searched at google.
Then each thread will create a new thread for each link it find in the page.
Each thread should write what it finds in a database. When each thread finishes reading the page, it terminates.
And there you have a very big database of links in your database.

Answer 8

Use wget, do a recursive web suck, which will dump all the files onto your harddrive, then write another script to go through all the downloaded files and analyze them.

Edit: or maybe curl instead of wget, but I am not familiar with curl, I do not know if it does recursive downloads like wget.

Answer 9

I'm using Open search server for my company internal search, try this : http://open-search-server.com its also open soruce.

Answer 10

i did a simple web crawler using reactive extension in .net.

https://github.com/Misterhex/WebCrawler

public class Crawler
    {
    class ReceivingCrawledUri : ObservableBase<Uri>
    {
        public int _numberOfLinksLeft = 0;

        private ReplaySubject<Uri> _subject = new ReplaySubject<Uri>();
        private Uri _rootUri;
        private IEnumerable<IUriFilter> _filters;

        public ReceivingCrawledUri(Uri uri)
            : this(uri, Enumerable.Empty<IUriFilter>().ToArray())
        { }

        public ReceivingCrawledUri(Uri uri, params IUriFilter[] filters)
        {
            _filters = filters;

            CrawlAsync(uri).Start();
        }

        protected override IDisposable SubscribeCore(IObserver<Uri> observer)
        {
            return _subject.Subscribe(observer);
        }

        private async Task CrawlAsync(Uri uri)
        {
            using (HttpClient client = new HttpClient() { Timeout = TimeSpan.FromMinutes(1) })
            {
                IEnumerable<Uri> result = new List<Uri>();

                try
                {
                    string html = await client.GetStringAsync(uri);
                    result = CQ.Create(html)["a"].Select(i => i.Attributes["href"]).SafeSelect(i => new Uri(i));
                    result = Filter(result, _filters.ToArray());

                    result.ToList().ForEach(async i =>
                    {
                        Interlocked.Increment(ref _numberOfLinksLeft);
                        _subject.OnNext(i);
                        await CrawlAsync(i);
                    });
                }
                catch
                { }

                if (Interlocked.Decrement(ref _numberOfLinksLeft) == 0)
                    _subject.OnCompleted();
            }
        }

        private static List<Uri> Filter(IEnumerable<Uri> uris, params IUriFilter[] filters)
        {
            var filtered = uris.ToList();
            foreach (var filter in filters.ToList())
            {
                filtered = filter.Filter(filtered);
            }
            return filtered;
        }
    }

    public IObservable<Uri> Crawl(Uri uri)
    {
        return new ReceivingCrawledUri(uri, new ExcludeRootUriFilter(uri), new ExternalUriFilter(uri), new AlreadyVisitedUriFilter());
    }

    public IObservable<Uri> Crawl(Uri uri, params IUriFilter[] filters)
    {
        return new ReceivingCrawledUri(uri, filters);
    }
}

and you can use it as follows:

Crawler crawler = new Crawler();
IObservable observable = crawler.Crawl(new Uri("http://www.codinghorror.com/"));
observable.Subscribe(onNext: Console.WriteLine, 
onCompleted: () => Console.WriteLine("Crawling completed"));