11

I'm interested in public facing sites (nothing behind a login / authentication) that have things like:

  • High use of internal 301 and 302 redirects
  • Anti-scraping measures (but not banning crawlers via robots.txt)
  • Non-semantic, or invalid mark-up
  • Content loaded via AJAX in the form of onclicks or infinite scrolling
  • Lots of parameters used in urls
  • Canonical problems
  • Convoluted internal link structure
  • and anything else that generally makes crawling a website a headache!

I have built a crawler / spider that performs a range of analysis on a website, and I'm on the lookout for sites that will make it struggle.

halfer
  • 19,824
  • 17
  • 99
  • 186
David Pratt
  • 703
  • 7
  • 16
  • Are you after list of challenging sites, or techniques to scrape challenging sites? I'm not sure what the query is here. – halfer Sep 20 '13 at 19:14
  • I am after a list of challenging sites. To give you some background, I have built a crawler / spider that performs a range of analysis on a website, and I'm on the lookout for sites that will make it struggle. – David Pratt Sep 21 '13 at 21:43
  • Ah righto. I'd say this is probably off-topic, as it's a request for external resources; however as it is an interesting question I'll not vote to close. I should be interested to see what comes up, though, as I'm working on a crawler project myself. – halfer Sep 21 '13 at 22:15
  • 2
    Found [this site](http://mosaic.taleo.net/careersection/10203/jobsearch.ftl) in [this question](http://stackoverflow.com/q/19068521/472495). That should be an interesting one to analyse: JavaScript links and pages rendered in POST operations. – halfer Sep 29 '13 at 14:55
  • Try to scrape some current prices from airline websites? Some that come to mind from my part of the world would be AirAsia and JetStar. They don't have any required authentication/login, but do use AJAX. In the past I've used Selenium WebDriver, but it was still challenging. – dodgy_coder Oct 14 '13 at 01:58
  • I am scraping this site www.interjet.com.mx, they has good firewall I just request 10 times and they block so challenging. If you find the way to do that hope you share for everyone. Thanks – giaosudau Oct 28 '13 at 08:19

1 Answers1

3

Here are some:

  • Content loaded via AJAX in the form of onclicks or infinite scrolling
    • pinterest
    • comments in such a page
      This is a Chinese commodity page and its comments is loaded by AJAX which is triggered by scrolling down the scrollbar in a browser or according to your browser's height. I must use PhantomJS and xvfb to trigger such actions.
  • Anti-scraping measures (but not banning crawlers via robots.txt)
    • amazon next page
      I have crawled amazon site in China and when I want to crawl the next page in such pages, it may modify the requests resulting in that you couldn't get the real next page
    • stackoverflow
      It has a limit of visit frequency. A few days ago, I wanted to get all of the tags in stackoverflow and set the spider's visit frequency to 10, but I was warned by stackoverflow...... Here's the screen shot. After that I have to use proxies to crawl stackoverflow.
  • and anything else that generally makes crawling a website a headache
    • yihaodian
      This is a Chinese e-commerce site and when you visit it in a browser, it will show your location and will offer some commodities according to your location.
    • etc.
      There're many sites like the above that will offer different contents according to your location. When you crawl such sites, what you get is not the same as what you see in a browser. It often needs setting cookie when emitting a request through a spider.

Last year I encountered a site which required http request headers and some cookies when emitting requests, but I don't remember that site....

Community
  • 1
  • 1
flyer
  • 9,280
  • 11
  • 46
  • 62
  • The Stack Overflow issue _may_ have been solved by putting a delay between each request. If you make a large number of requests per second to any site, you can expect to be blocked `:)`. – halfer Dec 15 '13 at 12:31