4

When sharing a link in major websites like Digg and Facebook; it will create thumbnails by capturing main images of the page. How they catch images from a webpage? Does it included loading the whole page (e.g. by cURL) and parsing it (e.g. with preg_match) ? To me, this method is slow and unreliable. Does they have a more practical method?

P.S. I think there should be a practical method for quick crawling the page by skipping some parts (e.g. CSS and JS) to reach src attributes. Any idea?

Googlebot
  • 15,159
  • 44
  • 133
  • 229
  • 1
    Yes, they'll load it using cURL or something similar, but they'll use an HTML parser to find images (and grab the `src` attribute), _not_ regular expressions. – Bojangles Sep 18 '11 at 14:19
  • Can you think of a "more practical" way of finding images from the page than parsing them out? – Eric Sep 18 '11 at 14:22
  • I was thinking of a more practical way to crawl the page instead f loading the whole page with something like cURL. For this purpose, they do not need to load CSS and Javascript codes. – Googlebot Sep 18 '11 at 14:34
  • CURL doesn't load CSS and javascript unless you point it to those URL's. If it's inline CSS and javascript, you'd have to read it anyway. It doesn't get interpreted by CURL. – CodeCaster Sep 18 '11 at 14:42

4 Answers4

2

They typcailly look for an image on the page, and scale it down on their servers. Reddit's scraper code shows a good deal of what they do. The Scraper class should give you some good ideas on how to tackle this.

JohnD
  • 3,884
  • 1
  • 28
  • 40
2

JohnD's answer shows that Reddit uses embed.ly as part of their Python solution. Really embed.ly does the hard part of finding the image and they're free under 10,000 requests/mo.

crizCraig
  • 8,487
  • 6
  • 54
  • 53
0

They generally use a tool like webkit2png.

ceejayoz
  • 176,543
  • 40
  • 303
  • 368
  • This tool creates an screenshot of the whole page. This is quite different from thumbnails which are created from images in the page (not the page itself). – Googlebot Sep 18 '11 at 17:07
-1

Some use 

 <link rel="image_src" href="yourimage.jpg" />

included in the head of the page. See http://www.labnol.org/internet/design/set-thumbnail-images-for-web-pages/6482/

Facebook uses

<meta property="og:image" content="thumbnail_image" />

see: http://developers.facebook.com/docs/share/#basic-tags

Gerben
  • 16,747
  • 6
  • 37
  • 56
  • These are the recommendations to developers. If you share almost any link in digg; it can catch the thumbnails (not only standard ones). – Googlebot Sep 18 '11 at 14:42