Ask facebook to re-scrape URL

Question

The og:image of the website has been changed recently. The site contains more than 100 pages, each containing their individual og:image. How can i ask or force facebook to re-scrape all the pages, so that the image gets updated? Using the facebook debugger tool will be too tedious task. Until facebook re-scrape's the site, i won't be able to submit collection for the app.

score 34 · Answer 1 · answered May 03 '13 at 06:35

34

You can force a re-scrape via API, as described here: https://developers.facebook.com/docs/opengraph/using-objects/#update:

POST /?id={object-instance-id or object-url}&scrape=true

(But if you don’t have a real list of the affected URLs, this is kinda moot. Then you only can wait until it happens automatically, I guess.)

answered May 03 '13 at 06:35

CBroe

91,630
14
92
150

This is great. The response includes the refreshed OG data as well, so you can check it yourself while scraping. – bloudermilk Apr 18 '14 at 21:04
This is the true answer and should be the chosen one. shame the other one is, since it's not even an answer at all. – vsync Jul 02 '14 at 10:01
2

This is definitely the correct answer. POSTing /?id=&scrape=true to https://graph.facebook.com/ will force a re-scrape of the og tags for hte url. One must remember that this must be a POST request, not a GET request, otherwise you just get the number of shares. And if you add another parameter called callback=, you can do this via an AJAX JSONP request within the browser, and no need to tie up your server. – Troy Morehouse Aug 05 '14 at 21:29
Hi , can this method be used to update dynamically created meta tags using javascript? ref : https://stackoverflow.com/questions/40868686/react-meta-tags-not-working-for-facebook – Haswin Aug 03 '17 at 09:21
@HaswinVidanage the answer to that question over there ... well, already answers that. – CBroe Aug 03 '17 at 09:23

score 14 · Accepted Answer · edited Jun 20 '20 at 09:12

14

You have two alternatives here in your situation

As per ysrb's answer loop through list of your URLs with the Open Graph Debugger tool
Or wait patiently for 30 days until Facebook re-scrapes your Pages as it says in documentation here

Why and when does Facebook scrape my website?

Facebook needs to scrape links shared to Facebook to know which to link preview information to show on Facebook.com or on Facebook for iOS and Android. This happens every 30 days to ensure the properties are up to date. The linked page is also scraped when the URL is entered into the Debugger Tool.

Facebook observes cache headers on your URLs - it will look at "Expires" and "Cache-Control" in order of preference. However, even if you specify a longer time, Facebook will scrape your page every 30 days.

edited Jun 20 '20 at 09:12

Community

1
1

answered May 03 '13 at 05:52

Anvesh Saxena

4,458
2
23
30

3

@ChrisHoughton - https://developers.facebook.com/docs/plugins/faqs/#scraperinfo (as of 2015.04.08) – changokun Apr 08 '15 at 19:15
This is no longer correct. There is an API endpoing that Doug and others explain in another answer. – Michael Khalili Feb 25 '16 at 20:15
my pages were scraped last time in 2012, and now im doing all of them manually :( – Aziz Jan 31 '17 at 18:14

score 5 · Answer 3 · answered Jan 23 '14 at 07:09

5

       $config = array(
            "appId" => 'APP_ID',
            "secret" => 'APP_SECRET');

        $fb = new Facebook($config);
        $r=$fb->api('/','POST',array(
            'id'=>PAGE_URL,
            'scrape'=>'true'
        ));

answered Jan 23 '14 at 07:09

dev.meghraj

8,542
5
38
76

score 3 · Answer 4 · answered Aug 08 '15 at 03:33

Be aware that Facebook requires the parameters be passed via POST. Facebook just ignores GET requests.

Here is the code in C#.

HttpWebRequest httpRequest = (HttpWebRequest)WebRequest.Create(
    "https://graph.facebook.com/?id="
    + HttpUtility.UrlEncode("http://www.example.com/index.html")
    + "&scrape=true");
httpRequest.Method = "POST";
using (HttpWebResponse httpResponse = (HttpWebResponse)httpRequest.GetResponse())
{
    using (Stream responsestream = httpResponse.GetResponseStream())
    {
        if (responsestream != null)
        {
            using (StreamReader bodyreader = new StreamReader(responsestream))
            {
                string fbResp = bodyreader.ReadToEnd();
            }
        }
    }
}

score 2 · Answer 5 · answered Apr 14 '15 at 19:30

2

curl --insecure "https://graph.facebook.com/?id=[YOUR-URL-TO-SCRAPE]&scrape=true"

curl should be ssl capable (as it is https://graph...)

answered Apr 14 '15 at 19:30

Dusan

49
2

I got this to work, thanks a lot. - You need to be sure to set the user agent, or else FB will decide not to honor the scrape request, however. – Trass Vasston Mar 26 '16 at 06:31

score 1 · Answer 6 · answered May 03 '13 at 05:23

1

You can try loop from the list of url and do

curl "http://developers.facebook.com/tools/debug/og/object?q=$url"

answered May 03 '13 at 05:23

ysrb

6,693
2
29
30

Ok. The exact list of url is unknown. Approx there are 310 urls as of now. Manually specifying each of them for the loop will be time consuming and we may miss some url due to humman error – curious_coder May 03 '13 at 05:44
You must have some kind of sitemap.xml or something. Even if it's static files, you can list it and create the url – ysrb May 03 '13 at 07:02
3

Tested - This method does not work. – GTCrais Sep 12 '14 at 13:45
@GTCrais what do you mean doesn't work? What's the test case? – ysrb Sep 12 '14 at 15:19
3

@ysrb I mean I tried it, and it doesn't work, probably because you can't access that URL without being logged into facebook. The data doesn't get scrapped by facebook with this method. – GTCrais Sep 15 '14 at 10:11

Ask facebook to re-scrape URL

6 Answers6

Linked