0

I have an app that encourages people to refer their friends by sharing the page on Facebook. When they click our share link, they are actually sharing example.com/?ref=REFCODE, where REFCODE is a unique string identifying them. Their total referrals are tracked via how many people sign up after visiting example.com/?ref=REFCODE.

The og data is set universally and I would like to update it for every REFCODE. However, Facebook seems to store the scraped data separately for every refcode.

Facebook's debugger only allows me to manually rescrape 20 URLs per hour.

I've tried passing a random additional variable at the end of the query string to make Facebook think it's a new URL, but it doesn't seem to have any effect. (Example: passing example.com/?ref=REFCODE&f=g rather than the original URL.)

I'm aware of the method found here, but I'm not sure how I would apply it to a large set of URLs at once: Is there an API to force Facebook to scrape a page again?

Is there any good way to do this?

Update: Thanks to Igy for leading me to one effective solution, a simple shell script loop. I also should have mentioned that the way I was passing the URL to Facebook was via sharer.php. It turns out that the random additional parameter trick wasn't working because I wasn't encoding my own URL, see the answer I just added for details on both methods.

Community
  • 1
  • 1
max norton
  • 168
  • 9
  • Is there a reason you can't just call the scraper via the API multiple times in a loop with a few seconds between calls? – Igy Sep 10 '14 at 23:54
  • No, in fact, that's exactly what I needed to do, I just didn't know enough about bash scripting to do it but now I do. Thanks. I'll add it as an answer. – max norton Sep 11 '14 at 01:24

1 Answers1

0

It turns out there are at least two ways two do this.

Method one: loop through them all in a bash script
Igy suggested this in the comments, it worked for me and it is the most universal approach. I put all the referral codes in a file called refs.txt, and then looped through the API endpoint discussed here. Here's what my script looked like:

#!/bin/bash
for refcode in $(cat refs.txt); do
    curl -X POST \
     -F "id=http://example.com/?ref="$refcode \
     -F "scrape=true" \
     "https://graph.facebook.com"
    sleep 5
done

Method two: append a random additional encoded parameter to the query string
As described above, I was passing a URL to Facebook that needed one query string parameter and I tried appending a second, meaningless parameter so that Facebook would interpret it as a new URL. This does work. My mistake was that I was passing my URL as a parameter of Facebook's sharer.php, but I had not encoded my own URL. The URL I called looked like this: http://www.facebook.com/sharer.php?u=http://example.com/?ref=REFCODEf&f=g

so predictably &f=g was interpreted as a parameter of sharer.php, not of example.com. The solution is to encode the example.com URL: http://www.facebook.com/sharer.php?u=http%3A%2F%2Fexample.com%2F%3Fref%3DREFCODEf%26f%3Dg

That way &f=g is read as a parameter of example.com and Facebook interprets it as an entirely new URL.

Community
  • 1
  • 1
max norton
  • 168
  • 9