0

I'm extracting hundreds of URLs from tweets. Most of these URLs are shortened by external services like bit.ly and I need to expand them for comparison purposes. Right now I'm using a method from a previous post: How can I unshorten a URL using python? My code is below:

r = requests.head(url)
if r.status_code / 100 == 3:
    expanded_url = r.headers['Location']
else:
    expanded_url = r.url

Most URL expansions take less than 3 seconds, but it adds up quickly for such a large set of URLs. Is there a faster way to do this?

Community
  • 1
  • 1
J.T
  • 226
  • 2
  • 11
  • Don't you think that the delay is more likely due to the service provider rather than to your Python code? IMVHO the only viable path is multiplexing, but spelling it correctly is the limit of my knowledge about multiplexing, sorry... – gboffi Feb 12 '15 at 23:02

2 Answers2

0

I suggest taking a look at the https://docs.python.org/2/library/threading.html module. The delay is probably not your code, but the server you're connecting to. This will allow you to make your calls and not have to wait for the response before making the next.

Daniel Timberlake
  • 1,179
  • 6
  • 15
0

Check the APIs of the shortening services. Some of them offer batch services, bit.ly has an option to send up to 15 urls to expand on one call according to this page.

up to 15 URLs can be handled in one API call using the /v3/lookup, /v3/expand and /v3/clicks endpoints.

Juan E.
  • 1,808
  • 16
  • 28