4

So I am trying to use an API (http://twittercounter.com/pages/api) to get some data from the net. While using my API key direcly via browser, I am getting the required results. But on using requests.get() function in python, I am getting an error, the traceback is given here.

code:

>>> import requests
>>> r = requests.get('https://api.twittercounter.com/?apikey=XXXX&twitter_id=57947109')

traceback:

 File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 382, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 485, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 372, in send
    raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='api.twittercounter.com', port=443): Max retries exceeded with url: /?apikey=XXXX&twitter_id=57947109 (Caused by <class 'socket.error'>: [Errno 111] Connection refused)

I made about 10 connections with this key, and the rate limit is 100, so I am sure I am not exceeding the limit. Can anyone please help. I am pretty much a noob with requests and http.

EDIT: Tried to set browser agent in the request headers

I tried this to change the browser agent, and it still does not work

>>> headers = {
...     'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0'}
>>> url = 'https://api.twittercounter.com/?apikey=09792d72d848c55a5b6b9a1bf3bb225a&twitter_id=57947109'
>>> response = requests.get(url, headers=headers)

Am getting the same traceback as last time.

Apoorv Ashutosh
  • 3,834
  • 7
  • 23
  • 24
  • Do you get this exception permanently or from time to time? I've faced with this problem recently, but I got this exception **from time to time**. So I just caught an exception and then retried connection. It was worked for me. – Dmitry Vakhrushev Jan 22 '14 at 05:59
  • 1
    @DmitryVakhrushev Am getting this permanently. I tried to catch an exception and retry, but its giving me the same error again :( – Apoorv Ashutosh Jan 22 '14 at 06:02
  • 1
    @DmitryVakhrushev, Could you paste your code, or some fragment here, maybe that might help. – Apoorv Ashutosh Jan 22 '14 at 06:45
  • You say you made _about_ 10 connections. Would you go so far as to say you made _exactly_ 10 connections? – Lukasa Jan 24 '14 at 13:16
  • @Lukasa Not exactly, I wasn't counting. But it was definitely in the 10-15 range. – Apoorv Ashutosh Jan 25 '14 at 05:48
  • The reason I ask is that Requests uses connection pooling, and the default size of the connection pool is 10. I wonder if we're encountering an odd interaction with it. Can you boil this down to a minimal test case that can reproduce it? – Lukasa Jan 25 '14 at 13:38
  • @Lukasa: Can only do so tomorrow..will post here after trying – Apoorv Ashutosh Jan 25 '14 at 15:44

2 Answers2

1

This method worked perfectly for me:

>>> import requests
>>> url = 'http://api.twittercounter.com/'
>>> payload = {'apikey': 'XXXXXX', 'twitter_id':53687449}
>>> requests.get(url, params=payload)
<Response [200]>
Sayan Chowdhury
  • 419
  • 4
  • 13
  • 1
    I don't know. But this seems to be a recurring issue in requests, and many people have told me about this error, mostly appearing temporarily, but at times like my situation, not being resolved at all. I had to use the httplib module for this. – Apoorv Ashutosh Jan 22 '14 at 10:47
  • What version of requests are you using? – Sayan Chowdhury Jan 22 '14 at 10:50
0

While I couldn't resolve this problem myself, I found a workaround to this, wherein this works, instead of using requests, I had to use httplib and then the ast module. The code I used is:

>>> import httplib
>>> conn = httplib.HTTPConnection("api.twittercounter.com")
>>> conn.request("GET", "?apikey=XXXX&twitter_id=15160529")
>>> r1 = conn.getresponse()
>>> a = r1.read()
>>> import ast
>>> b = ast.literal_eval(a)

Now, b is a dictionary that has whatever data I am looking for. However, if someone can tell me the proper solution to this error, it would be very useful.

Apoorv Ashutosh
  • 3,834
  • 7
  • 23
  • 24