3

I was running my code at work, and it just went smoothly, but on a different network (Home WiFi), I keep receiving 403 error when accessing CoinGecko V3 API. It can be observed that accessing the API on a private browser (to test it) will prompt a security check (hCaptcha) to be able to continue.

https://api.coingecko.com/api/v3/coins/bitcoin/market_chart?vs_currency=usd&days=30

with requests.get(url) as source:
    print(source.status_code)

I also tried pycoingecko (for checking) and still returns a similar error:

>>> from pycoingecko import CoinGeckoAPI
>>> cg = CoinGeckoAPI()
>>> cg.get_coin_market_chart_by_id(id='bitcoin', vs_currency='usd', days=1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\...\Python39\lib\site-packages\pycoingecko\utils.py", line 12, in input_args
    return func(*args, **kwargs)
  File "C:\Users\...\Python39\lib\site-packages\pycoingecko\api.py", line 169, in get_coin_market_chart_by_id
    return self.__request(api_url)
  File "C:\Users\...\Python39\lib\site-packages\pycoingecko\api.py", line 29, in __request
    response.raise_for_status()
  File "C:\Users\...\Python39\lib\site-packages\requests\models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://api.coingecko.com/api/v3/coins/bitcoin/market_chart?vs_currency=usd&days=1

I guess Cloudflare is suspecting my network activity, but of course I am using Python scripts to access the API, obviously. Is there a workaround to get past the security check?

Rod Maniego
  • 139
  • 3
  • 10
  • 1
    The rate limit is 50 requests/minute. Depending on how your ISP handles things, someone else might be using up your rate limit. Their [website](https://www.coingecko.com/en/api) says "Need more API calls with flexible rate limits? Contact us at hello+api@coingecko.com" - have you tried that? – user2357112 Jul 16 '21 at 03:59
  • 1
    I have tested your code and I get a status code of 200. I can trigger a rate limit status code of 429, but so far no 403 error. What are you doing to trigger the 403 error? – Life is complex Jul 18 '21 at 02:46
  • @user2357112supportsMonica, I'm not sure on the ISP handling side, but I'm sure I'm the only one accessing the API in my area. So far, the issue was no longer reproducible for several weeks already. – Rod Maniego Jul 19 '21 at 03:25
  • @Lifeiscomplex, my hypothesis would be my previously slow internet connection, Cloudflare must be detecting unusual retries / successive access to the API as a possible bot/threat. The issue was no longer reproducible as of the moment. – Rod Maniego Jul 19 '21 at 03:28
  • @RodManiego a lot of Python Requests get flagged by Cloudflare. You should add some basic header information to your requests, such as user-agent. – Life is complex Jul 19 '21 at 03:31
  • Websites don't usually like Python requests that much. Have you tried setting the `User-Agent` header to the one of your browser? Just open a developer console in your browser and type `navigator.userAgent` to get it. – Marco Bonelli Jul 19 '21 at 10:12

3 Answers3

1

I can replicate a call to the endpoint on the command line with:

curl -vv -H "user-agent: C" https://api.coingecko.com/api/v3/coins/bitcoin/market_chart?vs_currency=USD&days=30

-vv - extra verbose logging - show detail on requests/response from the server

-H - set an HTTP header

With the user-agent: header set to blank I receive no response. If I set the value to any non-null value like C, Chrome, Chrome/91.0.0 it works.

If you can replicate the same in python you should get the same results... HTTP is a stateless protocol. If it doesn't work, something else is at play.

Rob Evans
  • 2,822
  • 1
  • 9
  • 15
  • I'd also add it seems like if you just want the price of bitcoin at the current time there are other api's/tools available that would work for exactly this purpose. – Rob Evans Jul 22 '21 at 22:12
0

Just tried to do more than 50 requests per minute as said in CoinGecko docs and after approximately 200, I started to get error

urllib.error.HTTPError: HTTP Error 429: Too Many Requests

So the error you see is a result of particular activity.

In your case I guess the only way to through is to use a network proxy servers or to run parsing from some remote server like AWS EC2 instance, which is cheap and can work for parsing.

Peter
  • 11
  • 2
0

I think using scraperapi.com would be a good workaround for this. You get a few thousand requests to see if it works before having to pay and it's extremely easy to set up + designed to handle stuff like this where you might get a captcha/blocked etc.

Just tested it and works OK for me:

import requests
payload = {'api_key': 'YOUR API KEY',
           'url': 'https://api.coingecko.com/api/v3/coins/bitcoin/market_chart?vs_currency=usd&days=30'}
with requests.get('http://api.scraperapi.com', params=payload) as source:
    print(source.status_code)
osint_alex
  • 952
  • 3
  • 16