5

I am trying to parse/scrape https://etherscan.io/tokens website using requests in Python but I get the following error:

etherscan.io

Checking if the site connection is secure

etherscan.io needs to review the security of your connection before proceeding. Ray ID: 73b56fc71bc276ed Performance & security by Cloudflare

Now, I found a solution here: https://stackoverflow.com/a/62687390/4190159 but when I try to use this solution, I am still not being able to read the actual content of the website and getting a different error stated below.

My code as follows:

import requests
from collections import OrderedDict
from requests import Session
import socket

answers = socket.getaddrinfo('etherscan.io', 443)
(family, type, proto, canonname, (address, port)) = answers[0]
s = Session()
headers = OrderedDict({
    'Accept-Encoding': 'gzip, deflate, br',
    'Host': "grimaldis.myguestaccount.com",
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0'
})
s.headers = headers
response = s.get(f"https://{address}/tokens", headers=headers, verify=False).text
print(response)

Error for the above code as follows:

Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen chunked=chunked, File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 381, in _make_request self._validate_conn(conn) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 976, in validate_conn conn.connect() File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connection.py", line 370, in connect ssl_context=context, File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/util/ssl.py", line 390, in ssl_wrap_socket return context.wrap_socket(sock) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 407, in wrap_socket _context=self, _session=session) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 814, in init self.do_handshake() File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 1068, in do_handshake self._sslobj.do_handshake() File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 689, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:833)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/adapters.py", line 449, in send timeout=timeout File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 725, in urlopen method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/util/retry.py", line 439, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='172.67.8.107', port=443): Max retries exceeded with url: /tokens (Caused by SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:833)'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "label_scrapper.py", line 16, in response = s.get(f"https://{address}/tokens", headers=headers, verify=False).text File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/sessions.py", line 543, in get return self.request('GET', url, **kwargs) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/sessions.py", line 530, in request resp = self.send(prep, **send_kwargs) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/sessions.py", line 643, in send r = adapter.send(request, **kwargs) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/adapters.py", line 514, in send raise SSLError(e, request=request) requests.exceptions.SSLError: HTTPSConnectionPool(host='172.67.8.107', port=443): Max retries exceeded with url: /tokens (Caused by SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:833)'),)) Somdips-MacBook-Pro:Downloads somdipdey$ python3 label_scrapper.py Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen chunked=chunked, File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 381, in _make_request self._validate_conn(conn) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 976, in validate_conn conn.connect() File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connection.py", line 370, in connect ssl_context=context, File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/util/ssl.py", line 390, in ssl_wrap_socket return context.wrap_socket(sock) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 407, in wrap_socket _context=self, _session=session) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 814, in init self.do_handshake() File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 1068, in do_handshake self._sslobj.do_handshake() File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 689, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:833)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/adapters.py", line 449, in send timeout=timeout File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 725, in urlopen method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/util/retry.py", line 439, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='172.67.8.107', port=443): Max retries exceeded with url: /tokens (Caused by SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:833)'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "label_scrapper.py", line 16, in response = s.get(f"https://{address}/tokens", headers=headers, verify=False).text File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/sessions.py", line 543, in get return self.request('GET', url, **kwargs) Somdips-MacBook-Pro:Downloads somdipdey$ python3 label_scrapper.py Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen chunked=chunked, File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 381, in _make_request self._validate_conn(conn) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 976, in validate_conn conn.connect() File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connection.py", line 370, in connect ssl_context=context, File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/util/ssl.py", line 390, in ssl_wrap_socket return context.wrap_socket(sock) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 407, in wrap_socket _context=self, _session=session) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 814, in init self.do_handshake() File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 1068, in do_handshake self._sslobj.do_handshake() File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 689, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:833)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/adapters.py", line 449, in send timeout=timeout File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 725, in urlopen method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/util/retry.py", line 439, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='172.67.8.107', port=443): Max retries exceeded with url: /tokens (Caused by SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:833)'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "label_scrapper.py", line 16, in response = s.get(f"https://{address}/tokens", headers=headers, verify=False).text File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/sessions.py", line 543, in get return self.request('GET', url, **kwargs) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/sessions.py", line 530, in request resp = self.send(prep, **send_kwargs) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/sessions.py", line 643, in send r = adapter.send(request, **kwargs) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/adapters.py", line 514, in send raise SSLError(e, request=request) requests.exceptions.SSLError: HTTPSConnectionPool(host='172.67.8.107', port=443): Max retries exceeded with url: /tokens (Caused by SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:833)'),))

How to resolve this?

Somdip Dey
  • 3,346
  • 6
  • 28
  • 60
  • One of Stack Overflow's goals is to build a long-lived knowledgebase. Insofar as attempts to block scrapers are an ongoing cat-and-mouse game, moving the state of the art forward on one side or the other is likely to be countered by the opposing group in short order, _especially_ when it's done in a well-known public forum that Cloudflare's developers are prone to reading. As such, answers to such questions are unlikely to remain valid for long -- indeed, they're likely to be invalidated sooner when given here rather than self-discovered and kept private. – Charles Duffy Aug 15 '22 at 23:08

1 Answers1

2

The website is under cloudflare protection. So you can use cloudscraper instead of requests to get rid of the protection. Now it's working fine.

Example:

from bs4 import BeautifulSoup
import cloudscraper
scraper = cloudscraper.create_scraper(delay=10,   browser={'custom': 'ScraperBot/1.0',})
url = 'https://etherscan.io/tokens'
req = scraper.get(url)

soup = BeautifulSoup(req.content,'lxml')
for tr in soup.select('table#tblResult tbody tr'):
    d= list(tr.stripped_strings)
    print(d)

Output:

['1', 'Tether USD (USDT)', 'Tether gives you the joint benefits of open blockchain technology and traditional currency by converting your cash into a stable digital currency equivalent.', '$1.005', '0.000042\xa0Btc', '0.000534\xa0Eth', '0.40%', '$56,223,000,912.00', '$67,594,315,221.00', '$40,022,235,444.23', '4,384,118', '0.003%']
['2', 'USD Coin (USDC)', 'USDC is a fully collateralized US Dollar stablecoin developed by CENTRE, the open source project with Circle being the first of several forthcoming issuers.', '$1.006', '0.000042\xa0Btc', '0.000534\xa0Eth', '0.44%', '$7,028,538,289.00', '$53,707,876,130.00', '$46,882,045,425.04', '1,470,173', '0.132%']
['3', 'BNB (BNB)', 'Binance aims to build a world-class crypto exchange, powering the future\nof crypto finance.', '$316.6558', '0.013290\xa0Btc', '0.168135\xa0Eth', '-0.87%', '$1,131,380,099.00', '$51,088,380,246.00', '$5,250,000,297.97', '322,395', '0.001%']
['4', 'Binance USD (BUSD)', 'Binance USD (BUSD) is a dollar-backed stablecoin issued and custodied by Paxos Trust Company, and regulated by the New York State Department of Financial Services. BUSD is available directly for sale 1:1 with USD on Paxos.com and will be listed for trading on Binance.', '$0.9976', '0.000042\xa0Btc', '0.000530\xa0Eth', '-0.30%', '$6,219,951,485.00', '$17,920,238,921.00', '$17,532,299,450.15', '125,632', '0.197%']
['5', 'HEX (HEX)', "HEX.com averages 25% APY interest recently. HEX virtually lends value from stakers to non-stakers as staking reduces supply. The launch ends Nov. 19th, 2020 when HEX 
stakers get credited ~200B HEX. HEX's total supply is now ~350B. Audited 3 times, 2 security, and 1 economics.", '$0.0626', '0.000003\xa0Btc', '0.000033\xa0Eth', '-5.10%', '$22,598,794.00', '$10,856,229,132.00', '$36,158,058,204.94', '308,035', '-0.040%']
['6', 'SHIBA INU (SHIB)', 'SHIBA INU is a 100% decentralized community experiment with it claims that 1/2 the tokens have been sent to Vitalik and the other half were locked to a Uniswap pool and the keys burned.', '$0.00', '0.000000\xa0Btc', '0.000000\xa0Eth', '-9.20%', '$2,070,477,368.00', '$9,155,756,506.00', '$15,489,873,503.95', '1,206,115', '0.088%']
['7', 'stETH (stETH)', 'stETH is a token that represents staked ether in Lido, combining the 
value of initial deposit + staking rewards. stETH tokens are pegged 1:1 to the ETH staked with Lido and can be used as one would use ether, allowing users to earn Eth2 staking rewards whilst benefiting from Defi yields.', '$1,844.28', '0.077404\xa0Btc', '0.979260\xa0Eth', '-2.26%', '$3,408,944.00', '$7,909,446,933.00', '$3,418,574,006.52', '94,316', '0.215%']
['8', 'Matic Token (MATIC)', 'Matic Network brings massive scale to Ethereum using an adapted version of Plasma with PoS based side chains. Polygon is a well-structured, easy-to-use platform for Ethereum scaling and infrastructure development.', '$0.9397', '0.000039\xa0Btc', '0.000499\xa0Eth', '-6.39%', '$529,032,596.00', '$7,551,024,649.00', '$9,397,310,544.00', '466,641', '0.083%']
['9', 'Dai Stablecoin (DAI)', 'Multi-Collateral Dai, brings a lot of new and exciting features, such as support for new CDP collateral types and Dai Savings Rate.', '$1.005', '0.000042\xa0Btc', '0.000534\xa0Eth', '0.36%', '$635,956,564.00', '$6,800,555,162.00', '$9,848,650,590.65', '479,078', '-0.008%']
['10', 'Wrapped BTC (WBTC)', 'Wrapped Bitcoin (WBTC) is an ERC20 token backed 1:1 with Bitcoin.\nCompletely transparent. 100% verifiable. Community led.', '$23,983.00', '1.006567\xa0Btc', '12.734291\xa0Eth', '-1.26%', '$263,338,154.00', '$5,928,934,190.00', '$6,279,085,162.00', 
'51,459', '0.058%']

... so on

cloudscraper

Md. Fazlul Hoque
  • 15,806
  • 5
  • 12
  • 32
  • 1
    ... if you are ok to run a full fledge browser executing remote code, as: "cloudscraper requires a JavaScript Engine/interpreter to solve Javascript challenges. This allows the script to easily impersonate a regular web browser without explicitly deobfuscating and parsing Cloudflare's Javascript." – Patrick Mevzek Aug 16 '22 at 03:41
  • Thanks, I tried cloudflare before for my problem but it wasn't working, implementing scraperbot does however :) – Rivered Feb 20 '23 at 23:07