0

I want to scrape web results from google.com. I followed the first answer from this questions, Google Search Web Scraping with Python. Unfortunately I am getting connecting error. I happened to check with other websites too, its not connecting . Is it because of the corporate proxy settings?

Please note that i am using virtual env "Webscraping".

from urllib.parse import urlencode, urlparse, parse_qs

from lxml.html import fromstring
from requests import get

raw = get("https://www.google.com/search?q=StackOverflow").text
page = fromstring(raw)

for result in page.cssselect(".r a"):
    url = result.get("href")
    if url.startswith("/url?"):
        url = parse_qs(urlparse(url).query)['q']
    print(url[0])

raw = get("https://www.google.com/search?q=StackOverflow").text Traceback (most recent call last):

File "", line 1, in raw = get("https://www.google.com/search?q=StackOverflow").text

File "c:\users\appdata\local\programs\python\python37\webscraping\lib\site-packages\requests\api.py", line 75, in get return request('get', url, params=params, **kwargs)

File "c:\users\appdata\local\programs\python\python37\webscraping\lib\site-packages\requests\api.py", line 60, in request return session.request(method=method, url=url, **kwargs)

File "c:\users\appdata\local\programs\python\python37\webscraping\lib\site-packages\requests\sessions.py", line 524, in request resp = self.send(prep, **send_kwargs)

File "c:\users\appdata\local\programs\python\python37\webscraping\lib\site-packages\requests\sessions.py", line 637, in send r = adapter.send(request, **kwargs)

File "c:\users\appdata\local\programs\python\python37\webscraping\lib\site-packages\requests\adapters.py", line 516, in send raise ConnectionError(e, request=request)

ConnectionError: HTTPSConnectionPool(host='www.google.com', port=443): Max retries exceeded with url: /search?q=StackOverflow (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x0000021B79768748>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond'))

Please advise. Thanks

EDIT: I tried pining google.com, it fails.

import os
hostname = "https://www.google.com" #example
response = os.system("ping -c 1 " + hostname)

#and then check the response...
if response == 0:
  print(hostname, 'is up!')
else:
  print(hostname, 'is down!')

https://www.google.com is down!

DisappointedByUnaccountableMod
  • 6,656
  • 4
  • 18
  • 22
dhinar1991
  • 831
  • 5
  • 21
  • 40

1 Answers1

2

I think you are getting this error because of your proxy setting. Try run one of the following commands in command prompt 

set http_proxy=http://proxy_address:port
set http_proxy=http://user:password@proxy_address:port
set https_proxy=https://proxy_address:port
set https_proxy=https://user:password@proxy_address:port
Souren
  • 95
  • 2
  • 10