BeautifulSoup return ConnectionError

Question

I'm setting up a weather data, and I want weather(time, date, Temperature). I didn't found the solution.

from bs4 import BeautifulSoup
import requests

headers = {"User-agent": "Mozilla/5.0 (X11; Linux 
x86_64)AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 
Safari/537.36"}

url = 'https://www.accuweather.com/en/us/new-york-ny/10007/daily-weather-forecast/349727'
page = requests.get(url, headers=headers).text
soup = BeautifulSoup(page, 'html.parser')
weather = soup.find('div', 'five-day').text

time = weather.find('h3')
data = weather.find('h4')
temp = weather.find('div', 'info')
print(time)
print(data)
print(temp)

I have tried to change attribute and curly braces of weather variable name.

weather = soup.find('div', {'class': 'five-day'}).text
time = soup.select_one('div.five-day h3')

Output

During handling of the above exception, another exception occurred: Traceback (most recent call last):
File "first_scraping_solve.py", line 6, in <module> page = requests.get(url, headers=headers).text
File "C:\Users\ABC\AppData\Local\Programs\Python\Python37-32\lib\site packages\requests\api.py", line 75, in get return request('get', url, params=params, **kwargs)
File "C:\Users\ABC\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\api.py", line 60, in request return session.request(method=method, url=url, **kwargs)
File "C:\Users\ABC\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 533, in request resp = self.send(prep, **send_kwargs)
File "C:\Users\ABC\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 646, in send r = adapter.send(request, **kwargs)
File "C:\Users\ABC\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\adapters.py", line 498, in send raise ConnectionError(err, request=request) requests.exceptions.ConnectionError: ('Connection aborted.', 
TimeoutError(10060, 'A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond', None, 10060, None))

*I expect the output is Early Am July 21, 27 temperature, partly cloudy

You have a space here "daily-weather- forecast" before forecast, try to remove it and see if it works — Ron Serruya, Jul 21 '19 at 09:01
Still It's not working. I'm getting same error. It's just a mistake. I did fix it. — , Jul 21 '19 at 09:22
There is [accuweather API](https://developer.accuweather.com/) and it seems you can get 50 requests a day for free - so maybe you don't have to scrape it. — furas, Jul 21 '19 at 11:09
there is link to [getting started](https://developer.accuweather.com/getting-started) — furas, Jul 21 '19 at 11:16

Ajay · Answer 1 · 2019-07-21T11:10:46.420

0

I think the website is blocking the connection because of using same header within short span of time. You can use multiple headers or use a third party package as mentioned in this answer

To catch the exception use try and except block

 headers = {
    'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0',
    'X-Requested-With': 'XMLHttpRequest',
}


url = 'https://www.accuweather.com/en/us/new-york-ny/10007/daily-weather-forecast/349727'

try:
        response = requests.get(url, headers=headers)
except requests.exceptions.RequestException as e:

    print(e)
    exit()

soup = BeautifulSoup(response.text, "html.parser")

and slight modification in the code will get you the desired results

#print(soup)
weather = soup.find('div', 'five-day')
#print(weather)
time = weather.find('h3').text
data = weather.find('h4').text
temp = soup.find('div','temp').text
details=soup.find('div','details').text
print(time)
print(data)
print(temp)
print(details)

If you want to use weather and not soup then you need to use string slicing

temp = weather.find('div','temp').text[:4]
details=weather.find('div','details').text

edited Jul 21 '19 at 11:10

answered Jul 21 '19 at 10:05

Ajay

5,267
2
23
30

`https://stackoverflow.com/questions/17478731/whats-the-point-of-the-x-requested-with-header – Ajay Jul 21 '19 at 10:12
Can you make me understand the concept of try and except – Jul 21 '19 at 11:21
https://www.pythonforbeginners.com/error-handling/python-try-and-except – Ajay Jul 21 '19 at 15:30
ok, why we need this headers, what does it work and how to make this header (It's hard to remember this long code)? – Jul 21 '19 at 16:14
@RohitDalal Making your script look like a browser.You need not remember this,save this code somewhere and reuse when ever needed.for more on http headers `https://code.tutsplus.com/tutorials/http-headers-for-dummies--net-8039. Did my answer worked for you? – Ajay Jul 22 '19 at 07:33

BeautifulSoup return ConnectionError

1 Answers1