Web scraping issue

Question

I am trying to web scrape a page but I keep getting an error message. "raise HTTPError(req.full_url, code, msg, hdrs, fp) HTTPError: Not Found".

Can anyone see why I am getting this issue and how I can fix it?

Here is my code:

import bs4
from urllib.request import urlopen as uReq
from bs4 import BeautifulSoup as soup
U="https://llis.nasa.gov/search?organization=arc&page=1"


uClient=uReq(U)
page_html=uClient.read()
uClient.close()
page_soup=soup(page_html,"html.parser")
page_soup.h2

Replacing `urllib.request` with `urllib` in the imports works for me. — Daniel Flannery, Nov 15 '17 at 11:06
Could it be linked to this https://stackoverflow.com/questions/16627227/http-error-403-in-python-3-web-scraping — user115014, Nov 15 '17 at 11:35

score 1 · Answer 1 · answered Nov 15 '17 at 11:06

1

Use requests module as the more powerful and flexible for dealing with HTTP requests:

import bs4, requests

url = "https://llis.nasa.gov/search?organization=arc&page=1"
data = requests.get(url)
soup = bs4.BeautifulSoup(data.content, 'html.parser')

print(soup.body)

The output:

<body>
<script src="assets/vendor.js"></script>
<script src="assets/js/bootstrap.min.js"></script>
<script src="assets/llis.js"></script>
</body>

answered Nov 15 '17 at 11:06

RomanPerekhrest

88,541
4
65
105

Is there a way of passing bf over the webpage and scrapping off all the NASA lessons? For example: https://llis.nasa.gov/lesson/1347 – J. Smith Nov 15 '17 at 13:51
@J.Smith, what does mean your *passing bf over the webpage*? – RomanPerekhrest Nov 15 '17 at 14:53
Let me rephrase it. I want all the information from the following webpage: https://www.transit.dot.gov/regulations-and-guidance/list-project-management-oversight-pmo-lessons-learned – J. Smith Nov 17 '17 at 10:00
Is it possible for Beautiful Soup to go and and collect all the information from different links on a single page? Or can you only scrape words from that one page? – J. Smith Nov 17 '17 at 10:01
@J.Smith, what does mean your *all the information from different links* - should it parse only `` tag text values OR follow all the links and extract the whole web page presented by external link? – RomanPerekhrest Nov 17 '17 at 10:16

Web scraping issue

1 Answers1