-2

It's my first question here I started learning python, already watched many videos.

Appreciate if you can enlighten me on my code below,

I get below error "AttributeError: 'NoneType' object has no attribute 'find'"

I simply need to search a word (parrot in this instance) and scrape and list down the Titles in class:"snippet"

import requests
from bs4 import BeautifulSoup

page = requests.get("http://web.archive.org/web/*/parrot#")
soup = BeautifulSoup(page.content, 'html.parser')
container = soup.find("div", {"class":"search-result-container container"})
mysnippet = container.find("div", {"class":"snippet"})
print("List of Titles")
print(mysnippet)
Tomerikoo
  • 18,379
  • 16
  • 47
  • 61
Akein mandila
  • 1
  • 3
  • it means that your container variable is empty – MrHola21 May 30 '21 at 09:07
  • Its json data so you dont require `bs4` library to get output if your data is in html tags then `bs4 ` is required how to extract data from json look [here](https://www.w3schools.com/python/python_json.asp) – Bhavya Parikh May 30 '21 at 13:21
  • Please note that this is not a forum. Don't edit the question with follow-up questions. You can comment the relevant answer for that – Tomerikoo May 30 '21 at 20:56
  • 1
    Does this answer your question? [Why do I get AttributeError: 'NoneType' object has no attribute 'something'?](https://stackoverflow.com/questions/8949252/why-do-i-get-attributeerror-nonetype-object-has-no-attribute-something) – outis Sep 10 '21 at 07:42

1 Answers1

0

  1. If you print your soup you find that data is not present and even that div tag contain ::before means dynamically loaded

  2. So you can try this approach first go to chrome developer mode and then Network tab now refresh your browser

  3. In xhr you find links under Name tab in which that second link containing json data so you can grab that link by copy link address from that find title

 import requests
 res=requests.get("http://web.archive.org/__wb/search/anchor?q=parrot")
 main_data=res.json()
 for i in range(len(main_data)):
     print(main_data[i]['text'])

Output:

parrot
dead parrot society
parrot forum
parrot vids
....

Image:

enter image description here

Bhavya Parikh
  • 3,304
  • 2
  • 9
  • 19
  • it works like a charm, how ever can you clarify me why Beautifulsoup is not working on this? – Akein mandila May 30 '21 at 11:20
  • Great if you see div tag it is written `::before` most probably that contents are loaded via JS so you can use this method where data is present in API and just make request call to retrieve data if answer is usefull you can accept it or upvote it for better user engagement also refer this [post](https://stackoverflow.com/questions/8049520/web-scraping-javascript-page-with-python) – Bhavya Parikh May 30 '21 at 11:58