0

I'm trying to do some scraping on a website that contains information that I can save on it. After basic scrapping with bs4 on python I saw that the page was empty.

def getPage(url):
  source = requests.get(url).text
  soup = BeautifulSoup(source, 'html')
  article = soup.find('div', class_="row q-pa-sm")
  items = article.find_all("data-v-6838bbc6")
  return items

After some research I saw that information is saved in "Local storage" on my web browser and I don't know how to access to it with my script. My research only leads me to javascript and I don't know javascript at all, I can learn if it is the only solution. How could I do? Is there a folder on my computer with that "local storage" ? Or should I inject that information when I do my scraping ?

Edit : I tried the same with selenium

def getPage():
  url = "https://dofusdb.fr/fr/tools/craft-manager"
  options = Options()
  options.headless = True
  driver = webdriver.Chrome(options=options)
  driver.get(url)
  driver.get(url)
  page = driver.page_source
  driver.quit()
  return page

What I found after some research is that the driver open a blank instance of the browser despite the localStorage. I tried with Chrome and Firefox and i obtained the same result. (The best would be on opera but I haven't found webdriver for opera on selenium for now)

Noah Fraiture
  • 33
  • 3
  • [This answer](https://stackoverflow.com/questions/48603339/how-to-call-javascript-function-using-beautifulsoup-and-python) implies that you willl not be able to achieve the desired functionality with beautifulsoup. You might want to take a look at Selenium and the `.execute_script` method that is provided by the driver. – Palladium02 Nov 12 '22 at 12:52
  • Thank you but I tried with selenium chrome to do it and I have the same result. Execute script can be usefull but and I don't need it for now. I need to get the local storage on the page. – Noah Fraiture Nov 12 '22 at 13:51
  • If it is really the localstorage you are looking for `driver.execute_script("localStorage.get('<>')")` should do just fine. Some [further reading](https://stackoverflow.com/questions/46361494/how-to-get-the-localstorage-with-python-and-selenium-webdriver) from another user having the same/similar question. – Palladium02 Nov 12 '22 at 19:01

0 Answers0