0

First time posting here. Just finished my first Python course, so I've been trying to do some web scraping on my own but have failed to do so.

When I run my code I get an empty dictionary, and I don't really understand what should I do. I've been doing some research and it appears that the page I'm trying to scrape uses an AJAX application that maybe requires some other type of technique.

I've noticed that the piece of information that I'm trying to gather (a list of product prices) is under this tag: https://b2b.devir.cl/addmultipleproducts/cart/ajaxaddmultiple

This is my code and the end result is:

200 (which signals the page was accessed correctly as far as I know)

[]

from bs4 import BeautifulSoup as bs
import requests

URL = 'https://b2b.devir.cl/'
LOGIN_ROUTE= 'customer/account/login/'

HEADERS={'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36', 'origin':URL, 'referer':URL+LOGIN_ROUTE}

s=requests.session()

front_end=s.get(URL).cookies['frontend']

login_payload = {
        'login[username]': 'xxxx',
        'login[password]': 'yyyy',
        'form_key': front_end
        }

login_req = s.post(URL+LOGIN_ROUTE, headers=HEADERS, data=login_payload)

print(login_req.status_code)

cookies=login_req.cookies

r=requests.get(URL+'juegos-de-mesa.html?po_stock=1')
soup=bs(r.content,'lxml')

productlist=[]
productlist=soup.find_all('div', {'class': 'category-products'})
print(productlist)

I would very much appreciate to hear some piece of advice on this, and I'm sorry in advance for my ignorance on the topic.

Philippe Cazabon Kauak
  • 1
  • Welcome to SO! This login is blocked so it's basically impossible to help without a [mcve]. Very likely, the products are injected dynamically with JS, so if they're not part of the initial static HTML payload, BS can't do much for you. See [Web-scraping JavaScript page with Python](https://stackoverflow.com/questions/8049520/web-scraping-javascript-page-with-python#) – ggorlen Apr 27 '21 at 00:00
  • Does this answer your question? [Web-scraping JavaScript page with Python](https://stackoverflow.com/questions/8049520/web-scraping-javascript-page-with-python) – ggorlen Apr 27 '21 at 00:00
  • Hi! Thank you very much for your response. Would it help if I sent you the login details? I'm really struggling with this and I don't really understand much of the coding instructions you sent me. I'm pretty much limited to Youtube tutorials :( – Philippe Cazabon Kauak Apr 27 '21 at 02:57

0 Answers0