0

I've been trying to scrape kith.com search results but I get skeleton sample code. Tried to use scrapy, requests-html and selenium but I haven't managed to make them work.

Right now my code is:

from requests_html import HTMLSession

session = HTMLSession()
r = session.get("https://kith.com/pages/search-results-page?q=nike&tab=products&sort_by=created")

r.html.render()
print(r)

From what I've seen, render() should get the html code as it's seen in a browser but I still get the same "raw" code.

PD: kith.com is a shopify shop

NyTrOuS
  • 63
  • 1
  • 12
  • https://stackoverflow.com/q/2148493/11301900, https://stackoverflow.com/q/17608572/11301900, https://stackoverflow.com/q/8183682/11301900, https://stackoverflow.com/q/8049520/11301900 – AMC Feb 08 '20 at 00:40
  • Does this answer your question? [Web-scraping JavaScript page with Python](https://stackoverflow.com/questions/8049520/web-scraping-javascript-page-with-python) – AMC Feb 08 '20 at 00:40

1 Answers1

2

Selenium is suitable for a job like this

from selenium import webdriver
from selenium.webdriver.firefox.options import Options

options = Options()
options.headless = True
driver = webdriver.Firefox(options=options)
driver.get('https://kith.com/pages/search-results-page?q=nike&tab=products&sort_by=created')


item_titles = driver.find_elements_by_class_name("snize-title")

print item_titles[0].text
#NIKE WMNS SHOX TL - NOVA WHITE / TEAM ORANGE / SPRUCE AURA

Edit:

If you want to capture all item info, the div elements with snize-overhidden class will be what you want to capture. Then you may iterate through them and their sub elements

Kasem Alsharaa
  • 892
  • 1
  • 6
  • 15
  • How would I do this without the computer having to open any browser? My intention is to upload it when the project is finished to AWS so it can run once every couple of hours – NyTrOuS Feb 07 '20 at 23:53
  • 1
    The browser can operate in headless mode (it runs in the background). Check the updated answer @NyTrOuS – Kasem Alsharaa Feb 07 '20 at 23:56
  • Copied your code and I'm getting an error stating that 'Options' is not defined – NyTrOuS Feb 07 '20 at 23:59