How do I extract data from dynamic updating webpages

Question

I want to scrape the review from Sephora website. The review is dynamically updated.

After inspection I found the review is here in the HTML code.

<div class="css-eq4i08 " data-comp="Ellipsis Box">Honestly I never write 
reviews but this is a must if you have frizzy after even after straightening 
it! It smells fantastic and it works wonders definitely will be restocking once 
I’m done this one !!</div>

I want to write a python selenium code to read the review.

The code I wrote is here...

from selenium import webdriver
chrome_path = (r"C:/Users/Connectm/Downloads/chromedriver.exe")

driver = webdriver.Chrome(chrome_path)
driver.implicitly_wait(20) 
driver.get("https://www.sephora.com/product/crybaby-coconut-oil-shine-serum-P439093?skuId=2122083&icid2=just%20arrived:p439093")
reviews = driver.find_element_by_xpath('//*[@id="ratings-reviews"]/div[4]/div[2]/div[2]/div[1]/div[3][@data-comp()='Elipsis Box'])
print(reviews.text)

If I write find_element_by_class it gives me blank.

What is the best option?

I am trying to use xpath with attribute. The code is not working. Someone please help me on what is the best solution?

@JeffCDoes not work after I make the changes.Please guide me how I can access the above review?The path can anyone guide me? — Himanish Bhattacharjee, Jan 11 '19 at 05:27
The problem here is that, the reviews are loaded when you scroll down on the page. Simply scrolling the page down to where reviews are, explicitly waiting for the reviews to load on the page and then calling `find_element_by_xpath` will get you the review text. Also, the xpath in you question has issues. `//div[@id='ratings-reviews']//div[@data-comp='Ellipsis Box']` should do the trick. Its `Ellipsis` with 2 "l" — GPT14, Jan 11 '19 at 06:28
Thanks@GPT14.After scrolling down data is coming.Can we do the scrolling down in the code itself ? — Himanish Bhattacharjee, Jan 11 '19 at 09:08
The xpath which is working for me is //*[@id="ratings-reviews"]/div[5]/div[2]/div[2]/div[1]/div[3] — Himanish Bhattacharjee, Jan 11 '19 at 09:09

score 2 · Accepted Answer · answered Jan 11 '19 at 12:35

To scrape the reviews from Sephora website you have to induce WebDriverWait for the elements to be visible and you can use the following solution:

Code Block:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_argument("disable-infobars")
options.add_argument("--disable-extensions")
driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
driver.get("https://www.sephora.com/product/crybaby-coconut-oil-shine-serum-P439093?skuId=2122083&icid2=just%20arrived:p439093")
driver.execute_script("arguments[0].scrollIntoView(true);", WebDriverWait(driver,20).until(EC.visibility_of_element_located((By.XPATH, "//div[@id='tabpanel0']/div//b[contains(., 'What Else You Need to Know')]"))))
reviews = WebDriverWait(driver,20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@data-comp='GridCell Box']//div[@data-comp='Ellipsis Box']")))
for review in reviews:
    print(review.get_attribute("innerHTML"))

Console Output:

Honestly I never write reviews but this is a must if you have frizzy after even after straightening it! It smells fantastic and it works wonders definitely will be restocking once I’m done this one !!
I really like this product. I was looking for something to tame frizz and fly aways during the winter and this does the job. At first I was nervous it might give a greasy look but it makes my hair smooth and soft. Scent is actually a little subtle for me, but still nice.
This oil-serum is perfect for the right level of hydration without the feel of oil residue. Great for all hair types and my new go-to product.
I LOVE how weightless this oil feels in my hair.. takes away all of my flyaways without looking of feeling greasy.. the packaging is COOL (travel-friendly) and it smells wonderful!!
I tried this when it first dropped on their website. I’ve been using it for about 3 weeks now. And I have to say its just OKAY. Nothing super special about it. I haven’t noticed super smooth hair that isn’t given with other products that cost less. It’s just like any other smoothing serum. I also can’t figure out what the smell is. It doesn’t really smell as pleasant as their other products.
in love!! A tiny bit goes a long way. No more fly aways. No more frizz from touch or environment.

Thanks.@DebanjanB .Now i need another help.If I want to access the review by opening the inspect then go to network and check the url that it will hit?I can send my post/get request to that url?Is it possible to do this? — Himanish Bhattacharjee, Jan 14 '19 at 11:47
@HimanishBhattacharjee Can you raise a new question with your new requirement please? StackOverflow contributors will be glad to help you out. — undetected Selenium, Jan 14 '19 at 11:56

How do I extract data from dynamic updating webpages

1 Answers1

Linked