0

I am scraping the data from the website. 2 days ago the script still returned the data but today it returns nothing.

What is the problem with my code ?

from selenium import webdriver
import pandas as pd

op = webdriver.ChromeOptions()
op.add_argument('--ignore-certificate-errors')
op.add_argument('--incognito')
op.add_argument('--headless')
driver = webdriver.Chrome(executable_path='C:/Users/chromedriver.exe',options=op)
driver.get('https://www.conforama.fr/chambre-literie/literie/tous-les-matelas/c/030106?fromSearch=Matelas&limit=120')


sellers = driver.find_elements_by_class_name('sold-by')
titles = driver.find_elements_by_tag_name('h3')
links = driver.find_elements_by_class_name('awk-detail-product')

n = 5
data = [[] for _ in range(n)]

for x in sellers:
    data[1].append(x.text[10:])
    
for y in titles:
    data[2].append(y.text)
    
for z in links:
    data[3].append(z.find_element_by_css_selector('a').get_attribute('href'))
    data[4].append(z.find_element_by_css_selector('a').get_attribute('href').rsplit('/',1)[-1])

data = pd.DataFrame(data).T
mht
  • 133
  • 1
  • 2
  • 9
  • 1
    Ran your code here and if I do `print(driver.page_source)` I get `Request unsuccessful. Incapsula incident ID: 1227000050329793951-899015399063617868` with some HTML code that indicates that the page has detected your web-scraping attempt and has blocked it. –  Oct 02 '20 at 12:46
  • Oh thanks Justin. Is there any solution for that ? Change the code or something ? – mht Oct 02 '20 at 12:57
  • I do not have time today but see if you can find some tips here: https://stackoverflow.com/questions/33225947/can-a-website-detect-when-you-are-using-selenium-with-chromedriver –  Oct 02 '20 at 13:40

0 Answers0