store scraped data in format like csv, json, mysql/sql, txt from selenium scrip using python

Question

I was scraping data from a website and I want to store this in a format like JSON, excel, sqlite or text format so that the data looks organised and sensible. Please help me.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
driver.get('https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/dp/B08Z1HHHTD/ref=sr_1_2?dchild=1&keywords=skybags&qid=1627786382&sr=8-2')

product_title = WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.ID, "productTitle"))).text

print(product_title)

WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//a[@data-hook='see-all-reviews-link-foot']"))).click()
    
while True:
    for item in WebDriverWait(driver, 10).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "[data-hook='review']"))):
        reviewer = item.find_element_by_css_selector("span.a-profile-name").text
        review = ' '.join([i.text.strip() for i in item.find_elements_by_xpath(".//span[@data-hook='review-body']")])
        print(reviewer,review)

    try:
        WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "//*[@data-hook='pagination-bar']//a[contains(@href,'/product-reviews/') and contains(text(),'Next page')]"))).click()
        WebDriverWait(driver, 10).until(EC.staleness_of(item))
    except Exception as e:
        break

driver.quit()

This might help: https://stackoverflow.com/questions/7528801/how-do-i-write-data-to-csv-file-in-columns-and-rows-from-a-list-in-python — Ram, Aug 01 '21 at 08:46
this is about CSV only, my preference is either JSON or mysql — krishan, Aug 01 '21 at 09:02
sorry for the communication gap but it want to know json or MySQL also with csv — krishan, Aug 01 '21 at 09:12

Ram · Accepted Answer · 2021-08-01T12:10:35.667

1

Store the values product_title, review and reviewer in a dictionary and convert that to a Json format using json module.

You may store the data in this format and finally convert the list to JSON.

lst = [{"product_title": <title>, "reviews": [{"review": <review>, "reviewer": <reviewer>}, {"review": <review>, "reviewer": <reviewer>}....]

import json
json.dumps(lst)

To write the data to a JSON file

with open('data.json', 'w', encoding='utf-8') as f:
    json.dump(lst , f, ensure_ascii=False)

edited Aug 01 '21 at 12:10

answered Aug 01 '21 at 09:32

Ram

4,724
2
14
22

how i will use this in for loop – krishan Aug 01 '21 at 10:08
We're here to help you and not to solve your entire problem without any effort from your end. – Ram Aug 01 '21 at 11:01
Well what data do you want to store as JSON ? – Ram Aug 01 '21 at 11:03
sorry my friend , i love those contributors who help each other and you are one of them Ram and as per your suggestion I applied this to my code and its working – krishan Aug 01 '21 at 11:16
Glad to hear that. :) – Ram Aug 01 '21 at 11:18
one last question the output is coming in one line only I want to add property like next line how can I do this – krishan Aug 01 '21 at 11:19
and the numbers are appearing in different way \u20b92,010.00 like this I think I should use utf-8 – krishan Aug 01 '21 at 11:41
use of indent solved the next line property issue just want to know about the numerical error – krishan Aug 01 '21 at 11:56
Yes. Those are Unicode characters. Use ```ensure_ascii = False``` in ```json.dump()```. – Ram Aug 01 '21 at 12:04
UnicodeEncodeError: 'charmap' codec can't encode character '\u20b9' in position 1: character maps to displaying this error after adding above statement – krishan Aug 01 '21 at 12:10
Updated answer. – Ram Aug 01 '21 at 12:10
still displaying error 'ensure_ascii=False' is not working fine ERRR: UnicodeEncodeError: 'charmap' codec can't encode character '\u20b9' in position 1: character maps to – krishan Aug 01 '21 at 12:17

store scraped data in format like csv, json, mysql/sql, txt from selenium scrip using python

1 Answers1