How can I export this web-scripped data into a csv file?

Question

I'm quite new to coding and web-scraping, i've been watching plenty of tutorials on youtube but couldn't find a way to write those data in a csv file. Can someone help?

import pandas as pd
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from bs4 import BeautifulSoup


options = Options()
options.add_argument("window-size=1400,600")
from fake_useragent import UserAgent
ua = UserAgent()
a = ua.random
user_agent = ua.random
print(user_agent)
options.add_argument(f'user-agent={user_agent}')


driver = webdriver.Chrome('/Users/raduulea/Documents/chromedriver', options=options)
driver.get('https://www.immoweb.be/fr/recherche/immeuble-de-rapport/a-vendre')

import time
time.sleep(10)

html = driver.page_source
soup = BeautifulSoup(html, 'html.parser')

results = soup.find_all("div", {"class":"result-xl"})

for result in results:
    print(result.find("div", {"class":"title-bar-left"}).get_text())
    print(result.find("span", {"result-adress"}).get_text())
    print(result.find("div", {"class":"xl-price rangePrice"}).get_text())
    print(result.find("div", {"class":"xl-surface-ch"}).get_text())
    print(result.find("div", {"class":"xl-desc"}).get_text())

score 0 · Accepted Answer · answered May 17 '19 at 07:00

Use pandas DataFrame to add data in it. and then export to CSV file which is much easier.

    import pandas as pd
    from selenium import webdriver
    from selenium.webdriver.chrome.options import Options
    from bs4 import BeautifulSoup


    options = Options()
    options.add_argument("window-size=1400,600")
    from fake_useragent import UserAgent
    ua = UserAgent()
    a = ua.random
    user_agent = ua.random
    print(user_agent)
    options.add_argument(f'user-agent={user_agent}')


    driver = webdriver.Chrome('/Users/raduulea/Documents/chromedriver', options=options)

    driver.get('https://www.immoweb.be/fr/recherche/immeuble-de-rapport/a-vendre')

    import time
    time.sleep(10)

    html = driver.page_source
    soup = BeautifulSoup(html, 'html.parser')

    results = soup.find_all("div", {"class":"result-xl"})
    title=[]
    address=[]
    price=[]
    surface=[]
    desc=[]
    for result in results:
       title.append(result.find("div", {"class":"title-bar-left"}).get_text().strip())
       address.append(result.find("span", {"result-adress"}).get_text().strip())
       price.append(result.find("div", {"class":"xl-price rangePrice"}).get_text().strip())
       surface.append(result.find("div", {"class":"xl-surface-ch"}).get_text().strip())
       desc.append(result.find("div", {"class":"xl-desc"}).get_text().strip())


df = pd.DataFrame({"Title":title,"Address":address,"Price:":price,"Surface" : surface,"Description":desc})
df.to_csv("output.csv")

Output: Your csv file will be like this.

Thanks a lot ! You helped me so much in my projet, I'll try to advance by myself now, but I was blocked at this problem for a long time ! — mr-kim, May 17 '19 at 07:23

How can I export this web-scripped data into a csv file?

1 Answers1