1

I am currently conducting a data scraping project with Python 3 and am attempting to write the scraped data to a CSV file. My current process to do it is this:

import csv

outputFile = csv.writer(open('myFilepath', 'w'))
outputFile.writerow(['header1', 'header2'...])
for each in data:
     scrapedData = scrap(each)
     outputFile.writerow([scrapedData.get('header1', 'header 1 NA'), ...])

Once this script is finished, however, the CSV file is blank. If I just run:

import csv

outputFile = csv.writer(open('myFilepath', 'w'))
outputFile.writerow(['header1', 'header2'...])

a CSV file is produced containing the headers:

header1,header2,..

If I just scrape 1 in data, for example:

outputFile.writerow(['header1', 'header2'...])
scrapedData = scrap(data[0])
outputFile.writerow([scrapedData.get('header1', 'header 1 NA'), ...])

a CSV file will be created including both the headers and the data for data[0]:

header1,header2,..
header1 data for data[0], header1 data for data[0]

Why is this the case?

jdoe
  • 634
  • 5
  • 19
  • 1
    Try using a context manager. You are never closing your file, so it might not be flushing. A context manager should always be used when working with files, anyway. – juanpa.arrivillaga Aug 03 '18 at 23:33
  • You have to close the file after you finished writing into it. – DYZ Aug 03 '18 at 23:53

2 Answers2

2

When you open a file with w, it erases the previous data

From the docs

w: open for writing, truncating the file first

So when you open the file after writing scrape data with w, you just get a blank file and then you write the header on it so you only see the header. Try replacing w with a. So the new call to open the file would look like

outputFile = csv.writer(open('myFilepath', 'a'))

You can fine more information about the modes to open the file here

Ref: How do you append to a file?

Edit after DYZ's comment:

You should also be closing the file after you are done appending. I would suggest using the file like the:

with open('path/to/file', 'a') as file:
    outputFile = csv.writer(file)
    # Do your work with the file

This way you don't have to worry about remembering to close it. Once the code exists the with block, the file will be closed.

anon
  • 1,258
  • 10
  • 17
  • The OP is not appending to the file. They open it just once. The problem is with them not closing the file. – DYZ Aug 03 '18 at 23:53
  • I will update the answer. Good catch. OP still needs to be using the correct mode otherwise the problem would still exist. – anon Aug 03 '18 at 23:57
0

I would use Pandas for this:

import pandas as pd
headers = ['header1', 'header2', ...]
scraped_df = pd.DataFrame(data, columns=headers)
scraped_df.to_csv('filepath.csv')

Here I'm assuming your data object is a list of lists.

PMende
  • 5,171
  • 2
  • 19
  • 26