Write to CSV from sqlite3 database in python

Question

Ok, So I have a database called cars.db which has a table == inventory,

Inventory essentially contains

    ('Ford', 'Hiluz', 2),
    ('Ford', 'Tek', 6),
    ('Ford', 'Outlander', 9),
    ('Honda', 'Dualis', 3),
    ('Honday', 'Elantre', 4)

I then wrote this which is meant to edit that to the csv, however, I can't seem to work this out, in some cases I get stuff to print but its not right, and when I try and fix that, nothing prints. Any suggestions to get me on track?

#write table to csv

import sqlite3
import csv

with sqlite3.connect("cars.db") as connection:
    csvWriter = csv.writer(open("output.csv", "w"))
    c = connection.cursor()

    rows = c.fetchall()

    for x in rows:
        csvWriter.writerows(x)

`.writerows` as name suggests writes all rows and you want `.writerow` (no `s` at the end) which writes a single row. Also see Viktor's answer. — freakish, Sep 16 '13 at 11:49

score 9 · Accepted Answer · answered Sep 16 '13 at 11:41

9

You should just do:

rows = c.fetchall()
csvWriter.writerows(rows)

If the reason you iterate through the rows is because you wan't to preprocess them before writing them to the file, then use the writerow method:

rows = c.fetchall()
for row in rows:
    # do your stuff
    csvWriter.writerow(row)

answered Sep 16 '13 at 11:41

Viktor Kerkez

45,070
12
104
85

Great. That definitely helped. I also forgot that I needed to add a SELECT method in order to grab the data i.e. `code` c.execute("SELECT * FROM inventory") Then in combination with your suggestion it worked. Thanks – SScode Sep 16 '13 at 12:51

GERMAN RODRIGUEZ · Answer 2 · 2020-06-24T19:34:03.407

In order to put tittles in first row, dictionary approach is suggested for table inventory in cars.db

import sqlite3
import csv
import os.path
BASE_DIR = os.path.dirname(os.path.abspath(__file__))
db_path = os.path.join(BASE_DIR, "cars.db")
conn = sqlite3.connect(db_path)
c = conn.cursor()
c.execute("SELECT rowid, * FROM inventory")    
columns = [column[0] for column in c.description]
results = []
for row in c.fetchall():
    results.append(dict(zip(columns, row)))
with open("output.csv", "w", newline='') as new_file:
    fieldnames = columns
    writer = csv.DictWriter(new_file,fieldnames=fieldnames)
    writer.writeheader()
    for line in results:
        writer.writerow(line)
conn.close()

Powers · Answer 3 · 2020-09-18T20:35:54.193

Using Pandas should be more performant and requires less code. You can save the data from a sqlite table to a Pandas DataFrame and then use Pandas to write the CSV file.

df = pd.read_sql('SELECT * from cars', conn)
df.to_csv('cars.csv')

Here's the full code that creates your sqlite table with fake data:

import pandas as pd
import sqlite3

# create Pandas DataFrame
data = [('Toyota', 'Hilux', 2),
    ('Ford', 'Tek', 6),
    ('Ford', 'Outlander', 9),
    ('Honda', 'Dualis', 3),
    ('Honday', 'Elantre', 4)]
df = pd.DataFrame.from_records(data, columns=['make', 'model', 'age'])

# establish sqlite connection
conn = sqlite3.connect('../tmp/cars.db')
c = conn.cursor()

# create sqlite table
c.execute('''CREATE TABLE cars (make text, model text, age int)''')

# add data to sqlite table
df.to_sql('cars', conn, if_exists='append', index = False)

# write sqlite table out as a CSV file
df = pd.read_sql('SELECT * from cars', conn)
df.to_csv('../tmp/cars.csv')

Here's code to write out all the tables in a sqlite database as CSV files with a single command:

for table in c.execute("SELECT name FROM sqlite_master WHERE type='table';").fetchall():
    t = table[0]
    df = pd.read_sql('SELECT * from ' + t, conn)
    df.to_parquet('../tmp/' + t + '.csv')

See here for more info.

Write to CSV from sqlite3 database in python

3 Answers3

In order to put tittles in first row, dictionary approach is suggested for table inventory in cars.db

Linked