List of tables, db schema, dump etc using the Python sqlite3 API

Question

For some reason I can't find a way to get the equivalents of sqlite's interactive shell commands:

.tables
.dump

using the Python sqlite3 API.

Is there anything like that?

Answer 1

In Python:

con = sqlite3.connect('database.db')
cursor = con.cursor()
cursor.execute("SELECT name FROM sqlite_master WHERE type='table';")
print(cursor.fetchall())

Watch out for my other answer. There is a much faster way using pandas.

Answer 2

You can fetch the list of tables and schemata by querying the SQLITE_MASTER table:

sqlite> .tab
job         snmptarget  t1          t2          t3        
sqlite> select name from sqlite_master where type = 'table';
job
t1
t2
snmptarget
t3

sqlite> .schema job
CREATE TABLE job (
    id INTEGER PRIMARY KEY,
    data VARCHAR
);
sqlite> select sql from sqlite_master where type = 'table' and name = 'job';
CREATE TABLE job (
    id INTEGER PRIMARY KEY,
    data VARCHAR
)

Answer 3

The FASTEST way of doing this in python is using Pandas (version 0.16 and up).

Dump one table:

db = sqlite3.connect('database.db')
table = pd.read_sql_query("SELECT * from table_name", db)
table.to_csv(table_name + '.csv', index_label='index')

Dump all tables:

import sqlite3
import pandas as pd


def to_csv():
    db = sqlite3.connect('database.db')
    cursor = db.cursor()
    cursor.execute("SELECT name FROM sqlite_master WHERE type='table';")
    tables = cursor.fetchall()
    for table_name in tables:
        table_name = table_name[0]
        table = pd.read_sql_query("SELECT * from %s" % table_name, db)
        table.to_csv(table_name + '.csv', index_label='index')
    cursor.close()
    db.close()

Answer 4

I'm not familiar with the Python API but you can always use

SELECT * FROM sqlite_master;

Answer 5

Here's a short and simple python program to print out the table names and the column names for those tables (python 2. python 3 follows).

import sqlite3

db_filename = 'database.sqlite'
newline_indent = '\n   '

db=sqlite3.connect(db_filename)
db.text_factory = str
cur = db.cursor()

result = cur.execute("SELECT name FROM sqlite_master WHERE type='table';").fetchall()
table_names = sorted(zip(*result)[0])
print "\ntables are:"+newline_indent+newline_indent.join(table_names)

for table_name in table_names:
    result = cur.execute("PRAGMA table_info('%s')" % table_name).fetchall()
    column_names = zip(*result)[1]
    print ("\ncolumn names for %s:" % table_name)+newline_indent+(newline_indent.join(column_names))

db.close()
print "\nexiting."

(EDIT: I have been getting periodic vote-ups on this, so here is the python3 version for people who are finding this answer)

import sqlite3

db_filename = 'database.sqlite'
newline_indent = '\n   '

db=sqlite3.connect(db_filename)
db.text_factory = str
cur = db.cursor()

result = cur.execute("SELECT name FROM sqlite_master WHERE type='table';").fetchall()
table_names = sorted(list(zip(*result))[0])
print ("\ntables are:"+newline_indent+newline_indent.join(table_names))

for table_name in table_names:
    result = cur.execute("PRAGMA table_info('%s')" % table_name).fetchall()
    column_names = list(zip(*result))[1]
    print (("\ncolumn names for %s:" % table_name)
           +newline_indent
           +(newline_indent.join(column_names)))

db.close()
print ("\nexiting.")

Answer 6

If someone wants to do the same thing with Pandas

import pandas as pd
import sqlite3
conn = sqlite3.connect("db.sqlite3")
table = pd.read_sql_query("SELECT name FROM sqlite_master WHERE type='table'", conn)
print(table)

Answer 7

Apparently the version of sqlite3 included in Python 2.6 has this ability: http://docs.python.org/dev/library/sqlite3.html

# Convert file existing_db.db to SQL dump file dump.sql
import sqlite3, os

con = sqlite3.connect('existing_db.db')
with open('dump.sql', 'w') as f:
    for line in con.iterdump():
        f.write('%s\n' % line)

Answer 8

Some might find my function useful if you just want to print out all of the tables and columns in your db.

In the loop, I query each TABLE with LIMIT 0 so it just returns the header info without all the data. You make an empty df out of it, and use the iterable df.columns to print each column name out.

conn = sqlite3.connect('example.db')
c = conn.cursor()

def table_info(c, conn):
    '''
    prints out all of the columns of every table in db
    c : cursor object
    conn : database connection object
    '''
    tables = c.execute("SELECT name FROM sqlite_master WHERE type='table';").fetchall()
    for table_name in tables:
        table_name = table_name[0] # tables is a list of single item tuples
        table = pd.read_sql_query("SELECT * from {} LIMIT 0".format(table_name), conn)
        print(table_name)
        for col in table.columns:
            print('\t' + col)
        print()

table_info(c, conn)

Results will be:

table1
    column1
    column2

table2
    column1
    column2
    column3 

etc.

Answer 9

After a lot of fiddling I found a better answer at sqlite docs for listing the metadata for the table, even attached databases.

meta = cursor.execute("PRAGMA table_info('Job')")
for r in meta:
    print r

The key information is to prefix table_info, not my_table with the attachment handle name.

Answer 10

Check out here for dump. It seems there is a dump function in the library sqlite3.

Answer 11

#!/usr/bin/env python
# -*- coding: utf-8 -*-

if __name__ == "__main__":

   import sqlite3

   dbname = './db/database.db'
   try:
      print "INITILIZATION..."
      con = sqlite3.connect(dbname)
      cursor = con.cursor()
      cursor.execute("SELECT name FROM sqlite_master WHERE type='table';")
      tables = cursor.fetchall()
      for tbl in tables:
         print "\n########  "+tbl[0]+"  ########"
         cursor.execute("SELECT * FROM "+tbl[0]+";")
         rows = cursor.fetchall()
         for row in rows:
            print row
      print(cursor.fetchall())
   except KeyboardInterrupt:
      print "\nClean Exit By user"
   finally:
      print "\nFinally"

Answer 12

I've implemented a sqlite table schema parser in PHP, you may check here: https://github.com/c9s/LazyRecord/blob/master/src/LazyRecord/TableParser/SqliteTableDefinitionParser.php

You can use this definition parser to parse the definitions like the code below:

$parser = new SqliteTableDefinitionParser;
$parser->parseColumnDefinitions('x INTEGER PRIMARY KEY, y DOUBLE, z DATETIME default \'2011-11-10\', name VARCHAR(100)');