4

I'm trying to achieve the following. I want to create a python Class that transforms all tables in a database to pandas dataframes.

This is how I do it, which is not very generic... 

class sql2df():
    def __init__(self, db, password='123',host='127.0.0.1',user='root'):
        self.db = db

        mysql_cn= MySQLdb.connect(host=host,
                        port=3306,user=user, passwd=password, 
                        db=self.db)

        self.table1 = psql.frame_query('select * from table1', mysql_cn)
        self.table2 = psql.frame_query('select * from table2', mysql_cn)
        self.table3 = psql.frame_query('select * from table3', mysql_cn)

Now I can access all tables like so:

my_db = sql2df('mydb')
my_db.table1

I want something like:

class sql2df():
    def __init__(self, db, password='123',host='127.0.0.1',user='root'):
        self.db = db

        mysql_cn= MySQLdb.connect(host=host,
                        port=3306,user=user, passwd=password, 
                        db=self.db)
        tables = (""" SELECT TABLE_NAME FROM information_schema.TABLES WHERE TABLE_SCHEMA = '%s' """ % self.db)
        <some kind of iteration that gives back all the tables in df as class attributes>

Suggestions are most welcome...

bowlby
  • 649
  • 1
  • 8
  • 18

3 Answers3

5

I would use SQLAlchemy for this:

engine = sqlalchemy.create_engine("mysql+mysqldb://root:123@127.0.0.1/%s" % db)

Note the syntax is dialect+driver://username:password@host:port/database.

def db_to_frames_dict(engine):
    meta = sqlalchemy.MetaData()
    meta.reflect(bind=engine)
    tables = meta.sorted_tables
    return {t: pd.read_sql('SELECT * FROM %s' % t.name,
                           engine.raw_connection())
                   for t in tables}
    # Note: frame_query is depreciated in favor of read_sql

This returns a dictionary, but you could equally well have these as class attributes (e.g. by updating the class dict and __getitem__)

class SQLAsDataFrames:
    def __init__(self, engine):
        self.__dict__ = db_to_frames_dict(engine)  # allows .table_name access
    def __getitem__(self, key):                    # allows [table_name] access
        return self.__dict__[key]

In pandas 0.14 the sql code has been rewritten to take engines, and IIRC there is helpers for all tables and for reading all of a table (using read_sql(table_name)).

Andy Hayden
  • 359,921
  • 101
  • 625
  • 535
  • Thanks! Didn't know about sqlalchemy! The problem I'm having still is how to 'dynamically' set a class attributes, eg what you refer to as 'updating the class dict'. – bowlby Apr 09 '14 at 06:34
  • @bowlby for attributes: `class A: def __init__(self, d): self.__dict__.update(d)` where d is the dict above. Maybe `def __getitem__(self, key): return self.__dict__[key]` for good measure. – Andy Hayden Apr 09 '14 at 06:58
  • +Andy Hayden think you are missing the closing ) on the select – dartdog Apr 10 '14 at 19:15
  • How do I call sql2df? what is the expected argument? eg sql2df(?) and then how to access the tables? – dartdog Apr 10 '14 at 19:29
  • if i try t=[] sql2df(t) I get global name 'sqlalchmey' is not defined – dartdog Apr 10 '14 at 19:35
  • @dartdog shows how completely untested this answer is! The expected output of sql2df is a dict of dataframes... should choose a better name! – Andy Hayden Apr 10 '14 at 19:51
  • closer?? dunno about the reflect, my 1st attempt at getting sqlalchemy hooked up :-) but when I do "db_to_frames_dict(engine)" I get global name 'psql' is not defined (I'm using mysql (mariadb if that makes a diff..) – dartdog Apr 10 '14 at 20:17
  • @dartdog my guess is OP had imported it as `from pandas.io import sql as psql`... no need as it's available at top level! – Andy Hayden Apr 10 '14 at 20:26
  • nope ----> 5 return {t: sql.read_sql('SELECT * FROM %s' % t, engine.connect())} 6 # Note: frame_query is depreciated in favor of read_sql NameError: global name 'sql' is not defined Tried pd.sql as well... – dartdog Apr 10 '14 at 20:30
  • @dartdog try top level pd.read_sql (I updated post!) Dang, I really should test this properly! – Andy Hayden Apr 10 '14 at 20:35
  • think it may need the reflect? from my dim slow reading (remember this is my 1st go with sqlalchemy!) currently when I do with the updated code >>> frames=db_to_frames_dict(engine) frames I get {} (fwiw I am getting table results out of my connection so I have the base system working) – dartdog Apr 10 '14 at 20:40
  • Maybe the syntax is this: http://docs.sqlalchemy.org/en/rel_0_9/core/reflection.html#reflecting-all-tables-at-once (I vaguely recall from my attempt at pandas sql with SQLA, strange that it needs it but I think it does...) Thanks for testing/putting up with this! Updated post... – Andy Hayden Apr 10 '14 at 20:44
  • I'm adding what I have in a new answer it is erroring though You have a few edit issues MetaData and reflect... – dartdog Apr 10 '14 at 20:59
  • Meta needs to be MetaData I think? see my new post which I think is upto date with yours – dartdog Apr 10 '14 at 21:08
  • @dartdog goodness so many errors, I think the last is to use raw_connection... I'm hoping you can get away without closing here :s http://stackoverflow.com/questions/20401392/read-frame-with-sqlalchemy-mysql-and-pandas – Andy Hayden Apr 10 '14 at 21:23
  • Ok this db_to_frames_dict works but.. but SQLAsDataFrames not working class creation. I'm not exactly sure what the result of the db_to_frames is it seems to be a collection of all my data so I have not worked out how to address the resulting data yet? – dartdog Apr 10 '14 at 21:31
  • @dartdog it **should** be a dictionary of table names to DataFrames (for each table in the db). I have a feeling bug is should be using table name as key (i.e. should be `{t.name: ..}` rather that a Table object)? – Andy Hayden Apr 10 '14 at 21:44
1

Here is what I have now: Imports

 import sqlalchemy
 from sqlalchemy import create_engine
 from sqlalchemy import Table, Column,Date, Integer, String, MetaData, ForeignKey
 from sqlalchemy.ext.declarative import declarative_base
 from sqlalchemy.orm import relationship, backref
 import pandas as pd

engine = sqlalchemy.create_engine("mysql+mysqldb://root:password@127.0.0.1/%s" % 'surveytest')
def db_to_frames_dict(engine):
    meta = sqlalchemy.MetaData()
    meta.reflect(bind=engine)
    tables = meta.sorted_tables
    return {t: pd.read_sql('SELECT * FROM %s' % t.name, engine.connect())
               for t in tables}
# Note: frame_query is depreciated in favor of read_sql

Have not started to fiddle with this part yet! below:

class SQLAsDataFrames:
    def __init__(self, engine):
        self.__dict__ = db_to_frames_dict(engine)  # allows .table_name access
    def __getitem__(self, key):                    # allows [table_name] access
        return self.__dict__[key]

And the error: Which looks like at least it is trying to get the table names...

frames=db_to_frames_dict(engine)
frames

Error on sql SELECT * FROM tbl_original_survey_master
---------------------------------------------------------------------------
 AttributeError                            Traceback (most recent call last)
 <ipython-input-4-6b0006e1ce47> in <module>()
 ----> 1 frames=db_to_frames_dict(engine)
 >>>> more tracebck
 ---> 53             con.rollback()
 54         except Exception:  # pragma: no cover
 55             pass

 AttributeError: 'Connection' object has no attribute 'rollback'

Thank you for sticking with this!

dartdog
  • 10,432
  • 21
  • 72
  • 121
  • as above, Connection object is apparently not a connection! http://stackoverflow.com/questions/20401392/read-frame-with-sqlalchemy-mysql-and-pandas Apologies again for poor initial state of answer! – Andy Hayden Apr 10 '14 at 21:28
1

Thanks for all the help, this is what I ended up using:

import sqlalchemy
from sqlalchemy import create_engine
from sqlalchemy import Table, Column,Date, Integer, String, MetaData, ForeignKey
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import relationship, backref
import pandas as pd

engine = sqlalchemy.create_engine("mysql+mysqldb://root:123@127.0.0.1/%s" % 'mydb')

def db_to_frames_dict(engine):
    meta = sqlalchemy.MetaData()
    meta.reflect(engine)
    tables = meta.tables.keys()
    cnx = engine.raw_connection()

    return {t: pd.read_sql('SELECT * FROM %s' % t, cnx )
               for t in tables}

class SQLAsDataFrames:
    def __init__(self, engine):
        self.__dict__ = db_to_frames_dict(engine)  # allows .table_name access
    def __getitem__(self, key):                    # allows [table_name] access
        return self.__dict__[key]
bowlby
  • 649
  • 1
  • 8
  • 18