Pandas Write table to MySQL: "unable to rollback"

Question

I need help to get this working. I have a pd.DataFrame (df), which I need to load to a MySQL database. I don't understand what the error message means and how to fix it.

Any help will be highly appreciated.

This is what I tried:

    import MySQLdb
    from pandas.io import sql

    #METHOD 1 
    db=MySQLdb.connect(host="***",port=***,user="***",passwd="***",db="***")
    df.to_sql(con=db, name='forecast', if_exists='replace', flavor='mysql')
    ##Also tried
    sql.write_frame(df, con=db, name='forecast', if_exists='replace', flavor='mysql')

   **DatabaseError**: Execution failed on sql: SHOW TABLES LIKE %s
   (2006, 'MySQL server has gone away')
   unable to rollback


   #METHOD 2: using sqlalchemy
   from sqlalchemy import create_engine

   engine =   create_engine("mysql+mysqldb://**username***:**passwd**@***host***:3306/**dbname**")
   conn = engine.raw_connection()
   df.to_sql(name='demand_forecast_t', con=conn,if_exists='replace',    flavor='mysql',index=False, index_label='rowID')
   conn.close()

The error message is:

**OperationalError**: DatabaseError: Execution failed on sql: SHOW TABLES LIKE %s
(2006, 'MySQL server has gone away') unable to rollback

I think I know the issue. The dataFrame I am trying to write is too large to write at a time. If I take a subset, the above code works. I guess I will need to write some sort of iteration to do this job at once. — Amrita Sawant, Mar 30 '15 at 21:36

joris · Answer 1 · 2015-04-01T07:26:50.307

When using sqlalchemy, you should pass the engine and not the raw connection:

engine = create_engine("mysql+mysqldb://...")
df.to_sql('demand_forecast_t', engine, if_exists='replace', index=False)

Writing to MySQL without sqlalchemy (so with specifying flavor='mysql') is deprecated.

When the problem is that you have a too large frame to write at once, you can use the chunksize keyword (see the docstring). Eg:

df.to_sql('demand_forecast_t', engine, if_exists='replace', chunksize=10000)

score 0 · Answer 2 · answered Apr 01 '15 at 04:18

0

I was able to resolve this issue. I was trying to load a large table into MySQL and as a result of which was getting the error. A simple for-loop to upload data in chunks solved the issue ! Many thanks to everyone who replied.

answered Apr 01 '15 at 04:18

Amrita Sawant

10,403
4
22
26

You don't need a for loop for this. You can use the `chunksize` keyword. – joris Apr 01 '15 at 07:25
How to read sqlalchemy create engine parameters from a config file? Any help on this? – Viv Apr 26 '17 at 14:18

citynorman · Answer 3 · 2016-07-18T03:48:44.447

0

For me this was fixed using

MySQLdb.connect("127.0.0.1","root","","db" )

instead of

MySQLdb.connect("localhost","root","","db" )

and then

df.to_sql('df',sql_cnxn,flavor='mysql',if_exists='replace', chunksize=100)

edited Jul 18 '16 at 03:48

answered Jul 18 '16 at 03:23

citynorman

4,918
3
38
39

pyAddict · Answer 4 · 2016-09-04T14:45:40.133

You can write pandas dataframe in mysql table using mysql flavour(with DBAPI connection) in following ways

step1: install mysqldb module - $ sudo apt-get install python-dev libmysqlclient-dev then $ pip install MySQL-python

step2: make a connection with mysql import MySQLdb con = MySQLdb.connect("hostname","username","password","databasename")

step3: write pandas dataframe in mysql table by using df.to_sql df.to_sql('TableName',con = con,flavor='mysql',if_exists='replace', chunksize=100)

Pandas Write table to MySQL: "unable to rollback"

4 Answers4

Linked