7

The documentation for the Pandas function to_sql() available for DataFrame objects (see to_sql() documentation) does not state that a commit() call on the connection is needed (or recommended) to persist the update.

Can I safely assume that DataFrame.to_sql('table_name', con) will always automatically commit the changes (like in: con.commit())?

MaxU - stand with Ukraine
  • 205,989
  • 36
  • 386
  • 419
wstomv
  • 761
  • 1
  • 6
  • 13

1 Answers1

9

Yes, at the end of the day it will be commited automatically.

Pandas calls SQLAlchemy method executemany (for SQL Alchemy connections):

conn.executemany(self.insert_statement(), data_list)

for SQLite connection:

def run_transaction(self):
    cur = self.con.cursor()
    try:
        yield cur
        self.con.commit()
    except:
        self.con.rollback()
        raise
    finally:
        cur.close()

And due to the SQL Alchemy docs executemany issues commit at the end

MaxU - stand with Ukraine
  • 205,989
  • 36
  • 386
  • 419
  • The Pandas function `to_sql()` also works when SQLAlchemy is not installed, in which case it uses `sqlite` (if I understand the documentation correctly). Is the `commit` then also guaranteed? – wstomv Dec 10 '17 at 19:53
  • I cannot (unambiguously) find the SQLite connections are deprecated in Pandas. There are some discussions about this ([here](https://github.com/pandas-dev/pandas/issues/6900) and [here](https://github.com/pandas-dev/pandas/issues/6881)), and my understanding is that SQLite connections (via `sqlite3`) are still a supported feature (possibly under the flag 'legacy'). – wstomv Dec 10 '17 at 20:24
  • @TomVerhoeff, yes, that's correct. Sorry for the missleading information. I've mixed it up with the [deprecated `flavor` in to_sql](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_sql.html) – MaxU - stand with Ukraine Dec 10 '17 at 20:28
  • The links to the code lines are no longer applicable as they didn't refer to a specific git revision, so it's hard now to follow. It would be nice if we could control commit behavior explicitly through sqlalchemy api as an answer to this. – matanster Sep 29 '19 at 15:49
  • I think I don't see an explicit commit in https://github.com/pandas-dev/pandas/blob/c4489cbfa5f93e69301486e1ef867caed9f4eb9e/pandas/io/sql.py#L724 which is called by Pandas for this. It seems that the python API to sqlite does not auto-commit [if I'm looking in the right place](https://docs.python.org/3.7/library/sqlite3.html#controlling-transactions). On the other hand you can't `create_engine(isolation_level="AUTOCOMMIT")` for sqlite, so I'm not sure yet what is the default commit behavior with sqlite using sqlalchemy. – matanster Sep 29 '19 at 16:19