5

I am reading data from several csv sheets. After some transformation I am trying to write data to a MySQL database. I construct raw SQL insert statements and use sqlalchemy.text() to deal with string escaping issues. Writing line by line works fine, but I have to insert a lot of rows and therefore, I would like to construct a raw bulk insert statement. A full simplified version can be found at the end. In short I, for the following line by line python insert snippet I would like to have a something that is equivalent to a bulk insert using raw SQL and sqlalchemy.text().

Line by line text insert:

for row in rows:
  stm = text("INSERT INTO scrapping_data.foo (id, sheet, some_text) VALUES (:id,:sheet,:some_text)").\
      bindparams(id=row["id"], sheet=row["sheet"],some_text=row["some_text"])

  with engine.connect() as connection:
    connection.execute(stm)

Python code that is equivalent statement:

INSERT INTO scrapping_data.foo (id, sheet, some_text) VALUES (1,'sheet1','Lorem ipsum dolor sit amet'), (2,'sheet2','Lorem ipsum dolor sit amet') 

Full simplified version of my code:

Create table in MySQL

CREATE TABLE foo (
  id INT,
  sheet VARCHAR(255),
  some_text TEXT,
  PRIMARY KEY (id, sheet)
);

Insert script in python

from sqlalchemy import create_engine, text

DB_USER = "<YOUR_USER>"
DB_PASS = "<YOUR_PASS>"
DB_HOST = "<YOUR_HOST>"
DB_PORT = "<YOUR_PORT>"
DB_NAME = "<YOUR_NAME>"

engine = create_engine("""mysql+mysqlconnector://{db_user}:{db_pass}@{db_host}:{db_port}/{db_name}""".format(
    db_user=DB_USER,
    db_pass=DB_PASS,
    db_host=DB_HOST,
    db_port=DB_PORT,
    db_name=DB_NAME))

rows = [{'id':1,'sheet':'sheet1','some_text':'Lorem ipsum dolor sit amet'},
        {'id':2,'sheet':'sheet2','some_text':'consectetur adipiscing elit'}]

for row in rows:
  stm = text("INSERT INTO scrapping_data.foo (id, sheet, some_text) VALUES (:id,:sheet,:some_text)").\
      bindparams(id=row["id"], sheet=row["sheet"],some_text=row["some_text"])

  with engine.connect() as connection:
    connection.execute(stm)
Peanut
  • 803
  • 1
  • 11
  • 24
  • Take a look at https://stackoverflow.com/questions/8134602/psycopg2-insert-multiple-rows-with-one-query Yes, I know that thats for postgres. But some of the answers seem built around cursor.executemany, rather than anything postgres specific. You may be help to adopt one. – JL Peyret Apr 16 '20 at 17:33

0 Answers0