I am reading data from several csv sheets. After some transformation I am trying to write data to a MySQL database. I construct raw SQL insert statements and use sqlalchemy.text() to deal with string escaping issues. Writing line by line works fine, but I have to insert a lot of rows and therefore, I would like to construct a raw bulk insert statement. A full simplified version can be found at the end. In short I, for the following line by line python insert snippet I would like to have a something that is equivalent to a bulk insert using raw SQL and sqlalchemy.text().
Line by line text insert:
for row in rows:
stm = text("INSERT INTO scrapping_data.foo (id, sheet, some_text) VALUES (:id,:sheet,:some_text)").\
bindparams(id=row["id"], sheet=row["sheet"],some_text=row["some_text"])
with engine.connect() as connection:
connection.execute(stm)
Python code that is equivalent statement:
INSERT INTO scrapping_data.foo (id, sheet, some_text) VALUES (1,'sheet1','Lorem ipsum dolor sit amet'), (2,'sheet2','Lorem ipsum dolor sit amet')
Full simplified version of my code:
Create table in MySQL
CREATE TABLE foo (
id INT,
sheet VARCHAR(255),
some_text TEXT,
PRIMARY KEY (id, sheet)
);
Insert script in python
from sqlalchemy import create_engine, text
DB_USER = "<YOUR_USER>"
DB_PASS = "<YOUR_PASS>"
DB_HOST = "<YOUR_HOST>"
DB_PORT = "<YOUR_PORT>"
DB_NAME = "<YOUR_NAME>"
engine = create_engine("""mysql+mysqlconnector://{db_user}:{db_pass}@{db_host}:{db_port}/{db_name}""".format(
db_user=DB_USER,
db_pass=DB_PASS,
db_host=DB_HOST,
db_port=DB_PORT,
db_name=DB_NAME))
rows = [{'id':1,'sheet':'sheet1','some_text':'Lorem ipsum dolor sit amet'},
{'id':2,'sheet':'sheet2','some_text':'consectetur adipiscing elit'}]
for row in rows:
stm = text("INSERT INTO scrapping_data.foo (id, sheet, some_text) VALUES (:id,:sheet,:some_text)").\
bindparams(id=row["id"], sheet=row["sheet"],some_text=row["some_text"])
with engine.connect() as connection:
connection.execute(stm)