0

My mysql version is 8.0. I have an InnoDB table with more than 10 million rows of data, and the data file size is 2G.

I am using pd.to_sql to insert several thousand rows of data each time. When the index is enabled, the insertion speed is very slow.

How to temporarily disable the index? And after the insertion is complete, enable the index?

I tried the operation in How to disable index in innodb:

SET autocommit=0;
SET unique_checks=0;
SET foreign_key_checks=0;

But the index is still valid.

jaried
  • 632
  • 4
  • 15
  • 3
    *How to temporarily disable the index?* Noway. You may drop the index and recreate it after insertion completed. PS. Primary key (which is clustered) must be stored anycase - if you drop it with another indices then complete table rewriting will occur during its re-creation. – Akina Jun 10 '21 at 04:36
  • It seems that only this method is feasible. – jaried Jun 10 '21 at 04:43

2 Answers2

3
  1. Drop the index
  2. Insert your data
  3. Create your index
Albert Alberto
  • 801
  • 8
  • 15
3

No, something else is going wrong.

Please provide more details of the code involved. Inserting a few thousand rows in a 10M-row table will not benefit from the high cost of rebuilding the index.

Please provide SHOW CREATE TABLE so we can see the indexes. UNIQUE indexes are handled differently. FOREIGN KEYs add overhead. Batching the rows helps a lot, but 10K rows is on the high-end of what is efficient. What is the PRIMARY KEY? Are the rows being inserted in any predictable order?

Rick James
  • 135,179
  • 13
  • 127
  • 222