1

I have a import functionality in that I upload CSV/XLS file to import a data. I have a 30000 records (CSV) file.(size : 3.4 MB).

This file will take 40 to 50 MIN. to import a data.

As per each record I stored data into 3 tables.

I want to reduce that time to importing a data. What should I do please help me

Thanks In Advance

TechCompose
  • 53
  • 1
  • 8
  • With those numbers you are importing 10-12 records a second. First thing to check, if the data is going into 3 tables, do those tables have a lot of indexes? If so, you should try temporarily disabling indexing until the import is complete. Second thing to check, is the process importing to a local database, or are the imports going through some sort of network API? With a slow API, you could try splitting the input into two or more files and importing them in parallel (with the indexes disabled as well). – Roger Halliburton Mar 07 '15 at 04:55

1 Answers1

0

I'm going to assume that you are using tables in a database. Importing 3.4 MB shouldn't take that long to import. In on one of my recent projects, I had to import and parse 100MB+ of dictionary files and it only took a minute (python based).

The time really depends on the code that you have written. Although there are some things to look for that will help reduce the import time.

The first is don't print any values in loops. It generally uses up a good amount of time in any language.

Also only open the database once, no need to close it when your in the same function or problem space.

Use the executemany functionality when it is available. When you are ready to commit all of the changes then commit them all at once.

It would also be nice to see how you structured your import function, then I might be able to provide more details.

Edit: See Improve INSERT-per-second performance of SQLite?

Community
  • 1
  • 1
Blayze
  • 21
  • 7