Processing large csv file

Question

I have a large CSV file with a lot of data more than 75000 records. I need to insert it into a mysql database from my rails application. Using background processing using Sidekiq (https://github.com/mperham/sidekiq‎) is also taking a lot of time. Any better way to process the data ?

Thanks

This is a MySQL "load data" question, not Rails. Try: http://dev.mysql.com/doc/refman/5.5/en/load-data.html — Phlip, Nov 13 '13 at 16:08
if you need to read to file, then process something on the data and then save that proccessed data to the database you should try to optimize your code, I don't think it's sidekiq's fault, sidekiq only runs your code, you can try delayed_job or resque, but I think the problem is your processing algorithm (i.e. try to optimize validations, first validate the fields that will be faster, don't just run "valid?" on a model, if any fast validations fails just skip that record) — arieljuod, Nov 13 '13 at 18:57

score 1 · Answer 1 · edited May 23 '17 at 12:19

1

You can do it directly on mysql without using rails. It will be much faster that way. Have a look at this SO question for more information

edited May 23 '17 at 12:19

Community

1
1

answered Nov 13 '13 at 16:36

usha

28,973
5
72
93

score 0 · Answer 2 · answered Nov 15 '13 at 21:07

0

Use MySql Function:

LOAD DATA INFILE 'data.txt' INTO TABLE tbl_name
FIELDS TERMINATED BY ',' ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
IGNORE 1 LINES;

answered Nov 15 '13 at 21:07

Chadi

97
6

Processing large csv file

2 Answers2