0

My client has an e-commerce system with products (there is ~60k product items). The price list is placed on the URL address in ZIP format.

I need firstly download the price list, unzip that, then open the CSV file and start read line by line. In his database is updated price from the CSV (if is product's price different in the CSV than in his database, use the price from CSV and update database).

But, is possible to do this on Heroku? The download of almost 40MB file take some time, import data from CSV to database (the import on localhost takes about 17 minutes - but this is import of whole CSV to database) as well.

How would you solve this situation? What options do I have? Unfortunately, there is no way to change the CSV file, so I have to work with that file...

Thanks

user984621
  • 46,344
  • 73
  • 224
  • 412

2 Answers2

2

There's a common misconception about the file system on heroku. It's not that it's read-only, it's that it's ephemeral. You can start a process and start writing to the filesystem, but as soon as that process exits, the data is gone forever.

So, you can indeed download a zip file and then process it, but even if you run into an error while processing, causing your program to exit, the unzipped data will be lost. You might want to think about unzipping it, creating chunks of data that you put on S3, and then having any number of workers crunching that data and internalizing it with your own database.

hgmnz
  • 13,208
  • 4
  • 37
  • 41
1

This is possible by running the work on a delayed job on a worker process. For more information see: https://devcenter.heroku.com/articles/ps. Note that if this is a regular task you want to run on a scheduled time interval you can also use Scheduler (https://devcenter.heroku.com/articles/scheduler).

Kevin Sylvestre
  • 37,288
  • 33
  • 152
  • 232
  • Yes, this is a regular task. I know Scheduler, I have to run the task once a day. What I hesitate of how to do is, how to download ZIP file with ruby on Heroku, if I can't to save anything to Heroku server? – user984621 Aug 05 '12 at 21:06
  • @user984621 Heroku will allow you to store files temporarily if you just need to unzip an process them. If you want to keep them longer term (i.e. for record keeping or future access) you should store them on a cloud storage service such as Amazon Simple Storage Solution. For how to download and process check out http://stackoverflow.com/questions/5386159/download-a-zip-file-through-nethttp. – Kevin Sylvestre Aug 06 '12 at 03:47