1

My heroku web app has a feature to download images from S3. It works like this:

  1. There is one endpoint (A) to request downloading an array of images, that returns a task id.
  2. Those images are downloaded by A to the tmp Heroku folder of my app. And when all images are downloaded, a zip file is created.
  3. While images can still be downloading, the web client calls another endpoint (B) with the task id from point 1. This second endpoint checks how many images are already downloaded to return a progress percentage. When the zip is already created it "returns" the zip file and the images are downloaded.

This approach worked fine in Heroku with 1 dyno. Unfortunately, after scaling to 2 dynos, we have realised that it doesn't work anymore. The reason is that dynos in Heroku doesn't share the same file system, and endpoint A and B are managed by different dynos. Therefore, the dyno in endpoint B doesn't find any file.

Is there an easy way to make my approach work with multiple dynos?

If not, how should I implement the feature described? (downloading multiple images from S3 in a zip file)

Xavi Gil
  • 11,460
  • 4
  • 56
  • 71

2 Answers2

2

You could create a second S3 bucket and push the zip file to the second S3 bucket once it's done downloading. Then you can redirect the client to download the zip file directly from S3.

Then setup a process to run periodically to clean out anything old in that S3 bucket.

Daniel
  • 38,041
  • 11
  • 92
  • 73
  • Yes that could be one approach!! How could we monitor the progress of downloading a bunch of images though? In my solution I was "counting" the number of already downloaded images to return the download progress. – Xavi Gil Nov 04 '14 at 19:19
  • For the progress report, you should use Redis or Memcached. The dyno downloading the images would push progress to a key and any other dynos could pull that progress with the key value. Redis and memcached are great for synchronizing data between dynos. – Daniel Nov 04 '14 at 19:24
  • Thanks for those ideas. Tomorrow I'll think about this and accept your answer if appropriate. – Xavi Gil Nov 04 '14 at 19:26
  • Give us blog link or video I found that but i still stuck http://technomile.github.io/wordpress/setup.html – Abubakr Elghazawy May 05 '19 at 08:10
0

I think the solution is here , that may help http://technomile.github.io/wordpress/setup.html

Abubakr Elghazawy
  • 977
  • 14
  • 21