2

Hey Guys

At the moment I have a NodeJS webapp in the making which scrapes a website for data. Specifically, this webapp scrapes images for the purpose of downloading them. For example, all the image permalinks are scraped from the reddit front page. They are then sent to the client to download individually. My issue is with the website I am scraping there can be thousands of images.

This provides a horrible user experience if 1000+ images are downloaded to the download folder.
As a result I have two options.

A) Download to a temporary folder on the server. Zip. Send to client for download. Delete from server

B) Download files to browser cache. zip. download to specified download directory.

My question to you is this; Is option B even possible?

I am relatively new to this entire process and I can't find anything to actually zip the files in the browser cache. I can implement option A relatively easily however this requires a large amount of bandwidth, something I can find for around $5/MO on DigitalOcean. However this entire project is a learning experience and as a result I would love to be able to manage files in the browser cache instead.

I am using the following NPM Modules:

  • NodeJS
  • Express
  • Cheerio
  • Request

Further Update

I had come across a plugin for NPM called jsZip: https://stuk.github.io/jszip/

However, I was unaware it could be implemented on the client side as well. This was purely an error on my part. This brings up an interesting issue of WebStorage: https://www.w3schools.com/html/html5_webstorage.asp

the maximum storage size for the session is 5MB

From here I will attempt to implement this answer here: How do you cache an image in Javascript to my current code and will update this answer with the result for anyone else facing this issue.

Lachlan Young
  • 1,266
  • 13
  • 15
  • you should be able to achieve it with https://stuk.github.io/jszip/ - download them, zip them and then send to user - all from his browser :) – Michal Hatak Aug 30 '17 at 00:52
  • Hey @MichalHatak thanks for the quick response, I saw that but that focuses more on Option A. In fact, that was exactly the approach I was planning to take with Option A however I am really hoping to find some help with Option B at the moment. Do you know if it is even possible? Thanks! – Lachlan Young Aug 30 '17 at 01:01
  • @LachlanYoung Jszip works in the web browser... – Darkrum Aug 30 '17 at 01:23
  • I wasn't aware NPM Modules could be used in the browser? Only on the Node Server? – Lachlan Young Aug 30 '17 at 02:00
  • @LachlanYoung NPM can host any type of project. You need to work on your documentation reading skills. – Darkrum Aug 30 '17 at 02:37
  • Sorry @Darkrum I seem to be really misunderstanding you? I'm not looking to host the project on NPM, i have written the project in NodeJS and Express which currently exists on the server. I am trying to find a way to download the files to the browserCache, Zip them, and then send them to the download directory in a .zip format. Rather then flooding the downloads folder with hundreds of images. Jszip is an Npm module, thus can it not only be utilised on the Node Server and not the browser cache of the client? – Lachlan Young Aug 30 '17 at 03:13
  • @Darkrum I see what you mean, I was missing that it could be implemented on the clientSide without NPM. Thankyou for youre help, I'll update this question with my Progress – Lachlan Young Aug 30 '17 at 03:20
  • @LachlanYoung hey, have you find the solution? I have similar webapp and I want to do the same thing. I saw a website that can do that, and have been searching to achieve this but did'nt find any answer. – Ujang Karnadi Apr 12 '19 at 15:23

0 Answers0