I am looking at using a static generator to generate up to hundreds of thousands of pages (on S3) of data-driven content from json or csv files, each of which has an html form that posts to an external API. Is this a feasible undertaking?
Asked
Active
Viewed 553 times
1 Answers
3
It depends on your requirements, but at minimum, you might even get away with a simple node program that uses fs to read/write. Going up the complexity spectrum, you might do with a Gulp setup. Going even further up the spectrum, you can use static website generator to read/write your data files (but that's probably worth the trouble only if you already know static generator and/or you will want to have a blog on S3 as well, driven by .MD files, besides hundreds of thousands of data-driven pages).
If going simple node script route, you would create your local application in a js file, run it through command line in node. It would generate thousands of pages locally, then you would upload them to S3. You can either use standard fallbacks or fancier way using promises (like using Bluebird). This way is the most manual but you have the most control over the result.
For the record, you could whip up a script in any programming language that you are proficient, like for example, PHP. JavaScript is popular these days, that's why I'm assuming you would use JS.
If going Gulp route, I imagine a custom function that reads data files from the location, parses their contents into an array, and writes the contents into files.
If going Hugo route, simply use data driven content reference, getCSV function. You'll still need to work in the context of a website, this means the more you stray off the website's setup, the more you'll have to fight Hugo.
As I mentioned, arguments against static website generator would be if you don't need the website part, only to perform operations on data and write files, it might stand in a way.
Hugo is a good option for thousands of files because it's fast.
Solution also depends on if your CSV files are going to change, or is it one-off thing; also how much automation you need. Gulp approach might be handy even if you go Hugo route.
So, yes, it is a very feasible undertaking.
