I'm looking for some advice to run intensive jobs on demand in somewhere like AWS or Digital Ocean.
Here's my scenario:
- I have a template/configuration of a VM with dependencies (imagemagick, ruby, python etc)
- I have a codebase that runs a job, eg: querying a db and running reports, then emailing those reports to my user base
- I want to be able to run and trigger this job externally (i.e maybe via some webapp somewhere else, or via a command line somewhere - maybe some cron on another cloud instance somewhere)
- When I run this job, it needs to spin up a copy of this template on AWS or DO, run the job, which could run for any length of time, until all reports are generated and sent out
- Once the job has finished, shutdown the instance so I'm not paying for something to always be running in the background
- I'd like to not have to commit to one service (i.e AWS) but rather have a template that can be dropped in anywhere to test out the differences in cloud providers
Initially, I was thinking rubber but this seems more something you'd use for CI, rather than being able to spin up an instance, run a long running job, then shut down the instance once finished.
Does anything already exist for this, or would I need to build something myself hooking into the relevant APIs?
