0

I have a python script that queries some data from several web APIs and after some processing writes it to MySQL. This process must be repeated every 10 seconds. The data needs to be available to Google Compute instances that read MySQL and perform CPU-intensive work.

For this workflow I thought about using GCloud SQL and running GAppEngine to query the data. NOTE: The python script does not run on GAE directly (imports pandas, scipy) but should run on a properly setup App Engine Managed VM.

Finally the question: is it possible and would it be reasonable to schedule a cron job on a GApp Managed VM to run a command invoking my data collection script every 10 seconds? Any alternatives to this approach?

qcqp
  • 135
  • 5

2 Answers2

1

The finest resolution of a cron job is 1 minute, so you cannot run a cron job once every 10 seconds.

In your place, I'd run a Python script that starts a new thread every 10 seconds to do your MySQL work, accompanied by a cronjob that runs every minute. If the cronjob finds that the Python script is not running, it would restart it.
(i.e., the crontab line would look like * * * * * /command/to/restart/Python/script).

Worse-case scenario you'd miss 5 runnings of your MySQL worker threads (a 50 seconds' duration).

boardrider
  • 5,882
  • 7
  • 49
  • 86
0

If its Managed VM, you're able to use Task Queue as well as Cron Jobs. If you have a handler to run your script -- it will work.

If you would like to use internal cron, you should look on how to run cron job inside of docker environment.

Community
  • 1
  • 1
Dmytro Sadovnychyi
  • 6,171
  • 5
  • 33
  • 60
  • Thanks for the Docker-cron pointer: can't use it with 10 sec but will be needed for running the type of script @user1656850 suggested. – qcqp Mar 14 '15 at 19:06