0

I am currently developing a website that has to perform a task every 10 seconds. The website checks a database every ten seconds, finds rows that are to be processed, and then run a short script on each row.

The service is working great right now, but I just want to make sure that once the database gets very large, the service will still work efficiently.

I am using a cron job that runs every minute, with sleep functions every 10 seconds.

Thank you for your help!

user3647894
  • 559
  • 1
  • 4
  • 28
  • So... What is your question? – Noah May 20 '14 at 19:04
  • How much data are we talking about here? 10s is too short if you're script is not fast and optimized. I suggest that you look into DB triggers. – Zuzlx May 20 '14 at 19:04
  • How does the function know that it should die after a certain amount of time has elapsed? Did you hard code in a certain number of iterations? Is it possible that the code can be running twice, at around the 60 second mark because two processes are running simultaneously? – Lynn Crumbling May 20 '14 at 19:05
  • Obviously I'm not familiar with your application, but querying the database every 10 seconds seems very excessive. Why so often? Why wouldn't 60 seconds or 5 minutes be more reasonable? Does the change in the db really need to be picked up that quickly? What happens when your script notices a change? Does that code fire another function that does more intensive work? – Lynn Crumbling May 20 '14 at 19:06
  • well database entries must be accurate to 10 seconds...if i run it every minute, then an entry that is supposed to delete in 20 seconds, will really delete in one minute – user3647894 May 20 '14 at 20:56

1 Answers1

0

Just make sure your tasks don't overlap, i.e. wait till the previous task is over before launching a new one. Otherwise, if your tasks take more than a minute (or whatever timer you've set for them), you'll end up with tons of those running in parallel and your system will run out of available resources.

Oleg
  • 9,341
  • 2
  • 43
  • 58
  • And how do you suggest waiting? All i have is a cron job thats running a PHP file. Would the wait be something in the PHP or the cron job itself? – user3647894 May 20 '14 at 20:57
  • @user3647894 you might be interested in http://stackoverflow.com/questions/10552016/how-to-prevent-the-cron-job-execution-if-it-is-already-running – Oleg May 21 '14 at 07:29