1

I have a web-app, written in PHP, and I am using Symfony2 as the core framework.

I need to regularly run thousands of small network requests every 10 minutes or so and I am trying to find the best solution for asynchronously running these jobs, without conflicting or doubling up.

Currently I have a very basic and inelegant solution where a cron job executes a PHP command script. The command synchronously works through each entry in the database and sends a network request. When that request completes (or fails), it moves on to the next one. When it has iterated over all entries, it exists, to be executed again from the cron job.

For the rewrite, I have looked at php-resque and pcntl_fork as solutions for running jobs in parallel, thereby speeding up the execution significantly. I have also looked at running multiple non-blocking socket requests from PHP but, so far, have preferred the simplicity of isolated jobs.

KingCrunch
  • 128,817
  • 21
  • 151
  • 173
Ryall
  • 12,010
  • 11
  • 53
  • 77

2 Answers2

1

PHP can't do threading in the traditional sense, so what you're trying to do isn't really possible in the way that you're thinking. You've looked at the best solution (probably) in pcntl_fork, but that still won't be truly asynchronous. Have you considered using cron to accomplish this instead?

Madbreaks
  • 19,094
  • 7
  • 58
  • 72
  • I'm currently using cron to execute a PHP command, which then iterates through the database and works on each entry. – Ryall Jan 09 '13 at 17:56
  • Would `pcntl_fork` still be my best option? – Ryall Jan 09 '13 at 18:04
  • It depends on the details of what you're trying to do. You could also delegate the network operations to some other process via a system call, for example let Perl handle the network ops. – Madbreaks Jan 09 '13 at 18:05
  • I was considering that (writing a small C++/Java helper) but as I have limited time on this job, I would prefer to keep it in PHP if possible. – Ryall Jan 09 '13 at 18:09
  • Then yeah, look to `pcntl_fork` with the understanding that it's not *really* doing async. – Madbreaks Jan 09 '13 at 18:12
  • That's fine, so long as I can open several requests at once without having to worry about polling non-blocking streams then I'm happy. Thanks Madbreaks! – Ryall Jan 09 '13 at 18:18
  • In fact, I might still just go ahead with the non-blocking streams. That would have a much smaller memory footprint over forking all the time, I believe. – Ryall Jan 09 '13 at 18:25
1

http://php.net/pthreads

http://github.com/krakjoe/pthreads

Examples on github... and included in distribution, use code in github, it contains fixes not yet released.

Joe Watkins
  • 17,032
  • 5
  • 41
  • 62
  • Very interesting, this solves all my problems. Reading the documentation, it mentions that the library is still experimental. I'm assuming you are the author so how stable would you say it currently is? – Ryall Jan 10 '13 at 13:27
  • Also, how would I install the code on Github? I've never done any compilation of PHP modules before. – Ryall Jan 10 '13 at 13:27
  • http://pthreads.org/Building - this should get you started on building. There's some windows snapshots, sometimes at http://pthreads.org/snapshots. And windows dlls on pecl for releases http://windows.php.net/downloads/pecl/releases/pthreads/ ... it's remarkably stable, whatever I say is biased, get stuck in is my best advice :) – Joe Watkins Jan 10 '13 at 19:26
  • Thanks Joe, I'm currently using the Multiprocessing method but I would prefer threads so I'll give it a go. :) – Ryall Jan 11 '13 at 10:26