Non blocking functions PHP

Question

I have a project in which a user uploads an image through a form and the server does some thumbnails. The thumbnail making process is very slow so I thought that doing the image resizing with a non-blocking function could be a good solution. I mean: the server process the form (wich have more fields) gives the "ok" feedback to the user and then calls the thumbnailing function. How can I do that?

Thanks in advance

Answer 1

Your best option would be to implement Gearman. It's a Job Queue system where you can implement either synchronous of asynchronous jobs. http://gearman.org/

Answer 2

Better solution I usually go for: Create the thumbnails dynamically when needed, not upon upload.

You create a script that generates thumbnails on the fly, and all your image tags point to this script:

<img src="/thumbnail.php?image=foobar.jpg&size=150" />

This delays the thumbnail generation until it is needed and works "asynchronously". With some .htaccess rewrite magic you can even make it look like a normal image file and cache images in a way that the Apache server will serve them the next time without invoking the script.

---

To be a little more detailed, I use this for user profile images:

Image tags:

<img src="/img/users/123456/50.jpg" />

.htaccess:

<IfModule mod_rewrite.c>
    RewriteEngine On

    # Rewrites requests for user images to match directory structure.
    # E.g.: URL /img/users/123456/50.jpg -> /img/users/123/123456/50.jpg
    # Intermediate directory level is introduced to avoid cramming too many directories into the same directory.
    RewriteRule ^img/users/(\d{1,3})(\d*)/(\d+\.\D+)$ img/users/$1/$1$2/$3 [nocase,last]

    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteRule ^(.*)$ index.php?url=$1 [QSA,L]

</IfModule>

This first of all rewrites image requests to a deeper directory structure. If the image exists, Apache will serve it as usual. If it doesn't, my regular application is invoked. In the app, I route /img/users/... URLs to a module that ends up with two pieces of information: the user id 123456 and the requested size 50. It then generates a thumbnail roughly according to this logic:

1. Find profile image for user 123456
2. Generate thumbnail in requested size
3. Write thumbnail to /img/users/123/123456/50.jpg, where it will be picked up by Apache next time
4. Output image

Answer 3

You could have a cronjob that executes the thumbnailing script. You could add the image to be resized in some sort of queue (mysql database perhaps) and the thumbnailing script runs every minute to check if there is something in the que and then starts resizing.

Answer 4

On one system, I've seen an independent background process making the thumbnails:

• form is processed normally, without generating any thumbnail at all
• the image is given an unique name and copied to a special folder.
• a database entry is created in a thumbnails table, linking the original image and the new unique name, marked as "to be thumbnailed"
• the form processing script stops to care and continues with whatever else it needs to do.

There's an independent background process (and its watchdog), which continuously watches that special folder (most OSes have various tools that notify you when a folder's contents change); if it finds an image there, it will:

• make a thumbnail (we were using ImageMagick's CLI for that)
• save it somewhere else
• update the database, set the image status as "thumbnailed OK" (or "failed", if it couldn't make one)

When you need the thumbnail, check the thumbnails table - if the image is not "thumbnailed OK", show a placeholder, else get the correct thumbnail name and display this.

That worked great - most thumbnails were created within a few seconds, without slowing down the user-facing scripts. Note that you'll need to start the background script somehow - in this case, there was a watchdog in cron, which restarted the thumbnailing script if it died.

---

@yankee objects that some elements are uncommon:

• it is not necessary for the thumbnailer process to run as a background script - if you can live with a minute of latency before you get the thumbnails, you could run it as a cron script, getting rid of the watchdog altogether.
• ImageMagick was chosen over GD for specific performance reasons; the thumbnailer could use whatever method is available.

Edit: I checked the site, and there is one more mechanism - this one is not necessary and adds a bit of load, but looks cool, especially if you don't expect full page loads very often (e.g. on AJAX-driven sites):

• where a "not-failed-but-no-thumbnail" placeholder is output, the thumbnail is shown in an img with class="nothumb"
• a JS function checks for images with this class
• if any are found, it will periodically check if a thumbnail is available yet
• the static placeholders are replaced with "loading" placeholders
• if found, it will replace the placeholder with the thumbnail

This loads the thumbnails as soon as they are ready, at the cost of some resources. For a continuous background process, it's not really needed; but if you want to ensure that users will see the thumbnails as they become available instead of on their next pageload, this is an useful addition.

Answer 5

You can use http headers to tell the client that the output has ended after the "OK" message has been transfered, while keeping the script running on the server to process the thumbnails. I've once successfully used this code:

header("Connection: close");
@ob_end_clean();
ignore_user_abort();
ob_start();

//generate and print server response here
echo "everything OK";

$size = ob_get_length();
header("Content-Length: ".$size);
ob_end_flush();
flush();

//whatever you do here has no influence on the page loading time, as the client has already closed its connection.
generateThumbnail();

Answer 6

Suggestions:

1. Close your connection before script terminates as outlined here: http://www.php.net/manual/en/features.connection-handling.php#71172

2. Generate output in "real time" as explained here: How to echo output in real time, (before script finishes)? (won't work in all environments and the user will still get a loading bar until thumbnails finished to generate)

3. Start another process that creates the thumbnails for you. If you have the according privileges on the server use system() for that. If you don't have them, create another php script on your server which you call using URL and sockets. Then you can terminate the connection early and keep the script running to generate the thumbnails. Use ignore_user_abort() to stop the thumbnail generation to abort once you abort the tcp connection opened with your socket.

Answer 7

There is actually a way, If you send back the response headers and the response content you can actually keep the server thread going with your processing without keeping the client going. So you can send the headers back with the content type etc.. or a header redirect etc.. the browser will read the response. But that does not mean the server thread stops.,

You can use the connection status, the ignore user abort and keep the same thread going in the server here you can see this explained.

sample from the link provided:

<?php
ob_end_clean();
header("Connection: close\r\n");    
header("Content-Encoding: none\r\n");
ignore_user_abort(true); // optional
ob_start();
echo ('Text user will see');
$size = ob_get_length();
header("Content-Length: $size");
ob_end_flush();     // Strange behaviour, will not work
flush();            // Unless both are called !
ob_end_clean();

//do processing here
sleep(5);

echo('Text user will never see');
//do some processing
?>