Node.js thumbnailer using Imagemagick: nondeterministic corruption

Question

I have a Node.js server which dynamically generates and serves small (200x200) thumbnails from images (640x640) in a database (mongodb). I'm using the node-imagemagick module for thumbnailing.

My code works roughly 95% of the time; about 1 in 20 (or fewer) thumbnailed images are corrupt on the client (iOS), which reports:

JPEG Corrupt JPEG data: premature end of data segment

For the corrupt images, the client displays the top 50% - 75% of the image, and the remainder is truncated.

The behavior is non-deterministic and the specific images which are corrupt changes on a per-request basis.

I'm using the following code to resize the image and output the thumbnail:

im.resize({
  srcData: image.imageData.buffer,
  width: opt_width,
}, function(err, stdout) {
  var responseHeaders = {};
  responseHeaders['content-type'] = 'image/jpeg';
  responseHeaders['content-length'] = stdout.length;
  debug('Writing ', stdout.length, ' bytes.');
  response.writeHead(200, responseHeaders);
  response.write(stdout, 'binary');
  response.end();
});

What could be wrong, here?

Notes:

The problem is not an incorrect content-length header. When I omit the header, the result is the same.
When I do not resize the image, the full-size image always seems to be fine.
In researching this I found this and this StackOverflow questions, which both solved the problem by increasing the buffer size. In my case the images are very small, so this seems unlikely to be responsible.
I was originally assigning stdout to a new Buffer(stdout, 'binary') and writing that. Removing it ('binary' will be deprecated) made no difference.

Few questions: What's the node.js environment? Are the images corrupt in the db, or only on iOS? Can you create a test method that you can drive from a browser and output the resized image (to disk or browser) before db insertion? — k00k, Nov 01 '12 at 12:41
This happens consistently in a few different environments with various "recent" versions of Node, ImageMagick and mongodb. One is Node.js version v0.9.1-pre (OS X) and another is 0.8.6 (Ubuntu). I'm reasonably sure the source image is ok, since if I don't run it through ImageMagick it always displays fine in the iOS client. — mjh, Nov 01 '12 at 17:07
The plot thickens... when I cache thumbnails (in Node) so I can compare outputs when the identical source image is resized, there are sometimes radical differences (which would line up with client corruption) in thumb size. I'm beginning to suspect that the buffers being sent to IM are somehow getting borked by concurrent requests. — mjh, Nov 01 '12 at 18:05
Yes, concurrent reqs was my next thought. Does it work just fine if you take your time and do one request, let it finish, then do another? Is it a concurrency problem? — k00k, Nov 01 '12 at 18:08
Nope, that wasn't it either; I now copy the buffer before giving it to IM and get the same result. I'm also writing all the thumbs to disk; when I run 'identify' on them there are no problems reported. (But that's because identify only reads the header...) @k00k, I'll try your suggestion and limit requests to one at a time. — mjh, Nov 01 '12 at 18:24
@k00k, I modified my Node server to queue requests and only handle one at a time (a fun bit of code), and there is no more corruption! I was using "node-imagemagick" version 0.1.2. Found the solution: upgraded "node-imagemagick" to 0.1.3 and the problem seems to have gone away. Thank you for prompting me through debugging this. :) — mjh, Nov 02 '12 at 06:48

score 1 · Accepted Answer · answered Nov 02 '12 at 07:07

The problem seems to have been due to a slightly older version of node-imagemagick (0.1.2); upgrading to 0.1.3 was the solution.

In case this is helpful to anyone, here's the code I used to make Node.js queue up and handle client requests one at a time.

// Set up your server like normal.
http.createServer(handleRequest);
// ...

var requestQueue = [];
var isHandlingRequest = false;  // Prevent new requests from being handled.

// If you have any endpoints that don't always call response.end(), add them here.
var urlsToHandleConcurrently = {
  '/someCometStyleThingy': true
};

function handleRequest(req, res) {
  if (req.url in urlsToHandleConcurrently) {
    handleQueuedRequest(req, res);
    return;
  }
  requestQueue.push([req, res]);  // Enqueue new requests.
  processRequestQueue();          // Check if a request in the queue can be handled.
}

function processRequestQueue() {
  // Continue if no requests are being processed and the queue is not empty.
  if (isHandlingRequest) return;
  if (requestQueue.length == 0) return;

  var op = requestQueue.shift();

  var req = op[0], res = op[1];

  // Wrap .end() on the http.ServerRequest instance to
  // unblock and process the next queued item.
  res.oldEnd = res.end;
  res.end = function(data) {
    res.oldEnd(data);
    isHandlingRequest = false;
    processRequestQueue();
  };

  // Start handling the request, while blocking the queue until res.end() is called.
  isHandlingRequest = true;
  handleQueuedRequest(req, res);
}

function handleQueuedRequest(req, res) {
  // Your regular request handling code here...
}

Node.js thumbnailer using Imagemagick: nondeterministic corruption

1 Answers1