How to save segements of image in memory from server request for merging later

Question

I have a nodejs application that calls a server which returns a fragment of an image. I am trying to save this segment in a buffer so that I can later merge them together to make a full image

var req = http.request(options, function(res) {
        var segment = parseInt(res.headers["segment"]);
        res.setEncoding('binary');
        var data = "";

        res.on('data', function(chunk) {
            data += chunk;
        });

        res.on('end', function() {
            images[segment] = (new Buffer(data, 'binary'));
        });
    });

after I get all the segments

var totalImage = "";
for (image in images) {
    totalImage += image.toString('binary');
}
fs.writeFile('image.png', totalImage, function(e) {
    console.log('done');
});

I want to use node-pngjs but I am not sure how to stream the response binary into png so that I can save the pixel buffer instead of the binary itself for later consumption

I attempted to do the following:

res.pipe(new PNG()).on('parse', function(err, data) {
    buffers[segment] = data;
});

but this lead to an error 'Invalid file signature' during parse

I think you will need to show more of your code including how you manage the `buffers` variable, the index `i` and how you write all the data to a file. Right now, you aren't showing enough of your code for us to understand how you collect each part and how you write all the parts out to a file. — jfriend00, Feb 01 '15 at 19:09
If I were you, I'd add a `console.log(segment)` to your request handler to make absolutely sure that the segment numbers are coming in appropriately in your headers. Then, you need to see if the response is actually binary (not encoded some other way) which you can probably do by looking at the response and headers in the network tag of the Chrome debugger. — jfriend00, Feb 01 '15 at 20:26

score 0 · Accepted Answer · edited May 23 '17 at 12:32

There may well be more than one issue here, but this code is not correct:

for (image in images) {
    totalImage += image.toString('binary');
}

In that code, image is the index into the array (not the contents of the cell of the array) so all you're doing is add up the indexes 0+1+2+3+4 and so on.

Arrays should never for iterated with the for/in construct because that iterates all properties of the object, not just array elements. And, you weren't fetching the actual contents of the array cell which would have been images[image] either. Instead, you could use:

for (var i = 0; i < images.length; i++) {
    totalImage += images[i].toString('binary');
}

P.S. I suspect that you may have encoding issues also. You take the data from the http request, but it all into an array of Buffer objects, then you convert those to a "binary" string and add them together into a string variable. Since this is binary data, it really seems like you should only be using a Buffer to collect stuff, not a string.

After looking into Buffers in nodejs a little more, you can combine your Array of buffers like this:

var totalImage = Buffer.concat(images);

This keeps everything in binary buffer format and wouldn't have any encoding issues. Found this here and here.

How to save segements of image in memory from server request for merging later

1 Answers1