Maximum parallel pipes to fs

Question

i am quite new to the node.js world, but i wanted to give it a try.

I am trying to set up an Server with Express which exposes an public API.

my code is pretty much set up, but i got an problem where i don't know why it is happening.

this is what it does:

check if the passed URL is online.
if the target is an Image, download it locally
after the download finished, make a thumbnail from the source

.

function requestPromise(url, path) {
    return new Promise(function(fulfill, reject) {
        request(url, function(err, httpResponse, body) {
            if(err) {
                reject(err);
            } else {
                fulfill(path);
            }
        }).pipe(
            fs.createWriteStream(path)
        );
    });
}

function imageResizeToThumbnail(path) {
    return new Promise(function(fulfill, reject) {
        gm(path)
        .resize(300, 300, ">")
        .write(path, function(err) {
            if (err) return reject(err);
            fulfill(path);
        });
    });
}

function fetchImage(url) {
    return new Promise(function(fulfill, reject) {
        // creates a filesystem friendly string
        var path = './download/' + sanitize(url, {replacement: '_'});

        requestPromise(url, path).then(function(path) {
            return imageResizeToThumbnail(path);
        }).then(function(path) {
            fulfill(path);
        }).catch(function(err) {
            reject(err);
        });
    });
}

following test works perfectly, the image gets downloaded and after it is downloaded it gets shrunk down

var test1 = 'https://s.ytimg.com/yts/img/favicon_144-vflWmzoXw.png';

fetchImage(test1).then(function(response) {
    console.log(response, ' worked!');
});

then i've tried with a couple images at once, no problem at all

var test2 = [
    'https://s.ytimg.com/yts/img/favicon_144-vflWmzoXw.png',
    'https://s.ytimg.com/yts/img/favicon_144-vflWmzoXw.png',
    'https://s.ytimg.com/yts/img/favicon_144-vflWmzoXw.png',
    'https://s.ytimg.com/yts/img/favicon_144-vflWmzoXw.png',
    'https://s.ytimg.com/yts/img/favicon_144-vflWmzoXw.png',
    'https://s.ytimg.com/yts/img/favicon_144-vflWmzoXw.png',
    'https://s.ytimg.com/yts/img/favicon_144-vflWmzoXw.png',
    'https://s.ytimg.com/yts/img/favicon_144-vflWmzoXw.png',
    'https://s.ytimg.com/yts/img/favicon_144-vflWmzoXw.png'
]

Promise.all(test2.map(function(item) {
    return fetchImage(item).then(function(response) {
        console.log(response, ' worked!');
    });
})).then(function(result) {
    console.log('all Promises done!');
}).catch(function(err) {
    console.log('error: ', err)
});

then i took the same example as above and run it with 1K items!

the result was: imageResizeToThumbnail was getting an error, because the image wasn't downloaded yet... the file was created, but not finished (filesize was 0bytes)

so my question is... is there a limitation of how many Filestreams can be instantiated or piped to fs?

There are always operating system limits on file handles, but in your case there seems to be a concurrency issue that only appears under heavy load. There is an [old question](http://stackoverflow.com/questions/11447872/callback-to-handle-completion-of-pipe) that might give a clue. — Kenney, Jul 29 '15 at 18:15
@Kenney: That's it! You should've posted it as answer (if you want, I can reopen the question) — Bergi, Jul 29 '15 at 18:46
Some other points about your usage of promises: in `imageResizeToThumbnail` you should call `reject` when there's an `err`, in `fetchImage` you're using the [promise constructor antipattern](http://stackoverflow.com/q/23803743/1048572), and your `test2.map` callback misses a `return` statement. — Bergi, Jul 29 '15 at 18:49
after a lot of headache, i got it to work! first of all, i made a bad `request`. when piping requests, you need to listen for the `close` event - emitted by the pipe itself. but it wasn't working even after that little fix. So i tried downgrading node from 0.12.x to 0.10.x and now it works flawlessly! Anybody having the same problem with pipes not finishing should consider downgrading before getting frustrated :) — ChiefORZ, Aug 10 '15 at 13:33

Maximum parallel pipes to fs

0 Answers0