0

I have a Node.js application which makes a call to the pipe() function of a readableStream. I have to be able to wait for pipe() to complete before the calling function can return however.

var copyFile = function(paramSource, paramDestination, paramCallback) {
  var sourceFile = fs.createReadStream(paramSource);
  var destinationFile = fs.createWriteStream(paramDestination);
  sourceFile.on("error", function(error) {
    return paramCallback(error);
  });
  destinationFile.on("error", function(error) {
    return paramCallback(error);
  });
  destinationFile.on("close", function() {
    return paramCallback(null, paramDestination);
  });
  sourceFile.pipe(destinationFile);
};

This function is called from within other functions which can trigger multiple cases of copyFile running at once (which is not desirable in this case, even though paramCallback would imply it). The resulting requirement is that I need to prevent this function from returning until destinationFile emits a close event. Currently the function returns after pipe() is called as it's at the end of the function and any combination of while loops and boolean flags to wait at the end for completion just hangs the function.

EDIT

Here is the same code with a simple call to the function and some text sent to the console to follow the programs flow.

var copyFile = function(paramSource, paramDestination, paramCallback) {
  var sourceFile = fs.createReadStream(paramSource);
  var destinationFile = fs.createWriteStream(paramDestination);
  sourceFile.on("error", function(error) {
    return paramCallback(error);
  });
  destinationFile.on("error", function(error) {
    return paramCallback(error);
  });
  destinationFile.on("close", function() {
    console.log("close event has been emitted");
    return paramCallback(null, paramDestination);
  });
  console.log("pipe is being called");
  sourceFile.pipe(destinationFile);
  console.log("pipe has been called");
};

copyFile("/home/fatalkeystroke/testing/file1.txt", "/home/fatalkeystroke/testing/file2.txt", function(error, name) {
  if (!error) {
    console.log("callback has been called");
  }
});
console.log("copyFile has been called");

This produces the output:

pipe is being called 
pipe has been called 
copyFile has been called 
close event has been emitted 
callback has been called

What I want is:

pipe is being called
pipe has been called
close event has been emitted
callback has been called
copyFile has been called
FatalKeystroke
  • 2,882
  • 7
  • 23
  • 35
  • Why would `while` loop be necessary? `.pipe()` returns a `Promise`, yes? – guest271314 Jun 06 '17 at 02:47
  • According to the documentation, no. `.pipe()` does not return anything, it just triggers events to be emitted throughout it's process. – FatalKeystroke Jun 06 '17 at 02:50
  • _"it just triggers events to be emitted throughout it's process."_ Why do you not utilize the event handlers described at documentation? – guest271314 Jun 06 '17 at 02:51
  • That other question was too broad and I've narrowed down where the fault is, and it's directly relating to pipes return behavior. The other question people were suggesting promises as well, but pipe causes the function to exit regardless, which is where the issue is coming up – FatalKeystroke Jun 06 '17 at 02:52
  • I do. they're right there in the example. – FatalKeystroke Jun 06 '17 at 02:53
  • I'm with him on not understanding why you're using a while loop to wait. Are you under the impression that you have to poll yourself? – tsturzl Jun 06 '17 at 02:54
  • I'm not using a while loop, I was just stating that I've tried that and it didnt work. I need to wait for the close event to be emmited before returning so I tried a while loop waiting for the close event but it hung. – FatalKeystroke Jun 06 '17 at 02:55
  • I have read the stream documentation in nodes docs – FatalKeystroke Jun 06 '17 at 02:56
  • _"I need to wait for the close event to be emmited"_ Then what is issue? Are you trying to convey that the `close` event is not dispatched? – guest271314 Jun 06 '17 at 02:57
  • Oh well that wouldn't make sense, you'd just create a bunch of listeners on the event emitter. So your callback is never getting called on this method? – tsturzl Jun 06 '17 at 02:57
  • pipe() is the last line of the function. pipe() continues ont the next line before it does it's actions and emits events. this causes the function to return by default before I can act on the emitted event and call the callback properly. – FatalKeystroke Jun 06 '17 at 02:59
  • `.on` creates a listener with a callback to fire on an event, its not a method to poll an event. In fact there is no idomatic way to poll an event emitter in nodejs. – tsturzl Jun 06 '17 at 02:59
  • What do you mean by "this causes the function to return by default? Why are you trying to `return` a value from a function call from within an event handler? – guest271314 Jun 06 '17 at 03:01
  • it reaches the end of the function because it's the last line. So the function returns without having called the callback, which has to be called after the close event is emitted – FatalKeystroke Jun 06 '17 at 03:02
  • 1
    That isn't true. If I wrote a function that called another function, does that mean the stackframe for the function below it is removed? No. You're method isn't returning. Even if it did remove your stackframe you're stream is allocated on heap. – tsturzl Jun 06 '17 at 03:02
  • Read the [Streams Standard](https://streams.spec.whatwg.org/) – guest271314 Jun 06 '17 at 03:03
  • I don't think he needs to read the stream standard, I think he just needs to better understand idiomatic event driver programming and overall concurrency concepts. – tsturzl Jun 06 '17 at 03:04
  • @tsturzl Ok. Perhaps post an Answer detailing what you are describing at comments? Have not actually tried `nodejs` streams, only streams documented at the specification; you are probably best suited to provide an Answer given the context is `nodejs`. Not entirely certain if `nodejs` streams follow the specification. – guest271314 Jun 06 '17 at 03:05
  • That was actually a really poor explanation, and really over simplifies javascript memory management, but overall being a memory safe language, the parameter `paramCallback` will not be deallocated because that would cause a null pointer issue when your callback fires. – tsturzl Jun 06 '17 at 03:07
  • I don't think there is really an answer to be given. I think the flaw is more fundamental. If you're polling a stream for an emitted event, then I'm not sure he understands callbacks. It almost seems to me like he's expecting the method to `return` a result rather than provide it by a callback. – tsturzl Jun 06 '17 at 03:12
  • @tsturzl The stream standard uses `Promise` object to handle `.pipe()`. `nodejs` apparently does not use a `Promise`-based approach. Though there is [web-streams-polyfill](https://github.com/creatorrr/web-streams-polyfill), which, again, references the streams standard – guest271314 Jun 06 '17 at 03:12
  • 1
    No, but you can subscribe as many listeners as you want(within reason), so I honestly don't see any reason his above method isn't working. I really did think they added that to the nodejs standard, because they usually just piggy back off other standards anyway, apparently not. Would be nice if they finally starts pushing promises in the core modules. – tsturzl Jun 06 '17 at 03:15
  • @FatalKeystroke give me an example of how you invoke that function. – tsturzl Jun 06 '17 at 03:17
  • I'm putting in an example of the current programs flow and what I expect it to do (which is different). That should help clarify a lot more – FatalKeystroke Jun 06 '17 at 03:18
  • @FatalKeystroke You still have not clarified whether or not `paramCallback()` is called within `close` event? And if you are expecting a value to be returned from `copyFile()` function? – guest271314 Jun 06 '17 at 03:20
  • @guest271314 Wait, what? Line 11 he calls the callback from within the close event of the write stream. – tsturzl Jun 06 '17 at 03:23
  • @tsturzl Yes. There should not be an issue at `javascript` at Question unless OP is expecting that the `return` statement will return a value from `copyFile()` call [How do I return the response from an asynchronous call?](https://stackoverflow.com/questions/14220321/how-do-i-return-the-response-from-an-asynchronous-call/), from within an event handler – guest271314 Jun 06 '17 at 03:25
  • @FatalKeystroke What's wrong with the result you're getting? Why would you want the callback called before the pipe has completed? – tsturzl Jun 06 '17 at 03:28
  • I don't want to return any values from copyFile, just pass some into the callback function and trigger the callback. But I have to prevent the function from exiting until its ready to call callback – FatalKeystroke Jun 06 '17 at 03:29
  • @FatalKeystroke Don't worry about it, this is working. You're callback is calling after the pipe has finished. – tsturzl Jun 06 '17 at 03:30
  • I don;t want it called before the pipe is completed. I want the code after the copyFile call to not run until callback has been called. – FatalKeystroke Jun 06 '17 at 03:30
  • _"just pass some into the callback function and trigger the callback. But I have to prevent the function from exiting until its ready to call callback"_ What do you mean by "prevent the function from exiting"? Which function are you referencing? _"I want the code after the copyFile call to not run until callback has been called."_ What do you mean by "after the copyFile" call? What is not occurring in the sequence that you are expecting? Note, `copyFile` does not return a value. – guest271314 Jun 06 '17 at 03:30
  • 1
    @FatalKeystroke Ok ok ok, you're expecting this to work synchronously. That's not how this works, and probably not what you actually want. – tsturzl Jun 06 '17 at 03:32
  • Whats happening is the copy file stream is running, and polled later in the eventloop after your last `console.log` – tsturzl Jun 06 '17 at 03:33
  • There is nothing wrong with this, this is how concurrent applications behave. – tsturzl Jun 06 '17 at 03:33
  • @FatalKeystroke If you want logic to perform after the stream is finished you need to invoke that logic in your callback you pass to that function. – tsturzl Jun 06 '17 at 03:34
  • in other words put that console.log in your callback that you pass to your method. – tsturzl Jun 06 '17 at 03:35
  • @guest271314 is right, this is a duplicate technically. This is a pretty common misunderstanding with nodejs and new comers to event driven programming. You might want to consult some some [guides](https://blog.risingstack.com/node-hero-async-programming-in-node-js/), books, or courses on the matter. They will better explain the paradigms than someone could reasonably do on SO. – tsturzl Jun 06 '17 at 03:43

0 Answers0