2

I am writing a simple json database with transaction rollbacks. I need to append a line of text to a file and then log success or failure to another file depending on if my append was successful or not. The second file is used for rollbacks if required. So I need to know for sure that the write was successful before proceeding.

I am using stream.write to append my line of text which includes a callback that is supposed to verify success or failure of the write operation.

Then I read this unfortunate news in the NodeJS docs at the following URL https://nodejs.org/api/stream.html#stream_class_stream_writable

The writable.write() method writes some data to the stream, and calls the supplied callback once the data has been fully handled. If an error occurs, the callback may or may not be called with the error as its first argument. To reliably detect write errors, add a listener for the 'error' event.

So while the following code seems to work, the docs say it will not be reliable.

  // Wait till the file is open before trying to write to it.
  wStream.on('open', function() 
  {
    // Call function to append the string to the file with a new line character added at the end.
    wStream.write(stringData + '\n', function(error)
    {
      if(!error) // If the string was appended successfully:
      {
        callback(false); // Report back there was no error
      }
      else // The string was not appended successfully.
      {
        lib.log
        (
          'h0fceuftq8xkdkvh4dl9' + '\n' +
          'Error appending to file ' + fileName + '\n' +
          'This was the error:' + '\n' +
          JSON.stringify(error) + '\n' + 
          '\n'
        );

        callback('Error appending to file ' + fileName)
      } // End of: else The string was not appended successfully.

    }); //End of: wStream.write(...
    // End of: Call function to append the string to the file with a new line character added at the end.   

  }); // End of: wStream.on('open', function(){Do stuff}
  // End of: Wait till the file is open before trying to write to it.

The docs say to add a listener like this but my problem is that it is not part of the callback

  // Listen for errors on the write stream and log them.
  wStream.on('error', function(error)
  {
    lib.log
    (
      'pmazz7shsko8mfnc0gyz' + '\n' +
      'An error occured when appending to ' + fileName + '\n' +
      'This was the error:' + '\n' +
      JSON.stringify(error) + '\n' +
      '\n'
    );
  }); // End of: wStream.on('error', function(error)
  // End of: Listen for errors on the write stream and log them.

I have not been able to find any example code which shows how to do this. So I guess my question is: Can I somehow write the on error listener into the callback or is there some other way to callback whether or not the write was successful?

Thanks, John

John Shearing
  • 249
  • 3
  • 11

2 Answers2

0

i don't think you need to provide a callback for each write. If error event is not executed it means all your writes succeeded 

 writer.on('error', function(error){
    // handle all errors here
 });

you need to call writer.end('Last string to write') to let the stream know that is your last write then finish event will fire so you can say all writes are done.

I hope this helps

Jasper Bernales
  • 1,601
  • 1
  • 11
  • 16
  • Hi Jasper You wrote: >If error event is not executed it means all your writes succeeded.//// The way I read it, the documentation seems unclear on this. >If an error occurs, the callback may or may not be called with the error as its first argument. To reliably detect write errors, add a listener for the 'error' event./// – John Shearing Aug 29 '19 at 10:33
  • You wrote: >you need to call writer.end('Last string to write') to let the stream know that is your last write then finish event will fire so you can say all writes are done./// – John Shearing Aug 29 '19 at 10:34
  • I am only writing one line. Would I start with writer.end? If I am thinking clearly, I must have something in the callback which tells me that my write was a success before proceeding. Otherwise I must start the rollback process. Also, I think I do not wish to close the stream until the user logs out because there will be several times during the users work that they will be writing to these files. I thought that was one of the advantages of streams - that you don't need to keep opening and closing them. Thanks for your help. John – John Shearing Aug 29 '19 at 10:35
0

pipeline is what I was looking for.

The docs describe it as follows:

A module method to pipe between streams forwarding errors and properly cleaning up and provide a callback when the pipeline is complete.

My sources of information were as follows. Source1, Source2

Also, Anto Aravinth's course on SkillShare called "Understand NodeJS Streams By Example" was very helpful.

The following is the finished function for appending a json record to a file which provides a reliable way to callback success or failure of the append. (really whether or not there was an error during the append operation).

// Define function to append a string to a file  
// and create the file if it does not exist.  
lib.append = function(dir, fileName, appendObject, callback)
{
  // Convert the data object to a string.
  let stringData = JSON.stringify(appendObject);

  // Create a readable stream.
  const sourceStream = new Readable();

  // Load the readable stream with data.
  sourceStream.push(stringData + '\n');

  // Tell the stream no more data is coming.
  sourceStream.push(null);

  // Create a writable stream and specify the file which will receive the data from the readable stream.
  let destinationStream = fs.createWriteStream(lib.baseDir + dir + '/' + fileName + '.json', {flags : 'a'});


  pipeline
  (
    sourceStream,
    destinationStream,
    function(error)
    {
      if (!error) // If the string was appended successfully:
      {
        callback(false); // Report back there was no error
      } 
      else // The string was not appended successfully.
      {
        // Write the error to a log file
        lib.log
        (
          'h0fceuftq8xkdkvh4dl9' + '\n' +
          'Error appending to file ' + fileName + '\n' +
          'This was the error:' + '\n' +
          JSON.stringify(error) + '\n' + 
          '\n'
        );

        // This starts the rollback process for all files involved.
        callback
        (
          'h0fceuftq8xkdkvh4dl9' + '\n' +
          'Error appending to file ' + fileName + '\n' +
          'This was the error:' + '\n' +
          JSON.stringify(error) + '\n'
        );        
      }
    }
  );
}; // End of: lib.append = function(...
// End of: Append a string to a file.
John Shearing
  • 249
  • 3
  • 11