0

Im' working with a pelias project and using package wof-admin-lookup to handle data that is read from a file.

There is a case, there is no valid data for being pushed to stream. The wof-admin-lookup will never end.

Here is my code:

const stream = recordStream.create(filePath)
        .pipe(blacklistStream())
        .pipe(adminLookup.create())
        .pipe(model.createDocumentMapperStream())
        .pipe(peliasDbclient())

  stream
        .on('data', data => {
            count++
        })
        .on('finish', () => {
            console.log(`Imported ${count} addresses`)
            resolve()
        })
        .on('error', (e) => {
            reject(e)
        })

Here is the code in wof-admin-lookup:

module.exports = function(pipResolver, config) {
  if (!pipResolver) {
    throw new Error('valid pipResolver required to be passed in as the first parameter');
  }

  // pelias 'imports.adminLookup' config section
  config = config || {};

  const pipResolverStream = createPipResolverStream(pipResolver, config);
  const end = createPipResolverEnd(pipResolver);

  const stream = parallelTransform(config.maxConcurrentReqs || 1, pipResolverStream);
  stream.on('end', end);

  return stream;
};

Although the console logged "Imported 0 addresses" but the pipResolverStream will stay forever if I do not shut down it manually by Ctrl+C.

Update, this case only happens if there is no data passed through stream.

Nguyen Hoang Vu
  • 751
  • 2
  • 14
  • 35
  • 1
    The `end` event is meant to be used with readable streams when you reach the end of the stream when reading. It's not meant to be used with writables. The `finish` event is used for writable streams when `.end()` has been called and the last bit of write data has been flushed. All that said, you can send your own `end` event if you really want to with `stream.emit('end')`, but it does not occur naturally on a writable stream. – jfriend00 Sep 16 '21 at 03:07
  • Hi, I updated my post, this situation only happens if there is no data passed through the stream, the stream.on('end', end) never fire – Nguyen Hoang Vu Sep 16 '21 at 04:21
  • Not enough code shown here and not enough understanding of what all these things do for me to help further. Maybe someone else will come along. – jfriend00 Sep 16 '21 at 04:25

1 Answers1

0

"the end event will never trigger without something like < /dev/null to generate that EOF. Otherwise the program waits for the terminal to send a ^D."

node.js: How to detect an empty stdin stream?

Joel Pusanur
  • 34
  • 2