2

I am looking to read files in a folder using Node.js. The incoming files are text files, with a given name (for example, 'FishText1')

The folder will be having new files added to it constantly, and each of these new file will ALWAYS be called 'FishText' + int.

Currently, I am just using this to perform a task on the text file, in this case moving it to another folder (later in my project I will be looking to perform searches through these text files for certain strings, but will get to that later and have a plan for that), and at the moment have some code which does this perfectly, see below. [NB this was taken from code suggested on another thread, by user 'adotout', 30th July 2013]

My question is, is there a better way than using a forced 'while' loop, which will stop when the processed.js file gets to the end of its list - ie when a new text file is added, I want them to be processed too, without having to rerun processed.js.

Extra Info: I am aware that this isn't the most elegant way to do this, so any help in streamlining it would also be appreciated, as this is working up to be part of a bigger project. The files themselves are always going to be <1kb, however, could be several thousand per hour which need to be processed. The requirement is that they are processed 'live' rather than waiting until they have finished and then processing them all.

var fileExist = true;
var fileNumber = 1;

while (fileExist) {

** Not my actual code **
** Work PC is at work and I dont have a local copy **

if __DIR/FishText + fileNumber + .txt
   Write __NewDIR/NewText + fileNumber + .xml
   fileNumber++
else
   error message to console
}

Current results are as expected - I can move all of the files to my new destination folder, but if a new file is added AFTER I run the 'processor.js', then I have to run it again.

So, I have given the 'Chokidar' approach a try, and I cant seem to get any of the code to run - not even within Atom (which I am using to write my code). See full code below, with directories omitted.

When I run the code 'processor.js' the command line just jumps to a new line, as if it has run the process, but I see no output from my code, I would expect to see something, as I have purposefully includes a line which prints out the file to console.

var processedFileNumber = 1;
var chokidar = require('chokidar')
var watcher = chokidar.watch('__DIR', {ignored: /^\./, persistant: true})

watcher
  .on('add', function(path) {
    const file = readFile ('__DIR/file ' + processedFileNumber + '.txt', 'utf8');
    console.log("Print something");
    writeFile('__DIR/fileXML ' + processedFileNumber + '.xml', data);
    processedFileNumber++;
  })
  .on('error', function(error) {
     console.error('Error Happened', error)
  })
Jamal Abo
  • 472
  • 4
  • 16
Dan Edwards
  • 23
  • 4

1 Answers1

2

The "while" solution is high cpu demanding and quite rough.

A similar, but better, approach is the "interval" one:

const elaborateFiles = setInteval(() => {

    if (thereAreNewFiles()) {
        elaborateFile()
    }

    if (allFileAreElaborated()) {
        clearInterval(elaborateFiles)
    }
}, 1000)

But i think in these case you should aim for a event-driven approach.

You can find more info and some examples here: Watch a folder for changes using node.js, and print file paths when they are changed

In our case:

var chokidar = require('chokidar')
var watcher = chokidar.watch('file or dir', {ignored: /^\./, persistent: true})

watcher
    .on('add', function(path) {
        const file = readFile(path)
        elaborateFile(file)
     })
    .on('error', function(error) {
        console.error('Error happened', error)
    })

EDIT:

I think you have a problem with the ignored options. This work prefectly on my pc:

const fs = require('fs')
const chokidar = require('chokidar')
const watcher = chokidar.watch('./folder')

watcher
    .on('add', (path) => {
        console.log('path', path)
        const file = fs.readFileSync(path)
        console.log('\nfile')
        console.log(file.toString())
    })
    .on('error', (error) => {
        console.error('error', error)
    })
Andrea Franchini
  • 548
  • 4
  • 14
  • That sounds brilliant, first thing tomorrow will update the code at work and then post the results. Thanks for the speedy reply and code examples (and links too) – Dan Edwards Jul 20 '19 at 19:07
  • 1
    @DanEdwards - You might check this answer as `accepted` one. I've also voted for +1 – SuperStar518 Jul 22 '19 at 09:28
  • I edited my original post,but have got it working using the fs.watch function: Why not just use the old fs.watch? Its pretty straightforward. fs.watch('/path/to/folder', (eventType, filename) => { console.log(eventType); // could be either 'rename' or 'change'. new file event and delete // also generally emit 'rename' console.log(filename); }) For more info and details about the options param, see Node fs Docs shareedit answered Jun 16 '17 at 11:42 Kaushik Evani – Dan Edwards Jul 22 '19 at 09:34
  • I think is a problem with the `ignored` configuration, see the edits – Andrea Franchini Jul 22 '19 at 09:37
  • The code you have edited does seem to work, as in its showing the expected 'waiting' result in the cmd window, however, it doesn't seem to be outputting anything. Do I need to run anything in addition to the .js files for Chokidar to work? As per my previous comment, I did get something working with fs.watch, however, there are known issues that it triggers events twice, which causes issues for me as I dont want the files to double up. – Dan Edwards Jul 22 '19 at 12:34
  • I just run `node test.js` which contains that code and play with some files in the specified folder. I see all working as expected. For more particular management you shuold check their docs - like for understand when the `add` event is fired. I found (hands-on) on creation, moving and copy. – Andrea Franchini Jul 23 '19 at 07:17