How to query a directory tree without loading them in memory

Question

I have a root directory, for example events.

Every event is a directory containing files, and there are millions of events.

Directories in the events directory:

...
2018-01-17-18:40:51.151343-1bb809
2018-01-17-20:55:22.627688-26ada5
2018-01-17-20:55:27.919532-8243cd
2018-01-17-20:56:30.743072-94e913
2018-01-17-20:57:39.824845-64ccb7
...

What I am trying to do is:

Load the directories one by one, instead of doing it in one call e.g. fs.readDir
As the directory names are timestamped, I want to be able to query them with

e.g.

{ time: 
    $gte: t1
    $lte: t2
}

How would I do this query, with performance, without reading all the directories in memory at once?

I am using NodeJs 9

score 1 · Accepted Answer · answered Jan 18 '18 at 10:57

Using pure node.js File System functions you cannot do it.

fs.readdir exist but it will return all the names at once.

What you can do

Execute a command line using node.js :

So you can do find -regex "Here the regexp to match your dates"

Example from stack here by @hexacyanide :

const util = require('util');
const exec = util.promisify(require('child_process').exec);

async function ls() {
  const { stdout, stderr } = await exec('ls');
  console.log('stdout:', stdout);
  console.log('stderr:', stderr);
}

ls();

How to query a directory tree without loading them in memory

1 Answers1