3

I try to implement a list structure reading FileSystem Directory tree using node.js

DIR/file structure:

DIR1
     DIR2R
     file1
     file2
     file3
     …

-> list structure: 

("DIR1" ("DIR2" "file1" "file2" "file3" …))

 DIR()   //looks like a C function
 file    //looks like an Atom value

so, for instance, a given UNIX root DIR: 

root bin  cd
          ls

     lib  file1
          file2

     opt

     usr  file3

     file4

is equivalent to a list:

("root" ("bin" ("cd" "ls")
         "lib" ("file1" "file2") 
         "opt" ()
         "usr" ("file3")
         "file4"
        )
)

Having said that, I want to implement this code with some lazy async sequence (stream/ Infinite list?).

node.js fs.readdir recursive directory search is a good reference.

node.js has many excellent libraries such as file.walk etc. , but this is a very interesting subject, in fact, many code examples exsist.

In other words, a good subject to learn a basic concept of lazy async sequence (stream/ Infinite list?), and the code can be a good library, I want to implement from a scratch.

substack/stream-handbook covers the basics of how to write node.js programs with streams . Truly Excellent article.

so, here is my thought on this subject - Recursive Dir tree processing:

  • This subject deal with a recursive search, and there are 2 ways: async and sync , for various reasons, I chose async method.

  • The search result is equivalent to a list structure, where Functional programming paradigm does matter (Lisp/Scheme).

  • For JavaScript Functional programming, there are libraries such as underscore.js, and recently, lazy.js with one important difference: lazy evaluation (also known as deferred execution) where appears to outperform underscore.js.

  • lazy.js wraps Stream processing in Node.js as well. I also found stream.js says: an infinite number of elements. Their power comes from being lazily evaluated, which in simple terms means that they can contain infinite items. .

  • Stream data(infinite list) realizes Reactive Functional Programming(FRP) paradigm. lazy.js actually demonstrates a mouse event FRP implementation. RxJS is a FRP library: RxJS or Reactive Extensions for JavaScript is a library for transforming, composing, and querying streams of data.

  • async method of this subject is a FRP subject.

  • Therefore, lazy evaluation is a core factor of faster List processing Functional Programming, and expands list to infinite list(stream) which integrates async/events to Stream data source to process in Functional Programming paradigm (FunctionalReactiveProgramming).

  • node-lazy Introduction states clearly: Lazy comes really handy when you need to treat a stream of events like a list. The best use case currently is returning a lazy list from an asynchronous function, and having data pumped into it via events. In asynchronous programming you can't just return a regular list because you don't yet have data for it. The usual solution so far has been to provide a callback that gets called when the data is available. But doing it this way you lose the power of chaining functions and creating pipes, which leads to not that nice interfaces. (See the 2nd example below to see how it improved the interface in one of my modules.)

At last, here is my Question on this subject - Recursive Dir tree processing:

Are there any sample code to implement this subject exclusively in lazy evaluation or FRP manner?

I know lazy.js, stream.js, or linq.js is based on lazy evaluation, but any of them can not define node.js fs.readdir as a stream data for async processing (FRP ). According to http://danieltao.com/lazy.js/docs/AsyncSequence.html , it's not yet implement, correct?

RxJS probably can, but I don't know how.

Google rx.Observable fs.readdir hits nothing.

Any ideas?

Community
  • 1
  • 1

1 Answers1

0

Here's a start based on https://gist.github.com/edygar/ee0945a73c79182367df

Given a directory it produces a list of subdirectories.

(coffeescript)

Rx = require "rx"
fs = require "fs"

readdir = Rx.Observable.fromNodeCallback fs.readdir
stat = Rx.Observable.fromNodeCallback (pathName, cb) ->
    fs.stat pathName, (err, stats) ->
        stats.pathName = pathName
        cb err, stats

dirObservable = (dirPath) ->
    readdir dirPath
        .flatMap (items) -> Rx.Observable.from items
        .flatMap (item) -> stat item
        .filter (stats) -> stats.isDirectory()
        .map (stats) -> stats.pathName

module.exports = dirObservable


if not module.parent
    path = require "path"
    dirobs = dirObservable path.resolve __dirname, ".."
    dirobs.subscribe (data) -> console.log data
irickt
  • 1