create stream from another stream and join both streams in node.js

Question

I have a problem with dealing multiple streams. Consider this case, I have data.csv file whose content is

id,val
1,100
2,75

I want to produce an output like this, half should be 50% of val and we need to append same rows but multiplied by 10.

id,val,half
1,100,50
2,75,37.5
10,100,500
20,75,37.5

I am successful to read csv from file, transform it and process it. I am only able to produce 1 stream. thats is, In the beginning there are only 2 row, so my output is also containing 2 rows. I am able to produce result like this

id,val,half
1,100,50
2,75,37.5

but unable to add extra part. that is whole content to be multiplied by 10.

One solution is very simple, read the content in the file, have it in javascript Array/Object format, process it and write to disk. this do not deal with pipes. I want to write whole things in terms of pipes.

I already have code something like this.

// Initialize file streams where the output will be written to
const forecastcsv = fs.createWriteStream(forecastfile);
fs.createReadStream(argv.file).pipe(parser).pipe(transformer).pipe(stringifier).pipe(forecastcsv);

Unable to think how I can create two streams (one is doing half, other is doing multiply to 10) the join them to achieve double rows in final csv file.

Thanks

Did you look at the question http://stackoverflow.com/questions/16431163/concatenate-two-or-n-streams ? — kyrisu, Jul 06 '15 at 04:46
@kyrisu yes, this will solve my problem upto some extent, but I am unable to to create two streams. — Narendra Sisodiya, Jul 06 '15 at 06:01
Do you want a format where ALL the data that is 2x is after the original data? — Chris Anderson, Jul 06 '15 at 06:55
That's not really going to fit into the stream pattern. How would your stream know when you've finished writing all the first kind of data? You could read the data to a file and then, once you've finished, you could stream it out into the 10x function and append it to that file. — Chris Anderson, Jul 06 '15 at 07:31
@ChrisAnderson-MSFT I want something like this.. datasource = fs.createReadStream(argv.file).pipe(parser); var x = datasource.pipe(tranformer_for_AddHalf); var y = x.pipe(tranformer_for_DubleIndex)); var z = joinStream(x,y); x.pipe(writeTodisk) Syntax wise this is wrong, but I want this kind of functionality, Let me know If stream fits into it or not. — Narendra Sisodiya, Jul 06 '15 at 07:36
If you do this you'll get `a, 10*a, b, 10*b` etc. Order isn't guaranteed. If that's fine, just have your transform function emit two records one 1x and one 10x. Nothing forces you to emit 1:1. :) — Chris Anderson, Jul 06 '15 at 07:42

score 1 · Accepted Answer · answered Jul 06 '15 at 08:27

Thanks @Chris Anderson-MSFT I was having misunderstanding about 1:1, we can emit more data, So here is my transformer which is working. no need to multiple streams.

var transformer = csv.transform(function (record, callback) {
    record.half = record.value/2;
    var extra = JSON.parse(JSON.stringify(record));
    extra.id = extra.id * 10;
    extra.value = extra.value * 10;
    extra.half = extra.half * 10;
    callback.call(this, null, record, extra); 
});

create stream from another stream and join both streams in node.js

1 Answers1