Deduplicating using nodeJS

Question

My goal is to take in a CSV file which contains approximately 4 million records and process each record while scrubbing the data of a particular field. The scrubbing process we have actually creates a reversible hash but is a time consuming process (almost 1 second). What I would like to do since there are only about 50,000 unique values for that field is to set them as properties of an object. Here is a pseudo example of how the object will be built. You can see that for duplicates I plan to just overwrite the existing value (this is to avoid having to loop through some if based search statement.

    var csv = require('csv');
    var http = require('http');
    var CBNObj = new Object;
    csv()
    .fromPath(__dirname+'/report.csv',{
        columns: true
    })

    .transform(function(data){
        CBNObj[data['Field Value']] = data['Field Value'];
    });
    console.log(CBNObj);

This should create my object something like this.

myObj['fieldValue1'] = 'fieldValue1'
myObj['fieldValue2'] = 'fieldValue2'
myObj['fieldValue3'] = 'fieldValue3'
myObj['fieldValue1'] = 'fieldValue1'
myObj['fieldValue1'] = 'fieldValue1'

I have looked over some good posts on here about iterating over every property in an object (like this one Iterating over every property of an object in javascript using Prototype?) but I am still not exactly sure how to acccomplish what I am doing. How can I then take my object with 50k properties and essentially dump the values into an array so that I can end up with something like this?

myArray = ['fieldVaue1','fieldVaue2','fieldVaue3']

EDIT: I could also use some assistance on the first part here because I am getting a null value or undefined when I try and set the object properties. I also still need help then traversing through the object properties to build my array. Any help would be greatly appreciated.

It is important to note that the order of an object's keys is **[not guaranteed](https://developer.mozilla.org/en/JavaScript/Reference/Statements/for...in#Parameters)**. You could very well end up with `['fieldValue3', 'fieldValue1', 'fieldValue2']`, so you'll have to do extra work if order is important. — josh3736, Dec 06 '11 at 15:15

score 1 · Answer 1 · answered Dec 06 '11 at 05:26

You know that the keys of your object are the unique values you want. You just need an array. In node.js you can use Object.keys().

https://developer.mozilla.org/en/JavaScript/Reference/Global_Objects/Object/keys

It's a standard way to take all the keys of an object (that aren't provided by the prototype chain) and put them into an array. So your example looks like this.

var csv = require('csv');
var AcctObj = new Object();
var uniqueArray;

csv()
.fromPath(__dirname+'/report.csv',{
    columns: true
})
.on('data',function(data){
    AcctObj[data['Some Field Value']] = data['Some Field Value'];
})
.on('end', function(){
    uniqueArray = Object.keys(AcctObj);
});

Object.keys also does the hasOwnProperty check internally, so it's similar to the answer by @DvideBy0. It's just one step to the array you want.

Thanks for the reply. I didnt know about .keys. I will definitely have some use for this in the future :) — DvideBy0, Dec 06 '11 at 19:54

score 0 · Accepted Answer · answered Dec 04 '11 at 22:09

var csv = require('csv');
var AcctObj = new Object();
csv()
.fromPath(__dirname+'/report.csv',{
    columns: true
})
.on('data',function(data){
    AcctObj[data['Some Field Value']] = data['Some Field Value'];
})
.on('end', function(){
    for(var prop in AcctObj) {
        if(AcctObj.hasOwnProperty(prop))
        //Do something here....
    }
});

Deduplicating using nodeJS

2 Answers2