reformat large csv dataset using d3 for nvd3

Question

I have a csv dataset with multiple mentions of the same item (ie conflict name - england IRA, israel palestine etc), and values of dates and number_of_casualties for each specified date.

I want to express this using nvd3 stacked area chart: http://nvd3.org/examples/stackedArea.html

the format that it uses in the example is:

key : "category a"
values: [ [ date , amount ] , [ date , amount ] ....]

here's some fo the actual code in the example:

   {
            "key" : "Energy" ,
            "values" : [ [ 1138683600000 , 1.544303464167] , [ 1141102800000 , 1.4387289432421] , [ 1143781200000 , 0] , [ 1146369600000 , 0] , [ 1149048000000 , 0] , [ 1151640000000 , 1.328626801128] , [ 1154318400000 , 1.2874050802627] , [ 1156996800000 , 1.0872743105593] , [ 1159588800000 , 0.96042562635813] , [ 1162270800000 , 0.93139372870616] , [ 1164862800000 , 0.94432167305385] , [ 1167541200000 , 1.277750166208] , [ 1170219600000 , 1.2204893886811] , [ 1172638800000 , 1.207489123122] , [ 1175313600000 , 1.2490651414113] , [ 1177905600000 , 1.2593129913052] , [ 1180584000000 , 1.373329808388] , [ 1183176000000 , 0] , [ 1185854400000 , 0] , [ 1188532800000 , 0] , [ 1191124800000 , 0] , [ 1193803200000 , 0] , [ 1196398800000 , 0] , [ 1199077200000 , 0] , [ 1201755600000 , 0] , [ 1204261200000 , 0] , [ 1206936000000 , 0] , [ 1209528000000 , 0] , [ 1212206400000 , 0] , [ 1214798400000 , 0] , [ 1217476800000 , 0] , [ 1220155200000 , 0] , [ 1222747200000 , 1.4516108933695] , [ 1225425600000 , 1.1856025268225] , [ 1228021200000 , 1.3430470355439] , [ 1230699600000 , 2.2752595354509] , [ 1233378000000 , 2.4031560010523] , [ 1235797200000 , 2.0822430731926] , [ 1238472000000 , 1.5640902826938] , [ 1241064000000 , 1.5812873972356] , [ 1243742400000 , 1.9462448548894] , [ 1246334400000 , 2.9464870223957] , [ 1249012800000 , 3.0744699383222] , [ 1251691200000 , 2.9422304628446] , [ 1254283200000 , 2.7503075599999] , [ 1256961600000 , 2.6506701800427] , [ 1259557200000 , 2.8005425319977] , [ 1262235600000 , 2.6816184971185] , [ 1264914000000 , 2.681206271327] , [ 1267333200000 , 2.8195488011259] , [ 1270008000000 , 0] , [ 1272600000000 , 0] , [ 1275278400000 , 0] , [ 1277870400000 , 1.0687057346382] , [ 1280548800000 , 1.2539400544134] , [ 1283227200000 , 1.1862969445955] , [ 1285819200000 , 0] , [ 1288497600000 , 0] , [ 1291093200000 , 0] , [ 1293771600000 , 0] , [ 1296450000000 , 1.941972859484] , [ 1298869200000 , 2.1142247697552] , [ 1301544000000 , 2.3788590206824] , [ 1304136000000 , 2.5337302877545] , [ 1306814400000 , 2.3163370395199] , [ 1309406400000 , 2.0645451843195] , [ 1312084800000 , 2.1004446672411] , [ 1314763200000 , 3.6301875804303] , [ 1317355200000 , 2.454204664652] , [ 1320033600000 , 2.196082370894] , [ 1322629200000 , 2.3358418255202] , [ 1325307600000 , 0] , [ 1327986000000 , 0] , [ 1330491600000 , 0] , [ 1333166400000 , 0.39001201038526] , [ 1335758400000 , 0.30945472725559] , [ 1338436800000 , 0.31062439305591]]
        }

two questions pls:

1. How to I aggregate and format my csv data to that format?
2. What time format is that?

Answer 1

Answers to your questions,

1. To aggregate, slice and dice large datasets for d3 charts have a look at Crossfilter.

Crossfilter is a JavaScript library for exploring large multivariate datasets in the browser. Crossfilter supports extremely fast (<30ms) interaction with coordinated views, even with datasets containing a million or more records;

Here's an example taken from Crossfilter Tutorial by Rusty.io :

var livingThings = crossfilter([
  // Fact data.
  { name: “Rusty”,  type: “human”, legs: 2 },
  { name: “Alex”,   type: “human”, legs: 2 },
  { name: “Lassie”, type: “dog”,   legs: 4 },
  { name: “Spot”,   type: “dog”,   legs: 4 },
  { name: “Polly”,  type: “bird”,  legs: 2 },
  { name: “Fiona”,  type: “plant”, legs: 0 }
]);

// How many living things are in my house?
var n = livingThings.groupAll().reduceCount().value();
console.log(“There are ” + n + “ living things in my house.”) // 6

// How many total legs are in my house?
var legs = livingThings.groupAll().reduceSum(function(fact) { return fact.legs; }).value()
console.log(“There are ” + legs + “ legs in my house.”) // 14

You can find a few tutorials on how to get started here :

• Crossfilter Tutorial by Rusty.io
• Getting to know Crossfilter by Peter Cook

---

2. The date 1138683600000 is currently Unix Time Stamp.

You can define how you want the date to be displayed in your when passed into your chart. Check this answer on StackOverflow on how to Format the date in nvd3.js

Here is an example :

chart.xAxis.tickFormat(function(d) {
    // Will Return the date, as "%m/%d/%Y"(08/06/13)
    return d3.time.format('%x')(new Date(d))
});

Hope it helps.