How to transform data in angular to correct tensor format?

Question

I'm working on a machine learning project and I'm trying to deploy my algorithm with angular. I have already uploaded the pretrained model correctly and I managed to upload data from a csv, but now I have problems to properly transform the data I have to the correct format of tensors. My model is a LSTM Neural Network that expects timewindows with the length of 60 of accelerometer data (x-axis, y-axis and z-axis) to predict human activities so tensors of the format [any, 60,3] Down below you can find the important part of my code that I have up to now

Loading the model here

async loadModel() {
   this.model = await tf.loadLayersModel('assets/tfjs_model/model.json');
 };

Right now I have a placeholder with the tf.ones() function to simply test, if my prediction works (it does work!!)

 async predictProcess() {
   const output = this.model.predict([tf.ones([10, 60, 3])]) as any;    
   this.predictions = Array.from(output.dataSync());
   console.log(this.predictions);

};

This is the part of the code, where I load the data

getDataRecordsArrayFromCSVFile(csvRecordsArray: any, headerLength: any) {
let dataArr = [];

for (let i = 0; i < csvRecordsArray.length; i++) {
  let data = (<string>csvRecordsArray[i]).split(',');

  // FOR EACH ROW IN CSV FILE IF THE NUMBER OF COLUMNS
  // ARE SAME AS NUMBER OF HEADER COLUMNS THEN PARSE THE DATA
  if (data.length == headerLength) {

    let csvRecord: CSVRecord = new CSVRecord();

  //  csvRecord.timestep = Number(data[0].trim());
    csvRecord.xAxis = Number(data[1].trim());
    csvRecord.yAxis = Number(data[2].trim());
    csvRecord.zAxis = Number(data[3].trim());

    dataArr.push(csvRecord);
  }
}
return dataArr;
}

This is the CSVRecord class

export class CSVRecord {
  public timestep: any;
  public xAxis: any;
  public yAxis: any;
  public zAxis: any;
}

I acknowedge the answers and the time people took to answer my questions and I'm trying to work things out myself. But while doing that, new questions/problems come up. I'm simply seeking for help here, because I'm new to certain parts of programming... — DarkIudeX, Sep 10 '19 at 09:43

score 1 · Accepted Answer · answered Sep 10 '19 at 09:52

Instead of creating an object to populate dataArr, it will be better to use an array

if (data.length == headerLength) {

    let csvRecord: number[] = [];

  //  csvRecord.timestep = Number(data[0].trim());
    csvRecord.push(Number(data[1].trim()));
    csvRecord.push(Number(data[2].trim()));
    csvRecord.push(Number(data[3].trim()));

    dataArr.push(csvRecord);
  }

Then to create a tensor out of dataArr, you can use

tf.tensor(dataArr)

Using tf.tensor, it will create a tensor of shape [dataArr.length, 3]

But if dataArr is a big array, trying to create a tensor of it directly will cause memory issue as the whole data will be uploaded directly to the tensor backend when it starts to be used during computation. This answer discusses how to handle large data when creating a tensor.

Thanks, that helped a lot - also the link on how to deal with large datasets! :) Do you have an idea, how I can transform that tensor of this shape into a tensor of the shape [any,60,3]? So my algorithms expects batches of 60 data points with x-axis, y-axis, z-axis data. And any depending on the overall dataset (e.g. if I have 36o datapoints, the shape would be [6,60,3] — DarkIudeX, Sep 11 '19 at 06:06
you can use `tensor.reshape([-1, 60, 3])`. However, for the reshape to work, you need to have a number of data multiple of `60 * 3 = 180` — edkeveked, Sep 11 '19 at 07:40

How to transform data in angular to correct tensor format?

1 Answers1