how to dump csv data into mongoDB using angular and nodejs

Question

I've a csv file and want to dump the data in it into mongodb using angular and nodejs.

Need help to read csv file's data using angular and parse it and store in mongodb

import { Injectable } from '@angular/core';
import { Http } from '@angular/http';
import 'rxjs/add/operator/map';

@Injectable()
export class TemplateService {

  domain = 'http://localhost:3000';
  constructor(private http: Http) { }


  postCsvToMongo(csvJson: any) {
    console.log('CSV!!!' + csvJson);
    return this.http.post(this.domain + '/fu/fileupload', csvJson).map(res => res.json());
  }

}

this is the code i'm using to send json object over post routes.

<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="utf-8">
        <title>Error</title>
    </head>
    <body>
        <pre>TypeError: Converting circular structure to JSON
            <br> &nbsp; &nbsp;at JSON.stringify (&lt;anonymous&gt;)
            <br> &nbsp; &nbsp;at router.post (/home/agam/projects/AMEX/routes/fileUpload.js:10:23)
            <br> &nbsp; &nbsp;at Layer.handle [as handle_request] (/home/agam/projects/AMEX/node_modules/express/lib/router/layer.js:95:5)
            <br> &nbsp; &nbsp;at next (/home/agam/projects/AMEX/node_modules/express/lib/router/route.js:137:13)
            <br> &nbsp; &nbsp;at Route.dispatch (/home/agam/projects/AMEX/node_modules/express/lib/router/route.js:112:3)
            <br> &nbsp; &nbsp;at Layer.handle [as handle_request] (/home/agam/projects/AMEX/node_modules/express/lib/router/layer.js:95:5)
            <br> &nbsp; &nbsp;at /home/agam/projects/AMEX/node_modules/express/lib/router/index.js:281:22
            <br> &nbsp; &nbsp;at Function.process_params (/home/agam/projects/AMEX/node_modules/express/lib/router/index.js:335:12)
            <br> &nbsp; &nbsp;at next (/home/agam/projects/AMEX/node_modules/express/lib/router/index.js:275:10)
            <br> &nbsp; &nbsp;at Function.handle (/home/agam/projects/AMEX/node_modules/express/lib/router/index.js:174:3)
            <br> &nbsp; &nbsp;at router (/home/agam/projects/AMEX/node_modules/express/lib/router/index.js:47:12)
            <br> &nbsp; &nbsp;at Layer.handle [as handle_request] (/home/agam/projects/AMEX/node_modules/express/lib/router/layer.js:95:5)
            <br> &nbsp; &nbsp;at trim_prefix (/home/agam/projects/AMEX/node_modules/express/lib/router/index.js:317:13)
            <br> &nbsp; &nbsp;at /home/agam/projects/AMEX/node_modules/express/lib/router/index.js:284:7
            <br> &nbsp; &nbsp;at Function.process_params (/home/agam/projects/AMEX/node_modules/express/lib/router/index.js:335:12)
            <br> &nbsp; &nbsp;at next (/home/agam/projects/AMEX/node_modules/express/lib/router/index.js:275:10)
        </pre>
    </body>
</html>

this is the error i'm getting when I'm trying to debug the request received over the post request on node side. And how to receive the file over back end??

var csv = require('fast-csv');
var mongoose = require('mongoose');
var Bulletin = require('../models/bulletin');
var multer = require("multer");

module.exports = (router) => {

    router.post('/fileupload', (req, res) => {
        console.log('@@@ ' + req);
        res.send(req.files);
    });

    /*router.post('/fileupload', multer({dest: "./uploads/"}).array("uploads", 12), function(req, res) {
        res.send(req.files);
    });*/
    return router;
};

This is the nodejs code

1","Fed official says weak data caused by weather, should not slow taper","http://www.latimes.com/business/money/la-fi-mo-federal-reserve-plosser-stimulus-economy-20140310,0,1312750.story\?track=rss","Los Angeles Times","b","www.latimes.com","1394470370698
2","Fed's Charles Plosser sees high bar for change in pace of tapering","http://www.livemint.com/Politics/H2EvwJSK2VE6OF7iK1g3PP/Feds-Charles-Plosser-sees-high-bar-for-change-in-pace-of-ta.html","Livemint","b","www.livemint.com","1394470371207
3","US open: Stocks fall after Fed official hints at accelerated tapering","http://www.ifamagazine.com/news/us-open-stocks-fall-after-fed-official-hints-at-accelerated-tapering-294436","IFA Magazine","b","www.ifamagazine.com","1394470371550
4","Fed risks falling 'behind the curve', Charles Plosser says","http://www.ifamagazine.com/news/fed-risks-falling-behind-the-curve-charles-plosser-says-294430","IFA Magazine","b","www.ifamagazine.com","1394470371793
5","Fed's Plosser: Nasty Weather Has Curbed Job Growth","http://www.moneynews.com/Economy/federal-reserve-charles-plosser-weather-job-growth/2014/03/10/id/557011","Moneynews","b","www.moneynews.com","1394470372027
6","Plosser: Fed May Have to Accelerate Tapering Pace","http://www.nasdaq.com/article/plosser-fed-may-have-to-accelerate-tapering-pace-20140310-00371","NASDAQ","b","www.nasdaq.com","1394470372212

This is my sample csv file

> amex@1.0.0 start /home/agam/projects/AMEX
> node index.js

(node:6478) DeprecationWarning: current URL string parser is deprecated, and will be removed in a future version. To use the new parser, pass option { useNewUrlParser: true } to MongoClient.connect.
listening on port 3000
Conected to database: amex
@@@ [object Object]

this is the thing it is printing

// 1) Express for nodejs
const express = require('express');
const app = express();

const template = require('./routes/template');
const upload = require('./routes/upload');
const path = require('path');
// const fileUpload = require('express-fileupload');

const router = express.Router();
const fu = require('./routes/fileUpload')(router);

//const carts = require('./routes/carts')(router);

// 5) body parser
const bodyParser = require('body-parser');

const cors = require('cors'); // CORS is a node.js package for providing a Connect/Express middleware that can be used to enable CORS with various options.

// 2) Mongodb connection
var mongoose = require('mongoose');
const config = require('./config/database');
mongoose.connect(config.uri, (err) => {
    if (err) {
        console.log('error ' + err);
    } else {
        console.log('Conected to database: ' + config.db);
    }
});

app.use(cors({ origin: 'http://localhost:4200' }));
// 5) put bodyparser code before the routes (converts request to req.body)
app.use(bodyParser.urlencoded({ extended: true }));
app.use(bodyParser.json());


app.use(function(req, res, next) {
    res.header("Access-Control-Allow-Origin", "*");
    res.header('Access-Control-Allow-Methods', 'PUT, GET, POST, DELETE, OPTIONS');
    res.header("Access-Control-Allow-Headers", "Origin, X-Requested-With, Content-Type, Accept");
    next();
  });

// 3) connect node.js to angular 
app.use(express.static(__dirname + '/client/dist/'));

// app.use(fileUpload());

app.use('/fu', fu); // Use Product routes in application
//app.use('/carts', carts);
// 



app.get('*', (req, res) => {
    res.sendFile(path.join(__dirname + '/client/dist/index.html'));
});



app.listen(3000, () => {
    console.log('listening on port 3000');
});

this is my index.js file which you are expecting

were you able to read the csv file in angular and convert it to json , because that seems the obvious approach. — Vaibhav, Aug 16 '18 at 04:57
Actually I don't know the approach and even don't know how to read csv file in angular and convert it to json. Please help me out as I'm new in this field and learning — agamjain14, Aug 16 '18 at 05:32
https://stackblitz.com/edit/angular-mvq5ju Please use this link to access the code. I've created the project on stackblitz.com Please use the above link — agamjain14, Aug 16 '18 at 09:10
@NicholasKyriakides Please find the project on https://stackblitz.com/edit/angular-mvq5ju — agamjain14, Aug 16 '18 at 09:15
@VaibhavKumarGoyal Please use the link and suggest some approaches — agamjain14, Aug 16 '18 at 09:17
please visit and view the answer I have sent across, do send across the needful comment's hope it helps — Vaibhav, Aug 17 '18 at 17:12
import { Injectable } from '@angular/core'; import { Http } from '@angular/http'; import 'rxjs/add/operator/map'; @Injectable() export class TemplateService { domain = 'http://localhost:3000'; constructor(private http: Http) { } postCsvToMongo(csvJson: any) { console.log('CSV!!!' + csvJson); return this.http.post(this.domain + '/fu/fileupload', csvJson).map(res => res.json()); } } @VaibhavKumarGoyal how to read the json data over post routes... — agamjain14, Aug 18 '18 at 06:29
can you share the node api server code? ie.., the api can which is throwing this error , because this is a mongoose query execution error and thrown when you are trying to stringify the json with circular dependancies please visit https://stackoverflow.com/questions/28296600/querying-mongodb-subdocuments-error-converting-circular-structure-to-json — Vaibhav, Aug 21 '18 at 16:10
what is your `console.log('@@@ ' + req);` printing and second thing can you share your mongo calls because this file is probably your app.js / server.js, as i mentioned earlier this is mongo error of JSON.stringifying the JSON with circular depandancies which in your case `res.send(req.files); `automatically Jsonifies the data just for testing sake can you change this to `res.send({message:'done'})` — Vaibhav, Aug 21 '18 at 17:03
@VaibhavKumarGoyal I've other problems too.. I'll be glad if you are interested in helping... — agamjain14, Aug 28 '18 at 16:33

score 1 · Answer 1 · answered Aug 17 '18 at 17:10

Hi I have done a few changes in your StackBiltz example, so I'm able to achieve the payload to be posted via a post call to in the form of JSON array and created a service which accepts this payload and on subscription passes the payload to your node post api

Created a function named pass

test.component.ts

import { Component, OnInit } from '@angular/core';    
import { Router } from "@angular/router";
import { FileUtil } from './file.util';
import { Constants } from './test.constants';
import {TestService} from './test.service';

@Component({
  selector: 'app-test',
  templateUrl: './test.component.html',
  styleUrls: ['./test.component.css']
})
export class TestComponent implements OnInit {

  fileImportInput: any;
  data;
  csvRecords:any = [];

  constructor(private _router: Router,
    private _fileUtil: FileUtil,private testService:TestService
  ) { }

  ngOnInit() {
  }

  // METHOD CALLED WHEN CSV FILE IS IMPORTED
  fileChangeListener($event): void {

    var text = [];
    var target = $event.target || $event.srcElement;
    var files = target.files;

    if (Constants.validateHeaderAndRecordLengthFlag) {
      if (!this._fileUtil.isCSVFile(files[0])) {
        alert("Please import valid .csv file.");
        this.fileReset();
      }
    }

    var input = $event.target;
    var reader = new FileReader();
    reader.readAsText(input.files[0]);

    reader.onload = (data) => {
      let csvData = reader.result;
      let csvRecordsArray = csvData.split(/\r\n|\n/);

      var headerLength = -1;
      if (Constants.isHeaderPresentFlag) {
        let headersRow = this._fileUtil.getHeaderArray(csvRecordsArray, Constants.tokenDelimeter);
        headerLength = headersRow.length;
      }

      this.csvRecords = this._fileUtil.getDataRecordsArrayFromCSVFile(csvRecordsArray,
        headerLength, Constants.validateHeaderAndRecordLengthFlag, Constants.tokenDelimeter);
        this.passThisFunction(this.csvRecords);
        console.log('this.csvRecords',this.csvRecords);
      if (this.csvRecords == null) {
        //If control reached here it means csv file contains error, reset file.
        this.fileReset();
      }      
    }
    reader.onerror = function () {
      alert('Unable to read ' + input.files[0]);
    };
  };

  fileReset() {
    this.fileImportInput.nativeElement.value = "";
    this.csvRecords = [];
  }

  passThisFunction(records:any){
      console.log('inside function call');
      console.log('records',records);
      let json=JSON.stringify(records);
      console.log('jsonified',json);
      var str = json.replace(/},/g, "},\r\n");
        console.log('das' + str);
      this.testService.postCsvToMongo(str).subscribe((data) => {
                console.log(data);
      });
  }

}

test.component.html

<h4>Read .CSV File using Angular2, JQuery and Bootstrap</h4>

<table>
    <tr>
        <td>
            <input type="file" 
                #fileImportInput
                name="File Upload" 
                id="txtFileUpload" 
                class="btn btn-primary" 
                (change)="fileChangeListener($event)" 
                accept=".csv"
                />
        </td>

    </tr>
</table>

<div>
    <table class="table table-responsive table-hover" border="1" 
        style="width : 50%;">
        <tbody>
            <ng-container >
                <tr >                   
                        <td *ngFor="let csvRec of csvRecords;let i=index">{{csvRec}}</td>
                </tr>
            </ng-container>
        </tbody>
    </table>
</div>

file.util.ts

import { Injectable }       from '@angular/core';

@Injectable()
export class FileUtil {

    constructor() {}

    isCSVFile(file) {
        return file.name.endsWith(".csv");
    }

    getHeaderArray(csvRecordsArr, tokenDelimeter) {        
        let headers = csvRecordsArr[0].split(tokenDelimeter);
        let headerArray = [];
        for (let j = 0; j < headers.length; j++) {
            headerArray.push(headers[j]);
        }
        return headerArray;
    }

    validateHeaders(origHeaders, fileHeaaders) {
        if (origHeaders.length != fileHeaaders.length) {
            return false;
        }

        var fileHeaderMatchFlag = true;
        for (let j = 0; j < origHeaders.length; j++) {
            if (origHeaders[j] != fileHeaaders[j]) {
                fileHeaderMatchFlag = false;
                break;
            }
        }
        return fileHeaderMatchFlag;
    }

    getDataRecordsArrayFromCSVFile(csvRecordsArray, headerLength, 
        validateHeaderAndRecordLengthFlag, tokenDelimeter) {
          console.log(csvRecordsArray);
          console.log(headerLength);
          console.log(validateHeaderAndRecordLengthFlag);
          console.log(tokenDelimeter);
        var dataArr:any = [];

        for (let i = 0; i < csvRecordsArray.length; i++) {
            let data = csvRecordsArray[i].split(tokenDelimeter);

            if(validateHeaderAndRecordLengthFlag && data.length != headerLength){
                if(data==""){
                    alert("Extra blank line is present at line number "+i+", please remove it.");
                    return null;
                }else{
                    alert("Record at line number "+i+" contain "+data.length+" tokens, and is not matching with header length of :"+headerLength);
                    return null;
                }
            }
            dataArr=data;
            console.log('dataArr',dataArr);
        }   
        return dataArr;
    }

}

test.service.ts

import{Injectable} from '@angular/core';
import{Http} from '@angular/http';
import {map} from 'rxjs/operators';

@Injectable()
export class TestService{
  API_URL:string='your api url';
  constructor(private http:Http){}

  postCsvToMongo(csvJson:any){
    return this.http.post(this.API_URL,csvJson)
    .pipe(map((res)=>console.log(res)));
  }
}

Thanks for the help. I'm able to form json objects, but when i'm sending for post routes, it is giving bad request error — agamjain14, Aug 18 '18 at 06:28
whats the error can you share your node api calls? and the node server files code? — Vaibhav, Aug 18 '18 at 16:56

how to dump csv data into mongoDB using angular and nodejs

1 Answers1