How do I pass an uploaded CSV file, into my function?

Question

first of all, pardon me if this is not the correct way of asking a question but this is my first post.

I am working on a university project and am new to JavaScript (and subsequently, jQuery and NodeJS). I am trying to make a system that will take a CSV file, process it and load the results into a database ready to be used by the rest of the system.

I have made it so that I can upload the file to a directory and have a function to process the CSV file into a JSON object (we have only just started to cover databases so I will have to modify this to store in the database), but the problem I'm currently facing is this...

How do I pass/read the uploaded file into the function for processing?

I have searched around and have found similar questions, but these are either usually using PHP or cover different aspects of my issue, without actually touching upon my problem.

I'm sorry if this is something very basic that I am missing, but I'm pulling my hair out over this and can't progress to the rest of the tasks until i have the first part in place - upload, process and store the CSV file.

I will take any advice, either directly related to the issue or pointers from experience that you think I may face, and thank you for you time.

Upload

var express = require("express");
var multer = require("multer");
var fs = require("fs");
var path = require("path");
var app = express();
var upload = multer({ dest: './uploads/'});

app.use(express.static('public'));
app.use(upload);

app.post('/api/csvUpload',function(req,res) {
    upload(req,res,function(err) {
//    console.log(req.body);
//    console.log(req.files);
        if(err) {
            return res.end("Error uploading file");
        }
        else{   
            res.send("File is uploaded");
        }
    });
});

var server = app.listen(8081, function () {
    var host = server.address().address;
    var port = server.address().port;
    console.log("working on port 8081!");
});

<!DOCTYPE html>
<html>
    <head>
        <title>Home Page</title>
    </head>
    
    <body>
        
        <form id="uploadForm" action="/api/csvUpload" method="post" enctype="multipart/form-data">
            <input type="file" name="csvFile">
            <input type="submit" value="Upload File" />
        </form>
        
    </body>
</html>

CSV to JSON

//This will process and convert a CSV file in to a JSON objec 
function csvJson(csv) {
    var lines = csv.split("\n");
    var result = [];
    var headers = lines[0].split(",");
    
    for(var i = 1; i < lines.length; i++) {
        var obj = {};
        var currentLine = lines[i].split(",");
        
        for(var j = 0; j < headers.length; j++) {
            obj[headers[j]] = currentLine[j];
        }
        result.push(obj);
    }
    return JSON.stringify(result);
};

//Below is for testing purposes
var testCSV = "userName, firstName, lastName, studentID, course\n" +
    "username1,first1,last1,123456,Software Engineering\n" +
    "username2,first2,last2,234567,Software Engineering\n" +
    "username3,first3,last3,345678,Software Engineering\n" +
    "username4,first4,last4,456789,Software Engineering";

var processedCSV = csvJson(testCSV);
console.log(processedCSV);

Hi Richard, Welcome to Stackoverflow. You have explained your question very well, considering this is your first question. If you could post the code what you have done so far it would be easier for other to figure it out — WoodChopper, Nov 14 '15 at 14:47
So the upload part is working, and you want a function to read the file and save its contents into a database? Presumably, as you mention Javascript, this function is expected to run within a web page? The problem is, Javascript is not going to be able to communicate with a database. It can send requests and data to a web server, and read the web server's response, but to do anything with databases you will need a server-side script using a more powerful language like PHP / Perl / Ruby etc. Your Javascript would then send the data to that script. — Doug McLean, Nov 14 '15 at 15:01
Thank you for your feedback, I have added the code as requested. — Richard KeMiKaL GeNeRaL Denton, Nov 15 '15 at 12:06

score 3 · Answer 1 · edited May 23 '17 at 10:27

Because it's not completely clear for me how the files are uploaded, I'll start with the functions that gives the ability to load a (CSV)-file.

For loading a file or url you can use XMLHttpRequest, more on this can be found at W3Schools. This is a really powerful object, and I would really advice to read more into this.

 /**
 * Loads file using XMLHttpRequest
 * @param {String} path
 * @param {Function} success
 * @param {Function} error
 */
function loadFile(path, success, error) {
    var xhr = new XMLHttpRequest();
    xhr.onreadystatechange = function () {
        if (xhr.readyState === XMLHttpRequest.DONE) {
            if (xhr.status === 200) {
                if (success)
                    success(csvToJSON(xhr.responseText));
            } else {
                if (error)
                    error(xhr);
            }
        }
    };
    xhr.open("GET", path, true);
    xhr.send();
}

To use this function, simply execute the following code:

loadFile('Link_to_CSV_file.csv', function (data) {
        myFunction(data);
    }, function (xhr) {
        console.error(xhr);
    });

As soon the data is succesfully loaded, myFunction() will be executed with the data from loadFile(). This can be used in the function.

Finally, load the CSV-data in a function:

function myFunction(data){
    console.log(data);
}

PS: I would advice to convert the CSV file to JSON if you want to handle it in a javascript-function, if you're using Node.js, check out How to convert CSV to JSON in Node.js.

Thanks very much for a detailed answer, i have edited my question to include what i have so far, if that helps. I think i'm understanding your code, but i'm getting a ReferenceError on the XMLHttpRequest? — Richard KeMiKaL GeNeRaL Denton, Nov 15 '15 at 19:38

How do I pass an uploaded CSV file, into my function?

Upload

1 Answers1