Using a local file as a data source in JavaScript

Question

Background:

I want to make an "app" that uses JavaScript/HTML only and can be opened by a browser directly from the filesystem. This app must be able to read data from another file. I'll then use JS to parse it and render pages. As a simplified example, imagine I have a CSV file (download here):

Mark Rodgers,mark.rodgers@company.com,Accounting
[...]
Melissa Jones,melissa@company.com,CEO

I want to be able to read the file using JS and use data in it to generate my page.

What I've accomplished so far:

Demo (right-click -> "Save As" to save HTML to your computer). It's also available on jsfiddle in semi-broken fashion (layout is broken, but it should still be functionally correct).

Simply drag and drop the CSV text file into the drag and drop box, or select the text file using the file menu, and JavaScript will read, parse the file and populate the table.

This relies on the FileReader API; most of the heavy lifting is done by this function:

function handleFileSelect(evt) {
    evt.stopPropagation();
    evt.preventDefault();

    var files = evt.target.files || evt.dataTransfer.files; // FileList object.
    var file = files[0];

    // this creates the FileReader and reads stuff as text
    var fr = new FileReader();
    fr.onload = parse;
    fr.readAsText(file);

    // this is the function that actually parses the file
    // and populates the table
    function parse()
    {
        var table = document.getElementById('emps');
        var employees = fr.result.split('\n'); var c = 0;
        for (var i in employees)
        {
            var employee = employees[i].split(',');
            if (employee.length == 3)
            {
                var row = document.createElement('tr');
                row.innerHTML = "<td>" + employee.join("</td><td>") + "</td>";
                table.appendChild(row);
                c++;
            }
        }
        document.getElementById('result').innerHTML = '<span>Added ' + c + ' employees from file: ' + file.name + '</span>';
    }
}

This is almost OK, but it inconveniences the user into manually loading a file. Ideally it should be able to load it automatically, but for security reasons no browser will allow that... yet.

Solution Requirements:

• Must work offline; ie: it can't rely on any online service. This also includes HTTP servers running on the local machine. The idea is to have this run on any computer with just a browser installed.

• Must work when the page is opened using the file:/// protocol (ie: a HTML page on the hard drive).

• Should not rely on third party add ons (eg: Flash, Java, shudders ActiveX). I'm pretty sure these probably wouldn't work anyways if the page is in file:///

• It must be able to accept arbitrary data. This rules out loading a file in a well-behaved format that's ready for consumption like JSON.

• If it works on either (ideally both) Firefox or Chrome it's fine. It's also OK to rely on experimental APIs

I know what the file name is beforehand, so it could be coded in the HTML itself. Any solution that enables me to read a file from disk is fine, it doesn't have to use the FileReader API.

So if there's a clever hack to load a file into a page that's fine too (maybe load it into an invisible iframe and have JS retrieve the contents); that's OK too.

Answer 1

Here is the code I used for Firefox, which is not portable, but works:

As OP commented, enablePrivilege() has been deprecated, this should be considered usable. But as my Firefox using previous profile still work with my code, so I dig a little into the prefs.js (as about:config is hiding these settings,) And here is the settings you need you get it work.

user_pref("capability.principal.codebase.p0.granted", "UniversalXPConnect");
user_pref("capability.principal.codebase.p0.id", "file://");  // path to the html file.
user_pref("capability.principal.codebase.p0.subjectName", "");

And here goes the code:

var File = function(file) {
  netscape.security.PrivilegeManager.enablePrivilege('UniversalXPConnect');
  var ios = Components.classes["@mozilla.org/network/io-service;1"]
                            .getService(Components.interfaces.nsIIOService);
  if (!File.baseURI) {
    File.baseURI = ios.newURI(location.href.substring(0, location.href.lastIndexOf('/')+1), null, null);
    File.baseFolder = File.baseURI.QueryInterface(Components.interfaces.nsIFileURL).file.path;
  }
  var URL = ios.newURI(file, null, File.baseURI);
  this.fptr = URL.QueryInterface(Components.interfaces.nsIFileURL).file;
}

File.prototype = {
  write: function(data) {
    netscape.security.PrivilegeManager.enablePrivilege('UniversalXPConnect');
    var foStream = Components.classes["@mozilla.org/network/file-output-stream;1"]
                             .createInstance(Components.interfaces.nsIFileOutputStream);
    foStream.init(this.fptr, 0x02 | 0x08 | 0x20, 0666, 0);
    var converter = Components.classes["@mozilla.org/intl/converter-output-stream;1"]
                              .createInstance(Components.interfaces.nsIConverterOutputStream);
    converter.init(foStream, null, 0, 0);
    converter.writeString(data);
    converter.close();
  },
  read: function() {
    netscape.security.PrivilegeManager.enablePrivilege('UniversalXPConnect');
    var fstream = Components.classes["@mozilla.org/network/file-input-stream;1"]
                            .createInstance(Components.interfaces.nsIFileInputStream);
    var cstream = Components.classes["@mozilla.org/intl/converter-input-stream;1"]
                            .createInstance(Components.interfaces.nsIConverterInputStream);
    fstream.init(this.fptr, -1, 0, 0);
    cstream.init(fstream, null, 0, 0);
    var data = "";
    // let (str = {}) { // use this only when using javascript 1.8
    var str = {};
      cstream.readString(0xffffffff, str);
      data = str.value;
    // }
    cstream.close();
    return data;
  }
};

Answer 2

Here is an example that uses JSON data in an external file that works locally or on a server. This example just uses the browser's language setting to load a < script > with localized html and then processes its json object to reset the data in the indicated tags with localized content

<html><meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<head>
<script>
    function setLang(){
        for (var i=0;i<items.length;i++){
            term=document.getElementById(items[i].id)
            if (term) term.innerHTML=items[i].value
        }
    }
    var lang=navigator.userLanguage || navigator.language;
    var script=document.createElement("script");
    script.src=document.URL+"-"+lang.substring(0,2)+".js"
    var head = document.getElementsByTagName('head')[0]
    head.insertBefore(script,head.firstChild)
</script>
</head>
<body onload='setLang()'>
<div id="string1" class="txt">This is the default text of string1.</div> 
<div id="string2" class="txt">This is the default text of string2.</div>
</body></html>

The data files for this look like:

items=[
{"id":"string1","value":"Localized text of string1."},
{"id":"string2", "value":"Localized text of string2."}
];

but you can use any parameter to conditionally load the appropriate file (it will be inserted as the first tag in < head >, so it will be usable in anywhere) and the JSON format is capable of handling a large variety of data. You may want to rename the function setLang to something more appropriate and modify it to meet your needs such as ... for each i add a row, then add fields with the data (it looks like you already have a handle on that part) and your JSON would look like:

items=[
{"fname":"john","lname":"smith","address":"1 1st St","phone":"555-1212"},
{"fname":"jane","lname":"smith","address":"1 1st St","phone":"555-1212"}
];

if you need to preprocess your data, awk is pretty handy - it would be something like: (untested guestimate)

awk 'BEGIN{FS=",";print "items=[\n"}
{printf "{\"fname\":\"%s\",\"lname\":\"smith\",\"address\":\"1 1st St\",\"phone\":\"555-1212\"},\n", $1, $2, $3, $4}
END{print "];"}' file.csv > file.js

Edit: now that OP is more clear, only mozilla browsers allow XMLHttpRequest on file:// out of the box and chrome (possibly other webkit based browsers) can be configured to allow it. Knowing that it may NOT work on IE<10, you can:

var filePath = "your_file.txt";
xmlhttp = new XMLHttpRequest();
xmlhttp.open("GET",filePath,false);
xmlhttp.overrideMimeType('text/plain');
xmlhttp.send(null);
//maybe check status !=404 here
var fileContent = xmlhttp.responseText;
var fileArray = fileContent.split('\n')
var n = fileArray.length;
//process your data from here probably using split again for ','

I'm leaving the initial json-p variation for others that may have a similar issue, but have some control of their data format, since it will work on all javascript capable browsers. However, if anyone knows a way to make it work for IE (other than running a small web server), please edit.

Edit 2:

With mozilla browsers you can also use iframes

<html>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<head>
<script>
function showContents(frameObject){
    alert(frameObject.contentDocument.body.innerHTML);
    //replace with your code
}
</script>
</head>
<body onload='showContents()'>
<iframe id="frametest" src="data.txt" onload="showContents(this);" 
    style="visibility:hidden;display:none"></iframe>
</body></html>

Answer 3

Assuming the csv file is in the same directory as the app, I would load the file with AJAX. As far as I know, one can get the file in text format, and then parse it. This should work in IE and Firefox, but does not work in Chrome (unless one runs chrome with the --allow-file-access-from-files command line setting).

Answer 4

This can be done quite easily using javascript XMLHttpRequest() class:

function FileHelper()
{}
{
    FileHelper.readStringFromFileAtPath = function(pathOfFileToReadFrom)
    {
        var request = new XMLHttpRequest();
        request.open("GET", pathOfFileToReadFrom, false);
        request.send(null);
        var returnValue = request.responseText;

        return returnValue;
    }
}

...

var text = FileHelper.readStringFromFileAtPath ( "mytext.txt" );

Answer 5

As I understand, the content of the file is completely under your control, and it doesn't have to be a specific format? And you only need a way to read?

You could declare a global function "handleFile". In your external file the content would have to be like this:

handleFile('Mark Rodgers,mark.rodgers@company.com,Accounting');

To "read" the file, just add a script element with a corresponding src attribute. In your function "handleFile", you get your contents.

The location of the file would probably initally have to be set by the user, but after that you can save the location in localStorage or something like that.

Answer 6

Ensuring that the file is in the same directory or in a sub directory, load the file with AJAX.

Unlike a script tag, you will get access to the contents.