5

I have a NODE.JS express server which breaks down chunks of JSON into smaller sections and routes the request via URL's for mobile devices and various other apps I am creating.

I have just moved from test data to live data, but I am finding that NODE.JS is not using the latest version of the JSON, but caching and reusing the JSON that was inplace at the server runtime.

Here is (part of) the code

var express = require('express'),
    http = require('http');
    forever = require('forever');

var ppm = require('./data/ppm.json');
var stations= require('./data/stations.json');
var fgwstations= require('./data/fgwstations.json');

var app = express()
    .use(express.bodyParser())
    .use(express.static('public'));

app.all('/', function(req, res, next) {
    res.header("Access-Control-Allow-Origin", "*");
    res.header("Access-Control-Allow-Headers", "X-Requested-With");
    next();
});

app.get('/ppm/all', function (req, res) {
    res.json(ppm);
});
app.get('/*', function (req, res) {
    res.json(404, {status: 'datafeed not found please refer to documentation'});
});

http.createServer(app).listen(3000, function () {
    console.log("Data Server ready at http://localhost:3000");
});

There maybe typos in the code as I have just ripped chunks out as the proper artile is quite long as its does a LOT. I keep this server running permenantly using the FOREVER function so it will run without the shell command being open

Now I suspect I could do some kind of fs.filewatch type function that would restart the server every time the JSON file was updated, but this seems a bit iffy restarting a server especially when this data will be updated every 2-3 mins and future ones even more rapidly. It only takes one person to be doing a request or an app to be requesting data during this restart to cause a problem.

Is there a way of 're-reading' the JSON file assigned to var ppm (the others are fairlry static) or is restarting the server the only way?

Any good ideas on how to read that file and do this would be greatly appreciated as I am sure someone will have a much more efficient way of doing it.

The current dev server is open source (and very much WIP) and feel free to see what it does

http://54.194.148.89:3000
zvisofer
  • 1,346
  • 18
  • 41
MOLEDesign
  • 488
  • 8
  • 20
  • When you require a JSON file, it's not automatically updated when you change that file, you have to get the file again. – adeneo Feb 05 '14 at 20:55

1 Answers1

9

require() caches everything it loads, as it is designed for loading code modules which shouldn't be changing. If you want to reload something, you have to delete the cache entry.

A better approach would be to instead use a combination of fs.watch, fs.readFile and JSON.parse to reload the changing data. No need to fiddle with the cache or restart the server.

An even better approach would probably be to use a database of some sort instead of the filesystem.

OrangeDog
  • 36,653
  • 12
  • 122
  • 207
  • The database is coming next, for now I just want live data so I can work on some apps. This is proof of concept with the routing of data for swift retrieval with Angular.. I will have a read of the fs you suggest and JSON.parse to see what it can do in this case Thanks for the suggest – MOLEDesign Feb 05 '14 at 21:13
  • I used this pattern as a cache. I have a timer app that runs all the db queries in the evening and generates a static folder of json data. Angular app requests the static data via node. It is much faster than accessing the database. In my case data only needs to be updated once a day so it works in this case. – mbokil Jan 13 '16 at 14:47
  • Hey @OrangeDog. Thanks for your answer. I would like to touch on a point your made in your answer about the option of using a db over a fs. I want to agree to a certain level. I am currently working on a project where I need to store latitude and longitude information. Depending on the scope/size of the project, I realize that It would be faster and more cost-effective to just store all my geopoints in a file and then access it using `require()`. – AllJs Apr 14 '18 at 03:21
  • https://stackoverflow.com/questions/9210542/node-js-require-cache-possible-to-invalidate works for me ..but may be not for production purpose – Shailesh Vaishampayan Apr 19 '19 at 19:26