1

My app has a settings file. It can get pretty big (a few MB) because it also contains some history data in it.

When you do something in the app, it auto-saves. But this causes a big pause, because it takes so long to rewrite the file. I tried using async file saving, but if you close the app while it's saving, you lose all the data.

How can I safely autosave settings without lag?

My best idea is to use a local database, but then a user can't simply edit the settings file anymore (currently it's a simple json file, I don't want it to be more complicated to copy all your settings to a new computer for example)

Farzher
  • 13,934
  • 21
  • 69
  • 100
  • 1
    Architecturally, I'd say that if you're saving a single file that's several MB, you might want to revisit how you're going about it. That data can probably be written a small piece at a time as needed, you can spin it off to a backend process and have that do the write (so it wouldn't matter if the user closed the browser), or you can use localstorage to save a user's data (again, a piece at a time as needed). One way or another though, saving a file several MB just to persist some user prefs and history...yeah you want to rethink that. – Tim Consolazio Dec 19 '16 at 23:29
  • Not to mention, you probably don't want your users manually editing data files, lot of opportunity to break your app that way. – Tim Consolazio Dec 19 '16 at 23:30
  • 1
    I would use async Save as you do, and don't let the user exit from the app while it's in progress. Did you measure that the lag is because of the writing, not because of JSON.stringify? It can be slow sometimes. – balazs Dec 19 '16 at 23:35
  • Fast answer: When saving, set a process variable to busy, like following: `process.env.saving = true` and add a listener to an 'exit' (http://stackoverflow.com/questions/14031763/doing-a-cleanup-action-just-before-node-js-exits) and stop the exit till it's done saving Long answer: Listen to the comments above and start splitting ;) – CherryNerd Dec 19 '16 at 23:46
  • I would think carefully about depending on a user not exiting an app, or forcing a user to wait. You can usually kill an app regardless of what the programmer does. I implemented something like this for an app I wrote for google. You have a model, and you put a watch on it. Whenever there's a change to it, you save the changed data (async promises all that nice nice). This way you won't try to take a massive dump all at once and blow out your colon, if you know what I mean. – Tim Consolazio Dec 19 '16 at 23:53
  • @TimConsolazio How do you save only the changed data? That's impossible if all your settings are in 1 file, right? I have to rewrite the whole file – Farzher Dec 20 '16 at 00:07
  • Even if you can hold up normal exit of an app with a process listener, it won't save you in all cases. Power loss, app crashes, force quits, etc., would still be able to make the app stop running in the middle of a save. If you're writing critical data to disk, you should use atomic writes (like described in my answer). – mikl Dec 20 '16 at 02:21
  • This is probably why I wouldn't use a single file. I'd probably use localstorage, a datastore of some kind, or more than one file. I get the simplicity of the single file solution, but in the context you're trying to use it, you will run head first into these scalability problems. – Tim Consolazio Dec 20 '16 at 11:21

1 Answers1

1

Provided that you want to keep the model with a multi-MB settings file, the most reasonable solution would be to use atomic writes to avoid the problem of the app being quit in the middle of a save.

Assuming your file is called settings.json, atomic writes would work something like this:

  1. App decides to update settings, starts writing to a temporary file (to avoid overwriting the existing file halfway), say settings.json.tmp-1482198169 (1482198169 being current unix timestamp).
  2. Once writing to settings.json.tmp-1482198169 is complete, copy the current settings.json to settings.json.bak
  3. Rename settings.json.tmp-1482198169 to settings.json, overwriting the old one.

The basic idea is to construct a process where you always have a valid copy of settings.json, so it's only replaced with a complete copy.

npm has a number of implementations of this, like this one. I'd recommend you try one of those instead of writing your own, since any bugs in the implementation of the atomic write dance could cause you data loss, and it's tricky to get everything right when dealing with asynchronous code.

mikl
  • 23,749
  • 20
  • 68
  • 89