12

I want to use localStorage to store large amount of data(like 800GB), according to http://arty.name/localstorage.html and also I'm using Firefox, I changed my localStorage size and also the cache size. So the size isn't a problem. However, I write some jquery like the following:

 $("a[href]").each(function(){
    $(this).click(function(event){
       localStorage.test += "somenewinformation";
      ...

If this localStorage.test already have large amount of data like 400GB, so the storing information step would be extremely slow. When I click on a link,will the jquery wait for me to finish the appending new information to localStorage.test or it will just go to the next page and information in localStorage.test will all lost or localStorage.test will just remain the old value? What I dont understand is whether a new thread will be generated to do this storing in background or not and closing browser in the middle will affect it or not.

Sorry about the messy description and thanks in advance!

user2489547
  • 181
  • 1
  • 1
  • 9
  • 800Gb? Most browsers default to a local Storage limit of around 5Mb. You can change this, but this does rather suggest that localStorage is not designed for the size of data you are suggesting. You should look at another architecture. –  Jun 22 '13 at 22:18
  • 3
    You know, there's a *reason* `localStorage` is limited. – lonesomeday Jun 22 '13 at 22:18
  • What you are trying to store? Like others have said, there should rarely be a need to go over the storage limit. – Precastic Jun 22 '13 at 22:22
  • Hi, I'm using local storage to collect some user mouse movement traces and also informations of URLs on current web page that user is browsing. But I want to collect the trace for a whole year so this data might accumulate to more than 100GB. – user2489547 Jun 22 '13 at 22:36
  • If I use another way to store the data, like indexDB, if the questions I asked above is not solved, then it is not useful. I need to guarantee that when user clicks on a link, the new information will still be appended and stored back even the storage variable already contain large amount of data – user2489547 Jun 22 '13 at 22:39
  • 3
    There are tasks for which Javascript is suited. Then there is this task. Seriously, use server-side code and a proper database.. – lonesomeday Jun 22 '13 at 22:43
  • 15
    My harddrive cannot even store 800GB. – TimWolla Jun 22 '13 at 22:43
  • 1
    @ TimWolla This is done on a computer with harddrive 1TB so the size is not a problem. – user2489547 Jun 22 '13 at 22:47
  • 1
    @lonesomeday Thanks for the advice, but I have to solve it using just client side programming. – user2489547 Jun 22 '13 at 22:49
  • How about you put everything as one item, use many items, e.g. one a day. And think up some way to compress your data, e.g. if you have an integer, convert to string in base `36` (native) or another mapping. `23727593894..toString(36) === "awesome";` – Paul S. Jun 22 '13 at 23:08
  • @PaulS. Thanks. But I can't just retrieve the trace from participants everyday since this would bother them. But compression would be a good way to improve my script. – user2489547 Jun 22 '13 at 23:11
  • 1
    What happens when they clear their cache? You're ok with losing all that information? Is everyone using a browser that supports localstorage? Also, you're not appending the new string to the existing one. Strings in JS are immutable, so this operation is duplicating your data. http://stackoverflow.com/questions/51185/are-javascript-strings-immutable-do-i-need-a-string-builder-in-javascript – norepro Jun 23 '13 at 03:43
  • 3
    Seriously, relying on Javascript localStorage for this amount of data is nuts - it's not designed for it. I'd say using Javascript for this is nuts too, but if you're committed to a javascript approach look at the FileSystem API. At least that way you can use a proper filesystem to split the data up. –  Jun 23 '13 at 04:11
  • @Thr33Dii If you try var a = "aa"; a+="bb" and you will see a's value is "aabb" – user2489547 Jun 24 '13 at 16:53
  • 1
    @MikeW Using File API can read the file. But can you write content from the client side javascript to the local file system of the user? – user2489547 Jun 24 '13 at 17:04
  • @user2489547 No - a Javascript `FileSystem` is sand-boxed. You'd have to devise a way to retrieve your data, but you'd also have to solve that problem with your localStorage idea. A `FileSystem` approach gives you much greater flexibility to store and organise your data. Really, your best overall approach is to use a server-side approach to data storage. –  Jun 24 '13 at 19:54
  • @user2489547 yes, but it makes an entirely new string in the process. It doesn't simply tack "bb" on to the end. So in your situation, I hope you have 2*800GB of space while it makes that new copy. – norepro Jun 26 '13 at 02:59
  • @Thr33Dii I see. I misunderstood your previous post, but I have 1TB hard drive. – user2489547 Jun 26 '13 at 17:53

2 Answers2

17

You can't! The usual limit is 5 MB.

Some browsers such as Opera allow you to adjust the size, but this is purely dependent on the browser and is a user-initiated action, not a programmable one.

And even if you could remember that localStorage can only store strings, so anything else need to be stringified first. That together with this being an key-value storage array you will run into pretty poor performance at the end.

If you need large storage capacity, look into File API instead. This is made to work with large files (Blobs) and you can store anything as a Blob.

800 Gb is a large size and File API can only work as fast as the file system (at best, more likely a bit slower as it is sandboxed, ie. not a pure file system).

More about File System API (Note: discontinued as of 4/2014):
http://www.w3.org/TR/file-system-api/

Tutorial:
http://www.html5rocks.com/en/tutorials/file/dndfiles/

Blob:
https://developer.mozilla.org/en-US/docs/Web/API/Blob

Update As the Filesystem API has been discontinued there are essentially only two options left (besides from storing data to the server):

  • Indexed Database API aka Indexed DB (recommended)
  • Web SQL (deprecated but still supported in browsers such as Safari)

Also these are initially limited in size by the browser. It is possible to request a larger quote which the user is asked to accept.

See more details about this API here:
http://www.w3.org/TR/IndexedDB/

Roj
  • 995
  • 1
  • 8
  • 22
  • Can you write content from the client side javascript to the local file system of the user? – user2489547 Jun 24 '13 at 17:03
  • @user2489547 not directly. All file operations in the browser are sandboxed, ie. protected from the normal file system for obvious security reasons. But with a virtual file system you can make the experience appear as if the user used the files locally. –  Jun 25 '13 at 22:31
  • 1
    Plz tell me how any of this can be used to store files?!? Are you referring to the Filesystem API by any chance? – CyberFox Apr 10 '15 at 06:14
  • @CyberFox let me think back two years... yes, it was probably meant to be Filesystem API. Though, the name "File API" was used in the document (and still is) which is probably why the wrong link snuck in: see http://www.w3.org/TR/file-system-api/ (in any case, latter being discontinued) –  Apr 10 '15 at 06:46
  • Local storage will be deleted if the end user clear cookies and cache from browser settings – vicky Apr 16 '19 at 07:45
13

There is LargeLocalStorage which provides a cross-browser way to storage large amounts of data locally. You can store binary data or strings.

LargeLocalStorage uses the FilesystemAPI on Chrome, IndexedDB in IE and Firefox and WebSQL in Safari.

Matt Wonlaw
  • 12,146
  • 5
  • 42
  • 44
  • 1
    LargeLocalStorage requires the end-user to accept a prompt to allow "Store files on this device" (at least in Chrome) which is a big caveat in my opinion. – Micah Jan 27 '16 at 18:15
  • 6
    @Micah Umm, if a website could store 1GB+ of data without me giving explicit permission to do so, I'd be terrified. – Shien Apr 24 '17 at 12:19
  • 5
    Last activity in GitHub is 5 years ago. Does any one knows if this is still reliable? – Mark E Sep 15 '18 at 00:49
  • @Micah You do not want websites being able to allocate your entire disk without asking... – Cerin Jul 18 '20 at 12:53