0

I'm creating a table for getting web reports, everything is perfect, but the ram memory. With big data, the memory usage gets a real problem.

Any ideas on how could I store all data without using the memory? I saw that LocalStorage or IndexedDB might be an option. But I really don't know how to use them and if the performance is good.

My table just shows (elements) that can be seen on the screen, the rest keeps in the array, and all the time I scroll the rows (elements) are updated. So I need an option that is fast to access and get data from. And for sure, I need to sort, filter, and do stuff like that.

Any ideas?

Leonardo Buzinskas
  • 33
  • 1
  • 5
  • 2
    what sort of data, and how many elements? (localStorage will be no use because it only stores strings, and there are built-in limits as to how much you can actually store in localStorage). – Alnitak Oct 24 '18 at 12:43
  • 4
    ^ Have you actually had RAM issues or are you just hypothesizing? IndexedDB could be a good solution, but [JavaScript arrays](https://stackoverflow.com/a/6155063/2535504) are generally pretty rugged. – sheng Oct 24 '18 at 12:44
  • *"everything is perfect, but the ram memory"* how do you determine that? *"With big data, the memory usage gets a real problem."* again, what exactly is here the problem/symptom? – Thomas Oct 24 '18 at 12:51
  • In the array I store strings, numbers, bools and dates – Leonardo Buzinskas Oct 24 '18 at 13:04
  • About the RAM issue, it's real, I'm testing with and array[500.000][30] , the ram consumption gets to 2GB – Leonardo Buzinskas Oct 24 '18 at 13:06
  • You can store dates efficiently in a `Float64Array`. You could pack multiple boolean flags into a `Uint32Array`. Strings - umm, yeah, no idea just now! :p – Alnitak Oct 24 '18 at 13:10
  • Alnitak, I'm not sure how could I use that, my array is like [row][cells], and each cell has a different type. Anyway, most of the fields are strings and numbers. Would be IndexedDB a good idea? – Leonardo Buzinskas Oct 24 '18 at 13:26
  • It's a long time since I've used IndexedDB, and I don't recall its limitations. Numbers can also be stored in a `Float64Array` (or smaller, if their range is limited). As for your array being `[rows][cells]`, well that would just have to change. – Alnitak Oct 24 '18 at 13:36
  • I would be inclined to use a server to store the data, and use html/javascript to display chunks of it, having all the heavy lifting of sorting/filtering/etc done by the server database. – James Oct 24 '18 at 14:08
  • how crucial are the updates on the screen? I would think you are confusing DOM manipulations with the background (storing, calculating, etc) operations. You said you update when you scroll which calls event listeners etc... Get rid of your visual updates and compare. – Matt Oct 24 '18 at 16:21
  • The data is stored in a server, this will be the result from the queries. But I want a table to deal with this information on the client side, so I can manipulate it the way I want without a new query. The memory gets the same just by creating the array. Try to create an array of [500.000][50] with 15 chrs in each item and check what happens you your RAM...haha Sorting and filtering a huge array is veeery fast, like 1 sec. If I work with chunks of information I'll not have enough speed to update DOM while scrolling. – Leonardo Buzinskas Oct 24 '18 at 16:34

0 Answers0