how to store large amount of data in javascript

Question

I have to handle large amount of data retrieved from server using ajax and store it in JavaScript. I am currently using an array to store all the data. this is how I get data and store it in the javascript:

var buffer=new Array();
//when each ajax returns 
buffer=buffer.concat(JSON.parse(ajaxReq.responseText));

there are two questions in my mind:

Is using an array to store all the data efficient, is there an overflow issue with 1 million rows of data??
How is the concat performance? how can it be optimized here?

thanks for any input.

This question might be useful: http://stackoverflow.com/questions/4833480/is-this-asking-too-much-of-a-browser — Matt Ball, Dec 18 '11 at 18:49
I would avoid it, since you don't have any idea what the various client machines will do or can handle. I can't think of a valid reason to have that much data *in the browser*. I'd setup a method to offload the processing to the server. — Jared Farrish, Dec 18 '11 at 18:54

score 10 · Answer 1 · answered Dec 18 '11 at 18:53

10

I know, that I risk a downvote, but: The only valid answer to "How do I store 1 million of rows in JavaScript" is "Don't."

answered Dec 18 '11 at 18:53

Eugen Rieck

64,175
10
70
92

thanks, I see your point. but ajax retrieving takes time. users will observe the break points when serving data. I want to create good user experience. – bingjie2680 Dec 18 '11 at 18:54
I would agree. @bingjie2680, you're better off finding a way to cache and call what you need that is as efficient as possible. – Jared Farrish Dec 18 '11 at 18:56
2

well i wouldn't store a million of anything in javascript arrays . so i'd def. work on solving this another way – Scott Evernden Dec 18 '11 at 18:58
haha,,really interesting,,,how about 100,000 rows of strings. that won't be a problem. I guess. – bingjie2680 Dec 18 '11 at 18:59
@bingjie2680 - Why do you need that much data on the client? What does it represent? – Jared Farrish Dec 18 '11 at 19:02

score 4 · Accepted Answer · answered Dec 18 '11 at 19:03

It looks like you are dumping your whole database on the client at every pageload, so that at the particular page load he may or may not have to wait for quick ajax responses. This is honestly by far the most insane thing I have ever heard of, and hopefully you can see it as such as well.

Even if your users wouldn't have to wait for small ajax calls that don't really take long (look at google autosuggest), having to have them wait for 1 million database rows to transfer and load in memory on every pageload is even worse user experience.

What you are doing is only feasible if your data fits in localStorage (5MB), which it probably doesn't.

Not to mention the clients that can barely handle the operations on that data, operations that degrade the entire experience, instead of (intermittently) when requesting data from the server. — Jared Farrish, Dec 18 '11 at 19:05

how to store large amount of data in javascript

2 Answers2