0

I have non-sensitive data in an object array:

 data = [ { prop1: value, prop2: value}, ... ]

which contains about 200 objects and will probably grow to the low thousands. Reads are much more than writes, and I am using some Javascript to sort/extract from the array as needed. While I'd appreciate to automate sorting and filtering of data, using a database even for a few thousand records seems a bit of an overkill.

At what size would an object array begin to severely impact performance, and when would a database make more sense?

EDIT: My dilemma is, can I safely load the entire array client-side and let the browser do the heavy-lifting, saving me the trouble of managing a database for a simple set of data and operations?

vantage5353
  • 545
  • 1
  • 4
  • 13
  • The moment when you say "I only have a few thousand elements in the array" is the moment when you need to find a different way of storing your data. – lonesomeday May 24 '14 at 12:55
  • 2
    @lonesomeday I don't agree necessarily. This Q is likely OT however given that it's completely subjective and dependent a lot on the data and its usage. – Joe May 24 '14 at 13:05
  • I agree with Joe. The specifics of how many items are appropriate for an array depends upon a whole lot of factors such as: how big is the data and can your bandwidth and target system thrive if you load it all, what types of queries do you need to do on it, how many of the items in the data set will you actually use, what type of performance is needed, how much RAM you can afford to use, etc... This question can't be answered without fully understanding all those aspects. – jfriend00 May 24 '14 at 13:13
  • there is no definite answer. Just start with an array, and when you face problems, act accordingly. – vkurchatkin May 24 '14 at 13:15
  • to clarify a bit, since this is not sensitive data, I'm wondering whether it's OK to dump the array on the user's client and let client-side logic do any operations instead of the server. i.e., is a big array going to weigh on the average user's system? – vantage5353 May 24 '14 at 14:42

1 Answers1

0

There are obvious physical constraints on the length of an Array, because of the indexing constraints of 32 bit hardware systems (as far as I'm aware the JavaScript specification doesn't currently encourage utilising 64 bit architecture when working with Arrays).

According to this answer:

the longest possible array could have 232 - 1 = 4,294,967,295 = 4.29 billion elements

That's obviously a lot of records.

The burning question is: will it impact the user if I use large arrays? It clearly depends on the hardware running the code. And if you can't be sure on the hardware (which you can't when serving JavaScript to browsers on the web), you should assume that the technology is the lowest common denominator (in a derogatory sense, not a mathematical sense).

That means researching your demographic, looking into consumer statistics, browser statistics, testing on various architectures/setups, and then still worrying about the question after it's all finished...

Or.

Use a data-store that does one thing and one thing well.

I think, deep down, you know what you need to do. It's an interesting question, but a red-herring in terms of achieving peace-of-mind.

Community
  • 1
  • 1
shennan
  • 10,798
  • 5
  • 44
  • 79
  • given that a lot of users access the web through mobile devices, can I safely assume that the average Android user's browser would have no trouble loading a thousand objects in memory? – vantage5353 May 24 '14 at 14:48
  • I was kind of hoping that my last paragraph would spark a eureka moment. My main point was to highlight the fact that sending large amounts of data (I assume much of it superfluous) to the client is somewhat lackluster. I can't actually think of a scenario where a user would need that much data all on one page. Historically, whenever an application has needed that much embellishment, it would have been packaged as a local application. Can you justify the need for this so-called 'array'? – shennan May 25 '14 at 00:04
  • The justification is I wouldn't have to manage a database and add an extra layer of complexity; an object array is much simpler, and I'd use JavaScript to manipulate the data view instead of traditional database functions for filtering and sorting. 100 objects cost around 14KB. At 1000, the figure would go to 140KB. This sounds like a lot to load onto the client so I'm wondering at what size it would begin to impact performance. – vantage5353 May 25 '14 at 08:43
  • To be clear, my point is that performance will vary on different machines, so it is an unknowable variable. I understand *why* you want to do it, but I'm still struggling to understand *what* kind of application requires the client to download such a large array... – shennan May 26 '14 at 10:03