2

I would like to display to user a table which contains ~500,000 rows. The table should be calculated based on a file (i.e. the table is known at the beginning, and does not change). I guess that building the HTML text of this table is not a good idea in terms of memory performance. So how can I build such table ?

I would like user to be able to scroll the table using a vertical scroll bar. Is it possible to build on the fly only the visible part of the table ? I'm afraid to see delays because of this.

Is it a better idea to use server side programming rather than Javascript ?

I'm not restricted to any server side programming language, so any advise would be appreciated !

Misha Moroshko
  • 166,356
  • 226
  • 505
  • 746
  • A half-million rows doesn't sound very useable - usually when I have datasets this large I try and find ways to group and summarize the data (there is NO WAY I'm reading through 500,000 of data), usually I do that in Excel or some other app. How does the user find anything in your dataset? Could you send them a CSV file, or some sort of summary report? – FrustratedWithFormsDesigner Apr 26 '10 at 13:57
  • 3
    see this similar question for some good solutions - http://stackoverflow.com/questions/2402953/javascript-data-grid-for-millions-of-rows – Anurag Apr 26 '10 at 17:06

6 Answers6

7

Send the first 250-ish rows to the user, as he scrolls down, perhaps past row 200, fetch the next 250 rows, append to the table and so on.

This is a (ui) design pattern known as "Infinite scroll".

PatrikAkerstrand
  • 45,315
  • 11
  • 79
  • 94
4

Displaying 500,000 rows all at once to a user is a bad idea. Consider other options:

  • allow user to download file as CSV
  • show paginated (still not very useful)
  • provide filtering mechanisms (by date etc.) to allow the user to only see the data they need

If the users really needs to see all that data at once, then viewing it in the browser is one of the worst ways to do that - they should be using a tool made specifically for viewing data, like Tableau.

D'Arcy Rittich
  • 167,292
  • 40
  • 290
  • 283
  • 1
    +1, 500000 rows is just too many for a human to process. At best you're just letting them delude themselves into thinking they've seen a decent sample of the data. Another +1 for mentioning that paging is not a good answer either – Joel Coehoorn Apr 26 '10 at 14:02
  • 1
    To back this up: if the user cares enough about the data to glance at each row for just one second, to will take them over 5 days to get to the bottom. It is quite likely they are interested in particular rows (then add pagination and a filtering mechanism, think Google search results) or aggregate data (think Wolfram Alpha graphs and figures). – Douglas Apr 26 '10 at 14:02
  • I will not argue - it definitely seems silly to present that much information. However, the questioner did not ask whether or not it was a good idea. In fact, we have customers that will not even consider a solution that will not support at least 100000 rows. That is how their business has worked for decades, and they are not willing to change that. – moshefnord Jun 20 '18 at 19:31
3

Classic example of where to use ajax.

Use javascript in combination with a server side script.

zaf
  • 22,776
  • 12
  • 65
  • 95
2

This really is a case for server-side pagination, I would say, maybe in combination with Ajax. A HTML table with 500.000 rows is going to crash a lot of browsers.

You're not specifying which server side technology you work with. If you update your question, I'm sure people will be able to give you some pointers.

Pekka
  • 442,112
  • 142
  • 972
  • 1,088
1

Try this: Clusterize.js Tiny plugin to display large data sets easily https://clusterize.js.org/

aliawadh980
  • 468
  • 6
  • 5
0

Is it possible to build on the fly only the visible part of the table ?

If you build a "fake" scroll (e.g. jquery slider) you can retrieve parts of the table instead of the whole.

Rui Carneiro
  • 5,595
  • 5
  • 33
  • 39