1

When run with big datasets, my program produces large HTML files (i.e. sometimes in the range of 200mb). (These result files would only be viewed locally and not via the internet).

BUT browsers are unable to open large files (see this stackoverflow answer) - i.e. the browser simply crashes before loading the full file.

This results file is literally just a table (smaller version seen here) that can be sorted. These large files sometimes have more than 30,000 rows...

I would use Tablesorter with the Pager plugin, but the actual HTML file size is not reduced despite the table being split into pages. So this wouldn't solve the problem...

Thus I was planning on simply storing the table as a Json and then create smaller (paginated) tables via Javascript. I would also try to implement sorting using JS.

I have a feeling that there is probably a better way to do this (Js/Jquery will probably be to slow to sort 30,000+ rows of data etc.).

So are there any existing frameworks that I can use here?

Community
  • 1
  • 1
Ismail Moghul
  • 2,864
  • 2
  • 22
  • 28
  • Personally I would be leery of adding / relying on a framework... it took some time to run and complete (with jquery / javascript), but I had a similar issue with a recent project and I decided to write some javascript that would parse the html source and then use ajax to post to write to database record for each data element I wanted to store . I also thought about a flat json file but didnt like the idea of such a large flat file and knew that ultimately the mysql database would be really flexible and let me sleep well at night. – tamak May 21 '15 at 01:22
  • You source file is well formatted, and it can be easily converted from HTML to shorter version (like in JSON format) with about 50% of original size, plus some elements can be removed (like – agershun May 21 '15 at 04:06

1 Answers1

2

I think the answer you are seeking is actually to not do what you are attempting.

Displaying 200mb of pretty much anything in the browser is likely not going to be useful to the end user - even if it does load, scrolling and re-painting will make it painful to use.

I'd recommend either a pre-filter that lets/forces the user filter down the results to a more meaningful data set... and/or cap the results to 50/100/250? with the ability to page through the remaining.

For the scenarios where the user really needs all the data (or the client insists ;-) I'd recommend an option to download the data in CSV or a similar format that they can import into their favorite text editor/spreadsheet (or even database) application.

scunliffe
  • 62,582
  • 25
  • 126
  • 161