HTML table containing big data from elasticsearch

Question

I have an elastic search index with about 2 million records which I would like to view in a HTML table. The issue i`m facing is that I don't want to pollute the browser's memory with all that data, instead I would like to do the following:

1. Once the table is loaded, make a query to elastic search to find out how many records are in total (this number will be used to approximate the size of the scrollbar)
2. Make another query to obtain the first 1000 records and display them in the table.
3. Modify the table's scrollbar to give the impression to the user that all the records are loaded
4. Once the user starts scrolling down, add more records to the table and remove old records (to save memory)

I need advice regarding how to design this system and possible implementation solutions.

Thank you

Answer 1

elasticsearch offers the "from" and "size" options for all its queries. By default you get the first 10 results for your queries so you can set the "from" field to specify where in the results you want to start from and the "size" field to specify how many documents you wish to retrieve. Something along the lines of:

{
    from: 1,
    size: 1000,
    query: {
      "match": {
         "text":{
            "query": "search_text"
          }
       }
    }
}

Will give you the first 1000 documents which match "search_text" in the text field of the documents. You can use a variable instead of 1 in the query and keep updating the variable and calling the query to do what you described.

UPDATE

for the frontend check out this question: jQuery load more data on scroll

You have to do something like that and put in your changing query into the function.