Loading a large number of rows into a SlickGrid using async JQuery requests and RavenDB

Question

I need to load > 10k rows of data into a SlickGrid

I believe SlickGrid is designed for just this sort of use based on the following official example which loads 500,000 client-side rows: Optimizing The DataView

Since I cannot violate RavenDBs 'safety first' rule that stipulates returning less than 1024 rows per session query, I have used RavenDB's skip-take mechanism which allows the web-client to gather all 10K rows from the server in bites size chunks of 256 rows per http request.

My question is this: What is the best way to feed these rows into the SlickGrid as they are returned from the server? Does SlickGrid actively support the asynchronous loading of chunked up data-subsets like this?

I am looking for:

A SlickGrid callback that allows me to issue the next server request.
An optimised way of appending the resulting rows into the DataView in one go e.g.

grid.appendRows(rowSubset)

SlickGrid RemoteModel vs. Dataview Model suggests SlickGrid is designed to work either with large client-side datasets or in the more grid-usual page-on-demand server model. I want to use the former client-side approach as it results in a superior User Experience, imho.

Disclaimer: I understand that I am loading a large number of rows and that the usual approach would be to simply request each page from the server as required. No matter how laudable this approach may be, this is not what I am trying to achieve. Instead I am deliberately attempting to load the entire dataset from the server in such a way that it does not inconvenience the user.

Thanks

What purpose does presenting all 10k rows serve? If it's a presentation thing then surely displaying small sets would suffice. On the other hand, if it's for generating file contents then presenting in a webpages isn't likely to be the best solution. — CBusBus, Sep 30 '12 at 12:44
SlickGrid is clever enough to manage the optimised display of the correct rows as it adds / removes browser dom nodes under the hood. Caching all the rows client side means that I can take advantage of all of of that goodness plus slickgrid's filtering, sorting etc on the fast client, rather than being chatty with the server. I add the first 256 rows to slick grid, chunk-download the remaining, then update the grid. The user gets an instantly usable grid and about 5 secs later they have the complete dataset to work with. — biofractal, Sep 30 '12 at 13:33
I'd reconsider your approach. It kind of defeats the purpose of an ajax-based solution, given that the same could be achieved on page load but I'd say that a) You should never send 10k records to a user that needs only 10 (or x <= 10k), b)While it's probably safe to say that SlickGrid have done a good job of creating "all that goodness" but with a dataset of this size you'll never match server side (native language calls) database interaction, serving small - relevant to the user - packages. — CBusBus, Sep 30 '12 at 14:12
I hear your advice, normally I would agree, however in this instance my question remains: Given that SlickGrid is designed to work with large in-memory datasets (a premise derived from [this example](http://mleibman.github.com/SlickGrid/examples/example-optimizing-dataview.html) where 500,000 rows are loaded into client-memory), does it provide an intrinsic mechanism to help build such an in-memory dataset via some async batch-load process? — biofractal, Sep 30 '12 at 17:56

score 2 · Accepted Answer · edited May 23 '17 at 12:18

OK, SlickGrid does not have any built in functionality so we have to roll it ourselves. Here is one answer from stackoverflow:

SlickGrid AJAX data

And now here is my solution, which works a treat and I think is really easy. The technology stack for this is:

RavenDB
NancyFX
CoffeeScript
SlickGrid

The CoffeeScript

loadData=(grid)->
    take = 600
    loadingGlyph = $ "<img/>", {class:"loading", style:"float:right;margin-top:5px", src:"/Content/img/ajax-loader.gif", alt:"loading", width:"16", height:"16"}
    $(".slick-pager-status").after loadingGlyph
    for skip in [0..viewdata.UserCount] by take
        $.getJSON("/users/#{skip}/#{take}")
        .done (users) =>
            grid.addRows users
            $(".loading").hide() if users.length < take

$ -> 
    columns = [..your columns..]
    grid = [..your slick-grid..]
    loadData grid

First I stick in a fancy loading-glyph graphic to show the grid is busy. Note that I intend loading batches of data asynchronously. This will put the grid in a usable state very quickly (after the first batch returns) but the user needs to be told that the data is still downloading in the background.

To batch load the data I loop in chunks whose size is determined by the value of take and terminating at the viewData.UserCount value. This has been pre-populated when the page was requested and is the total number of items available (or the maximum you wish to load).

Finally I have made use of the addRows method I added to my slick-grid wrapper (not shown). This allows for the fast addition of row-batches and it looks like this:

Extend Slick-Grid

addRows:(rows)->
    dataView.beginUpdate()
    dataView.addItem row for row in rows 
    dataView.endUpdate()

For the sake of completeness, here is the rest of process, including the RavenDB service call that does the batch data retrieval.

NancyFX Module

Get["/users/{skip}/{take}"] = parameters =>
{
    var skip = (int)parameters.skip;
    var take = (int)parameters.take;
    var data = UserService.GetForGrid(Db, skip, take);
    return Response.AsJson(data);
};

RavenDB Service

public static List<UserGridDTO> GetForGrid(IDocumentSession db, int skip=0, int take=512)
{
    return db.Query<User>()
        .OrderBy(u=>u.Id)
        .Skip(skip)
        .Take(take)
        .ToList()
        .Select(user =>
                        new UserGridDTO
                        {
                                id = user.Id,
                                Email = user.Email,
                                Name = user.DisplayName,
                        })
                        .ToList();
}

score 0 · Answer 2 · answered Oct 01 '12 at 02:57

0

SlickGrid does not include any built-in functionality to load or save the data (not counting the sample slick.remotemodel.js that just serves as an example of how one might use the SlickGrid API). That is up to the developer implementing SlickGrid.

answered Oct 01 '12 at 02:57

Tin

9,082
2
34
32

Loading a large number of rows into a SlickGrid using async JQuery requests and RavenDB

2 Answers2