Dynamically Created Html table rendering slow for huge records

Question

This is how I am creating my table & rendering into the aspx page after getting the data from from database by ajax call. I checked the stored procedure execution Time which is only 1.5 seconds where table in my page renders about in 7- 8 seconds. I need to optimize the speed. Any suggestion how I can do it?

$.ajax({
    method: 'POST',
    //async: false,
    data: json_data,
    dataType: 'json',
    url: json_url
}).success(function (response) {
    if (response.IsError) {
        showError('Did not receive values from XRM.');
    } else {
        var webResponse = response;
        var dataSet = webResponse["Orders"];
        var statuses = [];

        //console.log(dataSet);

        if (table_role == 'pending') {
            statuses = [
                {
                    'id': 39,
                    'name': 'Pending Scheduling',
                    'class': 'schedule',
                    'color': '#7C44BC',
                    'order': 0
                },
                {
                    'id': 40,
                    'name': 'Pending Screening',
                    'class': 'screening',
                    'color': '#8e8e00',
                    'order': 1
                },
                {
                    'id': 41,
                    'name': 'Pending Pre-Auth',
                    'class': 'preauth',
                    'color': '#cc8500',
                    'order': 2
                },
                {
                    'id': 42,
                    'name': 'Ready For Confirmation',
                    'class': 'confirm',
                    'color': '#cc0070',
                    'order': 3
                },
                {
                    'id': 1,
                    'name': 'Open',
                    'class': 'open',
                    'color': null,
                    'order': 4
                },
                {
                    'id': 2,
                    'name': 'New Order',
                    'class': 'neworder',
                    'color': null,
                    'order': 5
                },
                {
                    'id': 3,
                    'name': 'Changed',
                    'class': 'changed',
                    'color': '#cd4e3e',
                    'order': 6
                },
                {
                    'id': 5,
                    'name': 'Approved',
                    'class': 'approved',
                    'color': '#6aa745',
                    'order': 7
                },
                {
                    'id': 6,
                    'name': 'On Hold',
                    'class': 'onhold',
                    'color': '#009acd',
                    'order': 8
                },
                {
                    'id': 11,
                    'name': 'In Queue',
                    'class': 'queue',
                    'color': null,
                    'order': 9
                }
            ];

            // For customers, change labels
            if (is_customer) {
                statuses[5]['name'] = 'Submitted to SMS';
                statuses[9]['name'] = 'Approved';
            }
        } else if (table_role == 'preauth') {
            statuses = [
                {
                    'id': 29,
                    'name': 'More Info Needed',
                    'class': 'moreinfo',
                    'order': 0
                },
                {
                    'id': 43,
                    'name': 'Phone Order',
                    'class': 'phone',
                    'order': 1
                },
                {
                    'id': 30,
                    'name': 'Received Order',
                    'class': 'received',
                    'order': 2
                },
                {
                    'id': 31,
                    'name': 'Waiting on Documentation',
                    'class': 'waiting',
                    'order': 3
                },
                {
                    'id': 32,
                    'name': 'Courtesy Call Pending',
                    'class': 'courtesy-pending',
                    'order': 4
                },
                {
                    'id': 33,
                    'name': 'Courtesy Call Complete',
                    'class': 'courtesy-complete',
                    'order': 5
                },
                {
                    'id': 34,
                    'name': 'In Pre-Authorization',
                    'class': 'preauth',
                    'order': 6
                },
                {
                    'id': 35,
                    'name': 'Additional Docs Needed',
                    'class': 'addtldocs',
                    'order': 7
                },
                {
                    'id': 36,
                    'name': 'Pending Authorization',
                    'class': 'pending',
                    'order': 8
                }
            ];
        }

        // Remove any "sorting" backgrounds from headers
        $th.removeClass('otable__sort');

        // If no orders available
        if ($.isEmptyObject(dataSet)) {
            $table.find('.otable__empty').css('display', 'table-cell');
        } else {
            // Loop through orders
            $.each(dataSet, function (key, value) {
                if (table_role == 'pending') {
                    var id = (value['ID'] == null) ? '' : value['ID'],
                        date = (value['ID'] == null) ? '' : value['ExamDate'],
                        billing = (value['BillCode'] == null) ? '' : value['BillCode'],
                        scanner = (value['ScannerCode'] == null) ? '' : value['ScannerCode'],
                        units = (value['Unit'] == null) ? '' : value['Unit'],
                        by = (value['EmployeeName'] == null) ? '' : value['EmployeeName'],
                        status = value['StatusID'],
                        customer = value['Customer'],
                        location = '';

                    // Build location string
                    if (value['Facility']) {
                        location += value['Facility'] + '<br>';
                    }
                    if (value['City'] && value['State']) {
                        location += value['City'] + ', ' + value['State'];
                    }
                } else if (table_role == 'preauth') {
                    var id = (value['ID'] == null) ? '' : value['ID'],
                        poid = (value['POID'] == null) ? '' : value['POID'],
                        site = (value['Site'] == null) ? '' : value['Site'],
                        patient = (value['PatientName'] == null) ? '' : value['PatientName'],
                        insurance_co = (value['InsuranceCo'] == null) ? '' : value['InsuranceCo'],
                        received_date = (value['ReceivedDate'] == null) ? '' : value['ReceivedDate'],
                        exam_date = (value['ExamDate'] == null) ? '' : value['ExamDate'],
                        updated_date = (value['UpdatedDate'] == null) ? '' : value['UpdatedDate'],
                        status = value['StatusID'],
                        priority = (value['Priority'] == null) ? 0 : value['Priority'],
                        custid = (value['custID'] == null) ? '' : value['custID'];
                }

                // Find status from ID
                var object = $.grep(statuses, function (element, index) {
                    return element['id'] == status;
                });
                object = object[0];

                // Get status name
                var status_name = object['name'],
                    status_pos = object['order'],
                    status_class = object['class'],
                    datestring = '',
                    received_datestring = '',
                    exam_datestring = '',
                    updated_datestring = '';

                // Build date strings
                if (date) {
                    datestring = splitDate(date);
                }
                if (received_date) {
                    received_datestring = splitDate(received_date);
                }
                if (exam_date) {
                    exam_datestring = splitDate(exam_date);
                }
                if (updated_date) {
                    updated_datestring = splitDate(updated_date);
                }

                // Build Time strings
                if (updated_date) {
                    updated_timestring = splitTime(updated_date);
                }
                else {
                    updated_timestring = '';
                }

                // Convert priority for sorting
                var priority_order;
                if (priority) {
                    if (priority == 0) {
                        priority_order = 2;
                    } else if (priority == 5) {
                        priority_order = 1;
                    } else if (priority == 10) {
                        priority_order = 0;
                    }
                }

                // HTML for table row
                var $row;

                if (table_role == 'pending') {
                    if (!is_customer && (is_scheduler || is_retail)) {
                        $row = $('<tr class="otable__row otable__row--' + status_class + '">' +
                                  '<td class="otable__id"><a href=/Customer/FDGOrders.aspx?oid=' + id + '>' + id + '</a></td>' +
                                  '<td class="otable__date" data-sort-value="' + date + '"><a href=/Customer/FDGOrders.aspx?oid=' + id + '>' + datestring + '</a></td>' +
                                  '<td class="otable__billing"><a href=#rptPendingGroupByOrder onclick=showDialog(\'/CustomerProfile.aspx?cid=' + customer + '\')>' + billing + '</a></td>' +
                                  '<td class="otable__scanner"><a href=/Customer/FDGOrders.aspx?oid=' + id + '>' + scanner + '</a></td>' +
                                  '<td class="otable__loc"><a href=#rptPendingGroupByOrder onclick=showDialog(\'/CustomerProfile.aspx?cid=' + customer + '\')>' + location + '</a></td>' +
                                  '<td class="otable__units"><a href=/Customer/FDGOrders.aspx?oid=' + id + '>' + units + '</a></td>' +
                                  '<td class="otable__by"><a href=/Customer/FDGOrders.aspx?oid=' + id + '>' + by + '</a></td>' +
                                  '<td class="otable__status" data-status-id="' + status + '" data-sort-value="' + status_pos + ' ' + date + '"><a href=/Customer/FDGOrders.aspx?oid=' + id + '>' + status_name + '</a></td>' +
                                '</tr>');
                    } else if (!is_customer && (!is_scheduler || !is_retail)) {
                        $row = $('<tr class="otable__row otable__row--">' +
                                  '<td class="otable__id"><a href=/Customer/FDGOrders.aspx?oid=' + id + '>' + id + '</a></td>' +
                                  '<td class="otable__date" data-sort-value="' + date + '"><a href=/Customer/FDGOrders.aspx?oid=' + id + '>' + datestring + '</a></td>' +
                                  '<td class="otable__billing"><a href=#rptPendingGroupByOrder onclick=showDialog(\'/CustomerProfile.aspx?cid=' + customer + '\')>' + billing + '</a></td>' +
                                  '<td class="otable__scanner"><a href=/Customer/FDGOrders.aspx?oid=' + id + '>' + scanner + '</a></td>' +
                                  '<td class="otable__loc"><a href=#rptPendingGroupByOrder onclick=showDialog(\'/CustomerProfile.aspx?cid=' + customer + '\')>' + location + '</a></td>' +
                                  '<td class="otable__units"><a href=/Customer/FDGOrders.aspx?oid=' + id + '>' + units + '</a></td>' +
                                  '<td class="otable__by"><a href=/Customer/FDGOrders.aspx?oid=' + id + '>' + by + '</a></td>' +
                                  '<td class="otable__status" data-status-id="' + status + '" data-sort-value="' + status_pos + ' ' + date + '"><a href=/Customer/FDGOrders.aspx?oid=' + id + '>' + status_name + '</a></td>' +
                                '</tr>');
                    } else {
                        $row = $('<tr class="otable__row otable__row--customer otable__row--' + status_class + '">' +
                                  '<td class="otable__id"><a href=/Customer/FDGOrders.aspx?oid=' + id + '>' + id + '</a></td>' +
                                  '<td class="otable__date" data-sort-value="' + date + '"><a href=/Customer/FDGOrders.aspx?oid=' + id + '>' + datestring + '</a></td>' +
                                  '<td class="otable__units"><a href=/Customer/FDGOrders.aspx?oid=' + id + '>' + units + '</a></td>' +
                                  '<td class="otable__by"><a href=/Customer/FDGOrders.aspx?oid=' + id + '>' + by + '</a></td>' +
                                  '<td class="otable__status" data-status-id="' + status + '" data-sort-value="' + status_pos + ' ' + date + '"><a href=/Customer/FDGOrders.aspx?oid=' + id + '>' + status_name + '</a></td>' +
                                '</tr>');
                    }
                } else if (table_role == 'preauth') {
                    var insurance_sort_val = priority_order;
                    if (exam_date && exam_date.indexOf('0001-01-01') < 0) {
                        insurance_sort_val = priority_order + ' ' + exam_date;
                    } else if (received_date) {
                        // Force items with null exam date to bottom, then sort by received date
                        insurance_sort_val = priority_order + ' Z ' + received_date;
                    }

                    $row = $('<tr class="otable__row otable__row--' + status_class + '">' +
                                  '<td class="otable__id"><a href="">' + id + '</a></td>' +
                                  '<td class="otable__site"><a href="/Customer/FDGOrders.aspx?oid=' + id + '">' + site + '</a></td>' +
                                  '<td class="otable__by"><a href="">' + patient + '</a></td>' +
                                  '<td class="otable__insurance otable__priority--' + priority + '" data-sort-value="' + insurance_sort_val + '"><a href="">' + insurance_co + '</a></td>' +
                                  '<td class="otable__date" data-sort-value="' + received_date + '"><a href="">' + received_datestring + '</a></td>' +
                                  '<td class="otable__date" data-sort-value="' + exam_date + '"><a href="">' + exam_datestring + '</a></td>' +
                                  '<td class="otable__status" data-status-id="' + status + '" data-sort-value="' + status_pos + ' ' + exam_date + '"><a href="">' + status_name + '</a></td>' +
                                  '<td class="otable__date--wide" data-sort-value="' + updated_date + '"><a href="">' + updated_datestring + ' ' + updated_timestring + '</a></td>' +
                                '</tr>');

                    $row.find('a').on('click', function (e) {
                        if (!$(this).parent('.otable__site').length) {
                            e.preventDefault();

                            showDialog('/Customer/FDGPatientOrder.aspx?oid=' + id + '&poid=' + poid);
                        }
                        if ($(this).parent('.otable__site').length) {
                            e.preventDefault();

                            showDialog('/CustomerProfile.aspx?cid=' + custid);
                        }
                    });
                }

                // Append to table body
                $table.find('tbody').append($row);
            });
        }

        // Fire event
        //$table.trigger('all-rows-added');
        //$table.off('all-rows-added');

        // Table sorting functionality
        if (table_role == 'pending') {
            initTableSorting($table, 7);
        } else if (table_role == 'preauth') {
            initTableSorting($table, 3);
        }

        // Status filter functionality
        var $list = $table.find('.otable__status__list');
        // Clear previous filters
        $list.find('ul>li:not(:first)').remove();

        // Loop through statuses to see which are contained in the table
        $.each(statuses, function (key, value) {
            if ($table.find('tbody').has('.otable__status[data-status-id="' + value.id + '"]').length > 0) {
                var $li;

                if (table_role == 'pending') {
                    $li = $('<li>' +
                              '<button data-filter="' + value.id + '" data-color="' + value.color + '">' +
                                '<div class="otable__legend"></div>' +
                                value.name +
                              '</button>' +
                            '</li>');
                } else {
                    $li = $('<li>' +
                              '<button data-filter="' + value.id + '">' +
                                '<div class="otable__legend"></div>' +
                                value.name +
                              '</button>' +
                            '</li>');
                }

                // Add the list item
                $li = $li.appendTo($list.find('ul'));

                // Set the legend color
                var $btn = $li.find('button'),
                    filter = $btn.data('filter'),
                    color = $btn.data('color');

                if (color !== null) {
                    $li.find('.otable__legend').css('background', color);
                }
            }
        });

        $th.on('click', function (e) {
            // Check if main filter is being clicked, otherwise don't sort
            if ($list.has(e.target).length > 0) {
                var $target = $(e.target).closest('button'),
                    filter = $target.data('filter');

                // Sort all rows by ascending status
                $table.find('th').eq(7).stupidsort('asc');

                // Style actively selected item
                $list.find('li').removeClass('otable__status__checked');
                $target.parent('li').addClass('otable__status__checked');

                if (filter == 'all') {
                    $table.find('tbody > tr').css('display', 'table-row');
                } else {
                    $table.find('tbody > tr').css('display', 'none');
                    $table.find('tbody > tr').has('.otable__status[data-status-id="' + filter + '"]').css('display', 'table-row');
                }

                // Force hide list when option clicked
                $list.css({
                    'clip': 'rect(0 0 0 0)',
                    'opacity': 0
                });
                // Restore default behavior once mouse is moved
                $table.mousemove(function () {
                    $list.removeAttr('style');
                });

                // Prevent main sorting action
                return false;
            }
        });

        // Sticky table header
        $table.stickyTableHeaders();

        // Remove loading indicator
        $table.find('.otable__loading').css('display', 'none');
    }
}).fail(function (jqxhr, textStatus, error) {
    var err = textStatus + ", " + error;
    console.log("Failed to get XRM initialization values. Message: " + err);
});

Answer 1

We ran out in similar scenarios, where we had more than 30000 records to render on the same page, API was doing good but UI became pathetic in terms of performance. Chrome and Firefox survived somehow and rendered records with delays of 10 to 15 seconds, but IE was a nightmare, where the browser just froze everytime we accessed that page. The solution to this problem was gradual but elegant, we had to switch to the basics. I would break down a subset of my solution here

1. Replace $.each and _.each loops with native for(var i = 0; i < lenght; i += 1) loops. Doing this created a major boost in terms of performance. To prove this concept, at that time I created a jsperf (check here) performance benchmark, which showed huge performance gains. In your case, for example, replace $.each(dataSet, function (key, value) { and $.each(statuses, function (key, value) { with a native for loops. See this image comparing the loops

2. Reducing the number of JQuery selectors also proved to be beneficial, and instead of creating an in memory jquery elements like $('<div>bla bla bla</div>');, use string concatenation and at the end using for example $el.html() proved to be very efficient. In your case instead of doing things like $row = $('<tr class ... and then binding function like $row.find('a').on('click', you only need to create a long string and append everything in it, and at the end append that long string to the DOM. If you are just doing this to bind the click event to every single row (OMG), you can achieve the similar behavior by using JQuery on click on the parent with child as a filter, like $('table').on('click', 'tr', myClickFunc);. This will only bind a single click event instead of thousands of click events, resulting in a great performance gain.

3. By avoiding JQuery filtering methods like $.grep or $.map or $.filter etc, and using native loops, also resulted in drastic improvements.
4. By using jQuery element caching and avoiding frequent queries made to the DOM significant gains were achieved in different pages. I think you are doing caching fine, but try avoiding frequent DOM querying.
5. Avoiding multiple method calling from the loops was also eating up the overall loading time, so combining multiple method calls to a single method call and doing common things within improved performance. Like you can make splitDate take an array and return an array, instead of multiple method calls.
6. Finally the gem of all, and it's my favorite, I was using custom data grids to populate tabular data, but since the time I have encountered W2UI Grid, I have fallen in love with it. It can render millions of records at a single time, and it works by rendering only few records to display and as the scroll happens it renders other records and removes the records from DOM that are no longer visible. It's super easy to configure, so you can also use this grid instead of your custom table (if you are not using pagination).

So this is just a brief summary of the immediate measures that you can make to improve the load-time and rendering performance.

One more important tip is that you should also check the size of the response coming from server, if its big and is in mega-bytes, then you will have to optimize your API also, and only provide UI with the things that it needs, don't provide excessive things, as it slows up stuff at the UI.

Answer 2

Applying pagination is best solution, else you need to optimise your code (though the performance is negligible, but help to prevent browser crash if you have more data)

For example :

$.each(dataSet, function (key, value) {

      // Append to table body
      $table.find('tbody').append($row);
}

should be write as

 var $row='"";
 $.each(dataSet, function (key, value) {
     $row +="something";
  }
     // Append to table body
      $table.find('tbody').append($row);

You are using .append() inside $.each() this mean append() trigger for each loop. It good if you append it outside loop.

---

Instead of $table.find('tbody').append($row) you can cache .find() in a variable and then direct append it