Qt widget for displaying large amount of data rows

Question

I am trying to display a large amount of columnar records in a scrollable view using Qt (5.1).

The number of rows I would like to be able to browse can vary from 100 million to 1 Billion, say.

The QTableWidget with a custom model works a few million rows, but the QTableWidget allocates data for each row because you can re-size the rows height, and so it must store data for this, which can use megabytes or even gigabytes of memory with 100M rows.

I do not require the re-sizeable rows functionality just a multi-column list would be ideal, but QTreeCtrl doesnt seem to work with many rows, and QList seems to only support single columns.

Should I be implementing a custom widget from QAbstractItemView for this purpose?

For those familiar with wxwidgets/wepython it can be done like this, and works well with billions of rows:

import wx

class VirtualList(wx.ListCtrl):

    def __init__(self, parent, id, pos, size, flags):
        wx.ListCtrl.__init__(self, parent, id, pos, size, flags)

    def OnGetItemText(self, item, column):
        return "Row %d, Column %d" % (item, column)

You don't need to allocate all 100 million or billion rows at once, allocate 100 or 200 and when user scrolls near the end of allocated 100 or 200 records, allocate new records, and free the ones, that are far away from active rows, that are viewed right now — Shf, Jan 16 '14 at 10:30
There is `QTreeView` that can handle multirow display and work faster than `QTableView`. It has `uniformRowHeights` property that holds whether all items in the treeview have the same height, which enables the view to do some optimizations. Also its allowed to load chunks of data. Not all data at once. — Pie_Jesu, Jan 16 '14 at 10:31
Can you destribe usecase where user can view (and really need to view) 1 billion rows in one time? You should think about data fetching. Because your can't run your program on x86 platforms. — Dmitry Sazonov, Jan 16 '14 at 15:08
There is a need to join several multi gigabyte text files together. This data coulbe be loaded into a database, but we want to avoid a long preprocess step if possible. It's probably true that these files won't be as big as I've said but — Spacen Jasset, Jan 17 '14 at 19:08
@Shf yes, that's a valid idea. I don't know if that would be easier than writing our own widget or not though. — Spacen Jasset, Jan 17 '14 at 19:10

score 2 · Accepted Answer · answered Jan 16 '14 at 13:14

If it's tabular data I would use a table. I would write a custom QTableView with a custom QAbstractTableModel. In the QTableView you have control of all visible items. I would put some kind of check or variable shared between QTableView and it's model to control how much data should be shown. By overriding the data method in QAbstractTableModel you can dictate how much data to show. You can also mess with the QTableView's scroll bar to make things look and feel nicer.

If you don't really care about editing or looks, you could use a simple QTextEdit/QTextBrowser that is set to read only.

Note: A QTableWidget with a custom model is somewhat pointless. The main difference between a QTableWidget and a QTableView is that the QTableWidget has it's own premade model.

Hi HashSplat. I have difficult to implement with your ideal. Could you please introduce more or give us a sample project for refering? Thanks! — ngvntoan, Apr 26 '16 at 03:15

score 1 · Answer 2 · edited May 23 '17 at 12:24

Qt views are known to be slow on large data amounts. QTableWidget or QStandardItemModel cannot be used here because they create new object for each table item. That causes large overhead. You should start from implementing your own fast QAbstractItemModel subclass and showing it in a standard QTableView. It's possible that it will work acceptably fast. You can also try to set fixed row height for the view to speed things up.

If it won't work, you can try to implement your own table view. Also you can use Graphics View Framework as described in this answer.

Qt widget for displaying large amount of data rows

2 Answers2

Linked