Using hibernate as backing for large dataset in JTable

Question

Have an application that has a Jtable with the data held in RAM, the table can upto be 1,000,000 rows by 100 columns so this just uses too much memory. So now I'm moving to backing the JTable from a database, but I think getting directly from JTable to database is not going to help much because either all the data from the database is going to get loaded into memory when jtable initilized , or as the user scrolls down the tabel I would do select staements to get the next data, which would be too slow, and how would I handle sorting.

So I think the correct solution is to stick Hibernate between the JTable and Database, but I still can't quite see how this is going to help with a really large JTable.

Can anyone point me to a good example / have experience of using this with large dataset to keep memory usage down.

EDIT:Ive read some comments on other threads that a table with this much data should have filters so that only a subset of data is ever shown. I agree with that as a general principle and I will provide a filter HOWEVER only the user can decide how they want to filter it, and I still need to provide a 'ALL' option, and this is where it could all blow up.

I also remember something about having one table on top of another showing some kind of subset of the data that changes as you scroll down, but not seen a concrete example of this idea.

See also [Do you know of a sophisticated spread sheet like component for Swing](http://stackoverflow.com/questions/7663990/do-you-know-of-a-sophisticated-spread-sheet-like-component-for-swing). The [JPA](http://en.wikipedia.org/wiki/Java_Persistence_API) layer can make paging easier. — trashgod, Nov 16 '11 at 02:34

score 0 · Answer 1 · answered Nov 28 '11 at 15:07

0

Hibernate will buy you following: - querying agains database returnuing you list of dehydrated proxies ( say, they contain only primary key, this consume less memory) - transparent on demand loading of entries with separate queries.

However, when hibernate session is open too long it becomes slow (it uses inetrnal caches which get clogged)

You may also consider some document oriented No-SQL Database (like mongodb, or countless others), which will behave better and provide loading on demand with less overhead)

Rule of thimb in you case is to track whether table rows are visible or not, and discard unused data.

answered Nov 28 '11 at 15:07

Konstantin Pribluda

12,329
1
30
35

Thanks, good points. I dont think a document based db would be so good for me though because then I would have to load the whole document that represents a row rather than just the visible data. – Paul Taylor Nov 30 '11 at 11:15
mongodb allows loading of subset of documents, but in any case use of nosql database has to be well justified, and there is less theory and knowlege than for relational databases – Konstantin Pribluda Nov 30 '11 at 12:12

score 0 · Accepted Answer · edited Jan 18 '21 at 12:30

Normally for a function that displays large amount of records in the table format ,in order to consume less memory, the records will be shown page by page just like the Stackoverflow reputation league

Hibernate 's Criteria and Query API provides the following functions to do the pagination:

setMaxResults(int maxResults) to limit the number of the row to be retrieved
setFirstResult(int firstResult) to set the first row number to be retrieved (ie. 0 means first row)

In the JTable , you should has the variables for storing the number of records shown per page (pageNum) and the current page number (pageNum). Whenever user changes the pageNum by navigating the page forward or backward , you fetch the records for this page:

Query q = entityManager.createQuery("from someTable tbl order by tbl.id asc");
q.setFirstResult((pageNum -1)*pagesize).setMaxResults(pagesize);

Important Points:

Without the "order by" clause , the order of the fetched row is unpredictable . So , it is important that the query has to be sorted by the "order by clause" such that the displaying order of the records can be maintained when the page is browsed forward and backward .

You can refer to this (Use Google translate to translate to English) and its Github for an example about JTable pagination using Hibernate or JDBC.

I never thought about pagination on a JTable, yes I think that could work and sounds alot easier than the table as a view over a larger table idea and the link is useful to, thankyou. — Paul Taylor, Nov 30 '11 at 11:13

Using hibernate as backing for large dataset in JTable

2 Answers2