Heap memory overflow when loading 500 Mb text file to JTable

Question

My task is to display data in text.DAT file in the form of tables.I am trying to do this with Java. I am using a Swing JTable for creating table but as the text file is 500 Mb, Java heap memory is overflowing. I understand this is due to every object that is being created for every row of table.

Is there any method to resolve this without increasing heap memory?

Relevant code is this:-

JFrame f = new JFrame();
        JTable tbl = new JTable();
        DefaultTableModel dtm = new DefaultTableModel(0, 0);
        line = br.readLine();
        String[] header = line.split(delimiter);
        dtm.setColumnIdentifiers(header);
        tbl.setModel(dtm);
        while ((line = br.readLine()) != null) { 
            String[] data = line.split(delimiter);
            dtm.addRow(data);
        }
        JScrollPane sp = new JScrollPane(tbl);
        f.add(sp);

with large data sets the only practical solution is: you can't load the whole file into memory at once. even if it did fit in memory, its going to take a long time to load. depending on what the data is, you might be able to dynamically load what you need. or you might have to load it into a traditional database and page through the rows you need — slipperyseal, Jul 08 '16 at 06:38
Once a file becomes that size it kind of screams 'use a DATABASE'. BTW - how many records are in that 500 Mb? — Andrew Thompson, Jul 08 '16 at 06:40
Side note: besides the loading-so-much-stuff-into-memory-could-probably-be-improved ... did you check the JVM startup options? You know, you can tell the JVM how much memory you allow it to consume ... — GhostCat, Jul 08 '16 at 06:44
Extend `AbstractTableModel`, as suggested [here](http://stackoverflow.com/a/25526869/230513). — trashgod, Jul 08 '16 at 06:47
@SlipperySeal In 2016, I would be careful about such statements. A lot of servers have half a TB, or even a full TB of memory nowadays. Memory price is going down exponentially; and people are building whole systems (like Alluxio) to do exactly that: **everything** in memory. Sure, a reasonable implementation of a in-memory-table might still require more thought; but don't just say: "never do this". — GhostCat, Jul 08 '16 at 06:48
If you want to load 500 mb into JTable, you have to [increase the heap size](http://stackoverflow.com/questions/1565388/increase-heap-size-in-java).Else you have to go with [pagination with lazy loading](http://stackoverflow.com/questions/1481138/how-to-provide-pagination-support-to-a-jtable-in-swing) — Madhan, Jul 08 '16 at 08:18

Heap memory overflow when loading 500 Mb text file to JTable

0 Answers0