3

I'm writing a little tail application using Eclipse RCP (4). In my main window I wish to display the contents of a file. My first idea was to use an SWT List component but my problem is when I open a really big file (more than 100000 lines) the List cannot cope with it. Do you have any idea which component should I use for this?

I want to filter and/or highlight the records depending on some criteria but I don't want to modify so it is read only.

One more thing: since I cannot load an arbitrarily large (say 10 GB) file into memory I just read the position of new line characters into a List and I load the concrete lines into a String only when they are going to be displayed. So I need some component which renders the lines only when they become visible.

Charles
  • 50,943
  • 13
  • 104
  • 142
Adam Arold
  • 29,285
  • 22
  • 112
  • 207
  • 1
    This might help: http://stackoverflow.com/a/2773698/1257372 – Nick Wilson Oct 30 '12 at 10:18
  • Will it work if I'm going to filter on the table? For example lazy content providers won't work. – Adam Arold Oct 30 '12 at 10:49
  • It will help, particularly for the cases where you're still displaying a large number of items, but it will need to be used in combination with some kind of indexing service to support filtering. Apache Lucene? – Nick Wilson Oct 30 '12 at 11:06
  • @NickWilson I used your link and my preliminary implementation is a complete success. It took 7436 ms to index an 1GB file and after that several us to load up the data. My question is how can I filter this table? – Adam Arold Oct 30 '12 at 12:40
  • The key is to avoid searching/loading items from the original file. If you used something like Lucene to create a separate searchable index and use that to map the filter string to a list of file entry references/line numbers. Once you have that list you can use what you already have and just load the subset of items that are visible in the table. – Nick Wilson Oct 30 '12 at 16:04

1 Answers1

0

You can use SWT.VIRTUAL, so that you can achieve lazy loading of the data. Here is an example with Table and SWT.VIRTUAL.

However 10GB is alot of information. I would somehow split the file into many chunks and load specific parts of it.

[EDIT]: Something that you should know:

As of 3.1 the TableViewer now supports the SWT.VIRTUAL flag. If the underlying table is SWT.VIRTUAL, the content provider may implement ILazyContentProvider instead of IStructuredContentProvider . Note that in this case, the viewer does not support sorting or filtering. Also note that in this case, the Widget based APIs may return null if the element is not specified or not created yet.

Users of SWT.VIRTUAL should also avoid using getItems() from the Table within the TreeViewer as this does not necessarily generate a callback for the TreeViewer to populate the items. It also has the side effect of creating all of the items thereby eliminating the performance improvements of SWT.VIRTUAL.

I've tried to implement TreeViewer with about 700k items. The filter/search function was very slow (I've used FilteredTree).

Community
  • 1
  • 1
aphex
  • 3,372
  • 2
  • 28
  • 56
  • It is unlikely that an user will try to open a 10 GB file I just want my program to be able to handle that kind of situation. – Adam Arold Oct 30 '12 at 11:59