What are the best ways to store this data?

Question

I'm creating a Desktop Java application which needs to store (what could become) reasonably large amounts of data. I initially thought-up my own format:

id;name;value,value,value...
id;name;value,value,value...

To be stored as plain text and read/written randomly. My initial thoughts were to use RandomAccessFile; however, I encountered the problem of data being overwritten, when inserting in arbitrary positions.

Using the method of shifting the existing data along, storing it temporarily, restoring it, etc, seems like a bad solution, particularly because I intend to read/write quite often, so it would become expensive. I have read that this is a limitation of the underlying OS rather than Java itself.

What would be the best way to neatly store such data for frequent access and writes? Using SQLite crossed my mind. I've also thought of perhaps creating one file per id/name, since data values will only be appended to the end for each, hence nothing will be overwritten.

This is what databases were made for. Trying to handle file reading/writing will be very slow and you will have huge performance hits if you need to search a file. If you store these in a database you can access by id/name and not have to worry about file access. — jzworkman, Mar 16 '12 at 17:55
Define "quite often". Without knowing more about the data (is it relational? Flat? Only and ever key/value(s)? How much data? How will you need to search? Etc.) it's difficult to provide meaningful advice. Looks more like a key/value DB more than file(s) at this point. — Dave Newton, Mar 16 '12 at 17:57

score 2 · Accepted Answer · answered Mar 16 '12 at 18:10

2

It is easy to use a database for solving this. You do not require a full fledged database. Better to use an embedded database.

Use a jvm embedded database such as apache derby or berkely-db

answered Mar 16 '12 at 18:10

ring bearer

20,383
7
59
72

This is the idea I've had in mind, which is why I thought SQLite, so it's self-contained. I'll have a look, thanks :) – c24w Mar 16 '12 at 18:13
1

@c24w, imho, apache derby or HSQL db are better fit for java. Also, see [this discussion](http://stackoverflow.com/questions/41233/java-and-sqlite) – ring bearer Mar 19 '12 at 00:52

score 1 · Answer 2 · answered Mar 16 '12 at 18:57

Using a database is going to be the most suggested method. I'll show you an optional way incase you or future readers want to easily store medium size data for extreamly fast access. Adjust exception handling as necessary.

public class PersistentMap extends HashMap<String, Object> implements Serializable {
    private static final long serialVersionUID = 0x42424242;
    private File file;
    protected PersistentMap(File file) throws Exception {
        this.file = file;
        if (file==null || file.isDirectory() ) {
            throw new RuntimeException("ProgramCheck: "+file);
        }
        if (!file.exists()) {
            if (file.getParent()!=null) {
                new File(file.getParent()).mkdirs();
            }
            save();
        }
        restore();
    }
    public void save() throws Exception {
        FileOutputStream fos = new FileOutputStream(file);
        ObjectOutputStream oos = new ObjectOutputStream(fos);
        oos.writeObject(this);
        oos.close();
        fos.close();
    }
    public void restore() throws Exception {
        FileInputStream fis = new FileInputStream(file);
        ObjectInputStream ois = new ObjectInputStream(fis);
        super.putAll((Map<String,Object>)ois.readObject());
        ois.close();
        fis.close();
    }
}

Usage:

PersistentMap map = new PersistentMap(new File("myData\\data"));
System.out.println(" "+map);
map.put("yoyo","mama");
map.save();
System.out.println(" "+map);

Output 1st run:

 {}
 {yoyo=mama}

Output Nth run:

 {yoyo=mama}
 {yoyo=mama}

Thanks for your contribution. I've done something similar to this before, but for data which did not require frequent read/writes - would it not be expensive calling save after many small data additions (to prevent data loss)? — c24w, Mar 16 '12 at 19:00
In terms of CPU% or File I/O I think the hit is minimal. The real hit is all data is in memory ( which is why it is so fast ). Add a JVM shutdown hook to do a final save and you will need less checkpoint saves. — Java42, Mar 16 '12 at 19:07

score 0 · Answer 3 · edited Dec 31 '12 at 16:08

0

I would try using a database. Databases are created for storing data. They're really easy to use and they are very simple. I like to use MySQL. It's really good for databases.

edited Dec 31 '12 at 16:08

Oliver Spryn

16,871
33
101
195

answered Mar 16 '12 at 18:06

John

3,769
6
30
49

Per the original question, he was already considering a database (SQLite). Standalone MySQL is almost certainly too heavyweight for a relatively simple schema like this in a desktop context (better to use an embedded DB like @ring bearer suggested). – mjk Dec 31 '12 at 16:10

What are the best ways to store this data?

3 Answers3