How to store an ArrayList in a file?

Question

I have a class which represents an ArrayList stored in a file, because I need an ArrayList with multiple gigabytes of data in it which is obviously too large to be stored in memory. The data is represented by a class called Field and the function Field.parse() is just for converting the Field into a String and the other way.

The Field class stores a list of (strange) chess pieces and their coordinates.

My class is working fine, but it takes a long time to add an element to the file and I need my program to run as fast as possible. Does anyone know a more efficient/faster way of doing things?

Also, I am not allowed to use external libraries/apis. Please keep that in mind.

This is the class which is responsible for storing Field objects in a temp file:

private File file;
private BufferedReader reader;
private BufferedWriter writer;

public FieldSaver() {
    try {
        file = File.createTempFile("chess-moves-", ".temp");
        System.out.println(file.getAbsolutePath());
    } catch (IOException e) {
        e.printStackTrace();
    }
}

public void add(Field field) {
    try {
        File temp = File.createTempFile("chess-moves-", ".temp");
        writer = new BufferedWriter(new FileWriter(temp));
        reader = new BufferedReader(new FileReader(file));
        String line;

        while((line = reader.readLine()) != null ) {
            writer.write(line);
            writer.newLine();
        }

        reader.close();
        writer.write(field.parse());
        writer.close();
        file.delete();
        file = new File(temp.getAbsolutePath());
    } catch (IOException e) {
        e.printStackTrace();
    }
}

public Field get(int n) {
    try {
        reader = new BufferedReader(new FileReader(file));
        for (int i = 0; i < n; i++) {
            reader.readLine();
        }
        String line = reader.readLine();
        reader.close();
        return Field.parse(line);
    } catch (IOException e) {
        e.printStackTrace();
    }
    return null;
}

And this is the Field class:

private WildBoar wildBoar;
private HuntingDog[] huntingDogs;
private Hunter hunter;

private int size;

@Override
public String toString() {
    String result = "Wildschwein: " + wildBoar.toString();
    for (HuntingDog dog : huntingDogs) {
        result += "; Hund: " + dog.toString();
    }
    return result + "; Jäger: " + hunter.toString();
}

@Override
public boolean equals(Object obj) {
    if (obj instanceof Field) {
        Field field = (Field) obj;
        HuntingDog[] dogs = field.getHuntingDogs();
        return wildBoar.equals(field.getWildBoar()) && hunter.equals(field.getHunter()) && huntingDogs[0].equals(dogs[0]) && huntingDogs[1].equals(dogs[1]) && huntingDogs[2].equals(dogs[2]);
    }
    return false;
}

public Field(int size, WildBoar wildBoar, HuntingDog[] huntingDogs, Hunter hunter) {
    this.size = size;
    this.wildBoar = wildBoar;
    this.huntingDogs = huntingDogs;
    this.hunter = hunter;
}

public WildBoar getWildBoar() {
    return wildBoar;
}

public HuntingDog[] getHuntingDogs() {
    return huntingDogs;
}

public Hunter getHunter() {
    return hunter;
}

public int getSize() {
    return size;
}

public static Field parse(String s) {
    String[] arr = s.split(",");
    WildBoar boar = WildBoar.parse(arr[0]);
    Hunter hunter = Hunter.parse(arr[1]);
    HuntingDog[] dogs = new HuntingDog[arr.length - 2];
    for(int i = 2; i < arr.length; i++) {
        dogs[i - 2] = HuntingDog.parse(arr[i]);
    }
    return new Field(8, boar, dogs, hunter);
}

public String parse() {
    String result = wildBoar.parse() + "," + hunter.parse();
    for(HuntingDog dog : huntingDogs) {
        result += "," + dog.parse();
    }
    return result;
}

Answer 1

Here's an MCVE to do what you want, based on the information you provided.

You can run it and see that it can save a Field to the file and get a Field by index very quickly.

The Fields are constant length, so you can get a Field by index by going to byte offset of index times field length in bytes. This would be significantly more difficult if the field were not constant length.

import java.io.Closeable;
import java.io.DataInput;
import java.io.DataOutput;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.RandomAccessFile;

public class FieldSaver implements Closeable {

    public static void main(String[] args) throws IOException {
        File f = File.createTempFile("chess-moves-", ".temp");
        try (FieldSaver test = new FieldSaver(f);) {
            for (byte i = 0; i < 100; i++) {
                test.add(new Field(8, new WildBoar(i, i), new Hunter(i, i), new HuntingDog[] {
                        new HuntingDog(i, i),
                        new HuntingDog(i, i),
                        new HuntingDog(i, i) }));
            }

            // Get a few Fields by index
            System.out.println(test.get(0));
            System.out.println(test.get(50));
            System.out.println(test.get(99));

            // EOF exception, there is no Field 100
            // System.out.println(test.get(100));
        }
    }

    private final RandomAccessFile data;

    public FieldSaver(File f) throws FileNotFoundException {
        data = new RandomAccessFile(f, "rw");
    }

    public void add(Field field) throws IOException {
        data.seek(data.length());
        field.write(data);
    }


    public Field get(int index) throws IOException {
        data.seek(index * Field.STORAGE_LENGTH_BYTES);
        return Field.read(data);
    }

    public void close() throws IOException { data.close(); }


    static abstract class Piece {
        protected byte xPos;
        protected byte yPos;

        public Piece(DataInput data) throws IOException {
            xPos = data.readByte();
            yPos = data.readByte();
        }

        public Piece(byte xPos, byte yPos) {
            this.xPos = xPos;
            this.yPos = yPos;
        }

        public void write(DataOutput data) throws IOException {
            data.writeByte(xPos);
            data.writeByte(yPos);
        }

        public String toString() { return "[" + xPos + ", " + yPos + "]"; }
    }

    static class Hunter extends Piece {
        public Hunter(byte xPos, byte yPos) { super(xPos, yPos); }
        public Hunter(DataInput data) throws IOException { super(data); }
    }

    static class HuntingDog extends Piece {
        public HuntingDog(byte xPos, byte yPos) { super(xPos, yPos); }
        public HuntingDog(DataInput data) throws IOException { super(data); }
    }

    static class WildBoar extends Piece {
        public WildBoar(byte xPos, byte yPos) { super(xPos, yPos); }
        public WildBoar(DataInput data) throws IOException { super(data); }
    }

    static class Field {
        // size of boar + hunter + 3 dogs
        public static final int STORAGE_LENGTH_BYTES = 2 + 2 + (3 * 2);

        private int size;
        private WildBoar boar;
        private Hunter hunter;
        private final HuntingDog[] dogs;

        public Field(int size, WildBoar wildBoar, Hunter hunter, HuntingDog[] huntingDogs) {
            this.size = size;
            this.boar = wildBoar;
            this.hunter = hunter;
            this.dogs = huntingDogs;
        }

        public String toString() {
            String result = "Wildschwein: " + boar.toString();
            for (HuntingDog dog : dogs) {
                result += "; Hund: " + dog.toString();
            }
            return result + "; Jäger: " + hunter.toString();
        }

        public static Field read(DataInput data) throws IOException {
            WildBoar boar = new WildBoar(data);
            Hunter hunter = new Hunter(data);
            HuntingDog[] dogs = new HuntingDog[3];
            for (int i = 0; i < 3; i++) {
                dogs[i] = new HuntingDog(data);
            }
            return new Field(8, boar, hunter, dogs);
        }

        public void write(DataOutput data) throws IOException {
            boar.write(data);
            hunter.write(data);
            for (HuntingDog dog : dogs) {
                dog.write(data);
            }
        }
    }
}

Answer 2

Use a Map implementation like Cache from ehcache. This library will optimize for you so you don't have to handle writing and reading to disk and manage when to keep it in memory or on disk. You can just use it as a normal map. You probably want a map instead of a list for faster lookup so the library can optimize even more for you.

http://www.ehcache.org/

CacheManager cacheManager = CacheManagerBuilder.newCacheManagerBuilder()
      .withCache("preConfigured",
           CacheConfigurationBuilder.newCacheConfigurationBuilder(Long.class, String.class,
                                          ResourcePoolsBuilder.heap(100))
           .build())
      .build(true);

  Cache<Long, String> preConfigured
      = cacheManager.getCache("preConfigured", Long.class, String.class);