Android fastest way of reading big arrays of variables

Question

In my application I need to find a faster way of loading big arrays of Integer and Float variables. What ave

Creating arrays directly using Java Code - This didn't work, because arrays used by my application are really big, 200 000+ float values, which cause java code too large error and I couldn't find any workaround for this.
Tried importing these arrays in project using xml which didn't work too, because file size is getting really big and I can't even build.
The thing which worked so far, I've added txt files in assets folder, read them using InputStream, parse them as floats / integers and create arrays dynamically, but this is too slow.

The reason which I have these big arrays is, because my project uses Vuforia for showing 3D Models, which needs to represent them as arrays of floats / integers (vertices,textcoords,normals,indices), but loading 12 Models is taking too much time.

I would love to get some advices / suggestions how can I speed up the loading process.

Thanks in advance!

have you tried to serialize/deserialize the data directly? (streams etc.) — Martin Frank, Sep 15 '14 at 10:31
then try to serialize it - make a 'createDataApp', create the binbary data and reLoad it within your original project - let me know if it works... (can't be much faster i think)... — Martin Frank, Sep 15 '14 at 10:58
just another question - you're having four 3D-Models wich are displayed in an android app - and each model has 16.000 Point3D ?! can't you simply use smaller Objects? (i know, this is not the solution for the question, but still looks very curious for me)... — Martin Frank, Sep 15 '14 at 11:03
No, I can't use smaller objects, because I am showing big buildings, which has complicated structure (museums, historical buildings and etc ), that's why the models are using big arrays of values. — hardartcore, Sep 15 '14 at 11:06
thank you for providing this information! it helps to categorize the problem! — Martin Frank, Sep 15 '14 at 11:08
is it possible to load the data in background? and distract the user until all data is loaded? — Martin Frank, Sep 15 '14 at 11:13
I didn't use serialization for this method, I am reading the file as string, than convert every string array using separator "," and convert items as float / integer. Using Serializable in Android as far as I know there is an issue loading big integer arrays. Loading data in background is possible, but for 12 models it will take much more time, which is not a good option for applications. — hardartcore, Sep 15 '14 at 11:15
The code that flankechen posted as final answer to save and load 300000 floats to file and back takes an eye whimper. http://stackoverflow.com/questions/22249483/write-read-float-array-in-java-fast-way You could maybe convert it to read from assets. Or otherwise copy from assets to file first. — greenapps, Sep 15 '14 at 11:25
well nice research done @greenapps - there are some good approaches listed in that article! — Martin Frank, Sep 15 '14 at 11:28

score 2 · Accepted Answer · edited May 23 '17 at 10:31

2

I have not benchmarked this, but (probably) the fastest simple way to read large numbers of integer and floating point values would be to use a DataInputStream and its various readXxx() methods.

If the data is homogeneous, then you can probably get faster transfer using FileChannel, ByteBuffer and (say) DoubleBuffer. This method is described here: "write/read float array in java fast way". (Kudos goes to @greenapps ...) However, it would be tricky to get that to work if the data is heterogeneous; i.e. a mixture of different primitive types.

edited May 23 '17 at 10:31

Community

1
1

answered Sep 15 '14 at 11:32

Stephen C

698,415
94
811
1,216

as mentioned in the article from @greenaps http://stackoverflow.com/questions/22249483/write-read-float-array-in-java-fast-way there are some other approaches described - i really recommend to read that article first... look at the accepted answer! – Martin Frank Sep 15 '14 at 11:35
do you think there is an even better way to access those data other than to directly read/write on file? just try to be openminded but i don't see any better way.... (nice answer, really) – Martin Frank Sep 15 '14 at 11:41
@MartinFrank - Yes, I agree he should read that article. However, I think the OP's problem requires reading *hereogeneous* data ... not just one big array of data that is all the same type. – Stephen C Sep 15 '14 at 11:41
@MartinFrank - Possibly one could map the file into memory. However, I have my doubts that it would make much difference. (Especially on an Android device). But like I said, I haven't benchmarked this ... or gone looking for benchmarks. – Stephen C Sep 15 '14 at 11:43
the data is never to be changed, so you can seperate different types of data - maybe a file for all int-values and another file for floats... but still: i think you should reduce the model, make a async task and load the data within an intro screen - another optimization would be to load only data required, so maybe just load the first model and in background loading the others...) – Martin Frank Sep 15 '14 at 11:44
i do think the same - i don't think it will make a big difference! +1 for all that information you shared and the input you added to this question! – Martin Frank Sep 15 '14 at 11:44
@MartinFrank - Obviously, the pragmatic solution is to make the data *appear* to load faster ... by doing something else to keep the user occupied :-) – Stephen C Sep 15 '14 at 11:45

score 1 · Answer 2 · answered Sep 15 '14 at 11:25

here are my loading/saving methods...

public Object loadData(String fileName, final Context con){

    assert fileName != null;
    assert con != null;

    Object data = null;
    try {
        final File dir = con.getFilesDir();
        final File file = new File(dir, fileName);
        if (!file.exists() ){
            return null;
        }
        ObjectInputStream ois = null;
        if (useCompression){
            ois = new ObjectInputStream(new GZIPInputStream(new FileInputStream(file) ) );          
        }else{
            ois = new ObjectInputStream(new FileInputStream(file) );
        }
        try {
            data = ois.readObject();
        } finally {
            ois.close();
        }           
    } catch (Exception e) {
        e.printStackTrace();
    } finally {

    }
    return data;
}

and for storing

public boolean saveData(String fileName, final Object data, final Context con){

    assert fileName != null;
    assert data != null;
    assert con != null;

    try {
        final File dir = con.getFilesDir();
        final File file = new File(dir, fileName);
        if (!file.exists() ){
            file.createNewFile();
        }

        ObjectOutputStream oos = null;
        if (useCompression){
            oos = new ObjectOutputStream(new GZIPOutputStream(new FileOutputStream(file)));
        }else{
            oos = new ObjectOutputStream(new FileOutputStream(file));
        }
        try {
            oos.writeObject(data);
        } finally {
            oos.close();
        }

    } catch (Exception e) {
        return false;
    }

    return true;
}

i guess it won't help you very much, but possibly make your data access easier and help you to get a start...

(yes i know - you don't have asserts in android, but simply ignore this)

right - not fast, but faster that reading Strings and convert them back again - as mentioned in that from greenapps there are some more good solutions to find! — Martin Frank, Sep 15 '14 at 11:29

Simon · Answer 3 · 2014-09-15T13:13:00.483

Another approach, not pretty but fast(ish).

I have several classes, as many as needed to stay under the 64K compilation unit issue which you will hit in addition to the "java code to large" error if you try to create one big static array.

public class structLatLongCellA {
    public static int[][] latLongCells = {
            {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 
            .....

public class structLatLongCellB {
    public static int[][] latLongCells = {
            {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 
            .....

Then in app initilisation:

public static int[][] latLongCells = new int[361][362];

....

int insertPoint = structLatLongCellA.latLongCells.length;
System.arraycopy(structLatLongCellA.lLatLongCells, 0, latLongCells, 0, structLatLongCellA.latLongCells.length);

System.arraycopy(structLatLongCellB.latLongCells, 0, latLongCells, insertPoint, structLatLongCellB.latLongCells.length);
insertPoint += structLatLongCellB.latLongCells.length;

System.arraycopy(structLatLongCellC.latLongCells, 0, latLongCells, insertPoint, structLatLongCellC.latLongCells.length);
insertPoint += structLatLongCellC.latLongCells.length;

etc...

They are all ints here but easy to convert to floats.

The classes containing the static arrays were generated using a quick GUI app I knocked up to read them from source files.

yes, that solution has been discussed as top bullet #1 ... i see you have a more sophisticated approach then Andoid-Developer i have just a slight improvment: make the final fields final ^^ but i must confess: i have used this approach as well when i was younger - i even wrote a code-generator for creating code snippets ^^ — Martin Frank, Sep 15 '14 at 12:15
@MartinFrank Hah! I just edited in the bit about the code generator. I have 16 of those puppies, each with thousands of values. I didn't hand crank them all ;) — Simon, Sep 15 '14 at 12:16
wow - i thought i was the only madMonkey doing things like that ^^ i like what you have proposed!!! — Martin Frank, Sep 15 '14 at 12:19

halxinate · Answer 4 · 2017-03-07T18:14:44.153

Just to add a bit of usage diversity, fast reading from resources:

void populateMyFloatsArray(Context c) {
    //long t0 = Calendar.getInstance().getTimeInMillis();
    try{
        InputStream inpStream = c.getResources().openRawResource(R.raw.floatsfile);
        ByteBuffer tempBBuf = ByteBuffer.allocate(mFloatArraySize*4); //size of floats in bytes
        tempBBuf.clear();
        inpStream.read(tempBBuf.array());
        tempBBuf.rewind();
        tempBBuf.asFloatBuffer().get(mFloatArray);
        inpStream.close();
    }
    catch (IOException ex) {
        //todo
    }
    //Log.d(TAG, "Popilated in "+(Calendar.getInstance().getTimeInMillis() - t0));
}

Android fastest way of reading big arrays of variables

4 Answers4

Linked