2

I need to open a csv file in more parts, each one by 5,000 samples and then plot them. To go back and forward on the signal each time I click a button I have to instantiate a new reader and than I skip to the point I need. My signal is big, is about 135,000 samples so csvReader.skip() method is very slow when I work with last samples. But to go back I can't delete lines, so each time my iterator needs to be re-instantiated. I noticed that skip uses a for loop? Is there a better way to overtake this problem? Here is my code:

    public void updateSign(int segmento) {
    Log.d("segmento", Integer.toString(segmento));
    //check if I am in the signal length
    if (segmento>0 && (float)(segmento-1)<=(float)TOTAL/normaLen)
    {
        try {
            reader = new CSVReader(new FileReader(new File(patty)));
        } catch (FileNotFoundException e) {
            e.printStackTrace();
        }

        List<Integer> sign = new ArrayList<>();
        //this is the point of the signal where i finish
        int len = segmento * normaLen;
        //check if i am at the end of the signal
        if (len >= TOTAL) {
            len = TOTAL;
            segmento=0;
            avanti.setValue(false);
            System.out.println(avanti.getValue());
        } else {
            lines = TOTAL - len;

            avanti.setValue(true);
            System.out.println(avanti.getValue());
        }
        //the int to i need to skip
        int skipper = (segmento-1)*normaLen;
        try {
            System.out.println("pre skip");
            reader.skip(skipper);
            System.out.println("post skip");
        } catch (IOException e) {
            e.printStackTrace();
        }
        //my iterator
        it = reader.iterator();
        System.out.println("iteratore fatto");
        //loop to build my mini-signal to plot
        //having only 5,000 sample it is fast enaugh
        for (int i = skipper; i < len-1; i++) {

            if (i>=(segmento-1)*normaLen) {

                sign.add(Integer.parseInt(it.next()[0]));

            }
            else
            {

                it.next();
                System.out.println("non ha funzionato lo skip");
            }
        }
        System.out.println("ciclo for: too much fatica?");
        //set sign to be plotted by my fragment
        liveSign.setValue(sign);
    }
}

Thanks in advance!

a_local_nobody
  • 7,947
  • 5
  • 29
  • 51
Sam
  • 51
  • 6
  • I see two options: Transform your CSV into a data structure supporting random access (e.g. ArrayList) or you remember where every file line starts and use RandomAccessFile, that's quite some effort though. CSVReader can't jump to a line because lines have different lengths and storing information would cost memory + traversing the whole file once. – peterulb Jul 01 '21 at 14:39
  • I thought to open my whole file into arraylist/array but it's too big, so it slows my app. Is there a way i can let it be faster? – Sam Jul 01 '21 at 14:54
  • How many entries do you have? From the code, you only read a single integer from position 0. Storing 135.000 Integers doesn't take much memory. Did you store the integers or the whole file? – peterulb Jul 01 '21 at 15:00
  • I open it and save in an ArrayList. It takes a while to open the file. By the way this is a easy case, because i can also have something like 5 millions of entries (signals are from ecg in polysomnography, so sampling at 125 Hz in a night gives a lot of samples) – Sam Jul 01 '21 at 20:58
  • You have a number of options. For example: 1. if you are not stuck to csv format, I'd suggest to use some binary encoding to produce a row with fixed length (8 bytes per long/double, some fixed amount of bytes for string value etc etc) and use RandomAccessFile. An example of such bin encoding can be found in jvm http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/00cd9dc3c2b5/src/share/classes/java/io/DataOutputStream.java, BTW. Since the size of each row is fixed (and the same), it's easy to calculate offset to read a row at any index 2. Use embedded storage like MapDB, H2 etc. – AnatolyG Jul 01 '21 at 22:42
  • I need so save in csv to share files in a human-readable way (and i think csv it's better than json in this case), but i can use MapDB to save a csv file? I use csv because I get via ble those files in another fragment and i save in a folder in the root named "Registrazioni". In this way i can easily share them and user can access to files (this application is developed to be used by researchers) – Sam Jul 02 '21 at 09:27

0 Answers0