Fastest way for Indexing Large Text File

Question

I want to index large text file about 1 gb, so I store in another file new line positions, to access the file later by RandomAccessFile, here is my code

               while (true) {
                raf.seek(currentPos);
                byte[] bytes = new byte[1000000];
                raf.read(bytes, 0, bytes.length);
                for (int i = 0; i < bytes.length; i++) {
                    if (bytes[i] == 10) {
                        rafw.writeInt(currentPos + i);
                    }
                }
                currentPos = currentPos + sizeOfPacket;
                if (currentPos > raf.length()) {
                    sizeOfPacket = (int) raf.length() - currentPos;
                } else if (currentPos == raf.length()) {
                    break;
                }
                bytesCounter = bytesCounter + 1000000;
                //Log.d("DicData", "Percentage=" + currentPos + " " + raf.length());
                int progress = (int) (bytesCounter * 100.0 / folderSize + 0.5);
                iDicIndexingListener.onTotalIndexingProgress(progress < 100 ? progress : 100);

Here I check all file bytes for value (10) which means "\n" new line, My big problem is: this proccess takes too much time, about 15 minutes, My question: Is there a way faster than this? Thanks

maybe this can help? https://stackoverflow.com/questions/16159183/java-concurrent-reads-on-an-inputstream — fantaghirocco, Jan 06 '21 at 00:54

score 0 · Answer 1 · answered Jan 06 '21 at 01:59

Writing and reading a 1 GB file with 1 Mio. lines takes < 10 secs each on my machine. I suspect your performance bottleneck is somewhere else.

public class Test {
  public static void main(String[] args) throws Exception {
    File file = new File("test.txt");

    System.out.println("writing 1 GB file with 1 mio. lines...");
    try(FileOutputStream fos = new FileOutputStream(file)) {
      for(int i = 0; i < 1024 * 1024; i++) {
        fos.write(new byte[1023]);
        fos.write(10);
        if(i % 1024 == 0) {
          System.out.println(i / 1024 + " MB...");
        }
      }
    }
    System.out.println("done.");

    System.out.println("reading line positions...");
    List<Long> lineStartPositions = new ArrayList<>();
    lineStartPositions.add(0L);
    long positionInFile = -1;
    byte[] buffer = new byte[1024 * 1024];
    try(FileInputStream fis = new FileInputStream(file)) {
      long read = 0;
      while((read = fis.read(buffer)) != -1) {
        System.out.println("processing MB: " + positionInFile / 1024 / 1024);
        for(int i = 0; i < read; i++) {
          positionInFile++;
          if(buffer[i] == 10) {
            lineStartPositions.add(positionInFile + 1);
          }
        }
      }

      // remove the last line index in case the last byte of the file was a newline
      if(lineStartPositions.get(lineStartPositions.size() - 1) >= file.length()) {
        lineStartPositions.remove(lineStartPositions.size() - 1);
      }
    }

    System.out.println("found: " + lineStartPositions.size());
    System.out.println("expected: " + 1024 * 1024);
  }
}

score -1 · Answer 2 · answered Jan 06 '21 at 00:58

-1

You can use the lib Scanner to pre-read file to index new line pos:

        File file = null;
        //init file here
        int newLineIndex = 0;
        int lineSepLength = System.lineSeparator().length(); // \r, \n or \r\n depend on OS
        Scanner sc = new Scanner(file);
        while(sc.hasNextLine()) {
            newLineIndex = sc.nextLine().length() + lineSepLength;
            //persist newLineIndex
        }

answered Jan 06 '21 at 00:58

Minh Đức Tạ

84
3

`"string".getBytes().length != "string".length()` – fantaghirocco Jan 06 '21 at 01:05

Fastest way for Indexing Large Text File

2 Answers2