Fastest way to read a large number from text file

Question

So I have a very large number (15 million digits) stored in a text file and I'm using this method to read the number

 BufferedReader Br1 = null;
    StringBuilder Final = new StringBuilder("");

    System.out.println("Loading......");

    Br1 = new BufferedReader (new FileReader("NumberFind.txt"));
    String Line = Br1.readLine();
    while(Line != null) {
        Final.append(Line);
        Line = Br1.readLine();
    }
    sum1 = new BigInteger(Final.toString());
    Br1.close();
            System.out.println("Loaded");

This works but it takes about 45 minutes to load the whole number, is there a faster way to load this?

score 0 · Accepted Answer · answered May 16 '17 at 18:16

0

Your file is only 14.3 megabytes if it's just got that one number in it. I don't know what quirk of BufferedReader, BigInteger, etc. causes a 45 minute load, but it's probably just that line-reading loop. You should be able to read the whole file into memory in seconds, not minutes.

Try reading the entire file (or possibly just the section containing the number) into a string without BufferedReader. See FileReader.readAsBinaryString() for accomplishing that.

After you have the number in memory as a string, you should be able to construct a new BigInteger by passing the string param to it like you are already doing in your above code sample.

If this doesn't solve everything, and you want more insights, I would recommend narrowing down where that 45 minutes of delay is happening. I'm guessing it's in your line-reading loop, but I could be wrong. If you are in some environment, e.g. embedded device, with unusual constraints on CPU, disk read time, etc. that may be a factor.

answered May 16 '17 at 18:16

Erik Hermansen

2,200
3
21
41

Ok here's the code I figured out, is this right? BufferedReader Br = new BufferedReader (new FileReader("test2.txt")); String Line = Br.readLine(); BigInteger number = new BigInteger(Line); System.out.println("Done"); – Samantha Clark May 16 '17 at 19:34
It seems pretty quick to turn the file into a string but it takes a long time to turn that string into a BigInteger – Samantha Clark May 16 '17 at 19:44
Then it seems that file-reading code isn't causing the delay. You might want to verify the line you are reading has expected length (e.g. console.log(Line.length) ) of around 15 million. But assuming your input from the file contains expected digits and it's not taking long, e.g. minutes, to complete, then the culprit is likely BigInteger(line). But what to do about that? Hmm. Not sure. – Erik Hermansen May 16 '17 at 20:12
I think I would verify for sure it is that particular line that takes forever. And if so, try out some other big integer libraries, e.g. strint, to see if another implementation parses in the very large string value faster. – Erik Hermansen May 16 '17 at 20:16
I double checked the length and it's 17,112,422 , also I've never used a BigInteger library before, is that a website or something I can download on eclipse? – Samantha Clark May 16 '17 at 20:27
Oh, you are in Java! I thought you were in javascript. Maybe add a "java" tag to your original question, and you may get some more eyeballs. In JS, there is no built-in BigInteger, which is why I assumed you were using a library/package. – Erik Hermansen May 16 '17 at 20:32
Oh lol I'm new to this site and I thought it was only for java – Samantha Clark May 16 '17 at 20:34

score 0 · Answer 2 · answered May 16 '17 at 21:20

One could create the StringBuilder with an initial capacity to the file size (maybe minus the line endings).

One can save considerable memory using a BigInteger instead of a StringBuilder to accumulate the result. Whether that indeed is faster I do not know.

Path path = Paths.get("NumberFind.txt");
BigInteger n = Files.lines(path)
        .reduce(// 1. the start value
                BigDecimal.ZERO,

                // 2. the accumulator adding the next line
                (num, line) ->
                num.scaleByPowerOfTen(line.length()).add(new BigDecimal(line)),

                // 3. The combiner for a parallel stream (irrelevant)
                (num1, num2) ->
                num1.scaleByPowerOfTen(num2.toString().length()).add(num2))
        .toBigInteger();

Reading a line it converts it to a BigDecimal. The prior lines were accumulated to one BigDecimal, that then has to be multiplicated by 10ⁿ where n is the line length.

I use BigDecimal as that has a nice scaleByPowerOfTen. At the end the BigDecimal is converted to a BigInteger.

This solution could be slower. I am curious.

Fastest way to read a large number from text file

2 Answers2