Writing a very large BigInteger to a .txt file quickly

Question

I am in need of a quicker way to output a 1.6 million digit BigInteger to a file. I am using this code right now.

FileStream fs1 = new FileStream("C:\\Output\\Final\\BigInteger.txt",  FileMode.OpenOrCreate, FileAccess.Write);
StreamWriter writer = new StreamWriter(fs1);
writer.WriteLine(big);
writer.Close();

This takes about 5 minutes to output the 1.6 Million digit number. Is there any way to speed this up?

You might want to [read this question](http://stackoverflow.com/questions/13154304/fastest-way-to-convert-a-biginteger-to-a-decimal-base-10-string) — Chris Haas, Oct 27 '15 at 20:27
Have you tried using [File.WriteAllText](https://msdn.microsoft.com/en-us/library/ms143375%28v=vs.110%29.aspx) to see if it made any difference? — John Odom, Oct 27 '15 at 20:27
Would you try converting the BigInteger to a string first, then writing it to the file? I'm curious where the bulk of the latency is. Then let us know how much time each step takes: conversion and writing to disk. — Lorek, Oct 27 '15 at 20:46
@M.kazemAkhgary That question has nothing to do with `decimal` the type, it's talking about converting a `BigInteger` to a base-10 number (decimal notation). — 31eee384, Oct 27 '15 at 20:46
Your problem is not write part. I have tested the code. The problem is `ToString` part that has to first convert biginteger into string. with debugger use `string result = big.ToString();`. for me it takes about 5 mins. actually `Write` part finishes in less than second. — M.kazem Akhgary, Oct 27 '15 at 20:57
In the question I linked to above, the accepted answer made a really great point that I agree with: _Spending extra effort to make the output "human readable" is a waste of time. No human is going to be able to make sense out of 450,000 digits regardless of whether they are base 10, base 16, base 2, or anything else._ — Chris Haas, Oct 27 '15 at 21:11
Without re-writing BigIntiger and rewriting your own [`FormatBigIntegar`](http://referencesource.microsoft.com/#System.Numerics/System/Numerics/BigNumber.cs,f8d6a47783f05d0c) I am not sure you can get much faster. — Scott Chamberlain, Oct 27 '15 at 21:20

score 6 · Answer 1 · answered Oct 27 '15 at 22:23

This is a very silly question with no practical usage. But always important to know exactly where the processor cycles are being used. You are complaining about writing to a file taking too long. Well, are you sure it is actually the file that's slow? Or is it BigInteger.ToString() that's slow?

Best way to find out is just doing the file writing so you can isolate the problem:

using System;
using System.Text;
using System.IO;

class Program {
    static void Main(string[] args) {
        var big = new StringBuilder(1600 * 1000);
        big.Append('0', big.Capacity);
        var sw = System.Diagnostics.Stopwatch.StartNew();
        // Your code here
        FileStream fs1 = new FileStream("BigInteger.txt",  FileMode.OpenOrCreate, FileAccess.Write);
        StreamWriter writer = new StreamWriter(fs1);
        writer.WriteLine(big);
        writer.Close();
        // End of your code
        sw.Stop();
        Console.WriteLine("That took {0} milliseconds", sw.ElapsedMilliseconds);
        Console.ReadLine();
    }
}

Output on my machine:

That took 13 milliseconds

Writing a file is very fast, the file system cache makes it a memory-to-memory copy. The operating system lazily writes it to the disk, long after your program stopped running. It is only ever not capable of hiding the slow disk write speed when you write more data than can fit in the cache. You are not close to that on any modern machine, they have lots of RAM and can easily store a gigabyte. 1.6 megabytes is dental floss.

So you know it is actually BigInteger.ToString() that is so slow. Yes, it is. It stores that Big Mother in base 2, makes the math as quick as possible. Processors like base 2, they count with 2 fingers. Converting to the human format, base 10, that's expensive. It requires division, one of the most expensive thing you can ever do with a processor.

Or rather, the base conversion is `O(N^2)` unless you use some fancy and non-trivial algorithms. — Mysticial, Oct 27 '15 at 22:35
That comment was posted by the guy that hold the world record for producing the largest number of digits in pi. [5 trillion of them](http://www.numberworld.org/misc_runs/pi-5t/announce_en.html). In decimal. As far as I could tell from his efforts, keeping the disk drives running long enough to store those digits was the major obstacle. YMMV. — Hans Passant, Oct 27 '15 at 23:15
As a dentist, I like the "dental floss" comment. I must use it myself, when appropriate. ;-) — Rudy Velthuis, Oct 28 '15 at 15:30

score 0 · Answer 2 · answered Oct 27 '15 at 20:53

You can try to convert the number to string, split into several parts and write part by part :

using (StreamWriter outfile = new StreamWriter("C:\\Output\\Final\\BigInteger.txt"))
    {      
         foreach var part in numberParts
         {
             outfile.Write(part);
         }
    }

Writing a very large BigInteger to a .txt file quickly

2 Answers2