3

I've got a .txt with 1 Billion digits of pi. I read in the file as a String but i get an OutOfMemoryError. It worked with 1 MIllion digits tho. I save the String as a char[] array. Is it possible to somehow stream the .txt when i cycle through the whole array? I simply need a way to have access to all the 1 Billion digits as an array.

Steakie
  • 57
  • 8
  • 3
    What kind of processing do you need to do with the read data ? – Arnaud Feb 20 '17 at 16:43
  • I am cycling through the entire char array to look for a certain number...so just a for loop through the array. Thats it – Steakie Feb 20 '17 at 17:10
  • What do you mean by "look for a certain number"? The number is pi. Do you mean "look for a specific sequence of digits"? If so, how many digits? – Klitos Kyriacou Feb 20 '17 at 17:28
  • yes, a sequence of digits. well about 4-9 or something like that. I need a way to have access to all the 1 billion digits as an array – Steakie Feb 20 '17 at 17:32
  • Why do you “need a way to have access to all the 1 billion digits as an array”? If you have a string, you can iterate over the `char`s, just as you can with an array. – Holger Mar 10 '17 at 16:22

4 Answers4

4

There is BufferedInputStream since java 1 or FileReader with 

public int read(char cbuf[], int offset, int length) throws IOException

I suggest you start from there

Pavel Niedoba
  • 1,554
  • 2
  • 19
  • 36
  • This is probably the solution for my problem. got problems implementing tho.. ._. Ill keep trying – Steakie Feb 20 '17 at 17:47
  • Where’s the relevance of `BufferedInputStream` here? – Holger Mar 10 '17 at 16:08
  • You right, I should have mentioned FileInputStream, which has buffer on it's own, so theres no there's no need for BufferedInputStream. This might be usefull here: http://stackoverflow.com/questions/5155226/fileinputstream-vs-filereader – Pavel Niedoba Mar 10 '17 at 16:52
0

It is not only possible: it is both highly recommended and done in practice. What is usually done is to reuse the same kind of interfaces as Java libraries (InputStream, etc).

In this case, this could mean a new IntegerInputStream class that outputs the digits as a stream. This class can itself forward calls to FileInputStream. Internally, you can use char[] arrays to store the buffer and improve performance, or have calls directed via BufferedInputStream as Pavel suggests, but it is best to isolate the consumer from the internal buffer management and keep the appropriate level of abstraction to the use case (decimals of pi).

Ghislain Fourny
  • 6,971
  • 1
  • 30
  • 37
0

You can open the file with a FileInputStream, and read it byte[] per byte[] to avoid the OOMError.

Jeremy Grand
  • 2,300
  • 12
  • 18
0

According to doc

You should be able to get a String of length Integer.MAX_VALUE (always 2147483647 (231 - 1) by the Java specification, the maximum size of an array, which the String class uses for internal storage) or half your maximum heap size (since each character is two bytes), whichever is smaller

This is why you get the Exception,

if you don't really need the whole 1B chars. you can try using buffer which doesn't load the whole thing into memory.

BufferedReader br = new BufferedReader(new FileReader(new File("path to file")));
char[] data=new char[1000000] ;//however many chars you want;
int i=0;
while ((c = br.read()) != -1 && i<data.length) {
    data[i++]= c;
}

br.close();
nafas
  • 5,283
  • 3
  • 29
  • 57
  • Why on earth are you reading up to 1000000 single characters instead of calling just `read(char[])` on the reader? Then, you don’t need a `BufferedReader` to work around the excessive `read()` invocations… – Holger Mar 10 '17 at 16:20
  • @Holger I guess that's another way of doing it. In terms of complexity both solution should be about the same. but yeah I agree that one liner solution is better – nafas Mar 10 '17 at 16:46