file size too big for java

Question

My program reads in text files of various sizes. It then takes the numbers from the text file and creates array lists based on the numbers. The largest file i plan on using is 286,040 KB. When I run my program and it reads the file, my program stops working.

How do I know what size is the maximum my java program can handle? Is there a way of computing how big a size a file my java program can handle?

Also, what are the best suggestions for making my program be able to hold array lists of such a large size? I have heard of hash tables, however; I have not been able to full understand the concept.

Per Request, I'm adding how I upload the file:

String name = getFileName();
Scanner scanDaily = new Scanner(new File(name));

public static String getFileName()
{ //getFileName
    Scanner getName = new Scanner (System.in);
    System.out.println("Please input File Name");
    String fileName = getName.nextLine();
    return fileName;    
}  //getFileName

Update : Thank you to those who responded, its been very helpful

New problem

I now want to read the numbers from the file into an arraylist

          String name = getFileName();
    FileReader f= new FileReader(new File(name));
        BufferedReader bf=new BufferedReader(f);
        Scanner sc=new Scanner(bf);

    ArrayList<Double> ID = new ArrayList<Double>();
    ArrayList<Double> Contract = new ArrayList<Double>();
    ArrayList<Double> Date = new ArrayList<Double>();
    ArrayList<Double> Open = new ArrayList<Double>();
    ArrayList<Double> High = new ArrayList<Double>();
    ArrayList<Double> Low = new ArrayList<Double>();
    ArrayList<Double> Close = new ArrayList<Double>();
    ArrayList<Double> Volume = new ArrayList<Double>();

    int rows = 8;
    int counter1 = 0;

    //Update code to prompt user for file
    ArrayList<Double> list = new ArrayList<Double>();

    while (scanDaily.hasNext())
    { //while
        double value = scanDaily.nextDouble();
        DecimalFormat df = new DecimalFormat("#.#####");
        df.format(value);
        list.add(value);
    }  //while

before I used a scanner to read my file, and that scanner was named scandaily. Now that I have a filereader and a buffered reader, which one do i use to go through my txt file?

You should adapt your program to be able to take actions as you are reading the file instead of loading the whole file before doing something. — Arnaud Denoyelle, Jun 11 '13 at 12:29

score 6 · Answer 1 · answered Jun 11 '13 at 12:32

6

Do you really need to have the whole file in memory ?

For simple treatment, you should consider using BufferedReader, especially BufferedReader.readLine

You can take actions for each line of the file so you don't need to load the whole file anymore.

answered Jun 11 '13 at 12:32

Arnaud Denoyelle

29,980
16
92
148

score 2 · Answer 2 · edited May 23 '17 at 11:57

2

You can increase the max memory size of the JVM via something like:

$ java -Xmx1024m ....

but you may want to be more efficient in how you read and store this data. e.g. are you reading the complete file into memory and then parsing/converting to a list of ints ? If so, why not simply read and parse each line without holding the complete file in memory.

e.g. see this answer for more info.

edited May 23 '17 at 11:57

Community

1
1

answered Jun 11 '13 at 12:27

Brian Agnew

268,207
37
334
440

What do you mean? It sounds like a good idea, hoewver; I do not understand how to implement it – Danny Jun 11 '13 at 12:30
@user2474459 what method are you using to load the file? can you add that part of code to your question? it matters.... – pinkpanther Jun 11 '13 at 12:32
@user2474459 You have to include the code where you are actually loading the file....) are you using `Scanner(new File(fileName))` sort of thing...? – pinkpanther Jun 11 '13 at 12:51
@user2474459 now check my answer – pinkpanther Jun 11 '13 at 13:01

score 1 · Accepted Answer · edited May 23 '17 at 11:49

When I run my program and it reads the file, my program stops working.

I thought the problem would be this and confirmed after you have added the code. I have faced the similar problem before.

Use of Scanner directly with File object causing the problem. Because that's not buffered. Use BufferedReader instead. Using scanner with big file object directly proved to be failed. Because, that's not buffered I guess.

Scanner scanDaily = new Scanner(new File(name));  //problematic for big files.

Use BufferedReader with using FileReader instead of that. It buffers the data from file as is needed but not at once.

Example:

     import java.io.BufferedReader;
     import java.io.FileReader;
     import java.util.Scanner;
     import java.io.File;
     ...............
     FileReader f=new FileReader(new File(fileName));
     BufferedReader bf=new BufferedReader(f);
     Scanner sc=new Scanner(bf);

So your code now becomes:

     String name = getFileName();
     FileReader f= new FileReader(new File(name));
     BufferedReader bf=new BufferedReader(f);
     Scanner sc=new Scanner(bf);

Your program hangs with your scanner code because, it is loading your big file all at once into memory and hence taking time.

Also, what are the best suggestions for making my program be able to hold array lists of such a large size? I have heard of hash tables, however; I have not been able to full understand the concept.

In this case, since the file size is large. I would suggest you using memory mapped file. So, that you can map the file in memory and use it access it like an array. See this link about memory mapping in java.

It seems you already know about ArrayLists.

I will brief about HashMap: HashMap uses key value pair to store the data, you have key based on which the value is stored. You will use key to store the data and get the data.

Example:

          HashMap<KeyType,ValueType> hm=new HashMap<KeyType,ValueType>

So this way you can use any type as key and any type as value.

          HashMap<Integer,String> hm = new HashMap<Integer,String>
          hm.set(0,"hello");
          hm.set(5,"bello");

          HashMap<String,String> sm=new HashMap<String,String>
          sm.set("USA","United States of America");
          sm.set("UK","United Kingdom");
          sm.set("IND","India");
          sm.set("AUS","Australia");              

          so, you can query `sm.get("AUS")` to get `"Australia"`,

To decide which data structure to use: When to use HashMap over LinkedList or ArrayList and vice-versa

I hope this will solve the problem.

Sorry for the late response. The problem with your code is that you cannot convert from a filreader to scanner (or so eclipse says). — Danny, Jun 11 '13 at 13:58
Awesome it worked. I hope I am not keeping you from anything important haha. Is there any type try/catch clause here? When i changed from scanner to filereader it made me remove the try/catch clause I had in. — Danny, Jun 11 '13 at 14:09
@user2474459 you have to catch `FileNotFoundException`, don't forget to import `java.io.FileNotFoundException` . please don't forget to accept the answer....an upvote also helps... — pinkpanther, Jun 11 '13 at 14:17
Ok, but now that I have a file reader and buffered reader, how would I appraoch this next problem - check question update - — Danny, Jun 11 '13 at 14:35
@user2474459 now also you are using `Scanner` but in a different way, FileReader->BufferedReader->Scanner, ultimately you got a scanner, that's what we want, so you can continue your normal operations with scanner... I hope you got it now... :) tell if you got it.... :) JUST CHANGE VARIABLE NAME FROM `sc` TO `scanDaily` and no further changes are required.. — pinkpanther, Jun 11 '13 at 14:54

score 0 · Answer 4 · answered Jun 11 '13 at 12:28

0

Since the max file size which you are using in < 3 GB, and I am assuming you are running it on a machine where RAM is >3 GB, you can run the program by using following argument

java -Xmx3046m -jar yourjarname.jar

answered Jun 11 '13 at 12:28

abhinav

1,252
10
27

can you explain in a little more detail what you mean. am i too resave my java file as a .jar? I am a little confused – Danny Jun 11 '13 at 12:29
I meant if you have made a jar from your java code. How are you running your java code. From IDE ?? – abhinav Jun 11 '13 at 12:40
i'm using eclipse right now, but was going to run it in CMD when i tested this method – Danny Jun 11 '13 at 12:44
In eclipse go run configurations and add -Xmx3046m in JVM arguments text box – abhinav Jun 11 '13 at 12:46

score 0 · Answer 5 · edited May 23 '17 at 12:05

0

You may try increasing memory allocation for the JVM. Check this post. Also try to track the exact exception/error you get if your program hangs before going to conclusions.

edited May 23 '17 at 12:05

Community

1
1

answered Jun 11 '13 at 12:31

Don Srinath

1,565
1
21
32

score 0 · Answer 6 · answered Jun 11 '13 at 12:35

0

If you store the numbers as int you could write the numbers to a memory mapped file (java.nio) IntBuffer. Depends on the usage scenario.

A fixed oversized int[] might be feasible.

answered Jun 11 '13 at 12:35

Joop Eggen

107,315
7
83
138

file size too big for java

6 Answers6

Linked