Read large input through Scanner

Question

I am trying to take some user input and process them in my program. For that using the Scanner class as below.

Scanner scan = new Scanner(System.in);  
System.out.print("Input : ");
String input = scan.nextLine();

// Process the Data

scan.close();

The program works fine with small amount of data. But if the input string length is huge (~100K characters), Scanner stops responding and fails to read any data.

Is there any different way or workaround to this problem ?

Note : These is no possible way to store the data in a file and read from there. It would be a nice option to read chunks of data from a file. But unfortunately there is no such implementation in my application.

EDIT : As mentioned earlier, I need to read data directly from user (not from a file). Anyway, already tried to use the BufferReader. But does not seem like can read some data of around 100K characters.

Here is the code below.

BufferedReader br = new BufferedReader(new InputStreamReader(System.in));

System.out.print("Input : ");

String input = br.readLine();

// Remaining Code

`BufferedReader br = new BufferedReader(new InputStreamReader(System.in));` — Pavneet_Singh, Oct 05 '16 at 12:35
Possible duplicate of [How to read a large text file line by line using Java?](http://stackoverflow.com/questions/5868369/how-to-read-a-large-text-file-line-by-line-using-java) — px06, Oct 05 '16 at 12:38
@px06 Thanks for the answer link. But as mentioned in the question, there is no option to read the data from a text file ([which the linked question describes](http://stackoverflow.com/questions/5868369/how-to-read-a-large-text-file-line-by-line-using-java)) — Biswajit, Oct 05 '16 at 12:42
@Biswajit yes I know, but that question contains answers which you can easily use to take input from user instead of file. — px06, Oct 05 '16 at 12:43
You mean you ahve 100K characters per line ? which mean that one line read will store a String of size 100.000 ? Also, could you put the Exception you get when you do this ? — Asoub, Oct 05 '16 at 13:23
@Asoub Got one line of data.. But that line has around 100K characters.. Yes, it mean one String variable will store 100.000.. `input.length() = 100000` Not getting any exception though.. But, it seems like, it reads some amount of data and processes them. — Biswajit, Oct 05 '16 at 13:46
So what do you mean by "Scanner stops responding and fails to read any data" ? It hangs on the nextLine() method ? — Asoub, Oct 05 '16 at 13:51
Yes.. It tries to read all data.. But it keep on reading through nextLine() and could not proceed further. — Biswajit, Oct 05 '16 at 13:57
Does the user presses "Enter" once it has entered all the characters ? — Asoub, Oct 05 '16 at 14:19

Asoub · Accepted Answer · 2016-10-05T15:28:05.837

Well this behavior might be normal, you'll have to wait for a moment before the Scanner code does anything. It's a big input and I guess the OS has to put it somewhere, so there is certain amount of time between the moment you press enter and the moment the JVM can read it.

You should benchmark it to see how much reading time evolves depending on you input.

I've made a code that read character per character from input. It is not a good solution as it surely is slower (and dirtier) than using nextLine(), but it will show how fast is the buffering and reading on your computer:

public static void main(String[] args) {
    Scanner scan = new Scanner(System.in);  
    Log.d("Input : ");

    StringBuilder test = new StringBuilder();

    String input = "";
    long measure = 0;
    while(input != null){
        input = scan.findInLine("."); // Should consume input one char per one char.

        measure++;
        if(input != null)
            test.append(input.charAt(0));

        if(measure == 1 || measure % 1000 == 0) // displays the reading time of first reads and the next thousand ones
            Log.d("Reading at:"+measure);

    }
    Log.d("End");
    scan.close();

    Log.d("size: "+test.toString().length());
    }
}

(note: you can replace Log.d(yourString) by System.out.println( new Date() +":" +yourString ) )

Test with this on your computer and gradually increase the input. I'm not sure the response is linear. It works well with 50000 char but it's in fact quite long with 100.000 (still, only a few minutes before I can read the first character).

edit: the time between reads also increases with input size ... it takes 80 seconds for 1000 char reads with an 100.000 chars input (and a few seconds with 50.000). At this rate, it will take more than 2 hours to read everything.

I'm not sure, looping on System.in.read(); might have better results (I don't know what the line end character is however ... \EOF maybe ?), but you can still try that too. Again, I don't think it's a good solution, but reading from the comment, it's all you got left.

Read large input through Scanner

1 Answers1