Stream Tokenizer in Java: Trying to implement

Question

I am new to Java and learning it. I have written a code to take user input and tokenize it. It compiles successfully but when run produces '-3' as o/p.

import java.io.*;


public class TokenTester{

    public static void main(String[] arguments){
    String name;
    StreamTokenizer token;
    BufferedReader reader;
    reader = new BufferedReader(new InputStreamReader(System.in));

     try{
            System.out.println("Enter a String  ");
            name = reader.readLine();
            System.out.println(name);
            token=new StreamTokenizer((new StringReader(name)));
           //token=new StreamTokenizer(name);
            // Using new StreamTokenizer(name) throws error
            //error: no suitable constructor found for StreamTokenizer(String)
             while(token.nextToken()!= token.TT_EOF)
            {
             System.out.println(token.nextToken());
                }
            }
        catch (IOException ioe){
             System.out.println("An unexpected error occured.");
        }

        }
    }

Also when I use new StreamTokenizer(name) throws error "error: no suitable constructor found for StreamTokenizer(String)"

When I use while(token.nextToken()!= token.TT_EOF) it shows infinite loop is -1 as o/p

Below code works absolutely fine:

import java.util.StringTokenizer;

 class TokenTester {

 public static void main(String[] arguments) {
 StringTokenizer st1, st2;

 String quote1 = "VIZY 3 -1/16";
 st1 = new StringTokenizer(quote1);
 System.out.println("Token 1: " + st1.nextToken());
 System.out.println("Token 2: " + st1.nextToken());
 System.out.println("Token 3: " + st1.nextToken());

 String quote2 = "NPLI@9 27/32@3/32";
 st2 = new StringTokenizer(quote2, "@");
 System.out.println("\nToken 1: " + st2.nextToken());
System.out.println("Token 2: " + st2.nextToken());
System.out.println("Token 3: " + st2.nextToken());
 }
 }

I am not able to figure out the issue in my code. Please guide.

I have refered:

http://tutorials.jenkov.com/java-io/streamtokenizer.html
http://www.java-samples.com/showtutorial.php?tutorialid=397
http://docs.oracle.com/javase/7/docs/api/

Thanks Gaurav

For reference, here is a working [example](http://stackoverflow.com/a/2082174/230513). — trashgod, Jan 31 '13 at 16:12

Andrii Tykhonov · Accepted Answer · 2013-01-31T16:22:34.057

1

In first example you use StreamTokenizer and in the second - StringTokenizer. So, in first example, to make it workable you should replace StreamTokenizer by StringTokenizer. And of course you should then use appropriate methods which are relevant to StringTokenizer.

edited Jan 31 '13 at 16:22

answered Jan 31 '13 at 16:16

Andrii Tykhonov

538
6
11

Thanks Andrey.. I spotted the mistake and corrected. It is working now – Gaurav K Jan 31 '13 at 16:43

score 0 · Answer 2 · answered Jan 31 '13 at 16:24

The StreamTokenizer class takes an input stream and parses it into "tokens", allowing the tokens to be read one at a time. The parsing process is controlled by a table and a number of flags that can be set to various states. The stream tokenizer can recognize identifiers, numbers, quoted strings, and various comment styles.

While, The string tokenizer class allows an application to break a string into tokens. The tokenization method is much simpler than the one used by the StreamTokenizer class.

The first example illustrated StreamTokenizer, while the second one illustrates StringTokenizer.

Guido · Answer 3 · 2013-01-31T16:48:03.363

The value of the tokens can be accessed via token.sval

Try this:

try{
        System.out.println("Enter a String  ");
        name = reader.readLine();
        System.out.println(name);
        token=new StreamTokenizer((new StringReader(name)));
        int numberOfTokens = token.nextToken(); //<----nextToken() returns an integer 
         while(numberOfTokens!= token.TT_EOF)
        {
         System.out.println(token.sval);  //<---- take sval
         numberOfTokens = token.nextToken();
            }
        }


    catch (IOException ioe){
         System.out.println("An unexpected error occured.");
    }

When I use while(token.nextToken()!= token.TT_EOF) it shows infinite loop is -1 as o/p

==> This infite loop happens because you increment 2x in one sinle loop:

first increment is here: while(token.nextToken()!= token.TT_EOF)
second increment is here: System.out.println(token.nextToken());

http://docs.oracle.com/javase/1.4.2/docs/api/java/util/StringTokenizer.html java.util.StringTokenizer has not property sval. So solution for 2nd part is not correct. — Andrii Tykhonov, Jan 31 '13 at 16:34

Stream Tokenizer in Java: Trying to implement

3 Answers3