Lexer Regular Expression

Question

I'm new to Java and have to compile lexer from the arithmetic exression for further calculation. I found useful code to check, but there is one problem while analizing string. When i have something like: 5-9.3 it determines as:

NUMBER 5

NUMBER -9.3

instead of

NUMBER 5

ADDSUBSTR -

NUMBER 9.3

Don't know how to fix it. Would appreciate your help. Want to solve it because it would be harder to cope with it while calculating. And sorry for my awful typesetting.

public class Lexer {

    public static enum TokenType {

        NUMBER("[-+]?[0-9]*\\.?[0-9]+([eE][-+]?[0-9]+)?"), ADDSUBSTR("[+|-]"), DIVMULT ("[/|*]"), WHITESPACE("[ \t\f\r\n]+");
        public final String pattern;

        private TokenType(String pattern) {
        this.pattern = pattern;
    }
    }


    public static class Token {
        public TokenType token;
        public String data;

        public Token (TokenType token, String data) {
            this.token = token;
            this.data = data;
        }

        @Override
        public String toString() {
        return String.format("%s %s", token.name(), data);
        }
    }

    public static ArrayList<Token> lexer(String s) {

        ArrayList<Token> tokens = new ArrayList<>();

        StringBuffer tokenBuffer = new StringBuffer();
        for (TokenType tokenType : TokenType.values())
          tokenBuffer.append(String.format("|(?<%s>%s)", tokenType.name(), tokenType.pattern));
        Pattern tokenPatterns = Pattern.compile(new String(tokenBuffer.substring(1)));

        Matcher matcher = tokenPatterns.matcher(s);
        while (matcher.find()) {
          if (matcher.group(TokenType.NUMBER.name()) != null) {
            tokens.add(new Token(TokenType.NUMBER, matcher.group(TokenType.NUMBER.name())));
          }
          else if (matcher.group(TokenType.ADDSUBSTR.name()) != null) {
            tokens.add(new Token(TokenType.ADDSUBSTR, matcher.group(TokenType.ADDSUBSTR.name())));
          } 
          else if (matcher.group(TokenType.DIVMULT.name()) != null) {
            tokens.add(new Token(TokenType.DIVMULT, matcher.group(TokenType.DIVMULT.name())));
          }
          else if (matcher.group(TokenType.WHITESPACE.name()) != null);
            }

    return tokens;

    }

    public static void main(String[] args) {

        try (Scanner in = new Scanner(System.in)) {
            String input = in.nextLine();
            ArrayList<Token> tokens = lexer(input);
            for (Token token : tokens)
                System.out.println(token);
        }
    }

}

Also, rather than apologising for "awful typesetting", why not just format your code correctly? (You're more likely to get help that way) ;) — Oliver Charlesworth, Jul 10 '17 at 10:37
The logic in your code would allow for something like `1 +-/* 12 12 12 *****` - is that intentional? — SamWhan, Jul 10 '17 at 10:41
@Oliver Charlesworth i tried to debug, but it didn't get me closer to result. Just can't understand. Even if i place check for operator "+-" befor checking number pattern it still adds - to the number. — Evillain, Jul 10 '17 at 10:41
@ClasG i know it, i'm trying to solve problems when they occur, i'll watch for some chekups for it, but not now — Evillain, Jul 10 '17 at 10:43
Maybe [this Q/A](https://stackoverflow.com/questions/20912455/math-expression-parser) could be of interest for you... — SamWhan, Jul 10 '17 at 10:44
...or (depending what you're trying to achieve, i.e. if you're simply trying to calculate the value) [this](https://stackoverflow.com/questions/3422673/evaluating-a-math-expression-given-in-string-form) or [this](https://stackoverflow.com/questions/4010674/looking-for-an-expression-evaluator). — SamWhan, Jul 10 '17 at 10:50

Lexer Regular Expression

0 Answers0