Recently, I've started learning ANTLR. I know that lexers/parsers together can be used to construct programming languages.
Other than DSLs or programming languages, have you ever directly or indirectly used lexer/parser tools (and knowledge) to solve real world problems? Is it possible for the average programmer to solve those problems without knowledge of lexers or parsers?
Yes, I've used them. Yes, you can do things with out them--but any time you choose the wrong tool for the job, you'll be making needless pain for yourself.
Some example of the non-standard uses I've personally put the technology to:
Syntax highlighting. The Scite text editor allows you to write your own lexer (in C++) to provide syntax highlighting for any custom language. I wrote my own custom lexer for Scite as a refresher on this topic (I studied it a while ago at my university).
Regular Expressions are often used as an alternative for pattern matching and simple language processing. This is even more common of recent years thanks to the improved RegEx support in frameworks such as .NET. In many cases developers may not even know of lexing/parsing techniques and so fall into usng Regex by default.
However, as another answer says, Regex can quickly become inefficient, slow and difficult to maintain for anything more than a simple grammar/language. In that situation parser/lexers are generally the best choice.
Any time there exists a static document (e.g., a file), or a dynamic document (e.g., a stream occuring over time), and that document has any kind of structure, you will find yourself needing some kind of parser. For simple enough structures, you can get by with ad hoc parsing (string hacking, regexes, etc.). For structures which do not nest, you can get by with a finite state machine; here a lexer generator is often helpful. For complex structures, you pretty much an organized parser. You can write parsers by hand if you are familiar with recursive descent style parsing. For really complex structures, a parser generator is almost always a big win.
If you want to process a computer langauge, you pretty much need lexers and parsers as a starting place. They aren't enough; you have to do something with the parser result.
A really spectacular usage of lexing and parsing that we did is to translate JOVIAL, a 1960s language, into C, for the B-2 stealth bomber. See http://www.semdesigns.com/Products/Services/NorthropGrummanB2.html
Yes, I've used them in real world stuff - but mostly the creation of custom languages that you use lexers and parsers for, has been supplanted by languages defined in XML. More verbose but then you don't have to do all that work...
Yes, I've used them. I'm a big fan of ANTLR. I give some tips and tricks on using ANTLR here and a brief endorsement of it here. It's possible to hand write your own parser using ad hoc methods but it is a lot harder and will take a lot longer to figure out how to make changes when you need to grow the language that your parser is supposed to parse.
Any place you handle text input ends up using some kind of lexer/parser although some times they end up being the degenerate case (lex anything but a comma as one token type and a comma as another. Parse A number, a name, a number and an end of line. That sort of thing) In one way of looking at it sscanf could be considered to be the most degenerate cases of a lexer/parser generator.
As for a full blown lex/yacc operation? I expect that gets used mostly for GPLs and for things that fall under the loose definition of DSLs
A great example of a lexer/parser that is in use in many systems exists in Apache Lucene (an open source Search Index library). Both the query parser and the document tokenizer use these techs. While I guess you could categorize the query parser in Lucene as a dsl parser, it is still being used to help solve a real world problem.
For that matter, I'm sure that Google is employing some sort of lexer/parser for it's own query syntax and document parsing.
This is interesting -
I just wrote a lexer/parser by hand to allow simple string-based query expressions to be handled by an IBindingListView implementation. That was the first useful thing outside of code that I have actually been able to use it for, and not just heard about it.
Pretty pedestrian example, but I'm pretty pedestrian in my experience with them.
I have not used one of the big guys to do any lexical analysis yet, I have however written my own lexer by hand for a project I worked on. We had to parse data that came back from a Near Space project's data computer and it was written to the SD card in binary. I had to pull the bits apart, convert them from binary to decimal and then write the entire contents out in a comma separated file.
It is a lot of fun to sit and think through it logically and write a state machine for the task at hand!
Yes! The team I work with has implemented a document generation framework, which among other things allows (mostly arithmetic) expressions to be evaluated. We're using a parser to extract expressions from the inputs/definitions for the generated documents and create expression trees for them. Afterwards those trees are evaluated and the evaluated results are written to the final document.
