How to use GNU readline with flex-lexer?

Question

I think that using the GNU Readline library for a command-line prompt is good and I want that functionality for my shell that I'm working on. Now readline works for me (my environment is CLion, CMake, Ubuntu, BSD, C, flex-lexer and lemon-parser) but I also need flex and yacc to work at the same time to scan and parse the input but the codes seem "incompatible" - are they really?

    params[0] = NULL;
   printf("> ");

    i=1;
    do {
        lexCode = yylex(scanner);

        /*  snprintf(shell_prompt, sizeof(shell_prompt), "%s:%s $ ", getenv("USER"), getcwd(NULL, 1024));
         Display prompt and read input (NB: input must be freed after use)...*/



        text = strdup(yyget_text(scanner));
        /*
        input = readline(text);

        if (!input)
            break;

        add_history(input);

        free(input);*/
        printf("lexcode %i Text %s\n", lexCode, text);
        if (lexCode == 4) {
            params[i++] = mystring;
            if (strcmp(text, "\'\0")) {
                params[i++] = mystring;
            }
        } else
        if (lexCode != EOL) {
                params[i++] = text;
                printf("B%s\n", text);
        }
        Parse(shellParser, lexCode, text);
        if (lexCode == EOL) {
            dump_argv("Before exec_arguments", i, params);
            exec_arguments(i, params);
            corpse_collector();
            Parse(shellParser, 0, NULL);
            i=1;
        }
    } while (lexCode > 0);
    if (-1 == lexCode) {
        fprintf(stderr, "The scanner encountered an error.\n");
    }

The above code has the parsing and scanning working and commented out the readline functionality that won't work if I want both at the same time. Can I make it work?

An *exemplary* answer would demonstrate *how* to do this, rather than skim over the available resources. — Thomas Dickey, Apr 25 '16 at 08:54
@ThomasDickey, a posted a quick flex + readline + history example. Suggestions welcome ☺. — JJoao, May 23 '16 at 16:11

score 5 · Answer 1 · edited Jun 20 '20 at 09:12

flex+readline example

Following a quick flex "shell" that just can ls and date

%{
  #include <stdlib.h>
  #include <readline/readline.h>
  #include <readline/history.h>
  #define YY_INPUT(buf,result,max_size) result = mygetinput(buf, max_size);

  static int mygetinput(char *buf, int size) {
    char *line;
    if (feof(yyin))  return YY_NULL;
    line = readline("> ");
    if(!line)        return YY_NULL;
    if(strlen(line) > size-2){
       fprintf(stderr,"input line too long\n"); return YY_NULL; }
    sprintf(buf,"%s\n",line);
    add_history(line);
    free(line);
    return strlen(buf);
  }   
%}

%option noyywrap    
%%
ls.*         system(yytext);
date.*       system(yytext);
.+           fprintf(stderr, "Error: unknown comand\n");
[ \t\n]+     {}
%%

build it with:

flex mysh.fl
cc -o mysh lex.yy.c -lreadline -lfl

How is this not the accepted answer? – Miguel Jan 27 '22 at 19:07 — Miguel, Jan 27 '22 at 19:07

Basile Starynkevitch · Accepted Answer · 2016-04-19T05:05:00.620

Read documentation of GNU readline. Read the tty demystified page. You'll want to use readline only when your stdin is a tty, so use isatty(3) as isatty(STDIN_FILENO) to detect that.

The basic behavior of a shell using readline is simply to use the

char *readline (const char *prompt);

function. So you want to adapt your parser to read from a buffer, not from stdin. This is usual practice. Don't forget to test (against failure) your call to readline and to free the resulting buffer when you are done.

Then, you want to add some completion. That is the hard part. Look at what other shells (zsh, bash, fish...) are doing, at least for inspiration. Notice that the default file-name completion could be enough, at least when starting.

BTW, I won't use both lemon and bison to parse the same input. Actually, for a shell (since its syntax is quite simple), I'll just use some hand-written recursive-descent parser.

How to use GNU readline with flex-lexer?

2 Answers2

flex+readline example