How to plot selectively from a column file (in C/C++)

Question

I am trying to selectively plot some data from a file with columns, which isn't .csv or .tsv A typical file is http://pastebin.com/pgSjezdh.

You can see that there is some info at first, which of course should be somehow skipped. Then there are some columns, from which I would like to plot one over the other. For instance the first column being x and the fourth being y.

The idea is to use CERN's root, which is using C/C++, so ideally this should be done in C/C++ so that root could handle it.

Apart from that, the main problem is to somehow get the desired data in a two-column format, without strings.

What is the most efficient way to do that?

Answer 1

You can use this Flex code, named cern.l:

%x _DATA_
%x _END_

%option noyywrap
%{
  #define NUMBER_OF_COLUMNS 6
  #define FILENAME_LENGTH 64

  // variables
  float data;
  int dataNumber;
  FILE *files[NUMBER_OF_COLUMNS];
  char fileNames[NUMBER_OF_COLUMNS] [FILENAME_LENGTH];

  // functions
%} 
NUMBER [0-9]*(\.[0-9]+)?(E(\+|-)[0-9][0-9])?

%% 
"-----------  ---------- ---------- ----------  ----------  ----------" {
            fprintf(stderr,"\nBEGIN DATA");
            BEGIN(_DATA_);
        }
<_DATA_>"-----" {
                    BEGIN(_END_);
         }
<_DATA_>{NUMBER} {
         data = atof(yytext);
         fprintf(files[dataNumber],"%f\n",data);
         dataNumber = (dataNumber+1)%NUMBER_OF_COLUMNS;
        }
<_DATA_>"----------" { 
      BEGIN(_END_);
      }
%%
int main(int argc, char* argv[])
{
  int i = 0;
  dataNumber = 0;
  for(i = 0; i < NUMBER_OF_COLUMNS; i++)
  {
    sprintf(fileNames[i],"/home/user/column%d.txt",i);
    files[i] = fopen(fileNames[i],"w");
  }
  yyin=fopen("/home/user/example.dat","r");

  yylex();
  for(i = 0; i < NUMBER_OF_COLUMNS; i++)
  {
    fclose(files[i]);
  }
  return 0;
}

You can compile it using make cern.l. Under Linux it will generate a binary executable cern. Calling ./cern it will generate 6 files (named column0.txt column1.txt etc) containing your data.

If you prefer to integrate this code as a function in your C++ program, you can simply compile it using flex cern.l. It will generate a file named lex.yy.c which contains the C code. Then, you can rename main into lex.yy.c with a function name, for example int parse(float *column0, float *column1, float *column2, float *column3, float *column4, float *column5) and call it from your program. Obviously in this case you should previously modify the original flex code in order to fill 6 float arrays with data.

Why using Flex? Because it generate an optimized scanner. If you need to parse large amount of data in a very small time it can be very useful. I also find it simpler than manually create a FSM using C language.

Answer 2

It's not quite clear from your description and the example input, but I assume that the data columns are in lines 25 to 51. It looks as if the row of hyphens just above that indicates the columns. The first column starts with the first character of the line. (There is a similar table above that with four columns and one row, which is indented, which should probably be skipped.) The data rows are terminated by another row of hyphens.

So the basic algorithm is: Read everything up to the row of hyphens, store column widths and starting points for each field, then read the subsequent data, cut out the columns you want using the information from the hyphens and stop reading when you encounter the next line.

That's probably something a script can do easily for you. The standalone C program below does that, too. You can call it from the command line like that:

./colcut data.txt 3 5 1

to print out columns 3, 5 and 1 (natural count, not zero-based) of the file "data.txt". Error handling is probably lacking - it doesn't check whether the columns are long enough, for example - but it looks serviceable:

#include <stdlib.h>
#include <stdio.h>



#define die(x) do {                             \
        fprintf(stderr, "Fatal: %s\n", x);      \
        exit(1);                                \
    } while (0)

#define MAX 10
#define MAXLEN 500

typedef struct {            /* Text slice into a char buffer */
    const char *str;        /* start pointer */
    int len;                /* slice length */
} Slice;

int main(int argc, char *argv[])
{
    FILE *f;
    int index[MAX];         /* column index */
    int nindex = 0;         /* number of columns to write */
    Slice cols[MAX];        /* column substrings */
    int context = 0;        /* Are we scaning columns? */
    int i;

    if (argc < 3) die("Usage: col file columns ...");

    for (i = 2; i < argc; i++) {
        int n = atoi(argv[i]);

        if (n < 1 || n > MAX) die("Illegal index");
        index[nindex++] = n - 1;
    }

    f = fopen(argv[1], "r");
    if (f == NULL) die("Could not open file.");

    for (;;) {
        char line[MAXLEN];

        if (fgets(line, MAXLEN, f) == NULL) break;

        if (context) {
            if (line[0] == '-') break;
            for (i = 0; i < nindex; i++) {
                int j = index[i];

                printf("    %.*s", cols[j].len, cols[j].str);
            }
            putchar(10);
        }

        if (line[0] == '-') {
            const char *p = line;
            int n = 0;

            while (*p == '-' || *p == ' ') {
                cols[n].str = p;
                while (*p == '-') p++;
                cols[n].len = p- cols[n].str;
                while (*p == ' ') p++;
                if (++n == MAX) break;
            }

            for (i = 0; i < nindex; i++) {
                if (index[i] >= n) die("Columns index out of range");
            }
            context = 1;
        }
    } 
    fclose(f);   

    return 0;
}