C++ string parsing ideas

Question

I have the output of another program that was more intended to be human readable than machine readable, but yet am going to parse it anyway. It's nothing too complex.

Yet, I'm wondering what the best way to do this in C++ is. This is more of a 'general practice' type of question.

I looked into Boost.Spirit, and even got it working a bit. That thing is crazy! If I was designing the language that I was reading, it might be the right tool for the job. But as it is, given its extreme compile-times, the several pages of errors from g++ when I do anything wrong, it's just not what I need. (I don't have much need for run-time performance either.)

Thinking about using C++ operator <<, but that seems worthless. If my file has lines like "John has 5 widgets", and others "Mary works at 459 Ramsy street" how can I even make sure I have a line of the first type in my program, and not the second type? I have to read the whole line and then use things like string::find and string::substr I guess.

And that leaves sscanf. It would handle the above cases beautifully

if( sscanf( str, "%s has %d widgets", chararr, & intvar ) == 2 )
      // then I know I matched "foo has bar" type of string, 
      // and I now have the parameters too

So I'm just wondering if I'm missing something or if C++ really doesn't have much built-in alternative.

Answer 1

sscanf does indeed sound like a pretty good fit for your requirements:

• you may do some redundant parsing, but you don't have performance requirements prohibiting that
• it localises the requirements on the different input words and allows parsing of non-string values directly into typed variables, making the different input formats easy to understand

A potential problem is that it's error prone, and if you have lots of oft-changing parsing phrases then the testing effort and risk can be worrying. Keeping the spirit of sscanf but using istream for type safety:

#include <iostream>
#include <sstream>

// Str captures a string literal and consumes the same from an istream...
// (for non-literals, better to have `std::string` member to guarantee lifetime)
class Str
{
  public:
    Str(const char* p) : p_(p) { }
    const char* c_str() const { return p_; }
  private:
    const char* p_;
};

bool operator!=(const Str& lhs, const Str& rhs)
{
    return strcmp(lhs.c_str(), rhs.c_str()) != 0;
}

std::istream& operator>>(std::istream& is, const Str& str)
{
    std::string s;
    if (is >> s)
        if (s.c_str() != str)
            is.setstate(std::ios_base::failbit);
    return is;
}

// sample usage...

int main()
{
    std::stringstream is("Mary has 4 cats");
    int num_dogs, num_cats;

    if (is >> Str("Mary") >> Str("has") >> num_dogs >> Str("dogs"))
    {
        std::cout << num_dogs << " dogs\n";
    }
    else if (is.clear(), is.seekg(0), // "reset" the stream...
             (is >> Str("Mary") >> Str("has") >> num_cats >> Str("cats")))
    {
        std::cout << num_cats << " cats\n";
    }
}

Answer 2

The GNU tools flex and bison are very powerful tools you could use that are along the lines of Spirit but (according to some people) easier to use, partially because the error reporting is a bit better since the tools have their own compilers. This, or Spirit, or some other parser generator, is the "correct" way to go with this because it affords you the greatest flexibility in your approach.

If you're thinking about using strtok, you might want to instead take a look at stringstream, which splits on whitespace and lets you do some nice formatting conversions between strings, primitives, etc. It can also be plugged into the STL algorithms, and avoids all the messy details of raw C-style string memory management.

Answer 3

I've written extensive parsing code in C++. It works just great for that, but I wrote the code myself and didn't rely on more general code written by someone else. C++ doesn't come with extensive code already written, but it's a great language to write such code in.

I'm not sure what your question is beyond just that you'd like to find code someone has already written that will do what you need. Part of the problem is that you haven't really described what you need, or asked a question for that matter.

If you can make the question more specific, I'd be happy to try and offer a more specific answer.

Answer 4

I've used Boost.Regex (Which I think is also tr1::regex). Easy to use.

Answer 5

there is always strtok() I suppose

Answer 6

Have a look at strtok.

Answer 7

Depending on exactly what you want to parse, you may well want a regular expression library. See msdn or earlier question.

Personally, again depending the exact format, I'd consider using perl to do an initial conversion into a more machine readable format (E.g. variable record CSV) and then import into C++ much more easily.

If sticking to C++, you need to:

1. Identify a record - hopefully just a line
2. Determine the type of the record - use regex
3. Parse the record - scanf is fine

A base class on the lines of:

class Handler
{
public:
    Handler(const std::string& regexExpr)
        : regex_(regexExpr)
    {}
    bool match(const std::string& s)
    {
        return std::tr1::regex_match(s,regex_);
    }
    virtual bool process(const std::string& s) = 0;
private:
    std::tr1::basic_regex<char> regex_;
};

Define a derived class for each record type, stick an instance of each in a set and search for matches.

class WidgetOwner : public Handler
{
public:
    WidgetOwner()
        : Handler(".* has .* widgets")
    {}
    virtual bool process(const std::string& s) 
    {
        char name[32];
        int widgets= 0;
        int fieldsRead = sscanf( s.c_str(),  "%32s has %d widgets", name, & widgets) ;

        if (fieldsRead == 2)
        {
            std::cout << "Found widgets in " << s << std::endl;
        }
        return fieldsRead == 2;
    }
};

struct Pred 
{
    Pred(const std::string& record)
        : record_(record)
    {}
    bool operator()(Handler* handler)
    {
        return handler->match(record_);
    }
    std::string record_;
};

std::set<Handler*> handlers_;
handlers_.insert(new WidgetOwner);
handlers_.insert(new WorkLocation);

Pred pred(line);
std::set<Handler*>::iterator handlerIt = 
     std::find_if(handlers_.begin(), handlers_.end(), pred);
if (handlerIt != handlers_.end())
    (*handlerIt)->process(line);