Parsing a std::vector as struct attribute with Spirit

Question

I've the following text:

[70000000:45]
4, 5, 6, 7

[60000000:60]
1, 2, 3, 4

[80000:90]
4, 5, 6, 7, 8, 9

The rows with square brackets contains a frequency and an angle in the form [freq:angle], while the subsequent row is a vector of number related to these parameters. I can have different sets of frequencies and angles, and a vector is defined for every of them.

I've the following structure:

struct Data {
  std::vector<int> frequencies;
  std::vector<int> elevations;
  std::vector<std::vector<double>> gains;
};

I need to store in this structure the file data: in the frequencies vector I'll have all frequencies in order, from top to bottom; in the elevations vector I'll have the same thing but for elevation data, and in the gains vector I'll have respective gain vectors.

The scope is that if I've an index, vector elements at this index will contain frequency, elevation and gains data related to them as in the file.

For examples, after the parsing, at index = 1 I will have

data.frequencies[1] = 60000000
data.elevations[1] = 60
data.gains[1] = {1, 2, 3, 4}

I need to parse the file in order to populate the structure. I was able to parse frequencies and elevations, and store them in vectors, but I cannot store gain data. I need to create a vector of vectors from these data.

Below you can find the code; after the parsing parsed is populated with frequencies and elevation, and I need gains. What I can do in order to parse them?

#include <boost/optional/optional_io.hpp>
#include <boost/date_time/posix_time/posix_time.hpp>
#include <boost/date_time/posix_time/posix_time_io.hpp>
#include <boost/fusion/adapted/struct.hpp>
#include <boost/spirit/include/qi.hpp>
#include <boost/spirit/include/phoenix.hpp>

namespace qi = boost::spirit::qi;
namespace px = boost::phoenix;

const std::string file1 = R"xx(
[70000000:45]
4, 5, 6, 7

[60000000:60]
1, 2, 3, 4

[80000:90]
4, 5, 6, 7, 8, 9
)xx";


struct Data {
  std::vector<int> frequencies;
  std::vector<int> elevations;
  std::vector<std::vector<double>> gains;

};

BOOST_FUSION_ADAPT_STRUCT(
  Data,
  (std::vector<int>, frequencies)
  (std::vector<int>, elevations)
  (std::vector<std::vector<double>>, gains)
)

template <typename It, typename Skipper = qi::space_type>
struct grammar : qi::grammar<It, Data(), Skipper> {

  grammar() : grammar::base_type(start) {

    auto frequencyParser = qi::int_[px::push_back(px::at_c<0>(qi::_val), qi::_1)];
    auto elevationParser = qi::int_[px::push_back(px::at_c<1>(qi::_val), qi::_1)];
    auto frequencyElevationParser = qi::lit('[') >> frequencyParser >> qi::lit(':') >> elevationParser >> qi::lit(']');
    auto gainsParser = qi::double_ % qi::lit(','); // Problem here where I want to parse vector rows

    start = *(frequencyElevationParser >> gainsParser);
  }

private:

  qi::rule<It, Data(), Skipper> start;
};

int main() {
  using It = std::string::const_iterator;
  Data parsed;
  bool ok = qi::phrase_parse(file1.begin(), file1.end(), grammar<It>(), qi::space, parsed);
  return 0;
}

1. Is there a reason for the specific modeling of data? I rather would have done `struct Data { int frequency, elevation; std::vector gains; }; std::vector data;`. 2. Though I find `boost` and `boost::spirit` impressive - in this case I had written the parser simply in "plain" C++ using `std::getline()`, `std::istringstream` and, may be, some `if`s (e.g. to check whether a line starts with `'['` or not). Is this "machine generated" text or something where users may insert any garbage? — Scheff's Cat, Feb 09 '18 at 13:21
The file template is fixed, so the class that I need to use. The file is also more complex and this is only a section for it so I can then assemble grammars. Also there can be other thinks between rows so `boost::spirit` for me is well suited in this case for matching parts to parse. I've tried to reduce complexity of class and file to a minimum in order to isolate the problem. — Jepessen, Feb 09 '18 at 13:34

score 2 · Answer 1 · answered Feb 09 '18 at 14:12

In the spirit of my oft-repeated mantra Boost Spirit: "Semantic actions are evil"? I'd use a trait, and parse each section into an Ast structure:

qi::rule<It, Data(), Skipper> start;
qi::rule<It, Ast::Data(), Skipper> section;

The whole parser doesn't need to be more complicated than this:

section = '[' >> int_ >> ':' >> int_ >> ']' >> double_ % ',';
start = *section;

The Magic:

Let's create an Ast structure, and ONLY adapt that, instead of the impractical data type you've been given:

namespace Ast {
    struct Data {
        int frequency;
        int elevation;
        std::vector<double> gains;
    };
}

BOOST_FUSION_ADAPT_STRUCT(Ast::Data, frequency, elevation, gains)

Now, all that is left to do is to tell Spirit how you can treat Data as a container of Ast::Data:

namespace boost { namespace spirit { namespace traits {
    template <> struct container_value<MyLib::Data> { using type = Ast::Data; };

        static bool call(MyLib::Data& c, Ast::Data const& val) {
            c.frequencies.push_back(val.frequency);
            c.elevations.push_back(val.elevation);
            c.gains.push_back(val.gains);
            return true;
        }
    };
} } }

That's a LOT simpler.

DEMO

Live On Coliru

#include <boost/fusion/adapted/struct.hpp>
#include <boost/spirit/include/qi.hpp>

namespace qi = boost::spirit::qi;

const std::string file1 = R"xx(
[70000000:45]
4, 5, 6, 7

[60000000:60]
1, 2, 3, 4

[80000:90]
4, 5, 6, 7, 8, 9
)xx";

namespace MyLib {
    struct Data {
        std::vector<int> frequencies;
        std::vector<int> elevations;
        std::vector<std::vector<double> > gains;
    };
}

namespace Ast {
    struct Data {
        int frequency;
        int elevation;
        std::vector<double> gains;
    };
}

namespace boost { namespace spirit { namespace traits {
    template <> struct container_value<MyLib::Data> { using type = Ast::Data; };

    template<> struct push_back_container<MyLib::Data, Ast::Data> {
        static bool call(MyLib::Data& c, Ast::Data const& val) {
            c.frequencies.push_back(val.frequency);
            c.elevations.push_back(val.elevation);
            c.gains.push_back(val.gains);
            return true;
        }
    };
} } }

BOOST_FUSION_ADAPT_STRUCT(Ast::Data, frequency, elevation, gains)

namespace MyLib {
    template <typename It, typename Skipper = qi::space_type>
    struct grammar : qi::grammar<It, Data(), Skipper> {

        grammar() : grammar::base_type(start) {
            using namespace qi;

            section = '[' >> int_ >> ':' >> int_ >> ']' >> double_ % ',';
            start = *section;
        }

      private:
        qi::rule<It, Data(), Skipper> start;
        qi::rule<It, Ast::Data(), Skipper> section;
    };
}

int main() {
    using It = std::string::const_iterator;
    MyLib::Data parsed;
    bool ok = qi::phrase_parse(file1.begin(), file1.end(), MyLib::grammar<It>(), qi::space, parsed);
}

score 1 · Answer 2 · answered Feb 25 '18 at 20:41

With X3 this becomes much simpler:

#include <boost/spirit/home/x3.hpp>
#include <iostream>

namespace x3 = boost::spirit::x3;

const std::string file1 = R"xx(
[70000000:45]
4, 5, 6, 7

[60000000:60]
1, 2, 3, 4

[80000:90]
4, 5, 6, 7, 8, 9
)xx";


struct Data {
  std::vector<int> frequencies;
  std::vector<int> elevations;
  std::vector<std::vector<double>> gains;
};

int main() {
  Data parsed;

  auto appender = [](auto& ctx)
  {
      Data& data = x3::_val(ctx);
      auto& entry = x3::_attr(ctx);
      data.frequencies.push_back(boost::fusion::at_c<0>(entry));
      data.elevations.push_back(boost::fusion::at_c<1>(entry));
      data.gains.emplace_back(std::move(boost::fusion::at_c<2>(entry)));
  };

  auto block = x3::rule<struct r_block, Data>{}  =
      (('[' >> x3::int_ >> ':' >> x3::int_ >> ']') >> (x3::double_ % ','))[appender];

  auto it = file1.begin();
  while (x3::phrase_parse(it, file1.end(), block, x3::space, parsed))
    ;

  if (it != file1.end()) {
      std::cout << "Not all input parsed" << std::endl;
  }

  assert(parsed.elevations.size() == parsed.frequencies.size()
      && parsed.frequencies.size() == parsed.gains.size());

  for (int i = 0; i < parsed.frequencies.size(); ++i) {
      std::cout << "frequency: " << parsed.frequencies[i] << std::endl;
      std::cout << "elevations: " << parsed.elevations[i] << std::endl;
      std::cout << "gains: ";
      std::copy(parsed.gains[i].begin(), parsed.gains[i].end(), std::ostream_iterator<double>(std::cout, ","));
      std::cout << std::endl;
  }

  return 0;
}

score 0 · Answer 3 · answered Feb 09 '18 at 13:50

I've found the solution by myself

I've added a qi::rule for the std::vector<double> in the grammar, and then I've used it for parsing the vector, appending the result in the structure like other fields. Below the corrected code:

#include <boost/optional/optional_io.hpp>
#include <boost/date_time/posix_time/posix_time.hpp>
#include <boost/date_time/posix_time/posix_time_io.hpp>
#include <boost/fusion/adapted/struct.hpp>
#include <boost/spirit/include/qi.hpp>
#include <boost/spirit/include/phoenix.hpp>

namespace qi = boost::spirit::qi;
namespace px = boost::phoenix;

const std::string file1 = R"xx(
[70000000:45]
4, 5, 6, 7

[60000000:60]
1, 2, 3, 4

[80000:90]
4, 5, 6, 7, 8, 9
)xx";


struct Data {
  std::vector<int> frequencies;
  std::vector<int> elevations;
  std::vector<std::vector<double>> gains;
};

BOOST_FUSION_ADAPT_STRUCT(
  Data,
  (std::vector<int>, frequencies)
  (std::vector<int>, elevations)
  (std::vector<std::vector<double>>, gains)
)

template <typename It, typename Skipper = qi::space_type>
struct grammar : qi::grammar<It, Data(), Skipper> {

  grammar() : grammar::base_type(start) {

    auto frequencyParser = qi::int_[px::push_back(px::at_c<0>(qi::_val), qi::_1)];
    auto elevationParser = qi::int_[px::push_back(px::at_c<1>(qi::_val), qi::_1)];
    auto frequencyElevationParser = qi::lit('[') >> frequencyParser >> qi::lit(':') >> elevationParser >> qi::lit(']');
    vectorParser = qi::double_ % qi::lit(',');
    auto gainsParser = vectorParser[px::push_back(px::at_c<2>(qi::_val), qi::_1)];

    start = *(frequencyElevationParser >> gainsParser);
  }

private:

  qi::rule<It, Data(), Skipper> start;
  qi::rule<It, std::vector<double>(), Skipper> vectorParser;
};

int main() {
  using It = std::string::const_iterator;
  Data parsed;
  bool ok = qi::phrase_parse(file1.begin(), file1.end(), grammar<It>(), qi::space, parsed);
  return 0;
}

Don't use auto with spirit. I know you probably gleaned it from an answer of mine, but you missed about the important things also there: https://stackoverflow.com/questions/22023779/assigning-parsers-to-auto-variables/22027181#22027181 — sehe, Feb 09 '18 at 13:53
PS. An important observation that crossed my mind after posting this: my approach is more correct. Using your "semantic-action-fest" approach you can end up with de-synced collections when e.g. the heading is parsed partially (try [this](http://coliru.stacked-crooked.com/a/88ecc519d65a19be)). The problem becomes inescapable with backtracking alternatives (imagine [a more interesting grammar like this](http://coliru.stacked-crooked.com/a/300f70ae8f3bb391)). It is one of the key reasons I give in the linked [Evil Actions](https://stackoverflow.com/questions/8259440/q) post — sehe, Feb 11 '18 at 16:19

Parsing a std::vector as struct attribute with Spirit

3 Answers3

The Magic:

DEMO