How can I prevent boost log to duplicate message

Question

I have code where I log different categories of information. Each piece of code has a specific tag that has been made for them. (e.g : database.message, database.odbc, event.cevent ...)

Moreover, I've made a function that reads a json file and their value which corresponds to their according severity filter

{
    "database" :
    {
        "message": "INFO",
        "odbc": "DEBUG",
    },
    "event" :
    {
        "cevent" : "INFO",
    },
}

My problem is, I want to set a "basic filter", (for instance only log messages that are "INFO" or above) for all items whose tags were not set in this file.

Right now, I'm adding filters this way:

logging::core::get()->add_global_attribute("Tag", attrs::constant<std::string>(""));
logging::add_common_attributes();
std::vector<std::string> tags; // Suppose it's already filled with the tags and values from the json

...
for (const auto& tag :tags)
{
    boost::shared_ptr<text_sink> sink(new text_sink(backend)); // same as the doc
    auto level = logging::trivial::info; // just an example for more clarity 
    sink->set_filter(expr::attr<std::string>("Tag") == tag && expr::attr<logging::trivial::severity_level>("Severity") >= level);
    logging::core::get()->add_sink(sink);
}

This piece of code works and correctly reads and sets filter according to the json file.

So to add this "basic filter", I also added this once every filter has been set:

boost::shared_ptr<text_sink> basic_sink(new text_sink(backend));
auto filter = logging::trivial::severity >= logging::trivial::info;
for (const auto& tag : tags)
{
    filter = filter && expr::attr<std::string>("Tag") != tag;
}
basic_sink->set_filter(filter);
logging::core::get()->add_sink(basic_sink);

But it duplicates messages that are defined in the json, when I thought this would filter out tags stored. Do you have any ideas on how to avoid such duplication or do I have to implement such a sink as mentionned in this post

sehe · Accepted Answer · 2022-12-28T04:12:09.937

You weren't duplicating messages. You were adding duplicate sinks.

As someone else posted, you want to combine into one filter instead of duplicating your sinks.

However, since the filter expression is a compile-time static template expression that describes a deferred invocation, you need a deferred function to work with it.

I'd use boost::phoenix::function to make it simple:

boost::phoenix::function matching = [tags](logging::value_ref<std::string> actual) {
    return tags.contains(actual.get());
};

Now you can use the single filter expression:

auto filter = matching(_tag) && (logging::trivial::severity >= level);

C++ 11

The above assumed C++17, but you can spell it out for older beasts:
struct match_impl {
    std::set<std::string> target_tags;

    using result_type = bool;
    result_type operator()(logging::value_ref<std::string> actual_tag) const {
        return 0 < target_tags.count(actual_tag.get());
    }
};
boost::phoenix::function<match_impl> matching{match_impl{std::move(tags)}};
If Boost Log supports C++03 I know how to get that done too, but hopefully that's not required.

Full Demo

Live On Coliru

#include <boost/log/sinks/text_file_backend.hpp>
#include <boost/log/sources/logger.hpp>

#include <boost/log/attributes/scoped_attribute.hpp>
#include <boost/log/trivial.hpp>
#include <boost/log/utility/setup.hpp>

#include <boost/phoenix.hpp>
#include <set>

namespace logging = boost::log;
namespace sinks   = logging::sinks;
namespace attrs   = logging::attributes;

void init_logging(logging::trivial::severity_level level, std::set<std::string> tags) {

    auto core = logging::core::get();
    using text_sink = sinks::synchronous_sink<sinks::text_ostream_backend>;

    core->add_global_attribute("Tag", attrs::constant<std::string>(""));
    logging::add_common_attributes();

    auto backend = boost::make_shared<sinks::text_ostream_backend>();
    auto sink    = boost::make_shared<text_sink>(backend);
    sink->locked_backend()->add_stream(
        boost::shared_ptr<std::ostream>(&std::clog, boost::null_deleter{}));

#if __cplusplus < 201703L
    struct match_impl {
        std::set<std::string> target_tags;

        using result_type = bool;
        result_type operator()(logging::value_ref<std::string> actual_tag) const {
            return 0 < target_tags.count(actual_tag.get());
        }
    };
    boost::phoenix::function<match_impl> matching{match_impl{std::move(tags)}};
#else
    boost::phoenix::function matching = [tags](logging::value_ref<std::string> actual) {
        return tags.contains(actual.get());
    };
#endif

    auto _tag   = boost::log::expressions::attr<std::string>("Tag");
    auto filter = matching(_tag) && (logging::trivial::severity >= level);

    sink->set_filter(filter);
    core->add_sink(sink);
}

int main() {
    init_logging(logging::trivial::severity_level::error, {"foo", "bar", "qux"});

    for (std::string tag : {"foo", "bogus", "bar"}) {
        BOOST_LOG_SCOPED_THREAD_ATTR("Tag", attrs::constant<std::string>(tag));
        BOOST_LOG_TRIVIAL(debug)   << "debug   tagged with " << tag;
        BOOST_LOG_TRIVIAL(error)   << "error   tagged with " << tag;
        BOOST_LOG_TRIVIAL(fatal)   << "fatal   tagged with " << tag;
        BOOST_LOG_TRIVIAL(info)    << "info    tagged with " << tag;
        BOOST_LOG_TRIVIAL(trace)   << "trace   tagged with " << tag;
        BOOST_LOG_TRIVIAL(warning) << "warning tagged with " << tag;
    }
}

Prints the expected

error   tagged with foo
fatal   tagged with foo
error   tagged with bar
fatal   tagged with bar

It's not exactly what I was looking for, but the process of thinking behind helped me figure it out. In your example, what I was looking for was also the levels of error of "bogus" (for error, fatal, and info), so just needed to add in this filter: ... || (!matching(_tag) && logging::trivial::severity >= logging::trivial::info ); Anyway a big thank you. — Forague, Dec 28 '22 at 08:36
@Forague So, add it? `init_logging(logging::trivial::severity_level::error, {"foo", "bar", "qux","bogus"});`. The key point is to fix the static type of the filter expression without fixing the tag selection, because you want to be able to combine those at runtime. — sehe, Dec 28 '22 at 16:04
My problem is there are some tags that are not provided in the JSON so I can't feed `init_logging` with those but I still want to be able to see their logs if they are somewhat necessary. That's what I meant by "basic filter". — Forague, Dec 29 '22 at 08:36
Can you be concrete? "When they are somewhat necessary" is not a technical requirement. "basic filter" doesn't explain it either. Do you *perhaps* want to be able to change the filtered tags at runtime (after initilization)? Then, take the set by reference: http://coliru.stacked-crooked.com/a/1be19cabbb1ed951 I took extra care to ensure thread safety. — sehe, Dec 29 '22 at 18:10
If you don't mind the efficiency loss, you can even "collect" the tags actually seen: http://coliru.stacked-crooked.com/a/652588daf1d136f9 — sehe, Dec 29 '22 at 18:29
Ah, I see what you mean by adding it, my bad. My issue is, I'm updating old code that has lots of tags. I'm unsure if adding tags every time I see one that doesn't belong to `target_tags` can scale very well to big projects with hundreds of tags and many logs without lagging too much (execution time is crucial for me). I'll give it a try and get back to you when it is tested (might take some time). — Forague, Jan 09 '23 at 13:21
@Forague :thumbsup: If the existing code already used string tags, it will not have been an issue. Of course, the map performance might not be optimal (due to cache/prefetch effects, see [e.g.](https://stackoverflow.com/questions/63049936/best-c-container-for-the-following-use-case/63057550#63057550)). If you can afford to replace the tags with hashes, consider a form of perfect hashing (https://stackoverflow.com/a/16141214/85371) — sehe, Jan 09 '23 at 14:44

How can I prevent boost log to duplicate message

1 Answers1

C++ 11

Full Demo