Protocol buffers logging

Question

In our business, we require to log every request/response which coming to our server. At this time being, we are using xml as standard implementation. Log files are used if we need to debug/trace some error.

I am kind of curious if we switch to protocol buffers, since it is binary, what will be the best way to log request/response to file?

For example:

FileOutputStream output = new FileOutputStream("\\files\log.txt");
request.build().writeTo(outout);

For anyone who has used protocol buffers in your application, how do you log your request/response, just in case we need it for debugging purpose?

Answer 1

TL;DR: write debugging logs in text, write long-term logs in binary.

There are at least two ways you can do this logging (and maybe, in fact, you should do both):

1. Writing your logs in text format. This is good for debugging and quickly checking for problems with your eyes.
2. Writing your logs in binary format - this will make future analysis much quicker since you can load the data using same protocol buffers code and do all kinds of things on them.

Quite honestly, this is more or less the way this is done at the place this technology came from.

Answer 2

We use the ShortDebugString() method on the C++ object to write down a human-readable version of all incoming and outgoing messages to a text-file. ShortDebugString() returns a one-line version of the same string returned by the toString() method in Java. Not sure how easy it is to accomplish the same thing in Java.

Answer 3

If you have competing needs for logging and performance then I suppose you could dump your binary data to the file as-is, with perhaps each record preceded by a tag containing a timestamp and a length value so you'll know where this particular bit of data ends. But I hasten to admit this is very ugly. You will need to write a utility to read and analyze this file, and will be helpless without that utility.

A more reasonable solution would be to dump your binary data in text form. I'm thinking of "lines" of text, again starting with whatever tagging information you find relevant, followed by some length information in decimal or hex, followed by as many hex bytes as needed to dump your buffer - thus you could end up with some fairly long lines. But since the file is line structured, you can use text-oriented tools (an editor in the simplest case) to work with it. Hex dumping essentially means you are using two bytes in the log to represent one byte of data (plus a bit of overhead). Heh, disk space is cheap these days.

If those binary buffers have a fairly consistent structure, you could even break out and label fields (or something like that) so your data becomes a little more human readable and, more importantly, better searchable. Of course it's up to you how much effort you want to sink into making your log records look pretty; but the time spent here may well pay off a little later in analysis.

Answer 4

If you've non-ASCII character strings in your messages, simply logging them by using implicit or explicit call to toString would escape the characters.

"오늘은 무슨 요일입니까?" becomes "\354\230\244\353\212\230\354\235\200 \353\254\264\354\212\250 \354\232\224\354\235\274\354\236\205\353\213\210\352\271\214?"

If you want to retain the non-ASCII characters, use TextFormat.printer().escapingNonAscii(false).printToString(message).

See this answer for more details.