4

I have an application server (Ubuntu 14.04) which has tomcat server running on top of it. This application server uses "rsyslog" which is configured to send the logs to a NXlog server (on Ubuntu 14.04).

Rsyslog sends all its logs, including the tomcat errors, exceptions & stack traces to syslog server, but there is a problem with multiline logs. When the log messages are stored in files or forwarded over the network without any encapsulation, the newline character present in messages spanning multiple lines confuse simple linebased parsers which treat every line as a separate event; & hence my exception logs get broken in new lines.

My rsyslog version is : 7.4.4

The rsyslog.conf file looks like this :

#################
#### MODULES ####
#################
$EscapeControlCharactersOnReceive off
$LocalHostName nishant-app

$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog   # provides kernel logging support (previously done by rklogd)i
#$ModLoad immark  # provides --MARK-- message capability
$ModLoad imfile
$ModLoad omrelp
#$ModLoad omhdfs

# provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514

# provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514

module(load="imfile" PollingInterval="10")
###########################
#### GLOBAL DIRECTIVES ####
###########################
#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++#

#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Filter duplicated messages
$RepeatedMsgReduction on

#
# Set the default permissions for all log files.
#
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog

#
# Where to place spool files
#
$WorkDirectory /var/spool/rsyslog

#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf           ## This includes all the conf files which tells rsyslog which logs need to be sent

So basically I need to send the tomcat stack-traces & exception such that exceptions don't get scattered in multiple lines.

  • I am looking to solve this at rsyslog end but also confused that weather this can be solved at Nxlog server side also ?
tink
  • 14,342
  • 4
  • 46
  • 50
Nishant Singh
  • 3,055
  • 11
  • 36
  • 74
  • Yes, this can be also solved at the NXLog side with xm_multiline. – b0ti Oct 19 '15 at 12:50
  • what exact configuration changes could be made on nxlog side ? My nxlog file : http://stackoverflow.com/questions/33233633/nxlog-ignores-multiline-tomcat-stacktraces-while-sending-to-papertrail – Nishant Singh Oct 21 '15 at 02:21

1 Answers1

5

I think the answer depends on how your logs end up in rsyslog. If there's an appender than sends stuff to the syslog socket, it's up to it. As far as I know, you can send multiline logs there, but if the appender breaks them before getting to rsyslog, there's not much you can do there. The same applies to UDP forwarding, each packet is a log, so rsyslog just takes it as it gets it.

If it sends via TCP, the default delimiter for messages is newline. Though rsyslog supports octet-delimited framing, this is again something to handle on the sender side.

If you're tailing files however, this is where you can do something. Though you'll probably need the latest version of rsyslog (there are Ubuntu packages here). With it, you'll get two important features for this usecase:

  • inotify mode (which is used by default). Much nicer than polling mode in terms of performance and playing nicely with log rotation
  • startmsg.regex (which allows you to specify a regex to figure out which line should belong to the current event and which should start a new one)

The point is, with multiline logs you (or rather, rsyslog) have to figure out another way of delimiting messages. With imfile, even in 7.4.4, there's the option of using ReadMode, which defaults to 0 (newline is a delimiter), but you can set it to 2 (if line begins with space/tab it belongs to the previous one).

You can find all the options around imfile here: http://www.rsyslog.com/doc/master/configuration/modules/imfile.html

Radu Gheorghe
  • 564
  • 4
  • 8
  • So basically i followed the upgradation procedure & did tweak my rsyslog file. My current version is 8.13.0 with LF frame delimiter which helps to get rid of space but it actually adds some more character in place of line breaks.. plus when the logs get dumped on NXLog server, the nxlog server just compltly ignores theses paragraphed logs and can't send to papertrail.. all other logs are being sent to papertrail though – Nishant Singh Oct 19 '15 at 12:23
  • Can you try global(parser.escapeControlCharactersOnReceive="off")? It's on by default and I think that's the reason for the line break escaping. – Radu Gheorghe Oct 21 '15 at 02:27
  • you mean $EscapeControlCharactersOnReceive off on rsyslog.conf .. right? – Nishant Singh Oct 21 '15 at 02:31
  • moreover does this will help me to solve http://stackoverflow.com/questions/33233633/nxlog-ignores-multiline-tomcat-stacktraces-while-sending-to-papertrail ? – Nishant Singh Oct 21 '15 at 02:33
  • Right, what you're quoting is the equivalent old-style config. Regarding your other question, it depends on how you send logs from rsyslog to nxlog. If you send over UDP, it should just work. If you send via TCP, it won't work (unless you use octet-delimited framing in rsyslog, which nxlog doesn't seem to support on the receiving end - at least this is what I get from the docs). You can also try TLS, which has octet-delimited framing by definition. Here's a howto for the rsyslog part: http://blog.sematext.com/2014/03/25/encrypting-logs-on-their-way-to-elasticsearch-part-2-tls-syslog/ – Radu Gheorghe Oct 26 '15 at 11:08
  • 1
    nxlog does support the octet-delimited framing with 'InputType Syslog_TLS'. This works with both TCP and SSL/TLS. – b0ti Oct 29 '15 at 22:25