0

This seems like an obvious request to me so I'm hoping others may have already solved this. I have app jboss logs with lots & lots of errors. In order to manage and address these I'd like to figure out a way to track them. after looking at How to retrieve unique count of a field using Kibana + Elastic Search I'm thinking I can use a similar approach. per es docs, it looks like facets have been replaced so I'm thinking I should dig into sum aggregation but not sure yet. I'm still not sure of best way to further break down my jboss log records. the field I'm most interested in is message field which has date/time stamp, hostname in front of each record. what's the best approach to tackle this? break the message field down further--ignore first 2 elements then sort & count next section of this field? I may need to ignore some of the end of this record as well but will deal with that next...

I'm pretty new to ELK stack but excited about its possibilities. Thx. Joe

Community
  • 1
  • 1
Joe Beck
  • 1
  • 3

2 Answers2

0

Logstash (part of E L K) comes up with a lots of filtering option. Most useful is Grok. It is best suited to parse the field from a long message in {key,value} pair. Also, you can delete/ignore the particuler data from the message in Logstash through different kinds of plugins avliable.You can explore it in http://logstash.net/docs/1.4.2/.

After you send those data Elastic, you can use the power of Kibana to create a dashboard based on your requirment.

Hence, ELK is perfectly suites for the requirement you have.

Nirdesh Sharma
  • 734
  • 5
  • 14
0

The best and easiest way to get your JBOSS output into ELK is through a socket connector. There are lots of tutorials but it will automatically give you your message breakdown for free.

See this for an example: http://blog.akquinet.de/2015/08/24/logstash-jboss-eap/

Please note that personally I have had to change the appenders and use documentation to get the correct fields. If you are using 2.0 elasticsearch than update the configuration. For simple debugging simple output to stdout.

Once you have the socket appenders working correctly you are laughing and go to kibanan, configure the dashboard with whatever aggregation you would like. I would not recommend breaking it down further as then you have a custom message breakdown that will not apply to a jboss implementation, feel free to add additional value/pairs such as appname.. etc.

SAMPLE: * jboss-eap-6.4.0.0 * elasticsearch-2.0.0-beta2 * kibana-4.2.0-beta2-windows * logstash-2.0.0-beta1


Create a file called log4j.conf under logstash/conf dir, i.e. "C:_apps\logstash-2.0.0-beta1\conf\log4j.conf" with the below content.

input {
  log4j {
    mode => "server"
    host => "0.0.0.0"
    port => 4712
    type => "log4j"
  }
}

output {
    elasticsearch {
    hosts => "127.0.0.1"
    #cluster => "myAppName"
    index => "logstash-%{+YYYY.MM.dd}"
  }
  stdout { codec => rubydebug }
}

Run Logstash with the following command prompt within dir: bin\logstash.bat -f conf\log4j.conf


Configuring Appenders:

JBOSS APPENDER

Within the profile:

<custom-handler name="Remotelog4j" class="org.apache.log4j.net.SocketAppender" module="org.apache.log4j">
            <level name="INFO"/>
            <properties>
                <property name="RemoteHost" value="localhost"/>
                <property name="Port" value="4712"/>
                <!--property name="BufferSize" value="1000"/-->
                <!--property name="Blocking" value="false"/-->
            </properties>
        </custom-handler>

within the root loggger configuration define your handlers:

<root-logger>
    <level name="INFO"/>
    <handlers>
        <handler name="CONSOLE"/>
        <handler name="FILE"/>
        <handler name="Remotelog4j"/>
    </handlers>
</root-logger>

Start JBOSS, note that your command prompt is printing out all the incoming messages from your standalone JBOSS instance.


Configuring Another Application with OLD Log4J

Log4J version log4j-1.2.15.jar

Inside the packaged WAR I created this simple additional log4j appender:

<appender name="log4jSocket" class="org.apache.log4j.net.SocketAppender" module="org.apache.log4j">
 <level name="ERROR"/>
 <param name="RemoteHost" value="localhost"/>
 <param name="Port" value="4712"/>
 <param name="threshold" value="ERROR" />
</appender>  

Again, add the appender to your application log4j loggers.

  <logger name="com.somepackage" additivity="false">
        <level value="error"/>
        <appender-ref ref="default"/>
        <appender-ref ref="event"/>
        <appender-ref ref="log4jSocket"/>
  </logger>

Now restart your jboss configuration and deploy/start your application inside JBOSS. You will get both jboss output and application output inside of logstash value/paired nicely.

GvD
  • 63
  • 2
  • 6
  • Please add the relevant code from the link you provided and how it solves OP's question instead of a link-only answer. Thanks! – Frakcool Oct 05 '15 at 21:49