Monitoring WSO2 Logs with Elasticsearch, Logstash, and Kibana (or Grafana)

I’ve recently been doing some work involving analytics dashboards and the subject proved to be really interesting. It has a wide range starting from Big Data to UX. Starting at a point where you determine what kind of data is collected and what kind information you need to get out of the dashboard, going to data analysis and data processing (probably on top an engine like Apache Spark), it finalizes at the visualization stage which is really fun.

WSO2 recently released the latest version of WSO2 ESB, with a new analytics solution that integrates seamlessly. Built on WSO2 Data Analytics Server, it packs a lot of power in terms of customizability when it comes to large scale data processing and visualization (plus all the benefits that come with WSO2 Carbon Platform). You just need to download ESB 5 Analytics, change minimal configuration options in the ESB deployment, and start it up. You’ll come across the following dashboard.

So I wanted to see how easily WSO2 products would integrate with a platform like Elasticsearch. And I followed through to setup log collection and monitoring for WSO2 ESB 5.0.0 with Logstash, Elasticsearch, and Kibana (LEK, preserve the order!).

I didn’t start at absolute zero knowledge, however I had no real experience with ELK as well. I went through a simple video explaining the concepts of ELK with a simple demo.

+0hrs

I downloaded the products (Elasticsearch, Logstash, Kibana, Filebeat) and started reading on Logstash. The initial configuration needed was minimal and then I got interested in the filter plugins, especially grok. And I spent nearly an hour just reading (advice: don’t do that).

All WSO2 products use log4j for logging and writes to <CARBON_HOME>/repository/logs/wso2carbon.log by default. If I was going to feed this log file line by line to Logstash, I was going to need to write a grok pattern. And it would likely to change across products when different products use different log4j patterns. I wasn’t setting up a platform wide log monitor, but I usually like simpler and more generic solutions.

+ 1hrs

Then I found another approach. Logstash has an input plugin for log4j. It’s basically a log4j appender which pushes log events to a specified Logstash instance. Udara has already done this and documented it really well so you can follow that guide.

+ 1.5hrs

After briefly reading through the basics of the Log4j appender and doing the configuration, I got the pipeline setup. With the following Logstash conf file and the log4j appender in ESB log4j.properties file, my log events are being published to Logstash, and being indexed by Elasticsearch.

<LOGSTASH_HOME>/wso2carbon.log.conf

input {
    log4j {
        mode => server
        host => "0.0.0.0"
        port => 6000
        type => "log4j"
    }
}

output {
    elasticsearch {
        hosts => [ "localhost:9200" ]
    }

stdout {
        codec => rubydebug
    }
}

<CARBON_HOME>/repository/conf/log4j.properties

# ELK stuff
log4j.rootLogger=INFO, CARBON_CONSOLE, CARBON_LOGFILE, CARBON_MEMORY,tcp

log4j.appender.tcp=org.apache.log4j.net.SocketAppender
log4j.appender.tcp.layout=org.wso2.carbon.utils.logging.TenantAwarePatternLayout
log4j.appender.tcp.layout.ConversionPattern=[%d] %P%5p {%c} – %x %m%n
log4j.appender.tcp.layout.TenantPattern=%U%@%D[%T]
log4j.appender.tcp.Port=6000
log4j.appender.tcp.RemoteHost=localhost
log4j.appender.tcp.ReconnectionDelay=10000
log4j.appender.tcp.threshold=DEBUG
log4j.appender.tcp.Application=esb500wso2carbon

+ 1.75hrs

With my logs being indexed now, visualization was the next step.

The UI is simple, but it took me sometime to get around and understand the concepts. I went through the Discover page, found useful information, and was able to navigate to the visualization creation page. Somehow the search didn’t feel natural to me. It took me several more tries to get a nice line chart going.

+ 2.5hrs

In the ESB server I had a few dummy Proxy Services which would have Log Mediators that log the message being processed along with a few other meta data. The following line chart tracks the number of those log entries made by the “TestProxyService2”.


Conclusion

In about effective 3 hours, I was able to learn a little bit of basics and get ELK to collect, index, and visualize the WSO2 Carbon based logs. I didn’t need to understand in depth the inner workings of the stack, but that would’ve certainly helped. I’m pretty sure if I had spent another 2 hours, I would have come up with a skeletal but better dashboard with more insights in to the logs.

However it’s worth to note that what ELK does here is different from what WSO2 Analytics for ESB does. In the latter, the metrics are being published to WSO2 DAS through Apache Thrift. The metrics collected include Synapse mediation statistics and API statistics (new in ESB 5.0.0) with various records like processing times, fail counts, and endpoint response times.

Logs do not necessarily reflect these information and most of the time it’s hard to track the path of an individual request. Furthermore, there doesn’t seem to be a readily available Thrift receiver for Logstash, to receive the detailed statistics from ESB.

So as I mentioned earlier, these two dashboards would look at the system at different points of views, and would serve different purposes. If you want an Ops view in to the system, ELK will provide that. With added JMX based statistics, ELK would probably provide a deeper insight. However if you want a detailed view in to the operations of the ESB, WSO2 Analytics is able to go deeper than ELK can. Starting from ESB 5.0.0 the statistics that are published from ESB are more detailed, and robust, providing an almost transparent view in to the system.

Bonus

+ 3.25hrs

I just completed a Grafana dashboard with two minimal line charts, that look way cooler than Kibana. The integration with Elasticsearch is simple and works out of the box.

---

Written on September 10, 2016 by chamila de alwis.

Originally published on Medium