Which Logging Tool is right for me?
With a plethora of logging tools available at a range of price points, it might be hard to decide on what to use. Rather than diving into a tonne of research, we’ve done it for you – a host of popular approaches to data processing have been fact checked and outlined for your convenience.
The theme of data processing is becoming increasingly popular. This means the arrival of new challenges in IT to enable automatic storage of data, processing and analysis. There’s also a whole range of logging tools available to facilitate the management of logging data. But which tool is right for you?
From cloud solutions like Splunk, Loggly, Sumo Logic or Papertrail to the classic options available such as GrayLog, XL TestView and Logstash, there are many different approaches providing different features or pricing models. Here’s a quick rundown of each option worth considering.
- Company: Splunk Inc.
- License: Freeware (500MB / day), Enterprise
- Current Version: Splunk 6.3 – www.splunk.com
Splunk is especially suitable for processing Big Data. Via its web interface, you have access to machine-generated data which is then indexed in real time, analysed and visualised if required. Splunk can also create reports and statistics for unstructured data. For Hadoop, the Hunk software is available to rapidly explore, analyse and visualise data. Splunk Cloud provides a corresponding SaaS solution.
Splunk is suitable for all applications operating with large amounts of data. Splunk Light is available for small IT environments to perform automated log analysis. Splunk is available for Windows, Mac and Linux and has been on the scene since 2003. The Freeware edition of the software allows for up to 500MB of data to be edited per day. Under the Enterprise license, machine data scaling and processing is unlimited.
- Company: Loggly Inc.
- License: Freeware (200MB / day), Standard, Pro, Enterprise
- Current Version: Generation 2 – www.loggly.com
Loggly not only offers a cloud version, but is itself a cloud-based logging service for real-time insight into used datasets. Loggly uses any copyrighted agents that need to be installed or maintained, but also works with standard protocols such as syslog or HTTP. Loggly can be used with all text-based logs, too. Recognition and parsing of popular web logs such as Apache, Nginx or JSON structured data takes place automatically. The index, where data is collected for analysis, renews itself automatically with each new log. In the Loggly dashboard users can put together search results and trend graphs in unlimited quantities. With a customisable alert function, specific log events can even be signalled.
For integration into existing workflows, there are a few DevOps-friendly functions available. These include a persistent workspace, where all saved searches with team members can be shared, different user roles, whose access and visibility can be defined individually, and even a RESTful API to integrate with the key functions of Loggly via your own applications. Loggly is available as a free ‘lite’ version (200MB / day), a Standard version for $45 USD a month, a Pro version for $99 USD and an Enterprise version starting from $349 USD. More about individual services can be found here.
- Company: Graylog Inc.
- License: Open Source
- Current Version: Graylog 1.2.1 – www.graylog.org
Graylog is an open source project for log management of data in any format. The Graylog platform looks after both structured and unstructured log data collected from various servers and applications and are consequently analysed. The use of components for metadata such as MongoDB or Elasticsearch as log data storage and text search is also an option. A dashboard is utilised to monitor analysis and these results can be added as widgets. Search and visualisation should be possible to execute in just a few minutes. Graylog has an open plugin architecture, a REST API and an open standard for log messaging (GELF). A video showcasing Splunk integration for Graylog has been recently released:
Graylog 1.2.1 can be found ready to download here.
- Company: Papertrail Inc.
- License: Freeware (5GB / first month, then 100MB / month), more models based on data volume and archival duration
- Current Version: Papertrail 4.0.0 – www.papertrailapp.com
The cloud log management system Papertrail gathers application logs, text logs and syslog data to examine in real time via the browser, command line or API. Thus, different log tools, flexible system groups, a split-team access function, long-term archival support, export of tables and analysis reports are available. It supports syslog, and among other things, Ruby on Rails, Apache, MySQL, Tomcat and Heroku apps. An alarm function allows for easy troubleshooting. In Papertrail, unrelated events can be considered together. Conventional tool integration is possible with AWS S3, Librato, Gecko Board or with Hadoop for long-term analysis. Combinations with web services have also been included.
Papertrail is available in a free version with a limited search duration (48 hours) and archive (7 days). After your first month, the available 5GB ‘bonus’ is reduced to 100MB per month. A staggered payment system can extend your search timeframe to two weeks and your archival time-period to one year, up to 1000 GB per month. In May 2015, Papertrail became part of the SolarWinds, a company focusing mainly on the development of DevOps solutions.
- Company: XebiaLabs
- License: Freeware (Community Edition), Pro, Enterprise
- Current Version: XL TestView 1.3.0 – xebialabs.com/products/xl-testview
XL TestView is a management and analysis tool for test results, offered up by XebiaLabs. XebiaLabs specialises in software solutions for DevOps, Continuous Delivery and is closely associated with the Jenkins CD-CI project. With XL TestView, users can have all relevant application tests defined, executed and analysed. A special feature is the central dashboard, where results are collected via different testing tools and analysed according to specific criteria. The overall quality of an application can be queried as well. The tool itself depends on existing applications from test recommendations.
Jenkins integration is offered, where test results are drawn in directly and can be set together with other results. XL TestView also supports Cucumber, FitNesse, Selenium, Cubumber.js, TestNG, Gatling and Apache JMeter. With xUnit, support becomes available for JUnit, Maven, Gradle, Karma and Appium.
A free Community Edition is available on Windows, Mac and Linux operating systems, allowing for one project plus a dashboard with standard reports. Flakiness and difference reports, together with OSS Test Tool plugins are also available. In the Pro version, which can purchased for $499 USD a month, LDAP Integration and Enterprise Test Tool plugins are available. The Enterprise version gives you an unlimited number of project slots. Outlined documentation covering every option is available on the XebiaLabs website.
- Company: Elastic
- License: Apache 2.0
- Current Version: Logstash 1.5.4 – elastic.co/products/logstash
Similar to XL TestView, the data pipeline in Logstash can centralise data processes of various types. It’s possible to parse different schemes and formats and convert them into common log formats. There are over 160 plugins available for Logstash with common data sources. Before version 1.5, Logstash included all plugins in each release, however it now houses all plugins independent of the Logstash core, meaning they’re developed and deployed independently. For more information, a separate guide for adding plugins has been made available. Various plugins also have their own GitHub repositories, where the use of JRuby version 1.9 is recommended.
According to Elastic, data from every imaginable source can be processed with Logstash. It’s also been recommended for use in conjunction with Elasticsearch and Kibana, as shown in the graphic above. Logstash is the meat in the sandwich between the data storage back-end (Elasticsearch) and Kibana for reporting in the front-end. Version 1.5.4 is licensed under the Apache 2.0 license.
- Company: Sumo Logic
- License: Freeware (500MB / day), Pro, Enterprise
- Current Version: Sumo Logic – www.sumologic.com
Sumo Logic is located in the cloud and specialises in the log management and analysis of machine-generated Big Data. In real-time, millions of log lines can be reduced to the viewpoint of safety-related or operational analysis factors. Sumo Logic has a distributed architecture, where all log data for immediate analysis is kept available and relies on Google’s search engine approach.
With advanced machine learning algorithms, enormous amounts of data are reduced to manageable groupings and thus categorised. This helps facilitate synthesising and analysis. Its ecosystem includes MySQL, Docker, Varnish, Chef, Puppet and AWS.
Sumo Logic claims to offer a deeper insight for DevOps teams with a scale-on-demand function and sharing options in the edited stack. This means an end-to-end visibility and collaborative dashboard. Like a host of other Continuous Delivery tools, monitoring for KPIs and statistics are part of the package as well as accelerated troubleshooting.
Sumo Logic offers a free trial which can be used by up to three people to process a maximum of 500 MB per day. The Pro version can be used by up to 20 team members for 1GB a day and can be purchased for $90 USD per month. An unlimited number of users with unlimited support is available via the Enterprise edition. More pricing information can be accessed here.