Using Logstash to correlate events

OpsGenie integration family has many members and still growing. The objective of this blog post is to explain using one of the newly added ones, Logstash. Logstash is a data pipeline that helps you process logs and other event data from a variety of systems.

A Logstash pipeline in most use cases has one or more input, filter, and output plugins. Logstash has a rich collection of input, filter, codec and output plugins. A filter plugin performs intermediary processing on an event. Filters are often applied conditionally depending on the characteristics of the event. An output plugin sends event data to a particular destination. Outputs are the final stage in the event pipeline.

The throttle filter is for throttling the number of events received. The filter is configured with a lower bound, the before_count, and upper bound, the after_count, and a period. All events passing through the filter will be counted based on a key.

The mutate filter allows you to perform general mutations on fields. You can rename, remove, replace, and modify fields in your events.

OpsGenie has a Logstash Output Plugin to send event data to OpsGenie. Also, OpsGenie has a Logstash Integration that execute alert actions (create, close, acknowledge and note) with incoming event data.

Using Throttle and OpsGenie Output Plugins

Let's say you're processing your service's logs and want to create alerts in exceptional situations. But exceptions mean a real problem when they exceed a certain limit, and you want to notify on-call people in that case. You can achieve this by configuring Logstash with Throttle Filter. Normal Throttle filter behaviour is not suitable for this requirement. It normally filters the events before before_count value and after after_count value. We should tweak throttle plugin to create alerts in OpsGenie with non-throttled events. The rest of this post will describe adding Logstash Integration in OpsGenie, installing OpsGenie Output Plugin in Logstash and configure your Logstash to fulfill the requirement.

Adding Logstash Integration in OpsGenie

1. Create an OpsGenie account if you haven't done already
2. Go to OpsGenie Logstash Integration page,
3. Specify who should be notified for Logstash alerts using the Teams and Recipients fields. Auto-complete suggestions will be provided as you type. 
4. Copy the API Key by clicking on the copy button or selecting and Save.

Installing OpsGenie Output Plugin in Logstash

1. OpsGenie Logstash Output plugin is available in RubyGems.org. Install plugin

   bin/plugin install logstash-output-opsgenie

2. Change your Logstash configuration as described in Configuring Logstash to Send non-throttled events to OpsGenie section
3. Run Logstash.

Configuring Logstash to Send non-throttled events to OpsGenie

To send non-throttled events to OpsGenie, we will configure throttle filter to add a tag called throttled to events and configure OpsGenie Output plugin only to send events that don't contain throttled tag. Also, OpsGenie expects the data in a certain structure, so we will add some additional fields to events. To get more information about the fields added to events and what they mean, please refer to inline code documentation of the plugin. Example configuration will be like following:

filter{
	if [message] =~ /.*EXCEPTION.*/ {
	      throttle{
			period => 600
			before_count => 4
			after_count => 5
			key => "%{message}"
			add_tag => "throttled"
		}

		 mutate{
		 	add_field => {
				"opsgenieAction" => "create"
				"alias" => "%{host}"
			}
		 }
	}
}

Using OpsGenie Output plugin with Throttle filter can also prevent alert storming if you don't use Alert Deduplication functionality. Even if you use alert deduplication, sending excessive amount of requests may result in getting blocked.