How to build highly expressive check results with sensu-wrapper

I’m gearing up to attend my first Sensu Summit this year, and have gone back and watched last year’s talks that I missed. There were a lot of great talks!

One technical talk that really caught my eye was Lee Briggs on sensu-wrapper. Lee introduced a wrapper utility he wrote to make it easier to use the Sensu client socket to monitor shell executables for correct operation. In this post, I’ll break down why his sensu-wrapper command is so useful.

Why is sensu-wrapper useful?

There are a lot of situations where you want to monitor the output of a command-line utility running outside of the Sensu client scheduler: sometimes you’re running a command as part of normal service operation in an unscheduled manner, or maybe you’re running a critical operational command from crond, atd, or as a systemd timer. Even though they’re running outside the control of the Sensu client scheduler, you may still want to drive their output and status into your Sensu event pipeline, firing off the appropriate remediation or alerting actions when they fail.

The Sensu 1.x client provides a handy client socket facility to quickly generate check result from custom application code. You write the appropriate JSON to the client socket, and the Sensu client sends it off as a check result, adding some client decoration information. Here’s a quick example showing how to send a check result via UDP to the default socket using ncat:

echo '{"name": "udp_socket_test", "output": "udp socket test", "status": 2}' | nc -4u localhost 3030

The Sensu client will use the JSON passed into the socket to fill in the check payload when publishing a check result. Here’s the corresponding snippet from the client log:

{
  "timestamp": "2018-07-17T03:56:22.064816+0000",
  "level": "info",
  "message": "publishing check result",
  "payload": {
    "client": "new-test-client",
    "check": {
      "name": "udp_socket_test",
      "output": "udp socket test",
      "status": 2,
      "executed": 1531799782,
      "issued": 1531799782
    }
  }
}

In practice however, using the socket for existing command-line utilities is easier said than done. Every time I want to monitor the return status of a standard command-line tool, I end up writing bespoke glue scripts to build the JSON I need. Trying to mangle shell command output into JSON using native shell pipelines or aliases can be frustrating — this skill was taught briefly at Hogwarts as part of the short-lived Dark Arts curriculum. Lee’s wrapper provides a great way to avoid writing that fragile glue code and saving my sanity.

What does Sensu wrapper do?

The sensu-wrapper command is a Golang utility that provides a thin wrapper around existing shell commands and outputs reasonable JSON to the client socket. Take a look in the github repository at the main.go and command.go files. I’m not Go fluent, but the logic for the wrapper is still pretty easy for me to follow.

Lee points out that Golang has a couple of great advantages. First, it has first-class JSON handling support, so it reduces the burden of working with JSON behind the scenes. Second, static compiled binaries are small and fast. There’s no need to pull an interpreted language engine into memory to reformat command output into a Sensu check result.

The sensu-wrapper command does a couple of very nice things when handling non-zero return status for the command it’s wrapping, namely:

• Coercing a non-zero return status into Nagios compliant status code range (0-3)
• Automatically appending the stderr with the stdout when return status is non-zero, to ensure all debugging information is available in the check result

It’s very easy to test the sensu-wrapper operation, using the dry-run option -d. Here’s a basic dry-run example that works on my Linux instance:

sensu-wrapper -d -n "fstab_stat" stat -c "%F" /etc/fstab

The dry-run output shows the JSON to be passed to the client socket:

{
  "name": "fstab_stat",
  "command": "stat -c %F /etc/fstab",
  "status": 0,
  "output": "regular file\n"
}

If you repeat that command and try to stat a file that does not exist, you’ll see the output will include the stderr message and the status will be non-zero.

Build highly expressive check results with Sensu wrapper

The sensu-wrapper exposes a set of options that allow you to build expressive check results as if they were generated from a valid check. The options include:

--ttl value, -t value       The TTL for the check (default: 0)
--timeout value, -T value   command timeout in seconds (default: 0)
--source value, -s value    The source of the check
--handlers value, -H value  The handlers to use for the check
--json-file value, -f value JSON file to read and add to output
--json value, -j value      JSON string to add to output

With these options, you can mimic most of the behavior of a scheduled check, like setting:

• A time to live (TTL) value, so that Sensu will alert if the named check result doesn’t repeat often enough
• A source, so the check acts as a proxy check
• Which handlers to use For any other metadata you need to add, you can merge it in either via the -j option or by reading in a JSON file with the -f option.

Here’s an example of sensu-wrapper using all the available options:

./sensu-wrapper -d -n "test" -s "proxy_client" -H "handler1, handler2" -t 120 -T 60 -j '{ "extra":"value"}' -f /tmp/more.json echo "test output"

The JSON file /tmp/more.json looks like this:

{
  "high_flap_threshold": 60,
  "low_flap_threshold": 20, 
  "more": {
    "att01": "val01",
    "att02": "val02"
  }
}

And the resulting JSON output of the dry-run:

{
  "command": "echo test output",
  "handlers": [
    "handler1, handler2"
  ],
  "high_flap_threshold": 60,
  "low_flap_threshold": 20,
  "more": {
    "att01": "val01",
    "att02": "val02"
  },
  "name": "test",
  "output": "test output\n",
  "source": "proxy_client",
  "status": 0,
  "ttl": 120
}

Sensu API supported as well!

Not running a Sensu client locally? No problem — the wrapper also implements Sensu API results endpoint access so you can send the check result directly to the Sensu API, even if you don’t have a Sensu client running. I love this! With this feature, I can now instrument systemd service unit failures at boot time for services that get run after the network is available even if the Sensu client isn’t running yet, or has failed to run! You can even inform the Sensu server that the Sensu client failed to start!

But I digress.

Like I said, sensu-wrapper is a really helpful utility. I hope this post has given you some food for thought on just how powerful it is, and how the extension of Sensu functionality can be a lot of fun. Feel free to leave any questions in the comments below, and don’t miss this year’s Summit to catch more great talks — including another one from Lee on monitoring Kubernetes resources and cluster components. I’m looking forward to being there in person this time!