Learn Sensu: Checks

Lesson 1: Handler sets

Thanks for your interest in our whitepaper on checks! Over the next several lessons, you’ll learn all about check definitions, including check tokens and hooks, proxy checks, check scheduling, and API keys.

Before we get started, a couple caveats: for the purposes of this course, we’ll assume you’ve gone through our Learn Sensu Go course (or have that baseline knowledge, such as how to create a check) and have downloaded Sensu (which you’ll need to do to follow along). Finally, don’t forget to register your download so you can start earning rewards on your Sensu journey!

With those out of the way, let’s get started! For our first lesson, we’ll go into handler set definitions, which allow you to use a single named handler set to refer to groups of handlers. If you’re going to be running checks, you need to decide where to route the results. That’s where handler sets come in. Check out our docs for the full overview:

Read Up on Handler Sets

As a monitoring event pipeline, Sensu lets you automate your monitoring workflows to suit your business’ needs, and handlers are a key part of that workflow. Standards introduced by the monitoring pipeline initiative use the following handler sets:

• alert (general alert mechanisms such as email, Slack, etc.)
• deregistration (handlers that deregister from cloud providers, configuration management, etc.)
• incident-management (PagerDuty, etc.)
• metric-storage (metrics handlers such as InfluxDB, etc.)
• remediation (handlers that do auto-remediation)

To use these handler sets, you’ll need to either create them with something like:

• sensuctl handler create alert --type set --handlers slack,email
• sensuctl handler create incident-management --type set --handlers pagerduty

Or you can edit existing checks with:

sensuctl edit handler <handler set name>

In the next lesson, we’ll dive into check tokens, which help you fine-tune your checks.

Lesson 2: Check tokens

In this lesson, we’ll discuss check tokens, which are check definition placeholders that the Sensu agent will replace with the corresponding entity definition attribute values (including custom attributes).

With tokens, you can fine-tune check attributes on a per-entity level with annotations and labels. Here’s a working example you can take from this monitoring checks template.

You can also adapt existing checks using that example.

Next time, we’ll get into check hooks, which let you lay the foundation for Sensu as a remediation tool.

Lesson 3: Check hooks

In this lesson, we’ll cover check hooks, commands run by the Sensu agent in response to the result of check command execution.

Learn How to Create a Check Hook

Uses for check hooks include:

• Capturing CPU usage in the event of a high alert
• Running a status command for a failed web server
• Introduction of check hooks as a simple remediation tool. Keep in mind, check hooks work best for remediation in simple use cases; if not used correctly, bad things can happen. For instance if your remediation restarts a failed service, and that service continues to fail, it will loop endlessly.

You can also add check hooks to an existing check. For example, assuming you have created the following hook definition for capturing the process tree (which would be useful during a high CPU usage event):

type: HookConfig
api_version: core/v2
metadata:
  annotations: null
  labels: null
  name: process_tree
  namespace: default
spec:
  command: ps aux
  stdin: false
  timeout: 60
  runtime_assets: null

You would then add that hook to your CPU check definition such that it gets executed whenever your check returns a critical event:

type: CheckConfig
api_version: core/v2
metadata:
  name: check-cpu
spec:
  command: check-cpu.rb -w 80 -c 90
  check_hooks:
  - critical:
    - process_tree
  runtime_assets:
  - sensu-plugins/sensu-plugins-cpu-checks:4.1.0
  - sensu/sensu-ruby-runtime:0.0.10
  subscriptions:
  - linux
  interval: 30
  timeout: 10
  ttl: 0
  stdin: false
  handlers:
  - slack

Next lesson, we’ll cover proxy checks, which let you monitor external resources (like a website).

Lesson 4: Proxy checks

In this lesson, you’ll learn about proxy checks, which let you monitor external resources on systems where a Sensu agent can’t be installed, like a network switch or a website.

Please note: proxy entities (created by proxy checks) do count towards your overall licensing count, so if you’re nearing the 100-entity limit for our free tier, proceed with caution. Learn more about the entity limit here.

With proxy checks, you can set up a website check with your site as the entity.

Learn How to Use Proxy Checks to Monitor a Website

And you can run an SNMP check with the network device as the entity:

type: CheckConfig
api_version: core/v2
metadata:
  name: check-router0-interface
  namespace: default
spec:
  command: check_snmp -H 192.168.0.1 -o 1.3.6.1.2.1.2.2.1.8.2
  interval: 60
  proxy_entity_name: router0
  publish: true
  round_robin: true
  runtime_assets:
  - sensu/monitoring-plugins
  subscriptions:
  - proxy

Learn More About Proxy Checks

That’s all for now! Next lesson, we’ll dive into check scheduling.

Lesson 5: Check scheduling

In this lesson, we’ll cover check scheduling, which — as the name implies — lets you schedule checks to go out at set times. You’ll use the interval, cron, and publish attributes to schedule checks; so, if you wanted to schedule a check to execute every 60 seconds, you’d set the interval attribute to 60 and the publish attribute to true.

The proxy check for monitoring your website also includes how to set up a round robin for proxy checks. For more on round robin, check out our docs.

Learn more about the differences between cron and interval checks in our docs.

Next lesson, we’ll teach you how to create and submit an event with API keys.

Lesson 6: API keys

In this (final!) lesson, you’ll learn all about API keys, including how to use them to create and submit an event.

API keys are long-lived authentication tokens that make it more convenient for Sensu plugins and other Sensu-adjacent applications to authenticate with the Sensu API. API keys are persistent, so they don’t need to be refreshed every 15 minutes like authentication tokens.

Read Up on API Keys

One use case of API keys is to create events without actually running a check. For example if you have a job that runs nightly and you want to create an event based on its success or failure, you could do so by creating an event for it. For this use case you would need to create a user that can submit events (and for security purposes, only submit events) via Sensu’s role-based access control (RBAC):

sensuctl user create event-submit --password='password' --groups=ops
sensuctl role create event-submit-only --verb create,update,delete \
--resource events
sensuctl role-binding create event-submit-only \
--role=event-submit-only --user=event-submit
sensuctl api-key grant event-submit

export SENSU_API_KEY=$(sensuctl api-key list --format json | \
jq -r '.[] | select(.username == "event-submit") | .metadata.name') 

From there, that user can submit an event to the Sensu API with curl:

curl -X POST -H "Authorization: Key ${SENSU_API_KEY}" -H 'Content-Type: application/json' -d '{ "entity": { "entity_class": "agent", "metadata": { "name": "'$(uname -n)'", "namespace": "default" } }, "check": { "output": "'$(echo OK && /usr/bin/true)'", "status": '"$?"', "handlers": ["slack"], "interval": 60, "metadata": { "name": "server-health" } } }' \
http://127.0.0.1:8080/api/core/v2/namespaces/default/events

You’ll also want to set the TTL for events submitted via the API so you can detect silent failures (creating what’s commonly known as a dead man’s switch).

That’s all for this course! If you haven’t already, we invite you to sign up for our email course on external services, where we dive into handlers, secrets management, mutators, and more. We hope you found the course useful, and either way, we’d love to hear what you think.