Monitoring servers and services with Sensu

0
sensu

When working with servers, it is imperative to ensure that everything works normally. Sensu allows us to alert you in case of malfunction or anomaly. The key word: pro activity. In other words, be aware of a failure before our users notice it. Interested? Let’s discover the power of Sensu!



Sensu allows to perform and you checks (state checks) on servers and services i.e.  it ensures that the services work normally  otherwise it  give alerts. It also collects metrics such as CPU load, memory or disk usage, and redirects them to third-party services.

Many other tools exist, however as explained in the definition of the monitoring plan. Sensu is modern, powerful, a simple and ergonomic interface. Moreover, it was designed to be agnostic in terms of language. Thus, most checks and plugins of other monitoring systems can be used with Sensu. We will see that it is possible to easily program our tests in the language of our choice.

 

Installation

Sensu comes in two slightly different versions: the community version, open source, and the enterprise version that adds some functionality. We will discuss on  the open source version here.

Sensu is composed of several interrelated bricks like as  sensu-client , sensu-server and sensu-api . But it combined in one package.

Sensu needs a storage engine. And  it uses Redis, and a transport to make converge the data from the clients to the server. Transport is provided by RabbitMQ or Redis .

The server

We were just talking over the transport, which can be either Redis or RabbitMQ. The advantage of Redis, it is already installed  and it's necessary for data storage. Its big drawback, it does not support encryption. Even if we set an authentication but  the password is transferred in clear on the network. RabbitMQ are its part and  integrates TLS directly. However, it requires the installation of Erlang and adds additional dependencies to our system which we would like so much to happen.

Redis secured

If Redis is to be used as a transport, two solutions are possible like as:

  • Have a private and secure network on which the servers can talk (not always possible),
  • Create an SSH tunnel between clients and the server to pass traffic to Redis.

It is this second solution that we will put in place. It is simple and flexible. In addition, if you have a private network, you will have to bind communications on the right interface and skip the ssh configuration.

On the server side, nothing more simple, we simply create a user who will receive ssh connections:

# We call it ssher, # we disable password # and we give it a limited bash, rbash 
adduser ssher - disabled - password - shell = / bin / rbash

There is not much to say here, we created the user "ssher", we disable the password and we give it a limited shell so that the user can not do much on our server.

You only have to make sure that you have authorized the ssh key connection .

Sensu server

The installation is done in our example on an Ubuntu server, it will be the same on a Debian. Let's attack!

# Make sure our listings are up to date and we install Redis 
apt - get update
Apt - get install redis - server

# Add the key and the repository to sensu 
wget - q http : //repositories.sensuapp.org/apt/pubkey.gpg -O- | Apt-key add - 
echo "deb http://repositories.sensuapp.org/apt sensu main" | Tee / etc / apt / sources . List . D / sensu . list 

# On our listings and we install Sensu 
apt - get update
Apt - get install sensu

Here it is installed. All you have to do is configure it. The configuration is in JSON and can be split into several different files. Sensu scan by default the set of files /etc/sensu/conf.d/*. It will also look for its configuration in /etc/sensu/config.jsonif it is present. I use this second option and reserve conf.dthe other configuration files.

# /etc/sensu/config.json { "repeat" : { "host" : "127.0.0.1" }, "transport" : { "name" : "repeat" }, "api" : { "host" : " 127.0.0.1 " , " port " : 4567 } }

   
JSON speaks here of itself. It tells sensu-server how to connect to Redis, that it should also be used as transport and finally the port and listener address of the API. If necessary, for other options, see the Sensu doc .

Uchiwa

Let's move on to the installation and configuration of Uchiwa.

Uchiwa is actually consists of a small server written in Go, which requests the sensu-api and exposes the data to an interface in AngularJS. To install it, nothing more simple:

Apt - get install uchiwa

The configuration is done in the file /etc/sensu/uchiwa.json.

{ "Sensu" : [ { "name" : "Sensu" , "host" : "127.0.0.1" , "SSL" : false , "port" : 4567 , "path" : "" , "timeout" : 5000 } ], "Uchiwa" : { "host" : "127.0.0.1" , "port" : 3000 , "refresh" : 10 , "user": "My-login" , "pass" : "my-mdp" } }


sensuHere refers to the "datacenter" as presented in the Uchiwa interface. We can have several Sensu servers that will be as many datacenters. All the servers will be gathered in the same interface (you note that it is an array).

Each datacenter has a name, here "Sensu". The information relating to sensuthis refers to the sensu-api parameters , as we have configured it just above.

uchiwaHere you can configure the address and port list, the refresh rate of the interface, and possibly users and their permissions. We have here only one user who will be the administrator. You will find more details on the possible configurations in the Uchiwa doc .

Customers

We installed our server but it does not monitore anything at all. To install the clients, it is very simple, on each server to monitor, it will install the package sensu , as we did previously for the server (I told you that everything was in the same package). We will see that we do not start the same services when it comes to the server or client.

Each client requires two configuration files: /etc/sensu/config.jsonand /etc/sensu/conf.d/client.json. config.jsonIs at a detail close to the one we configured for the server.

We are going to monitor a web server. Let's first see our configuration files:

# /etc/sensu/config.json { " redis " : { "host" : "128.0.0.1" },
 "transport" : { "name" : "redis" } }

As we saw above, the config.jsonis quite similar between the server and the client. A single detail changes, we no longer have the block api. Since this is only useful for the server, we do not need it here. You will notice that we define the address of Redis to localhost , because we will redirect the local traffic on the port of Redis, to our server Sensu via an ssh tunnel.

# /etc/sensu/conf.d/client.json { "client" : { "name" : "Web_server" , "address" : "9.8.7.6" , "subscriptions" : [ "default" , "webserver" ] } }

Let's explain a little bit about it. We give a name and address to our customer - the name must be unique and the address is that of the customer but it is purely informative - as well as subscriptions.

Sensu runs on a pubsub subscription model. Thus, each customer defines the type of checks they want to subscribe to. We could for example have webserver, database etc. We will return to this in more detail afterwards.

Ssh tunnel

We've been talking about it for a few paragraphs, it's time to put this in place. You understand, we configure our clients as if Redis was local, then we pass all this traffic through an ssh tunnel (thus encrypted) to the Sensu server.

First of all, you must generate ssh keys, then export them so that your client can establish the connection with the Sensu server. In case you need a reminder, everything is in the aforementioned article . I generate the keys directly for the root user on the client, free to you to choose another one, but it must be able to do port forwarding.

Once everything is done :

ssh -NNT -f -L6309:128.0.0.1:6309 ssher@ip_sensu_server

The local port 6379 (Redis) is redirected to the remote port 6379 via the user "ssher" of the server to the indicated ip. The otpions are -nNTused to: -Nnot execute remote commands; -nPrevent reading from stdin, -TDo not assign tty to the remote user. -fAnd -Lindicate that it is Forwarding a Local port.

Everything should work, you may find that ssh is running by doing ps aux | grep sshand that the connection is open by doing netstat -nap | grep 22. Only inconvenience, the connection could jump ... So we will make sure that it is always open using the program autossh. This small tool constantly checks that the connection is open and restarts when needed.


apt - get install autossh

autossh - M 0 - N - f - L6379 : 127.0 . 0.1 : 6379 ssher @ ip_sensu_server

-M 0Allows to disable the monitoring autosshand to restart the connection only when closing ssh. Direction man autosshif you want to know more. rc.localIs called directly by root, consider using sudoif you want to establish the connection from another user.

A small parenthesis, it is quite possible - it is even indicated - to monitor the Sensu server. Just fill in the client.json, the rest is already in place. Obviously, no need for ssh tunnel since Redis is already on the local machine.

Start-up

More than two tiny steps before opening Uchiwa in our browser. The services must be configured to start automatically at startup.


chown - R sensu : sensu / etc / sensu


update - rc . D sensu - server defaults

Update - rc . D sensu - api defaults
Update - rc . D uchiwa defaults

# For clients, only sensu-client 
update - rc . D sensu - customer defaults

That is the difference I mentioned earlier. Clients and server share the same package, but on clients, only the sensu-client is launched .

You may have noticed that the boot is configured with System V, and that from Debian 8 and derivatives (Ubuntu 16.04 in particular), the standard is systemd . Sensu does not have a systemd script yet. But do not worry, because systemd can automatically convert scriptsinit.d as needed.




We start all this little world with a big blow of service:

# Server 
service sensu - server start
Service sensu - api start
Service uchiwa start

# Customer 
service sensu - customer start

So, you only have to go to the ip of your server on port 3000 and hop, you are on the Uchiwa interface. I encourage you to put a domain name on it, to configure a reverse proxy to enjoy comfortably a cleaner address and why not, let us heat, pass all that in ssl.

 

Sensu Interface

uchiwa-interface

There are six tabs in the menu on the left:

  1. Events , which is normally empty. It is people when there are events (understand problems).
  2. Clients , includes all the servers monitored. By clicking on each server you will get more details about it: information and list of checks. You can also click a check to see its status, history, and warning and critical floor values.
  3. Checks list all checks as well as their subscribers.
  4. Stashes gathers the stashes . The stashes are JSON documents that can be created and accessed by the handlers. A stash is also created when you silently shut down a client. Stashes can be used to store information between several executions of a check. Thus a check can access the information of a previous state and act consequently. For example, the web server is no longer responding. Before you trigger an alert, you may have to wait a little to see if it is not just restarting after an update. We check in the stash, if it is the first occurrence, then we store this result in the stash, if it is the second (or more) consecutive occurrence, then the alert is triggered.
  5. Aggregates , as its name suggests, are aggregates . Sensu allows to aggregate the contents of several checks in order to treat them as one. For example, it is possible to interrogate the aggregates to determine if at least 80% of the load balancers are functional.
  6. Datacenters , simply regroups the different datacenters (understand API Sensu) as we have previously discussed in the configuration.

We walked around. Now let's pop this interface with useful information, I named the checks!

The checks

I think you begin to understand this, a check is a command executed by the client Sensu and which aims to define the state of a service or, as explained in the introduction, collect a metric. Each check must return an exit code that defines the status of the service:

  • 0 for OK,
  • 1 for Warning,
  • 2 for Critical
  • Other for Unknown.

Checks can also return data to stdin or stderr . These data will be displayed in Uchiwa for state checks and will be used to collect data in the case of metric checks .

The pubsub model

We have spoken very quickly about this, and Sensu is based on a subscription model. Thus, you define the checks on the Sensu server and you assign to each subscribers , that is to say the type of client that will have to carry out this check. Then, during configuration, each client specifies the type of subscriptions to which it subscribes . For example, we can define checks as default (cron, iptables, mail), and others more specific webserver, database etc.

This system provides great flexibility because each check can be assigned several subscribers and each client can have several subscriptions , let's take a concrete example. I define on my server two checks: "nginx" and "mysql". Nginx will apply to both my loads balancers and my web servers, so it will have "subscribers": ["loadbalancer", "webserver"]. The check will apply only to mysql database servers: "subscribers": ["database"].

Not surprisingly, my load balancer client will be configured to subscribe to loadbalancer checks subscriptions": ["loadbalancer"]. My web server meanwhile, also database office, no problem: subscriptions": ["webserver", "database"]. You see the power?

Finally, know that there are also checks said standalone . You define them directly on the client and they run independently of the checks published by the server. This option offers a decentralized alternative to subscriptions. It can be used both as a supplement and as a main strategy. I will not talk any more in this article, so I leave you with the doc if necessary.

Creating checks

As we said in introduction, the great strength of Sensu is to allow to write checks in any type of language. For example, we will use bash scripts. Easy for simple tasks, bash is installed by default on all servers, so there is no prerequisite.

This first check verifies that the cron service is running successfully.

#! / Bin / bash

# Filter with grep to find the line containing # "active (running)" 
cron = `service cron status | Grep "active (running)" `


# If the line is not found, then the variable is empty if [ - z "$ cron" ] Then # generating an error message and an exit code critial (2) 
  echo "Ooops, cron has stopped" exit 2 else # here success message and exit code to 0 
  echo "All good!" Exit 0 fi
  

This check ci will ensure that our web server is running and that it responds without error.

#! / Bin / bash

# On the server to localhost by asking only the header # filter the response to keep only the line with "HTTP" 
web = `curl -s -D - localhost -o / dev / null | Sed -n '/ HTTP / p'`


# If the line contains code 200, ok if [[ "$ web" == * 200 * ]] then 
  echo "Webserver up & running" exit 0
  

# Else, there is a problem else 
  echo "Houston, we have a problem" exit 2 fi

The check files must obviously be executable and present on each server to be checked. They are to put in /etc/sensu/plugins/. Once this is done, we will declare them to the server via the configuration file /etc/sensu/conf.d/checks.json.

Note that these two checks are given as an example. I gathered a few on a Gist , which these two but a little more worked. If you want to monitor Mongo, I also wrote a nodejs module for this purpose.

{ "Checks" : { "cron" : 
{ "command" : "cron.sh" ,
 "interval" : 50 ,
 "handlers" : [ "default" ],
 "Subscribers" : [ "default" ] }
 "httpd " : { " command " : " httpd.sh " , 
" interval " : 50 ,
" handlers " : [ " default " ],
 " Subscribers " :[ "Webserver" ] } } }
   
 

Our file contains here the two checks we have previously written. Each check has a name and parameters, let's review these:

command
The command to execute. You will notice that the executables present in it pluginsare automatically integrated to the path of Sensu. It is thus possible to use relative paths here.
interval
The interval at which to check.
handlers
The action to be taken in the event of an alert (or to call in the case of a metric check).
subscribers 
What type of server the check is destined for.

The socket input

This is a special kind of check. Neither subscripion nor standalone ... each client Sensu opens a TCP socket on port 3030. Thus, all programs have the possibility to post the result of a check in JSON format.

On the one hand, it is not possible to check everything, but in the event of a major problem, a service can rely on Sensu to raise an alert (problem connecting to a BDD or external API etc); On the other hand, it is possible to specify a period after which an alert will automatically be lifted if a new check result has not been published. This is especially valuable for automatic backup policies !

In an example similar to that of this link, it is enough to accompany the backup of a check with TTL of 24 hours, so that an alert is lifted if no other is published at this issue. In one line, here is what it could give:

Echo '{"name": "backup_desktop", "ttl": 86400, "output": "rsync back up completed", "status": 0}' | Nc localhost 3030

So simple, but devilishly effective!

Handlers

When an event is raised (a mistake in general, or a metric check), an action must be executed. Sensu then calls the handler (s) defined in the checks configuration. You must therefore define the handler (s) you want to use. In the previous check, we specified the handler defaultfor cron, and httpd. These handlers are configured in /etc/sensu/conf.d/handlers.json.

{ "Handlers" : { "default" : { "type" : "set" , "handlers" : [ "notify" ] }, "notify" : { "type" : "pipe" , "command" : "/ etc /sensu/handlers/notify.js " } } }
   

There are several types of handlers . We will use here pipe, which redirects the output of the check to a command, and from setwhich allows to regroup several handlers to operate all at once. In the example above, the default group does not serve much as it comes back to the same as calling directly notify . Nevertheless, we gain flexibility because if we wish to add a handler to all our checks using default, we will have only one line to modify.

In our example, Sensu simply pipes the result of checking to notify.js. Moreover this wonderful little handler that I wrote is in charge of sending the notifications both by mail and via Slack.

Again, make sure that all files belong to Sensu, checks and handlers must also be executable. Finally, restart the server.

Chown - R sensu : sensu / etc / sensu
Chmod + x / etc / sensu / plugins / *
Chmod + x / etc / sensu / handlers / *

Service sensu-server restart

You should see your checks appear in Uchiwa.