Prometheus : Set Alert Notification (Email)2021/09/17 |
This is the Alert Notification Settings on Prometheus.
There are many way to receive Alerts like Slack, HipChat, WeChat and others, though, on this example, Configure Alerting with Email Receiver.
For more details of Alerting, Refer to the official documents.
⇒ https://prometheus.io/docs/alerting/configuration/ |
|
[1] |
For Email notification, it needs SMTP Server.
On this example, it based on the environment that SMTP Server is running on localhost. |
[2] | Install Alertmanager on Prometheus Server Host. |
root@dlp:~# apt -y install prometheus-alertmanager |
[3] | Configure Prometheus Alert Settings with Email notification. |
root@dlp:~#
mv /etc/prometheus/alertmanager.yml /etc/prometheus/alertmanager.yml.org root@dlp:~# vi /etc/prometheus/alertmanager.yml # create new global: # SMTP server to use smtp_smarthost: 'localhost:25' # require TLS or not smtp_require_tls: false # notification sender's Email address smtp_from: 'Alertmanager <root@dlp.srv.world>' # if set SMTP Auth on SMTP server, set below, too # smtp_auth_username: 'alertmanager' # smtp_auth_password: 'password' route: # Receiver name for notification receiver: 'email-notice' # grouping definition group_by: ['alertname', 'Service', 'Stage', 'Role'] group_wait: 30s group_interval: 5m repeat_interval: 4h receivers: # any name of Receiver - name: 'email-notice' email_configs: # destination Email address - to: "root@localhost" # create new # for example, monitor node-exporter's [Up/Down] groups: - name: Instances rules: - alert: InstanceDown expr: up == 0 for: 5m labels: severity: critical annotations: description: '{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes.' summary: 'Instance {{ $labels.instance }} down'
root@dlp:~#
vi /etc/prometheus/prometheus.yml # line 14 : confirm - (Alertmanager Host):(Port) alerting: alertmanagers: - static_configs: - targets: ['localhost:9093'] # line 23 : add alert rules created above rule_files: # - "first_rules.yml" # - "second_rules.yml" - "alert_rules.yml"root@dlp:~# systemctl restart prometheus prometheus-alertmanager root@dlp:~# systemctl enable prometheus-alertmanager |
[4] | If [node-exporter] is down, following Email is sent. (mail body is HTML) |
root@dlp:~# mail Return-Path: <root@dlp.srv.world> X-Original-To: root@localhost Delivered-To: root@localhost Received: from localhost (localhost [IPv6:::1]) by dlp.srv.world (Postfix) with ESMTP id E6B7AA2166 for <root@localhost>; Thu, 16 Sep 2021 22:47:07 -0500 (CDT) Subject: [FIRING:1] InstanceDown (node01.srv.world:9100 Hiroshima example critic al) To: root@localhost From: Alertmanager <root@dlp.srv.world> Message-Id: <1631850427945847646.13718895160637918488@dlp.srv.world> Date: Thu, 16 Sep 2021 22:47:07 -0500 Content-Type: multipart/alternative; boundary=50fcf05622e22fc2a3dd0a59dbdcefd36 5940b2cfc18f77d141da42ca370 MIME-Version: 1.0 --50fcf05622e22fc2a3dd0a59dbdcefd365940b2cfc18f77d141da42ca370 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=UTF-8 ..... ..... |
Sponsored Link |