Alertmanager

• Introduction
• Step 0: Setting Up Telegram Integration
• Step 1: Setting Up Pushover Integration
• Step 2: Setting Up PagerDuty Integration
• Step 3: Configuring Juju Alertmanager
• Step 4: Test The Receivers

Introduction

In this tutorial, we’ll guide you through the process of integrating the Juju Alertmanager Charm with external notification services. These integrations enable you to receive alerts and notifications from COS Lite directly on your preferred messaging platform. For the sake of simplicity, we’ll cover Telegram, Pushover and PagerDuty integrations, but the steps demonstrated will be fairly similar to any other receiver.

Before proceeding with the integration steps, ensure that you have COS Lite deployed in your environment. If you haven’t deployed COS Lite yet, you can follow the detailed instructions provided here.

Step 0: Setting Up Telegram Integration

To integrate Alertmanager with Telegram, you need two pieces of information: your Telegram bot API token and chat ID . These will be used to create an Alertmanager Telegram Receiver, which routes alerts to your Telegram chat.

Set Up Your Telegram Bot

1. Open the Telegram app and find BotFather.
2. Type /newbot to create a new bot and follow the prompts.
3. Once created, you’ll receive a message containing your API token key.

Obtain the Chat ID

1. Add your bot to a group chat.
2. Interact with the bot by sending a dummy message starting with /.
3. Send an HTTP request to your bot’s API endpoint: https://api.telegram.org/bot<bot_api_token>/getUpdates.
4. Retrieve the chat ID from the response JSON.
5. If the request returns an empty response, remove and re-add the bot to the group chat, then retry the request.

Step 1: Setting Up Pushover Integration

Integrating Alertmanager with Pushover requires obtaining your Pushover User Key and Token. These credentials are necessary to create an Alertmanager Pushover Receiver, enabling notifications to be sent to your Pushover account.

Obtain Pushover User Key and Token

1. Sign up for a Pushover account if you haven’t already.
2. Log in to your Pushover dashboard.
3. Create a new application and note the generated API token.
4. Retrieve your User Key from your Pushover dashboard.

Step 2: Setting Up PagerDuty Integration

Before diving into integrating Alertmanager with PagerDuty, it’s essential to grasp a few concepts within PagerDuty:

1. PagerDuty Service: This is where your alerts are directed. Services have integration types; for our purposes, we’ll focus on Prometheus (using PagerDuty EventsV1) or EventsV2, offering more control over alerts.
2. PagerDuty Event Orchestration: This automates the flow of incoming alerts. It allows you to set up flows for each alert, routing them to services without the need for routing through AlertManager. Advanced rules can be set, but we’ll keep it simple for this tutorial.

Integration via PagerDuty Service

1. Navigate to the Services menu and select Service Directory.
2. If adding to an existing service, click its name, then the Integrations tab, and Add a new integration. For a new service, follow the documentation for creating a new service.
3. Choose Events API v2 or Prometheus from the Integration Type menu. For this tutorial, we’ll select Events API v2.
4. Save the integration by clicking Add Service or Add Integration. Copy the Integration Key provided for later use.

Integration via Event Orchestration

For this method, we’ll create two PagerDuty services and utilize Event Orchestration to route alerts:

1. Create two PagerDuty services (e.g., Juju Alertmanager Global Service 1 and Juju Alertmanager Global Service 2) as explained in the previous step.

2. In the PagerDuty web app, navigate to Automation > Event Orchestration and select or create your desired orchestration.

3. Create a service route for each service. For example, route alerts containing “hola” in the event summary to the first service, and route the rest to the second service.

4. Go to Integrations, copy your default Integration Key, and store it safely for later use.

Step 3: Configuring Juju Alertmanager

To configure Juju Alertmanager to work with Telegram, Pushover and PagerDuty, you’ll need to draft an alertmanager.yml configuration file and a templates.tmpl template file, replacing the placeholders with the information retrieved from previous steps.

Drafting alertmanager.yml

Below is a basic alertmanager.yml configuration file to integrate Juju Alertmanager with Telegram, Pushover, and PagerDuty. Here’s an overview of the main settings:

• Routing Configuration:
  • receiver: Specifies the default receiver for alerts, set to ‘telegram’.
  • group_by: Groups alerts based on Juju model UUID, application, and model.
  • continue: Determines whether to continue processing additional routes after a match is found.
  • group_wait, group_interval, repeat_interval: Control grouping and repetition of alerts; unresolved alerts are sent out every 1 hour.
• Receiver Configuration:
  • Configures four receivers: ‘telegram’, ‘pushover’, ‘pagerduty_service’ and ‘pagerduty_orchestration’.
  • Specifies settings for Telegram, Pushover, and PagerDuty, including API tokens, chat IDs, user keys, and tokens.

global:
  resolve_timeout: 5m

route:
  receiver: 'telegram'
  group_by:
    - juju_model_uuid
    - juju_application
    - juju_model
  continue: false
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 1h
  routes:
    - receiver: 'telegram'
      group_wait: 10s
      match_re:
        severity: critical|warning|none
      continue: true
    - receiver: 'pushover'
      group_wait: 10s
      match_re:
        severity: critical|warning|none
      continue: true
    - receiver: 'pagerduty_service'
      group_wait: 10s
      match_re:
        severity: critical|warning|none
      continue: true
    - receiver: 'pagerduty_orchestration'
      group_wait: 10s
      match_re:
        severity: critical|warning|none
      continue: true

receivers:
  - name: 'telegram'
    telegram_configs:
      - bot_token: YOUR_TELEGRAM_BOT_TOKEN
        chat_id: YOUR_CHAT_ID
  - name: 'pushover'
    pushover_configs:
      - user_key: YOUR_PUSHOVER_USER_KEY
        token: YOUR_PUSHOVER_TOKEN
  - name: 'pagerduty_service'
    pagerduty_configs:
      - routing_key: YOUR_PAGERDUTY_ROUTING_KEY_1
  - name: 'pagerduty_orchestration'
    pagerduty_configs:
      - routing_key: YOUR_PAGERDUTY_ROUTING_KEY_2

Drafting templates.tmpl

This file defines message templates for notifications sent to Telegram and Pushover. Each template includes information about the alerts, such as summary, description, severity, status, start and end times, and a URL. These templates can be customized as needed, but for the sake of simplicity in this tutorial, they are kept straightforward.

{{ define "telegram.default.message" }}
{{ range .Alerts }}
*Alert:* {{ .Annotations.summary }}

*Description:* {{ .Annotations.description }}

*Severity:* {{ .Labels.severity }}
{{ end }}
{{ end }}

{{ define "pushover.default.message" }}
{{ range .Alerts }}
Alert: {{ .Annotations.summary }}
Description: {{ .Annotations.description }}
Severity: {{ .Labels.severity }}
{{ end }}
{{ end }}

Applying Configuration

Run the following command to apply the configuration to the Alertmanager charm:

juju config alertmanager \
  config_file='@/path/to/alertmanager.yml' \
  templates_file='@/path/to/templates.tmpl'

Replace /path/to/alertmanager.yml and /path/to/templates.tmpl with the actual paths to your configuration files.

Step 4: Test The Receivers

To ensure that the integration with Telegram, Pushover and PagerDuty is functioning correctly, you can utilize the existing watchdog alert that periodically fires. This alert serves as a convenient way to verify that notifications are being delivered to both platforms.

Pushover Results

Please check your Pushover app on your device to view the notification.

Telegram Results

Please check your Telegram chat on your device to view the notification.

PagerDuty Results

Please check your PagerDuty app on your device to view the notification.

Thank you for following along with this tutorial. If you have any questions or need further assistance, feel free to reach out. Happy monitoring!