PagerDuty

Although a turnkey integration is available for email based notifications, organizations may wish to send notifications to other platforms in use internally. This recipe documents how organizations can customize the "Alert Manager" system addon configuration and send the alerts to Pager Duty

Configure PagerDuty ¶

If you would like to setup notifications via PagerDuty, you will need an existing PagerDuty account or create a trial for testing here. To setup the alerting to your PagerDuty, you will need a Service Integration Key

• Go to PagerDuty > Services > Service Directory

• Create a New Service

• Configure the new service with name, select Integration Type "Prometheus" and other settings

• Go to the created service > Integrations and copy the "Integration Key" to use in later steps

Configure Alert Manager ¶

Follow the below steps to customize the Alert Manager "System Addon" for integration with Pager Duty.

• Navigate to Infrastructure -> Addons

• Create a new addon with using "Customize System Addon" option.

• Create a new version for the customized Alert Manager addon.

• Download the default configuration manifest.

• Add a custom receiver for PagerDuty to the "receivers" section of the configuration. Use the "service_key" from the Integration Key that you copied from the "Configure PagerDuty Integration" step.

receivers:
- name: default-receiver
- name: rafay-webhook
  webhook_configs:
  - http_config:
      tls_config:
        ca_file: /etc/config/certs/ca.crt
        cert_file: /etc/config/certs/client.crt
        insecure_skip_verify: true
        key_file: /etc/config/certs/client.key
        server_name: '{{{ .V2EventFrameworkFQDN }}}'
    url: https://{{{ .V2EventFrameworkFQDN }}}/event/v1/alert/raise
- name: myalerts
  pagerduty_configs:
  - send_resolved: true
    service_key: 04exxxxxxxxxxx33975

• Next add the custom receiver for PagerDuty alerts to the routes configuration in "route" section

route:
  group_interval: 5m
  group_wait: 10s
  receiver: rafay-webhook
  repeat_interval: 3h
  routes:
  - continue: true
    match_re:
      severity: .*
    receiver: rafay-webhook
  - match_re:
      severity: .*
    receiver: myalerts

• Upload the modified configuration to the addon

• Save the custom Alert Manager addon version

Update Cluster Blueprint ¶

Follow the steps described below to deploy the customized Alert Manager configuration for PagerDuty to your clusters.

• Go to Infrastructure > Blueprints

• Add the custom Alert Manager addon to the custom blueprint for the cluster

• Update the new blueprint version with the customized Alert Manager addon to the clusters

• The new system Alert Manager addon will be deployed to the clusters

In a few minutes, the blueprint will be published successfully and the customized Alert Manager addon is deployed to your cluster.

Verify Configuration ¶

Once there are alerts generated in the clusters, you should see the notifications in PagerDuty. An illustrative example is show below.