On-call and incidents

Connecting Grafana and Alertmanager

Getting monitoring alerts straight to the person on call.

Your monitoring system posts an alert to our URL; an incident opens and the responder is notified. When the alert clears, the incident closes by itself.

Getting the URL

Duty → rotation → “Integration” → “Generate webhook token”. You are given a URL like this:

POST https://taskfly.az/api/duty/incidents/<token>

Grafana Alerting

Alerting → Contact points → Add contact point → Integration: Webhook. Put the URL above in the URL field, method POST. Nothing else needs configuring — Grafana's default body is accepted as-is.

Prometheus Alertmanager

receivers:
  - name: taskfly-duty
    webhook_configs:
      - url: 'https://taskfly.az/api/duty/incidents/TOKEN'
        send_resolved: true
send_resolved: true matters — without it the incident stays open after the alert clears and has to be closed by hand.

How an alert is read

Alert fieldIn the system
annotations.summaryThe incident title (falls back to labels.alertname)
labels.severityPriority: critical / high / warning / info
annotations.descriptionThe incident body — together with labels and the dashboard link
fingerprintThe dedup key — a repeated alert does not open a second incident
statusfiring → opens, resolved → closes
Every alert in a request becomes its own incident — alerts in a group do not overwrite each other.