Alerts

FlareWatch sends a message when a monitor goes down and again when it comes back up. Set the channels in notification.webhook in packages/config/src/worker.ts. Each entry needs a url. Give it a template and FlareWatch writes the message for you, or write the payload yourself as in Anything else.

notification: {
  webhook: [
    { url: 'https://hooks.slack.com/services/...', template: 'slack' },
    {
      url: 'https://api.pushover.net/1/messages.json',
      template: 'pushover',
      options: { token: 'app-token', user: 'user-key' },
    },
  ],
},

Webhook URLs usually contain a secret, and a fork of a public repo is public too. To keep them out of git, put them in the FLAREWATCH_WEBHOOKS GitHub secret instead, as JSON in the same shape:

[{ "url": "https://hooks.slack.com/services/...", "template": "slack" }]

Each deploy copies the secret to the monitor Worker. It alerts these webhooks as well as the ones in worker.ts. Tokens in headers or options can go in the secret too. The Worker skips a broken entry and logs why. The others still alert. If an entry has a timeout out of range, the Worker uses the default timeout. The entry still alerts. The same goes for a field the Worker doesn’t know: the Worker logs it, ignores it and still alerts the entry. The Worker removes a monitor ID in monitors that isn’t in your config and logs how many it removed. If none of the listed IDs remain, the entry gets no alerts.

A monitor in an active maintenance window sends no down alert. If it’s still down when the window ends, it alerts then. An outage that alerted before the window still sends its recovery.

Routing

By default every channel gets every monitor’s alerts. Give a channel monitors to send it only those monitors’ alerts:

webhook: [
  { url: 'https://hooks.slack.com/services/...', template: 'slack' },
  { url: 'https://ntfy.sh/db-team-topic', template: 'ntfy', monitors: ['db', 'backup'] },
],

Here Slack gets everything and ntfy gets only db and backup. An empty list sends a channel nothing. The config check rejects an ID that isn’t one of your monitors. A monitor that no channel takes doesn’t alert at all. skipNotificationIds wins over monitors: a skipped monitor sends nothing, even to a channel that lists it.

Summary alerts

Set notification.summaryAfter to an integer from 2 to 50 to combine a busy run’s alerts. Leave it out to keep one message per alert, as long as the check run can afford one request for each.

FlareWatch counts alerts after routing to each webhook. At or above the threshold, that webhook gets one message listing its alerts, grouped as down, recovered, still down, reminders and expiry warnings. A webhook below the threshold keeps individual messages. When the run can’t afford them all, the ones that don’t fit go out as one summary, with or without summaryAfter. Summaries use each template’s existing message or reason text, with no new payload fields. The title names FlareWatch and the alert count. The target field is empty and the duration is zero.

If a webhook accepts a summary, every listed alert counts as delivered. A refused summary keeps the existing rules for each alert: down alerts retry, and recoveries, error changes, reminders and expiry warnings do not. A summary costs one subrequest per webhook. A channel can refuse a summary that is over its message limit, as it can an individual alert.

Dependencies

Some monitors reach their target through something else, like a reverse proxy, a VPN or one server that runs several apps. When that goes down, every monitor behind it fails too. Put it in dependsOn and you get one alert, not one per monitor:

{ id: 'proxy', name: 'Reverse proxy', method: 'GET', target: 'https://proxy.example.com/health' },
{ id: 'app', name: 'App', method: 'GET', target: 'https://app.example.com', dependsOn: ['proxy'] },
{ id: 'wiki', name: 'Wiki', method: 'GET', target: 'https://wiki.example.com', dependsOn: ['proxy'] },

• While the proxy is down, App and Wiki don’t alert. The proxy’s alert ends with Also down: App, Wiki, naming the ones already down when it goes out. A long list ends with “and N more”.
• A monitor with dependsOn waits one extra check, about a minute, before its first alert. So an app that fails a minute before its proxy is still covered.
• When the proxy is back, anything still down behind it alerts on its own.
• A monitor that already sent a down alert still sends its recovery, even while the proxy is down.
• The status page and History show every monitor as it is. Only alerts change.
• This holds across channels. A channel whose monitors lists App but not the proxy gets nothing while the proxy is down, because App’s alert is held back and the proxy’s goes to other channels. Add the proxy to that channel’s monitors too.

Dependencies can have their own dependencies, and heartbeats can use dependsOn too. The config check rejects unknown ids and loops.

A monitor that fails again within 15 minutes of recovering reopens the same outage, so a flapping target doesn’t fill History. It sends a new down alert once it has been down for the whole grace period again. A reopened outage ends only after the monitor has stayed up for 15 minutes. Its recovery alert goes out then, and the outage counts as ending when the monitor came back up.

If no webhook accepts a down alert, FlareWatch tries it again on each check run, up to 10 times. After that it stops alerting about that outage. A check run can make a fixed number of requests, and the checks use most of them. When more down alerts are due than the run can send, the rest go out on the following runs, in config order. Waiting doesn’t count as a try. Recovery alerts, error changes and reminders are sent once, without retries. A recovery that a run has no requests left for waits for a later run. So does the recovery of an outage that ends while its down alert is still going out. A waiting recovery is dropped if the monitor goes down again first, stops alerting, or every webhook is removed. An outage sends at most 5 error changes, and 5 more each time it reopens. An error change that no webhook accepts doesn’t count toward the 5.

A recovery alert only follows a down alert that went out. If you remove every webhook while a monitor is down, its recovery goes unannounced.

Channels

Anything else

Leave out template and write the payload yourself. $MSG is replaced with the message. This reaches Signal, WhatsApp or SMS through a gateway such as CallMeBot or Twilio.

{
  url: 'https://example.com/webhook',
  payloadType: 'json',
  payload: { text: '$MSG', channel: '#alerts' },
}

payloadType can also be 'param' (query string) or 'x-www-form-urlencoded'. Both send the fields of payload, so it must be an object.

Settings