From 44 security mails in a minute to one summary a day

What happened on 25 October

Skald runs on its own server for eight websites. On 25 October we shipped a fix: security mail had only been reaching the team of the first site on the server, because each other site looked for its team in an empty file of its own. Once the fix went live, every site found its team at last.

The next run did exactly what it was built to do. It found every open security item on every site, saw each one as new for these people, and mailed each one. One mail per item, per site. One person got 44 mails in a minute, across 6 sites. Skald's 2.36.0 release notes (25 Oct 2026) record it in one line, and it is the reason that release exists.

Each of those 44 mails named a real open item: a possible exposure (an address on the site that answered with what looks like a private file, such as .env), or an advisory that matched a site's software. That is what makes alert fatigue hard to see from the inside. Every single alert named something real, and the pile is still useless.

Why alert fatigue costs more than a full inbox

A mail that says "something needs you" asks for a decision: open it now, or later. Forty-four of them in a minute ask for forty-four decisions, and the easy one to make is to stop reading them. A filter rule follows, or a muted folder. The next alert that matters lands in the same folder.

So the problem with too many alerts is less the time they take than what they teach. Each mail that turns out to be one of many makes the next one easier to skip. An alert system earns attention only while the alerts are rare enough to read and full enough to act on without opening ten tabs.

That gave us two rules for the redesign. First, the unit of mail is the person and the day, never the item. Second, each line in the mail says what it is, what to do, and links straight to it, so one reading is enough.

One mail per person, per day, at 01:00

Since 2.36 each person's security mail comes by default in a daily summary. Here is how it decides, read from the code that sends it.

• Who. On each site, the people on its team whose account is confirmed and who have security alerts on. Owners and admins have them on by default; viewers get none until they switch them on. A person on several sites gets one mail covering all of them.
• When. Once a day at 01:00 in the schedule's time zone. On Docker the scheduler runs it at SKALD_SUMMARY_AT; off stops it.
• What. A section per site, only for sites with something open, each in three parts: the open security items, the open to-dos from Overview › Insights (less the ones marked done or "not now"), and set-up problems the server's own health check finds, such as the daily run, the access log or the disk.
• How much. At most 10 lines per part and site, then "and N more" with a link to the rest.
• Nothing open. No mail.
• Once. Each send is claimed for the person and the day before it goes out. A restart or a second run the same day sends nothing; a send that fails is released again, so the next run retries.

Security items are marked New the first time a person sees them in any mail, and Still open after that, so the summary doubles as a reminder without a second kind of mail. Each send is written to the organisation's audit log: to whom, the day and how many items, never what was found.

What the summary says, for three sample sites

The mail is plain text, the same words in the HTML copy. Pick a day below and the summary is written by the same rules the code uses: the subject line's count, a section per site with something open, New and Still open, the 10-line limit, and no mail at all on a quiet day.

pick a day

Your Skald summary for 26 Oct 2026: 5 things to look at on 3 sites.

== example.com ==

Security (2, 1 new)
- New: Possible exposure on example.com: example.com/.env answered 200 (1,284 bytes)
  What to do: Open it yourself and check what it returns. Deny dotfiles and key files at the web server, for example in nginx: location ~ /\. { deny all; }  Then rotate any secret the file held.
  Open it: https://app.example.com/traffic/example.com/#threats/exposed?item=e3b8c41d07
- Still open: Advisory for example.com: PHP 8.3.11 on example.com: Update needed
  What to do: Update to the fixed release (Security > Your stack lists it).
  Open it: https://app.example.com/traffic/example.com/#threats/stack?item=a91f0c6e52

== northwind.example ==

To-dos to fix (2)
- Send old pages somewhere useful. 14 old page addresses answered 'not found' from 26 Sep to 25 Oct: people asked 41 times, search engines 63 times, AI 9 times.
  What to do: Redirect each to the closest live page, or answer 410 gone. Site > Not found suggests the page and gives the server line to copy, most asked first.
  Open it: https://app.example.com/traffic/northwind.example/#overview/insights?todo=lost-pages
- Give AI the Markdown copies it asks for. AI fetchers asked for the Markdown copy (.md) of 6 pages you have, 23 times from 26 Sep to 25 Oct, and got 'not found'.
  What to do: Serve a Markdown copy of each page at its .md address, or redirect .md to the page. One server rule covers all of them.
  Open it: https://app.example.com/traffic/northwind.example/#overview/insights?todo=ai-markdown

== shop.example ==

Set-up problems (1)
- Access log: /logs/access.log is readable but was last written 26 h ago
  What to do: Is this the site's live log? Check logs.dir and logs.base, and that the web server writes there.
  Open it: https://app.example.com/traffic/shop.example/#settings/integrations

You get this once a day (01:00 in the schedule's time zone), only when something is open, for every site you get security alerts for. Change it on Security > Alerts, Your security alerts: off, or the security items right away.

Right away, without the flood

Some people want a possible exposure the minute it is found, and that is a fair choice for a site holding customer data. So Right away stays, on Security › Alerts under "Your security alerts", as each person's own choice.

What changed is the shape. Before 2.36, each run's security step mailed each item on its own. Now the runs only keep the list of open items up to date, and after every site of a run has been read, one pass sends one mail per person with every item due on all of their sites. The 44 items of 25 October would arrive as a single mail.

Each item comes once. A reminder follows only while the item stays open, every 3 days by default (security.remind_days), and a summary that carried the item counts as a mail. Someone who chose Right away before the upgrade keeps it, now batched.

The other alerts were built quiet from the start

Security mail was the loud one. The site alerts (site down, logging stopped, server errors, a surge of not-found pages, a crawler burst, a new crawler, a traffic drop) already followed a quieter rule, and the summary leaves them as they are.

• An alert is mailed once, when it opens.
• While it stays open, the checks that find it again are counted, not mailed.
• If it reopens within 6 hours of its last mail, it is counted, not mailed.
• The counts go out in one daily digest at 08:00.
• Site down and logging stopped send one resolved line when they clear.

Alerts are also off until someone switches them on for a site, and each rule can be switched on or off by itself. The day-based alerts wait for figures that have been counted twice; a day the second count withholds never fires one. The rules and their thresholds are on the web analytics alerts page, and posting the same alerts to a channel is covered by Slack analytics alerts.

How to set it up, test it, and switch it off

The summary needs accounts (people signing in) and a mail server set in Settings › Mail. On Docker there is nothing else to do: the scheduler sends it at 01:00. Before trusting any new mail, print it first. This shows each person's summary without sending anything:

docker exec skald skald summary-mail --dry-run

That dry run earned its place on day one. We ran it on our own server, and it showed every site's links opening the first site's dashboard: a site with no dashboard address of its own had been taking the first site's. The fix shipped as 2.36.1 (release notes, 25 Oct 2026) the same day, before the first summary went out. The 2.35.0 security mails, the 44 among them, had carried the same wrong links.

Without Docker, add one hourly cron line, the scheduler's job on a Linux server. It sends once the site's local time is past 01:00, whatever the server's clock and summer time do (this is the line from the public repo's DEPLOY.md, with the paths of a default install):

0 * * * * skald . /etc/skald/env && cd /opt/skald && ./bin/skald --config /etc/skald/config.yml summary-mail --after 01:00 >>/var/log/skald/summary.log 2>&1

To change the hour on Docker, set -e SKALD_SUMMARY_AT=06:30; to stop it, set it to off. Each person can also turn their own security alerts off, or move them to Right away, on Security › Alerts.

What the summary leaves out, on purpose

A summary that held everything would become the next pile, so it keeps to what someone has to act on.

• Traffic figures are left out. The tiles and "What changed" live on Overview › Summary, where a period and a comparison give them meaning; see what changed this week.
• The site alerts keep their own road, as above.
• Set-up problems come only from checks that read files on the server; the one check that asks the hook receiver over the network is left out, and a problem that belongs to the whole server, such as the disk, is said once per mail.
• At 01:00 the to-dos are the ones the last daily run found; the security items are current from the live runs.

If one mail a day still feels like one too many, that is the point of the quiet day: when nothing is open, nothing comes. To see the whole set of daily questions this summary closes, read from reactive to proactive. To run it on your own sites, see the plans and the 30-day trial.