No deploy without a monitor.

On 29 May, a feature for extra quota policy checks reached Service Control without a feature flag and without error handling on the new path. On 12 June, a policy change produced blank fields, and the binaries crashed worldwide with a null pointer error. Google wrote that its first incident report came about an hour after the crashes began, because the Cloud Service Health infrastructure was itself down. It also wrote that for some customers, the monitoring infrastructure they ran on Google Cloud was failing too, leaving them without a signal of the incident.

What a check from outside showsThat last sentence makes the case for external monitoring in the operator’s own words: monitoring on the same platform fails with it. Perstat checks from outside, from up to 6 regions with a quorum, and its probe infrastructure does not run on the platform it watches. A status page on separate infrastructure would have informed your customers during that hour.

Source: Google Cloud incident report, 2025-06-12 · The Register, 2025-06-16

A script meant to delete instances of a retired app received site IDs instead of app IDs and deleted whole customer sites over 23 minutes, starting 07:38 UTC. The first customer ticket arrived at 07:46 UTC, 8 minutes in, and the major incident process started at 08:17 UTC. The first status page update came at 09:03 UTC, 85 minutes in, and the first broad external statement on social media on 7 April, 41 hours in. Restoring took up to 14 days and was only partly automated.

What a check from outside showsAn HTTP check of your own tenant URL from several regions reports the outage after 2 consecutive failed checks, without waiting for a support ticket. Some customers lost Statuspage and Opsgenie with their sites, so the status page and alerting must not sit with the provider whose outage they are meant to show. No monitoring would have shortened the 14 days.

Source: Atlassian post-incident review, 2022-04-29

The CEO wrote that the company renews thousands of domains every year with precision, yet the auto-renew process for its main domain failed. The statement named human and process error as the cause. Customers complained publicly on Twitter.

What a check from outside showsA domain check reports the expiry date days ahead, from a system that does not depend on the auto-renewal that failed. A DNS check from several regions would have reported the loss of resolution after 2 consecutive failed checks. A status page on another domain would have carried the message while marketo.com did not resolve, but no check can renew the domain.

Source: Marketo knowledge base (Adobe), P1 July 25, 2017 · The Drum, 2017-07-26

While repairing replication, an engineer removed the data directory on the primary instead of the secondary. The pg_dump backups did not exist, because the script ran pg_dump 9.2 against PostgreSQL 9.6 and failed. GitLab wrote that notifications for failing cron jobs were sent by email, but DMARC was not enabled for those emails, so the receiver rejected them. Disk snapshots were not enabled for the database servers, and all that remained was a manual LVM snapshot, 6 hours old.

What a heartbeat showsA heartbeat works as a dead man’s switch: the backup job reports after success, and Perstat alerts when the report stays away, whether or not an error email arrives. The case also shows that alerting must not depend on one channel that can fail itself, so escalation should run over several paths and require an acknowledgment. A heartbeat does not check whether the backup can be restored, which stays a job for a restore test.

Source: GitLab post-mortem, 2017-02-10 · The Register, 2017-02-01, on the 5 backup techniques