Nobody plans an outage.

Every case is publicly documented, with a date and a source. Each one shows what a check from outside measures, and what it cannot prevent.

57%
of respondents said their most recent major outage cost more than 100,000 USD. Uptime Institute, Annual Outage Analysis, 2026
2 of 3
publicly reported outages over 9 years occurred at third-party providers. Uptime Institute, Annual Outage Analysis, 2026
Over 90%
of mid-size and large enterprises reported downtime costs of more than 300,000 USD per hour. ITIC, Hourly Cost of Downtime Survey, 2024
8 min
until the first customer ticket in Atlassian’s April 2022 outage. Atlassian, Post-Incident Review, 2022

Certificates expire on a schedule

An expiry date is known in advance, yet expired certificates still take down networks, chat, and phone systems. A check from outside sees a certificate on a public endpoint, not one inside vendor software or on an internal device.

  1. Ericsson certificate expiry took O2 and SoftBank offline

    In the UK, about 25 million O2 customers and about 7 million on other providers using the O2 network were affected for most of a day. SoftBank was down across Japan for 4 h 25 min, and the same fault appeared at operators in 11 countries. O2 credited contract customers 2 days of charges.

    Ericsson confirmed the same day that an expired certificate in 2 software versions of its SGSN-MME core network nodes caused the outage. SoftBank said the software had run for 9 months. Recovery meant rolling back to the older version.

    What a check from outside shows The certificate sat inside vendor software, so a TLS certificate check would not have seen it. Probes from outside do not measure the mobile data path either. A status page carries the customer message, and the availability record gives the credits a basis.

    Source: Ericsson press release, 2018-12-06 · SoftBank press release, 2018-12-06 · MoneySavingExpert, 2018-12-07

  2. Microsoft Teams: expired certificate stopped sign-in

    A service with 20 million users, about 3 hours from the first reports to the fix.

    Microsoft 365 Status wrote that an authentication certificate had expired and that a fix would apply a new certificate. Users could not sign in.

    What a check from outside shows Microsoft did not publish whether that certificate sat on a public endpoint, so it is not known whether a certificate check would have caught it. An HTTP check of the sign-in flow from several regions can show the failure after 2 consecutive failed checks, wherever the certificate lived. A status page answers the question your customers ask first: is it us or is it Teams.

    The monitor that measures thisExample
    {
      "name": "Sign-in page",
      "type": "http",
      "interval_seconds": 60,
      "config": { "url": "https://app.example.com/login", "expected_status": 200 }
    }

    Source: TechCrunch, 2020-02-03, quoting Microsoft 365 Status

  3. Google Voice: front-end certificate expired

    The certificate expired at 23:51 Pacific on 15 February, 07:51 UTC on 16 February. New SIP connections failed for 4 h 22 min. Only clients with an existing, uninterrupted SIP connection kept working.

    Google’s incident report names an issue with updating certificate configurations that let the active certificate in the Google Voice front ends expire.

    What a check from outside shows A TLS certificate check on the public endpoint reports the remaining lifetime days ahead, from a system outside the rotation that failed. The rotation itself stays the operator’s job.

    The monitor that measures thisExample
    {
      "name": "Voice front-end certificate",
      "type": "ssl_cert",
      "interval_seconds": 900,
      "config": { "host": "voice.example.com", "port": 443, "warn_days": 21 }
    }

    Source: Google Workspace incident report (PDF) · BleepingComputer, 2021-02-28

  4. Equifax: expired certificate left traffic uninspected

    The intrusion ran for 76 days and affected at least 145.5 million people. It was noticed after the certificate was renewed on 29 July 2017.

    The U.S. Government Accountability Office found that a certificate on the device inspecting encrypted network traffic had expired about 10 months before the intrusion began. Until it was renewed, the traffic was not inspected.

    What a check from outside shows The device was not a public endpoint, so a certificate check would not have seen it. The case belongs here only because it shows that an expired certificate can go unnoticed for months when nobody watches expiry dates. The smallest possible answer is an inventory of every certificate you own, with the public ones checked from outside.

    Source: GAO-18-559, 2018-08-30

DNS and routing fail for everyone at once

When a name stops resolving, every other measurement is meaningless. The tools you would use to debug it often depend on the same name.

  1. Facebook, Instagram, WhatsApp: 5 h 30 min offline

    All Meta services worldwide. DNS for facebook.com stopped resolving at about 15:50 UTC and returned at 21:20 UTC.

    During maintenance, a command meant to assess backbone capacity took every backbone connection down, and an audit tool failed to stop it. Meta’s authoritative DNS servers withdraw their BGP announcements when they cannot reach the data centers, so they vanished from the Internet while still running. The internal diagnosis tools went with them, and engineers had to enter the data centers in person.

    What a check from outside shows After 2 consecutive failed checks, DNS checks from several regions report that the domain no longer resolves anywhere, and traceroute checks show that the routes are gone. Meta lost its own tools together with its network, and a view from outside keeps working in exactly that situation. A status page has to live on another domain, or at least with another DNS operator, or it vanishes too.

    The monitor that measures thisExample
    {
      "name": "Apex A record",
      "type": "dns",
      "interval_seconds": 60,
      "config": { "name": "example.com", "record_type": "A" }
    }

    Source: Meta Engineering, 2021-10-05 · Cloudflare, 2021-10-04

  2. Salesforce: DNS change took down products and status page

    About 4.5 hours worldwide, from shortly before 22:00 UTC to 02:20 UTC the next day.

    An engineer used the emergency break-fix process for a global DNS rollout instead of the staged one. A 4-year-old script hit a timeout under load, the DNS servers did not come back after restart, and the recovery tools depended on those servers. The status page was only intermittently reachable, which Salesforce attributed to missing auto-scaling on the page, not to DNS.

    What a check from outside shows DNS checks from several regions report the loss of resolution after 2 consecutive failed checks. More important, a status page is needed exactly when everyone looks at once, so it has to run outside your own platform and handle incident traffic. That is a property of a separately hosted status page, not a merit of the check itself.

    The monitor that measures thisExample
    {
      "name": "App host record",
      "type": "dns",
      "interval_seconds": 60,
      "config": { "name": "app.example.com", "record_type": "A" }
    }

    Source: The Register, 2021-05-12 · The Register, 2021-05-19

  3. AWS us-east-1: cleanup deleted DynamoDB’s DNS plan

    For about 14.5 hours, DynamoDB in us-east-1 and the services built on it were disrupted: EC2 launches, Lambda, and container services, as well as STS and the console sign-in. A large number of customer applications on top of them were affected too.

    DynamoDB’s DNS management runs a planner and several enactors. One enactor was applying an old plan while a second applied a newer one and then deleted the old plan in its cleanup step. AWS wrote that as this plan was deleted, all IP addresses for the regional endpoint were immediately removed. Knock-on effects on EC2 and load balancers stretched the recovery into the afternoon.

    What a check from outside shows DNS and HTTP checks of your own endpoints from several regions report the outage after 2 consecutive failed checks and show whether your region is affected. The status page belongs outside the region and the platform your services run on. The availability record gives you your own time series for customer questions and for a credit request, where you have to prove the downtime.

    The monitor that measures thisExample
    {
      "name": "API endpoint record",
      "type": "dns",
      "interval_seconds": 60,
      "config": { "name": "api.example.com", "record_type": "A" }
    }

    Source: AWS post-event summary, October 2025

  4. Akamai Edge DNS: customer zones stopped resolving

    Up to an hour until the rollback took effect, in the DNS component of the Secure Edge CDN.

    Akamai reported that a software configuration update at 15:45 UTC triggered a bug in the DNS system and ruled out an attack.

    What a check from outside shows A DNS check from several regions shows that your own zone no longer resolves, without waiting for the provider’s statement. The status page must not depend on the same DNS.

    The monitor that measures thisExample
    {
      "name": "Zone name servers",
      "type": "dns",
      "interval_seconds": 60,
      "config": { "name": "example.com", "record_type": "NS" }
    }

    Source: Akamai, 2021-07-22

Status pages failed along with the platform

Four operators documented it in their own reports: the page that should have told customers what was happening failed during the outage.

  1. AWS S3 outage froze its own status dashboard

    S3 in us-east-1 and the services built on it were disrupted for 4 h 17 min.

    During a standard procedure, an authorized engineer entered a parameter incorrectly and removed more servers than intended. The servers included capacity of the S3 index and placement subsystems, and both subsystems had to be restarted in full. AWS wrote that until 11:37 PST it could not update the individual services’ status on the dashboard, because the dashboard console depended on S3. AWS communicated over Twitter and a banner for 2 hours and later spread the dashboard administration across regions.

    What a check from outside shows The case makes one rule clear: never host the status page on the infrastructure whose status it shows. Perstat serves your status page from its own infrastructure, not from the platform being watched. HTTP checks from several regions would have reported S3 endpoints as failing after 2 consecutive failed checks. AWS knew that itself, so the gain here lies entirely in independent communication.

    The monitor that measures thisExample
    {
      "name": "Asset storage",
      "type": "http",
      "interval_seconds": 60,
      "config": { "url": "https://assets.example.com/health", "expected_status": 200 }
    }

    Source: AWS post-mortem, March 2017

  2. Cloudflare: proxy and status page failed together

    Core traffic recovered after 3 h 10 min, and the incident was fully resolved after 5 h 46 min. CDN and security services were affected, as were Turnstile, Workers KV, Access, and the dashboard. Cloudflare called it its worst outage since 2019.

    A permission change on a ClickHouse cluster made a query return duplicate column metadata. The bot management feature file doubled in size and exceeded a hard limit of 200 features in the proxy, and the process failed with a Rust panic. The independently hosted status page went down at the same time for an unrelated reason, which at first made the team suspect a coordinated attack.

    What a check from outside shows If you use Cloudflare, HTTP checks from several regions with a quorum report 5xx responses from your site everywhere after 2 consecutive failed checks. That answers whether the problem is yours or the provider’s. Your own status page on independent infrastructure serves your customers while the provider’s dashboard and status page are unavailable. Only Cloudflare could have fixed anything.

    The monitor that measures thisExample
    {
      "name": "Website behind the CDN",
      "type": "http",
      "interval_seconds": 60,
      "config": { "url": "https://www.example.com/", "expected_status": 200 }
    }

    Source: Cloudflare blog, 2025-11-18

  3. Google Cloud: API layer down worldwide

    The core disruption lasted about 3 hours and occurred globally, and recovery in us-central1 took about 2 h 40 min. Affected products ranged from IAM-dependent services to BigQuery, Cloud Storage, and Vertex AI. Workspace and customers such as Cloudflare were affected too.

    On 29 May, a feature for extra quota policy checks reached Service Control without a feature flag and without error handling on the new path. On 12 June, a policy change produced blank fields, and the binaries crashed worldwide with a null pointer error. Google wrote that its first incident report came about an hour after the crashes began, because the Cloud Service Health infrastructure was itself down. It also wrote that for some customers, the monitoring infrastructure they ran on Google Cloud was failing too, leaving them without a signal of the incident.

    What a check from outside shows That last sentence makes the case for external monitoring in the operator’s own words: monitoring on the same platform fails with it. Perstat checks from outside, from up to 6 regions with a quorum, and its probe infrastructure does not run on the platform it watches. A status page on separate infrastructure would have informed your customers during that hour.

    The monitor that measures thisExample
    {
      "name": "Public API",
      "type": "http",
      "interval_seconds": 60,
      "config": { "url": "https://api.example.com/health", "expected_status": 200 }
    }

    Source: Google Cloud incident report, 2025-06-12 · The Register, 2025-06-16

  4. Atlassian: script deleted 883 customer sites

    The outage affected 775 customers for up to 14 days, until the last site was restored. Jira, Confluence, and Access were unavailable for them, and so were Opsgenie and Statuspage. No customer lost more than 5 minutes of data.

    A script meant to delete instances of a retired app received site IDs instead of app IDs and deleted whole customer sites over 23 minutes, starting 07:38 UTC. The first customer ticket arrived at 07:46 UTC, 8 minutes in, and the major incident process started at 08:17 UTC. The first status page update came at 09:03 UTC, 85 minutes in, and the first broad external statement on social media on 7 April, 41 hours in. Restoring took up to 14 days and was only partly automated.

    What a check from outside shows An HTTP check of your own tenant URL from several regions reports the outage after 2 consecutive failed checks, without waiting for a support ticket. Some customers lost Statuspage and Opsgenie with their sites, so the status page and alerting must not sit with the provider whose outage they are meant to show. No monitoring would have shortened the 14 days.

    The monitor that measures thisExample
    {
      "name": "Tenant site",
      "type": "http",
      "interval_seconds": 60,
      "config": { "url": "https://yourteam.example.com/", "expected_status": 200 }
    }

    Source: Atlassian post-incident review, 2022-04-29

Domains lapse

An expired domain takes every service under it down at once. The process that should have renewed it is the one that failed.

  1. Marketo: domain renewal failed, customers complained publicly

    Sign-in, embedded forms, and images and links in emails were disrupted for every customer, and so were the Salesforce integration and activity tracking. The outage was largely resolved by 12:00 PDT, with propagation effects for 24 to 48 hours.

    The CEO wrote that the company renews thousands of domains every year with precision, yet the auto-renew process for its main domain failed. The statement named human and process error as the cause. Customers complained publicly on Twitter.

    What a check from outside shows A domain check reports the expiry date days ahead, from a system that does not depend on the auto-renewal that failed. A DNS check from several regions would have reported the loss of resolution after 2 consecutive failed checks. A status page on another domain would have carried the message while marketo.com did not resolve, but no check can renew the domain.

    The monitor that measures thisExample
    {
      "name": "Company domain",
      "type": "domain",
      "interval_seconds": 300,
      "config": { "domain": "example.com", "warn_days": 30 }
    }

    Source: Marketo knowledge base (Adobe), P1 July 25, 2017 · The Drum, 2017-07-26

  2. Microsoft: passport.com and hotmail.co.uk lapsed

    Hotmail was unusable on 25 December 1999 because the authentication domain had expired. An outsider paid the 35 USD renewal. In 2003, hotmail.co.uk expired and was registered by a private person, and Microsoft reacted only after a press inquiry on 5 November.

    Whether users were affected in 2003 is not documented, so that case counts as a loss of control, not an outage.

    What a check from outside shows A domain check with an expiry warning on every domain you own, including the ones nobody thinks about.

    The monitor that measures thisExample
    {
      "name": "Secondary domain",
      "type": "domain",
      "interval_seconds": 300,
      "config": { "domain": "example.co.uk", "warn_days": 30 }
    }

    Source: Slashdot, 1999-12-25 · The Register, 2003-11-06

Jobs and backups stopped silently

A cron job that fails does not call anyone. The failure shows up months later, on the one day the backup is needed.

  1. GitLab.com: database deleted, 5 backup paths failed

    About 18 hours of downtime, most of it restoring. Six hours of data were lost: about 5,000 projects, 5,000 comments, and 700 new accounts. Repositories and wikis were not affected.

    While repairing replication, an engineer removed the data directory on the primary instead of the secondary. The pg_dump backups did not exist, because the script ran pg_dump 9.2 against PostgreSQL 9.6 and failed. GitLab wrote that notifications for failing cron jobs were sent by email, but DMARC was not enabled for those emails, so the receiver rejected them. Disk snapshots were not enabled for the database servers, and all that remained was a manual LVM snapshot, 6 hours old.

    What a heartbeat shows A heartbeat works as a dead man’s switch: the backup job reports after success, and Perstat alerts when the report stays away, whether or not an error email arrives. The case also shows that alerting must not depend on one channel that can fail itself, so escalation should run over several paths and require an acknowledgment. A heartbeat does not check whether the backup can be restored, which stays a job for a restore test.

    The monitor that measures thisExample
    {
      "name": "Nightly database backup",
      "type": "heartbeat",
      "config": { "period_seconds": 86400, "grace_seconds": 1800 }
    }

    Source: GitLab post-mortem, 2017-02-10 · The Register, 2017-02-01, on the 5 backup techniques

  2. OVHcloud SBG2: fire, backups in the same building

    Netcraft counted 3.6 million websites on 464,000 domains offline. Among them were banks, webmail, and news sites, as well as shops and government sites in several countries. The commercial court of Lille awarded 101,102 EUR and 153,837 EUR to 2 customers who had paid for backups that sat in the same building as production.

    A fire in the night of 10 March 2021 destroyed the SBG2 data center in Strasbourg and part of SBG1, and SBG1 to SBG4 went offline. In 2 rulings in 2023, the court found that the contractually promised physical separation of the backups was not given. OVHcloud appealed the first ruling and, according to Blocks and Files, intended to appeal the second.

    What a heartbeat shows A heartbeat proves that a backup job ran, not where the copy is or whether it can be restored at another site. HTTP checks from several regions would have reported the outage after 2 consecutive failed checks. A status page hosted with another provider would have reached the customers.

    The monitor that measures thisExample
    {
      "name": "Off-site backup copy",
      "type": "heartbeat",
      "config": { "period_seconds": 86400, "grace_seconds": 1800 }
    }

    Source: Netcraft, 2021-03-09 · Blocks and Files, 2023-03-23

  3. RBS: overnight batch failed, 56 million GBP fine

    More than 6.5 million customers in the UK faced weeks of wrong balances, delayed mortgage payments, and unpaid salaries. In November 2014, the FCA fined the banks 42 million GBP and the PRA fined them 14 million GBP, after a 30% early-settlement discount.

    The central IT function updated the software for nightly account processing, saw problems, and uninstalled the update without testing the consequences. The versions were incompatible, batch processing stalled, and bookings piled up for weeks. The FCA named the failure to put in place adequate systems and controls to identify and manage IT risk.

    What a heartbeat shows One heartbeat per batch run reports the missing nightly close during the same night. HTTP checks on the customer portals give the view from outside, and the availability record gives the regulator a time series instead of estimates. No monitoring would have changed the faulty rollback or the weeks of cleanup. The size of the fine shows what a regulator means by missing controls.

    The monitor that measures thisExample
    {
      "name": "Nightly batch close",
      "type": "heartbeat",
      "config": { "period_seconds": 86400, "grace_seconds": 3600 }
    }

    Source: Financial Conduct Authority, 2014-11-20

Alerts need someone to acknowledge them

An alert goes to one person, has to be acknowledged, and escalates to the next person after a defined time if it is not. An email to a group is not an alert.

  1. Knight Capital: nobody acted on 97 emails

    In 45 minutes, the system sent more than 4 million orders to fill 212 customer orders and traded more than 397 million shares. The loss was more than 460 million USD, and the SEC fined the firm 12 million USD in 2013.

    New code was deployed to 7 of 8 servers. The eighth kept old code that reactivated a function unused since 2003. Before the open, an internal system sent 97 emails referencing the router error to a group of staff, and according to the SEC, nobody acted on them.

    What a check from outside shows This was not an availability outage, but it makes the case for alerting that requires an acknowledgment. Perstat would not have detected the software fault, so the case stands here for the alerting logic only.

    Source: U.S. Securities and Exchange Commission, 2013-10-16

Cases from Germany and the EU

DORA has required EU financial entities to report major ICT incidents since 17 January 2025. Article 5 of Delegated Regulation (EU) 2025/301 makes the first report due 4 hours after classification and no later than 24 hours after awareness.

  1. Postbank: migration disruption, BaFin special commissioner

    About 12 million customers and 19 million contracts migrated to the Deutsche Bank platform. BaFin saw significant impairments from the turn of the year 2022/2023 and appointed a special commissioner on 29 September 2023.

    BaFin named outages of online and mobile banking and poor phone access to customer service. It also named long processing times for garnishment and estate matters, account closures, and repayment of savings, as well as significant impairments for garnishment protection accounts. The commissioner was to oversee the swift and complete removal of the restrictions and report regularly.

    What a check from outside shows HTTP checks from several regions record the part that is measurable from outside: the online and mobile banking outages. An availability record that states its boundaries gives regulators and customers numbers instead of assurances. The larger part, processing backlogs and phone service, is not a monitoring topic. Whoever claims availability should be able to prove it before the regulator asks.

    The monitor that measures thisExample
    {
      "name": "Online banking login",
      "type": "http",
      "interval_seconds": 60,
      "config": { "url": "https://banking.example.com/login", "expected_status": 200 }
    }

    Source: BaFin, 2023-09-04 · BaFin, 2023-10-02 · heise online, 2023-10-02 · onvista (Reuters), 2023-07-03, on the migration figures

  2. ELSTER: property tax rush took the portal down

    Declarations for about 36 million properties were due from 1 July 2022, in most cases through ELSTER. The operator counted well over 100,000 simultaneous requests. The portal was intermittently unreachable on Sunday. On Monday it was restricted and then fully offline for maintenance from 13:00 CEST, and heise reported it back at 16:32 CEST.

    The Bavarian tax office spoke of very high demand and ruled out an attack. The interface for tax software kept working.

    What a check from outside shows HTTP checks with response times from several regions show the degradation before the outage and leave the time series. A status page with a notice reduces the number of reload attempts. This was a capacity problem the operator knew about, not a detection problem, so the case stands here as a public-sector example of reach and the need to communicate.

    The monitor that measures thisExample
    {
      "name": "Tax portal",
      "type": "http",
      "interval_seconds": 60,
      "config": { "url": "https://portal.example.com/", "expected_status": 200 }
    }

    Source: heise online, 2022-07-11 · Haufe, 2022-07-13

  3. Deutsche Telekom: about 900,000 lines disrupted

    Internet, VoIP, and IPTV were disrupted across Germany. A firmware update normalized the situation within days, and Telekom offered a free mobile day pass.

    According to the BSI’s 2017 situation report, a worldwide attack by a Mirai variant on port 7547 made the Telekom routers crash. The routers were not infected, but they responded to the requests with a fault.

    What a check from outside shows These were consumer devices, so the case has no relation to the product. It stands here only as a German example of the reach of an infrastructure failure.

    Source: BSI, Die Lage der IT-Sicherheit in Deutschland 2017, page 15 · The Register, 2016-11-28

None of these cases was prevented by monitoring, and Perstat does not claim otherwise. With the default policy, a check from outside reports an outage after 2 consecutive failed checks from 2 regions. Perstat warns days before a certificate or a domain expires, notices a job that stopped reporting, and keeps the status page reachable when the platform is not. It records the detected, confirmed, and resolved timestamps as data for a credit request or a regulator, but replaces no DORA report and no classification. Survey figures are not measurements and must not be added up.