{"repo":"danluu/post-mortems","free":true,"listed":false,"github":"https://github.com/danluu/post-mortems","clone":"git clone https://github.com/danluu/post-mortems.git","description":"A collection of postmortems. Sorry for the delay in merging PRs!","language":null,"stars":12240,"topics":["post-mortem","debugging","hacktoberfest"],"license":null,"category":"dev-tools","readme_excerpt":"A List of Post-mortems! Table of Contents - Config Errors - Hardware/Power Failures - Conflicts - Time - Database - Uncategorized - Other lists of postmortems - Analysis - Contributors Config Errors Allegro. E-commerce site went down after a sudden traffic spike caused by a marketing campaign. The outage was caused by a configuration error in cluster resource management which prevented more service instances from starting even though hardware resources were available. Amazon. A configuration change in the Seoul region incorrectly removed the setting that specifies the minimum healthy hosts for the EC2 DNS resolver fleet, so the system fell back to a very low default. The fleet's healthy host count dropped and in-VPC DNS queries from EC2 instances failed for 84 minutes until capacity was manually restored. AWS added semantic config validation and per-hour throttling on host removal as remediations. CircleCI. A deploy changed the type of a field in the PostgreSQL database used by the job-distribution service. Newly-written rows used the new type and old rows used the old type, so the distributor's strict schema validation failed on each scan and it stopped distributing work. Rolling back made the rows written between the two deploys unreadable, so distribution kept failing; recovery required hand-deploying a build that ignored the offending field, plus manual scaling. CircleCI. An IAM-role gap permitted out-of-band changes to AWS WAF outside of CircleCI's Terraform pipeline; an","default_branch":null,"files":null,"tree":[],"storefront":"/r/danluu","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/danluu/post-mortems/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}