Anne Baretta found a great video describing the October 2018 GitHub failure. Here’s the TL&DW:
- The failure was caused by a short (~ 1 minute) disconnect of the primary data center
- The database replicas failed over to the secondary data center, but that failover was never tested and of course some stuff didn’t work.
- In the meantime, batch jobs modified data in the primary data center, making the two replicas out-of-sync.
- It took them over 24 hours to clean up the mess.
You REALLY SHOULD watch the video – it nicely proves two points I’ve been making for ages (not that anyone would listen):