Here’s some mandatory reading in case you still believe redundant networking infrastructure cannot fail:

• The network is reliable – a fantastic collection of real-life failures, including all sorts of split-brain scenarios caused by hare-brained schemes to stretch a cluster just a bit too far;
• More stuff on impacts of network partitions from the same author;
• Notes on Distributed Systems for Young Bloods. A must-read for anyone who thinks that ignoring 40 years of hard-learned lessons and controlling a distributed system from a central controller makes perfect sense. Not that it would ever help.

Finally, If you're serious about deep-diving into distributed systems, here's the list of materials to master (thanks to Yuriy Babenko).