Most Difficult Outages

Most challenging outages

In this chapter we will detail some of the most challenging outages over many years, how we responded to these, and what was learned.

List of most challenging outages

1. Production keytab invalidated
2. Production home folder deleted
3. All boxes rebooted on New Years morning
4. Shared infrastructure oddly slows down
5. Overloaded databases (deserves a chapter.)

Kafka specific outages faced

1. Kafka home disks full
2. Kafka not enough file descriptors
3. Kafka not enough memory
4. Kafka not enough networking bandwidth