I did so rather quickly though the project has been up for a while and thus not at a safe, clean break point.
Except the replica database needed to be resync'ed. Neither crashed as much as the kernels got into some sort of dead lock and couldn't be wedged - in both cases we got the people down at the colocation facility to reboot the machines for us and all was well.
First oscar (our main mysql server) crashed, and then a couple hours after that so did carolyn (the replica). However last week we had some scary, unexpected server crashes. Hello again! Once again I'm emerging from a span of time where I was either out of the lab or in the lab working on non-newsworthy development, and realizing it's been way too long since I drummed up one of these reports.We had our usual Tuesday outage again today.
