6" WATER MAIN - Do NOT assume that it is EMPTY!!!

So it started out as a nice peaceful day at work. I had several things on my schedule, but nothing too bad, just some server installs for a new system (that is what I do at work). Then during a conference call to support, we all of a sudden lost power. That happens quite a lot though in my offices, so I didn't think anything of it. Then we slowly started to realize that the power was out throughout the entire building.

I hear someone talking outside about a water main, and so I decided I better go investigate. Upon investigation, I walked into the back door of the warehouse that occupies the other end of our building, only to find about 4" of water being shoved out the door by crazed warehouse workers and secretaries.

I guess there was someone there working on the fire suppression system that had assumed that the lines were drained and the valves shut when he tried to remove a broken section of the water main, he was wrong! OOPS!!!

So there was a man made geyser in the warehouse for several minutes as he scrambled to figure out how to shut it off, hence the 4" of standing water on the warehouse floor.

So the fun part; the geyser (so powerful that it pealed the paint off the wall) was pointed directly at all of the power distribution panels for the building, which also included the power distribution panels that feed the Data Center/Server Room for over 70 buildings spread throughout the valley and 100,000 users).

Miraculously the data center was still functioning, while the rest of the building was down. Several minutes after the water stopped someone went to check on the breaker boxes, and found that they were full of water. The one that we use for the Data Center however only had about 6" of standing water in it and was still passing power through it???

The servers were up for the time being, but as the water soaked into the electronics, it started to freak out, and caused the power to fail, and then kicked on the generator. The generator feeds through that same box, so I don't know how it was working but everything, except for the cooling units, stayed on in the server room.

Without the cooling units, the server room started to get hot quick, so we were forced to shut down everything that was not marked as Critical. We left the main router up as well as the security devices that allow for Internet access to and from the site. Other than that we shut about everything down and went to help squeegee an estimated 5,000 gallons out of the warehouse, FUN!!!

We all walked around with wet shoes for the rest of the day, and are still waiting for repairs to be done on the power and AC units. In all though we never lost Internet connection throughout all of the pain and chaos that we faced all day, that was awesome as it was our priority to keep these services open for the thousands of users that depended on them.

Quite the day at work!!!