20 hours

…is roughly about how much I worked this weekend. I got to see what happens when all of our production, dev, test, etc servers go down hard. Here’s what NAU said happened to the power on campus Saturday of move in weekend:

Construction equipment working on the University Union dining hall expansion struck a main power line on Saturday morning, causing a power outage on central campus and disrupting Internet and e-mail service campuswide.

The Fieldhouse opened for food service, and other locations are expanding their hours to accommodate students.

Buildings affected by the power outage were: Allen, Cowden , Gateway Student Success Center, Raymond, McDonald, Tinsley, Wilson, Information Technology Services and the University Union (the Union Fieldhouse has and will continue to have power).

These same buildings will lose power again sometime between 6 and 7 p.m. today while permanent power is restored. Expect Internet and e-mail disruption. The Fieldhouse will remain open for dining.

Some construction dude took a 2ft auger and blasted through the tunnel containing power and fiber to the Union. Even though there were hand dug holes showing where the pipe was. This power hit took out all of central campus, including my work, ITS. People weren’t called in enough time, so the server room lasted roughly 50 min on battery backup. We got in just in time to watch all the machines go down. Then we waited for an emergency power splice to be put into effect. From there we got critical machines back up, made sure DB’s and disks weren’t corrupt, and shut down all non critical machines. Then the power was taken down around 7:20pm Sat to put in a real fix for power. When power came back up, it blew a circut, thus putting power back down for another hour or so. When power came back up, we made sure all critical services were running, and were told to wait since a real fix would be put into place on Sunday.

Sunday, power wend down at the scheduled time, for roughly an hour. Once back up, we got all services, including dev and test instances up by 11:30.

Since many departments depend on our LDAP servers being up (everything from peoplesoft, to resnet) alot of commotion was raised. Maybe we’ll get some headway made on a real generator now.

Even though it was draining, it was good to see what/and how things break, etc. I’m surprised at how graceful a lot of machines and services came back up. Looks like I’ll have some days off in the near future. Whoo! 😀

No TweetBacks yet. (Be the first to Tweet this post)
Written by ray in: work |

No Comments »

RSS feed for comments on this post. TrackBack URL

Leave a Reply

Template: Wordpress Themes | Cheap Hosting, Direkt Depot