07/14/2008 Post By: Peter Andres
From Peer 1:
Dear Customer,
There has been a massive power outage that has affected the downtown Vancouver area, including the Harbor Center building, where our PEER 1 Vancouver data centers are located.
Shortly before 9am PST, primary power was lost due to a fire on Richards Street, which caused primary power to fail in a large portion of downtown. The main building generator, Generator 7, started and maintained power but failed 20 minutes after the primary power was lost. Initial diagnosis is that the generator failed due to over-heating. Once the generator failed, PEER 1 UPS systems failed after several minutes due to drained batteries.
PEER 1 facilities and building maintenance have been working on the generator continuously to restore power to the data centers. The first attempt to bring emergency power back on-line failed 15 minutes after the generator was restarted. It experienced a similar over-heating situation as caused the initial failure. PEER 1 will continue to work on this issue around the clock until power is restored.
We are keeping up to date information on forums.peer1.com. We sincerely apologize for this disruption in your business. Every effort is being made by PEER 1 and the Harbor Center building maintenance team to restore power as quickly as possible.
Andrew - 07/14/2008
http://www.k1technology.com
If someone wants to have some fun and follow the full conversation between anxious network people and the technical staff at Peer 1 - try this link: http://www.peer1.com/forums/viewtopic.php?f=37&t=50
Also, if you have questions about our network structure... etc... we are willing to talk with you.
Jason - 07/14/2008
http://www.k1technology.com
Here is Peer1's official Post Mortem.
http://www.peer1.com/forums/viewtopic.php?f=37&t=52
I was on holiday's today and came in to make sure everything was operational after the power came back, we have noticed one issue with mass email not going out, this has been resolved as of 10:40pm. All mail that was sent from orders since 4:00pm will start going out now.
