OVH Community, your new community space.

What just happened in RBX2?


yonatan
01-10-2010, 15:57
Quote Originally Posted by LawsHosting
Ive no servers in RBX2...... yet......... I did have one but I let that go.

I'm sure karma will bite me in the butt and it'll happen to RBX1 & 3 sooner or later!
Oi i got server in all four locations, keep your bad karma away :P

LawsHosting
01-10-2010, 10:44
Ive no servers in RBX2...... yet......... I did have one but I let that go.

I'm sure karma will bite me in the butt and it'll happen to RBX1 & 3 sooner or later!

darkfyre
01-10-2010, 10:41
I was 1 hour 57~ minutes
SLA claim emailed to support :-D

HandsomeChap
01-10-2010, 09:58
Mine stayed off for between 50mins and 2 hours, openvz take a while to get all the containers back online, especially combined with raid10 re-initialising which powercut also forced

hokapoka
01-10-2010, 09:25
Ahem...

It appears my server came backup quite quickly, but the ESXi didn't restart any of the machines.

To make matters worse, I'd forgotten the password I'd set for the ESXi service so I couldn't gain access!

Note to self : Test power failure on the server & remember the damn password!

Thanks goes to the support for sorting the issue.

yonatan
30-09-2010, 18:58
That's not the first power outage on room26 this year... - look http://travaux.ovh.com/?do=details&id=3819

I think something there is not good from the base up and need to be examined closer...


anyway my server in that room is toast ( literally )
Code:
Sep 30 14:01:01 server mcelog: Processor 1146431872 heated
above trip temperature. Throttling enabled.
Sep 30 14:01:01 server mcelog: Please check your system
cooling. Performance will be impacted
Waiting for the cooling to be fixed, it wont boot properly unless... so im on rescue-pro ATM.

Ticket number 555250

Andy
30-09-2010, 18:02
Mine was down for less than 10 minutes and hasn't gone off again since.

Neil
30-09-2010, 16:13
Quote Originally Posted by darkfyre
any ETA on this ?

been 30+ minutes so far... are these things not usually sorted within minutes.
Best to look at http://travaux.ovh.net/vms/index_rbx2.html - we do have a lot of servers down at the moment but we have alot of datacentre staff on it and the servers are being all brought up as I type.

fozl
30-09-2010, 16:12
Quote Originally Posted by hokapoka
------------------------------- SNIP -------------------------------

It looks like the room was being put on to UPS/Generators for some maintenance which have in turned failed under the load. Would have thought the Generator was tested prior it manually using it.
The generators wouldn't be the issue, the UPS must bear the load while the generators swing into play.

darkfyre
30-09-2010, 16:03
any ETA on this ?

been 30+ minutes so far... are these things not usually sorted within minutes.

hokapoka
30-09-2010, 16:01
Hummm reading this:

FS#4638 - Roubaix 2: swing of source station

Details
We will carry out a swing of source station on our site of Roubaix 2.
During handling, the site will function on generators.
However, we will maintain a solution fast of flashback in case
of problem.

The operations will take place in the morning of Thursday September 30, 2010.

FS#4656 - rbx2 room 26

Details
One of the inverters of room 26 was at fault during a few seconds.
It is up. And the waiters are returning.

We put the resources on the waiters at fault.
In looks in parallel the origin of the problem


FS#4657 - Room 26

Details
a defect on the UPS7 with RBX2 caused the cut of several bays 26xxx.
The manufacturer moves to diagnose the breakdown.


------------------------------- SNIP -------------------------------

It looks like the room was being put on to UPS/Generators for some maintenance which have in turned failed under the load. Would have thought the Generator was tested prior it manually using it.

Neil
30-09-2010, 15:52
Hi

We have a problem with a UPS and a power supply in room 26, details are on http://travaux.ovh.net and will be added to status,ovh.net shortly.

hokapoka
30-09-2010, 15:49
OMG!

100's of servers are out!

One UPS dies and all of that goes out... where's the backup generators / UPS?

fozl
30-09-2010, 15:47
http://status.ovh.net/?do=details&id=610

HandsomeChap
30-09-2010, 15:40
My servers all stopped responding, I checked the VMS and something massacred 1/4 of the DC by the looks of it!