[Thelen]

Well that is because usually the applications handle the load balancing, so the HW failure rate differences between Dell/supermicro/corner-shop is very little in the scheme of things. It more applies though to the OP, where the specific HDD matters, not just a general HDD Failure rate..

In other words, OVH has issues like this when it comes to specific data, because they don't replicate where others might. Although, I don't know of any dedicated provider that can guarantee data on anything less than RAID6 and 8 drives (rackspace and co do this). (Actually even then it isn't a guarantee, it is just the cost you pay and the SLA refund you get is so extremely high, I've never heard of rackspace and co paying out).

[DigitalDaz]

Well, it is consumer grade vs even supermicro let alone dell, but no I more mean the truly unmanaged aspect.

I no longer even think of consumer grade vs commercial. That's probably because in general I find all hardware very reliable these days and servers tend not to have a long lifespan due to cheap upgrades.

BTW OVH has a better track record for global outages than google microsoft and AWS, although they are not as big and as complicated systems, but still... :P (and yes I know OVH has far more packet loss type issues, but they aren't global impact).

That's interesting

[Thelen]

Why, do you think their kit is inferior or something? This new box will probably run for another four years now.

Well, it is consumer grade vs even supermicro let alone dell, but no I more mean the truly unmanaged aspect.

with "things like these" you mean misunderstandings of what the OVH SLA covers? we're very open and clear about it, there should be any misunderstanding of what's covered and when SLA1 kicks in.

It's true that the fact that the servers are unmanaged creates some confusion about who is responsible depending of the problem.

Yes that is what I mean.

That's a nonsense statement of course. No one has to "expect" that, regardless of the price. Had you merely said that expectations are often based on assumptions, then yeah.

Er yea, perhaps "consider" that things like that might happen on a monthly basis, I meant 'expect' with an aspect of risk assesment, vs expecting the inevitability.

Anyway, what OVH provides in the market they are targeting is by far the best.

BTW OVH has a better track record for global outages than google microsoft and AWS, although they are not as big and as complicated systems, but still... :P (and yes I know OVH has far more packet loss type issues, but they aren't global impact).

[Myatu]

There are many customers at OVH that understand perfectly what's exactly covered and what the customer's responsibilities include. And with discussions like this, I hope more customers will understand it better.

The OVH website could really benefit from a proper FAQ, especially for things like these. It would save everyone some confusion (and recurrence in the forum!). I had different thoughts about the SLA as well, tbh.

[Myatu]

They can see it from the clients side, just that the client has false expectations. At OVH prices, you should expect things like this literally every month, let alone once in 4 years..

That's a nonsense statement of course. No one has to "expect" that, regardless of the price. Had you merely said that expectations are often based on assumptions, then yeah.

[marks]

It may even come out of this that its actually better to get rid of any OVH servers and use Kimsufis. Spread clients more thinly across them and at the same time spread the risk.

you're right, and there are some customers that have taken that route. Opposed to 1 or 2 big servers, they have several kimsufis to implement their services.

It's a perfectly respectable approach, with it's up and down sides to it. It all depends of what you can afford, the tech requirements of your project, your skills, ...

We will still recommend to use pro servers as a base for your projects, as each of the individually are more reliable and better covered than kimsufis.

[marks]

They can see it from the clients side, just that the client has false expectations. At OVH prices, you should expect things like this literally every month, let alone once in 4 years..

with "things like these" you mean misunderstandings of what the OVH SLA covers? we're very open and clear about it, there should be any misunderstanding of what's covered and when SLA1 kicks in.

It's true that the fact that the servers are unmanaged creates some confusion about who is responsible depending of the problem. This is specially true when it comes to hardware problems that we cannot investigate and troubleshoot because we don't have access to the server.

There are many customers at OVH that understand perfectly what's exactly covered and what the customer's responsibilities include. And with discussions like this, I hope more customers will understand it better.

Now, DigitalDaz's server had got a commercial gesture applied to, for the time the server could be used, equal to double that time. We believe quite fair, through it's not the SLA Level 1 compensation (which is for 20h delayed is 100% of the server back).

Because of the perception I mentioned above, I now have to reevaluate all the projects and definitely change the way we operate some.

For example VOIP. Absolutely no way in this world can this remain on any servers where this situation could arise without an alternative solution. Moreover, probably more importantly, we are giving OUR clients these guarantees that we now can no longer meet.

Also, email, we have a couple of clients who we will definitely have to move into some High Availability scenario now.

The Level 2 with a GTI of 12 hours and no displayed GTR to all intents and purposes means there is NO SLA that is of any use whatsoever.

yes, you're doing the assessment that any project would need to do:
-what kind of guarantees I want to offer to my customers
-what are the guarantees that I'm offered with m products
-anything that's not offered to me, I have to implement it myself

so, if you're looking at 100% uptime, you'll never get that with a single dedicated server (this can always go down for many reasons), not even near that.

A high uptime service is most likely involving more than one server, even if you do the basic setup of having a master server (working most of the time) and a slave server that is always ready to take over if something is wrong with the main one (kimsfi or a small pro server for example).

That's not because OVH doesn't offer reliable services, which most of you can certify it does, but because there are problems that cannot be fix within SLA for several of the reasons commented before.

[DigitalDaz]

They can see it from the clients side, just that the client has false expectations. At OVH prices, you should expect things like this literally every month, let alone once in 4 years..

Why, do you think their kit is inferior or something? This new box will probably run for another four years now.

[Thelen]

They can see it from the clients side, just that the client has false expectations. At OVH prices, you should expect things like this literally every month, let alone once in 4 years..

[DigitalDaz]

Mark,

At the end of the day, non of this is about compensation though it will definitely help

What the thread is all about, at least for me is clarification and we are definitely getting some of that now. I just wish you could see it more from the client's side off the fence than the OVH one.

I've been with OVH now, by the looks of it, four years and I 'perceived' in all of them four years and that is the key, I perceived that the incident that has just happened to me would be treated as Level 1.

Its certainly not going to stop me using OVH. Now that it is fixed I'm receiving the excellent performance that I would expect from that piece of kit.

When I said above in this thread that I believed myself lucky I was not joking. I support a number of different clients on quite a few different projects. Over this last four years I have encouraged everyone, whenever possible, to move to OVH.

Because of the perception I mentioned above, I now have to reevaluate all the projects and definitely change the way we operate some.

For example VOIP. Absolutely no way in this world can this remain on any servers where this situation could arise without an alternative solution. Moreover, probably more importantly, we are giving OUR clients these guarantees that we now can no longer meet.

Also, email, we have a couple of clients who we will definitely have to move into some High Availability scenario now.

Also, there are at least four websites that I know of from the top of my head that have sales teams selling on them that again this could not be allowed to happen to. They need to be handled the same way.

Some may go to PCC some other cloud providers. Many new costs may have just appeared that now need refactoring. One option that may definitely need to be evaluated now is colocation in a local datacenter or even not so local. It may well work out now that it could be cheaper to colocate and pay for some sort of contract from a company like HP that will give you guaranteed response times. All this will need looking into.

The Level 2 with a GTI of 12 hours and no displayed GTR to all intents and purposes means there is NO SLA that is of any use whatsoever.

It may even come out of this that its actually better to get rid of any OVH servers and use Kimsufis. Spread clients more thinly across them and at the same time spread the risk.

Anyway, a lot of this is to let others weigh up and decide, not me

[marks]

Your web page said and still does with regards to SLA, Level 1 "faulty component" not "faulty component that our systems detect" or anything similar. I fully understand and accept that components fail and that if you are not aware of it then in that case I don't think anybody would expect you to be responsible. In this case I was doing absolutely everything within my power to alert you to this and you refused to treat it as a Level 1.

if you're saying a breach on the SLA to get the 5% refund per delayed hour, I'm afraid that there are several reasons why it cannot be a Level 1 SLA:

-the problem didn't appear in our first installation of the server, only when you reinstalled it.

-you didn't provide definite proofs that the RAID card was broken.
Our engineers had to investigate (we provide Reparation SLA of 2 hours, but that doesn't include Investigation + reparation. If a problem requires investigation, it's not Level 1 incident but Level 2 - remember that the servers are unmanaged, even though when the problem appears to be during the installation, you still have rescue mode to check the hardware).

-even though when the investigation was done, we still have to confirm that we can intervene (the engineers asked you in the ticket whether they could program the intervention and waited for your reply). Hence, we cannot assure that the replacement will be done within the SLA of 2 hours, because we're waiting for your green light.


To offer you an example of when a problem discovered by you can apply for Level 1 incident, consider the following case:

if you have a dead harddrive, but the server is still running (when in the RAID1), you can open a ticket and choose:

"hardware fault" >> "My hard disk is damaged"

and accept the condition:

By checking this box, you confirm the request for intervention on the server and authorise OVH staff to intervene on the dedicated server. I also have been warned by OVH the possible consequences of this intervention, including the risk of losing all or part of the data stored on the server. OVH can not be liable for loss or unavailability of such data.

then you have a Level 1 intervention subject to SLA Level 1 and subject to compensation. This programs an intervention directly with our engineers, and they'll have to do it within 2 hours (For EG servers, as it's your case) no matter what. Mind that through this, the basic boxes are ticked:
-faulty hardware is reported, and you take the responsibility that this is correct, no investigation required
-you've given the green light to intervene explicitly
-accepted that you won't hold OVH responsible for any lost of data (<= legally we need this!)

Well, hope this example and the explanation help to understand it. That's the reason why it's not a breach of SLA but we will treat the case nonetheless for a commercial gesture.

[DigitalDaz]

If I now told you one of the two virtual macs I have on this is broken you would probably call me a liar. You have to laugh or you would cry, I hope this new one works I've just ordered as then its only a matter of changing dns, reconfiguring network, submitting another ticket......

[Andy]

+1...

[DigitalDaz]

You've got no chance mate, I say it was a breach of your SLA and I stick by that, never mind any commercial gesture.

At 3:30 AM on Sunday morning I submitted a ticket stating that I believed there was a harware fault on the RAID controller. At 5:00AM I contacted your 24/7 incident support line which was answered after and hour and reported it by telephone.

Your web page said and still does with regards to SLA, Level 1 "faulty component" not "faulty component that our systems detect" or anything similar. I fully understand and accept that components fail and that if you are not aware of it then in that case I don't think anybody would expect you to be responsible. In this case I was doing absolutely everything within my power to alert you to this and you refused to treat it as a Level 1.

[marks]

ok. Could you close the ticket to confirm that the issue is sorted and send us an email requesting so in the customer support.

Just for clarify: the only thing we can guarantee through SLA is the intervention times, we cannot give you guarantees on the hardware not failing at some point. That be 3 years down the line or 2 days after delivery. The servers are checked before assigned to a customer, and no hardware error came up then, but it started showing problems as soon as you started reinstalling the server (it installed well when we delivered it to you).

So we give SLA to ensure that when the server is down, we send an engineer to deal with it (e.g.: if the server doesn't ping and within after 6 hours no one has looked at it, then the SLA Level 1 kicks in).

It's true that there are some issues that can be more tricky to find out and don't bring the server down. Even if it's a hardware issue, that won't be covered by Level 1 SLA, because we cannot anyhow detect it!

In any case, it's obvious the problem here, and what we can offer you is a Commercial Gesture to compensate for these days. Adn that's what we'll be looking at with your email to the support.

hope the explanation of the real nitty-gritty behind the scenes helps to understand how the SLA and compensations works

[DigitalDaz]

Pfft! Well, now you can claim your SLA!

Thats the next job

[Myatu]

It turns out it was a faulty raid controller, well, replacing it has certainly fixed it and the machine is now running as expected.

Pfft! Well, now you can claim your SLA!

[DigitalDaz]

It turns out it was a faulty raid controller, well, replacing it has certainly fixed it and the machine is now running as expected.

[DigitalDaz]

Well, apparently an intervention is going on at the moment, but you could try mpt_msi_enable_sas=1 (enabled, instead of disabled).

You do this by editing the /etc/modprobe.conf file and adding:

options mptbase mpt_msi_enable_sas=1

Then you need to run mkinitrd and reboot.

Another thing to try is noacpi in grub, but hopefully they'll figure out what's going on with the intervention.

The RAID controller is being replaced at the moment. Thanks anyway.

[DigitalDaz]

For that amount of customers you should probably have two servers - for if - no - when, one of them goes down. + backups obviously.

We have multiple servers knocking about, that is not a problem.

I'm counting myself very, very lucky that on this occasion apart from the inconvenience, stress, and some money, no harm has been done.

We run VOIP services with OVH, forget the websites, what about my VOIP

I have about 50 clients so far and its rising all the time.

I really need to think this all through. I think the answer, like you say, is spare server.

[Myatu]

Sorry, I should have posted that anyway

SCSI storage controller: LSI Logic / Symbios Logic SAS1064ET PCI-Express Fusion-MPT SAS (rev 08)

Well, apparently an intervention is going on at the moment, but you could try mpt_msi_enable_sas=1 (enabled, instead of disabled).

You do this by editing the /etc/modprobe.conf file and adding:

options mptbase mpt_msi_enable_sas=1

Then you need to run mkinitrd and reboot.

Another thing to try is noacpi in grub, but hopefully they'll figure out what's going on with the intervention.

[Mark1978]

Why do you say that?

For that amount of customers you should probably have two servers - for if - no - when, one of them goes down. + backups obviously.

[DigitalDaz]

I think part of the problem here has been that the ticket was opened and was completely ignored for a long period of time. A ticket should be answered almost immediately no matter the time of day. OVH is a huge company, why don't you have 24/7 support yet? We were told we were getting this 2 years ago, and where is it? Nowhere to be seen.

Andy,

This could very well be coming to a server near you soon. I've just had a lengthy conversation on the phone that has really clarified the SLA position.

At various points around the OVH site you may see, "SLA Hardware intervention and repair 24/7".

On the OVH server site you see:
Level 1
(Unavailable server, faulty component)

- Intervention (GTI)

1 hour

- Repair (GTR)

2 hours

This may be some sort of legal loophole to be able to put this up but essentially this is how I have been told it works:

Say in my case, it appears to be a faulty raid controller. Anyone with an OVH server may believe that under the above ie "faulty component" that would be a Level 1.

The reality is that because their system cannot detect the problem, it is treated as Level 2. If it later turns out that it is faulty then you may be eligible for compensation under the Level 1 clause.

But do not for one minute think that a faulty component is going to be treated as Level 1. Only those that the system can detect will be treated as Level 1. Do not think either that you can ring the 24x7 incident line it will still be handled as Level 2 if the OVH systems are unable to detect it.

Now for many people this may be perfectly acceptable that they are later compensated but for those of you hoping for the quick fix, you are bang out of luck. What that means is you have a potential 12 hour wait for an intervention on a hardware fault.

Here's a scenario that many of us will have seen: a stick of RAM goes faulty, the server then goes into an endless reboot cycle. It may respond to pings enough that that doesn't trigger the auto Level 1 intervention. Even if you take it into rescue mode and the rescue verifies the faulty stick of RAM, your ticket will STILL be treated as Level 2. That was just confirmed to me.

The only possible other way round is VIP support.

[Andy]

I think part of the problem here has been that the ticket was opened and was completely ignored for a long period of time. A ticket should be answered almost immediately no matter the time of day. OVH is a huge company, why don't you have 24/7 support yet? We were told we were getting this 2 years ago, and where is it? Nowhere to be seen.

[marks]

I can see your ticket on this ongoing issue.

Right now, the last installation you've launched has failed, and an intervention is on it's way.

Keep in mind that the 1 hour intervention time is only when the server is down (unfortunately, we don't have any way to detect other problems as quick as necessary to offer this SLA, therefore this SLA only applies on that case).

When there is an installation issue - which may end up being a hardware or software bug - the best thing to do is to open a ticket and reply to the engineers as quick as possible having done whatever they ask. It's a level 2 intervention in any case (as we cannot offer 1 hour SLA) and, if anything, after the issue is sorted, we can look into offering a compensation depending on the cause of the issue.

In a few minutes, there should be an intervention on your server and then we'll know more.

[DigitalDaz]

Well unless they are for free or some crazy low amount, that many websites you ought to be paying at least twice as much, and probably using a 'proper' web hosting company (say vps.net or something like that), rather than messing around with dedicated server.

Anyway, don't mean to criticise your business model, just saying the flaw is in your choice of providers.

An interesting observation. They are tiny, as in a homepage, about us where to find us, contact. And you probably get more traffic to your front door

[Thelen]

Why do you say that?

Well unless they are for free or some crazy low amount, that many websites you ought to be paying at least twice as much, and probably using a 'proper' web hosting company (say vps.net or something like that), rather than messing around with dedicated server.

Anyway, don't mean to criticise your business model, just saying the flaw is in your choice of providers.

[DigitalDaz]

Sorry, forgot to ask what lspci gives as well (just the line for the RAID controller).

Sorry, I should have posted that anyway

SCSI storage controller: LSI Logic / Symbios Logic SAS1064ET PCI-Express Fusion-MPT SAS (rev 08)

[Myatu]

Sorry, forgot to ask what lspci gives as well (just the line for the RAID controller).

[DigitalDaz]

Myatu, here's that info:

# modinfo mptbase
filename: /lib/modules/2.6.32.12-0.7.1.xs5.6.100.323.170596xen/kernel/drivers/message/fusion/mptbase.ko
version: 4.22.00.00
license: GPL
description: Fusion MPT base driver
author: LSI Corporation
srcversion: 7C0F039026E1ACBAB9BD36A
depends:
vermagic: 2.6.32.12-0.7.1.xs5.6.100.323.170596xen SMP mod_unload modversions Xen 686
parm: mpt_msi_enable_spi: Enable MSI Support for SPI controllers (default=0) (int)
parm: mpt_msi_enable_fc: Enable MSI Support for FC controllers (default=0) (int)
parm: mpt_msi_enable_sas: Enable MSI Support for SAS controllers (default=0) (int)
parm: mpt_channel_mapping: Mapping id's to channels (default=0) (int)
parm: mpt_debug_level: debug level - refer to mptdebug.h - (default=0)
parm: mpt_fwfault_debug:Enable detection of Firmware fault and halt Firmware on fault - (default=0)

That's on Xen 5.6 that I was asked to install after an intervention earlier. The intervention was caused after the server hung whilst trying to go into rescue mode. I thought that may have helped move it on but I was mistaken.

There still seems no acceptance by support that its hardware related and so falls under no SLA even though its now failed with:

VMWare 5
Xen 5.6
Xen 6
Proxmox 2.1
FreeBSD 9
OVH Rescue Pro

I'm at the end of my tether and feel I have got absolutely no further with support. On top of that I've had to shell out another £70 to renew the server I was going to allow to expire after I had done the transfer.

With all OS and Rescue mode, I end up roughly in the same place, just slightly different messages depending on OS:

mptbase: ioc0: ERROR - Doorbell ACK timeout (count=14999), IntStatus=ffffffff!
mptbase: ioc0: Initiating recovery
mptbase: ioc0: WARNING - Unexpected doorbell active!
mptbase: ioc0: ERROR - Failed to come READY after reset! IocState=f0000000
mptbase: ioc0: WARNING - ResetHistory bit failed to clear!
mptbase: ioc0: ERROR - Diagnostic reset FAILED! (ffffffffh)
mptbase: ioc0: WARNING - NOT READY WARNING!
mptbase: WARNING - (-1) Cannot recover ioc0
mptscsih: ioc0: host reset: FAILED (sc=ed7cba40)
sd 0:1:0:0: Device offlined - not ready after error recovery

[DigitalDaz]

I interrupted the install after 3 hours.

I decided to try a freeBSD install, that would rule out driver errors??

Same problem:

Sep 2 18:35:30 rescue-bsd kernel: mpt0: request 0xffffff8000a58320:291 timed out for ccb 0xfffffe000d439800 (req->ccb 0xfffffe000d439800)
Sep 2 18:35:30 rescue-bsd kernel: mpt0: attempting to abort req 0xffffff8000a58320:291 function 0
Sep 2 18:35:30 rescue-bsd kernel: mpt0: handshake aborted - invalid doorbell state
Sep 2 18:35:30 rescue-bsd kernel: mpt mailbox: (0xffffffff) State Unknown WhoInit Unknown
Sep 2 18:35:30 rescue-bsd kernel: mpt0: soft reset failed: device not running

[DigitalDaz]

dmesg gives more or less the same as above.

I currently can't do anything with it, I'm reluctant to forcibly reboot it because OVH support will consider that some form of indication of a success and simply restart the installation I assume.

We're now 110 minutes into the installation and I'm being asked to wait patiently until it finishes, though I don't know whether it will be an hour, tomorrow or a week.

mpt-status when I could get to it gave something like ioctl:bad address

There where also messages when running under proxmox like "controller disabled"

I was thinking bad driver but if it is its bad in Proxmox, Vmware and the OVH installer for Xen too.

I wish now that I hadn't ordered the hardware raid but VMware will not install without hardware raid which also indicates its either a hardware fault or that no one i using VMware at OVH with more than one disc, or that the installer/install has recently changed.

[Myatu]

Hmm, could be an initialization issue with the driver. What does /var/log/dmesg show? Also, what does modinfo mptbase give (version and options)?

[DigitalDaz]

They refuse to escalate the ticket as they do not think its a hardware problem, their systems would detect it, and, adter all, it still responds to ping.

Third attempt at installing a different hypervisor. Xen is now stuck installing and the KVM shows the following.



I can only assume they're now taking the p*** because even after adding the image to the ticket, I receive this:

Dear Customer,

I see that you reinstalled the server.
It's still in progress.

Kind Regards

[DigitalDaz]

So OVH, I raised the ticket, I telephoned your 24x7 reporting service and STILL the ticket shows new and nothing has been done?

[DigitalDaz]

233 client websites... on a 200GBP server? something wrong there mate...

Why do you say that?

[Thelen]

233 client websites... on a 200GBP server? something wrong there mate...

[DigitalDaz]

Well here I am again guys and gals but this time I've learnt my lesson, no more the devil you know and all that crap because I clearly didn't know.

This weeks exercise was to upgrade a clients webserver from a Kimsufi to an EG SSD Max with all the bells and whistles, harware raid, professional use.

From one extreme to another I know but the customer base had grown and we were going to use it for other things too. It currently has 233 client websites on it according to the backup I did before starting the move.

The server was delivered about eight last night, I'd installed it with VMWare, I thought I would give it a whirl as I've been getting used to it on our other clients private cloud.

Then things didn't seem to be quite right, the console was locking up and a VM hung that was formatting, something wasn't right.

I decided to do a reinstall but with Proxmox as I would then have a familiar interface and toolset that would enable me to spot anything untoward.

It was a nightmare getting it reinstalled as it failed a couple of times.

By now I was expecting the worst but I did eventually get a clean install and it appeared to be working. Then it broke! I believe there is a fault on the hardware raid controller.

I submitted a ticket at 3.30am, not a disaster I thought, these things have SLAs.

Unless I've got it completely wrong, for this server the following applies:


Level 1
(Unavailable server, faulty component)

- Intervention (GTI)

1 hour

- Repair (GTR)

2 hours


My understanding of this is that it will be looked at within 1 hour, fixed within two??

By 5am, the ticket was still showing as new and I thought I would try the 24 hour fault reporting service, they actually answered the call at 6:03, I'd been waiting just over an hour and was just about to put the phone down. I'd only left it waiting whilst I was typing this.

The first thing the guy said was that if it was a hardware fault their systems should have detected it. An interesting statement because in my control panel it did say, though it doesn't now, that the rtm wasn't available with this service.

What the guy then told me has just left my head in a complete spin: When I asked the guy about the SLA he told me that they do not apply out of hours. I didn't mishear it, I even asked him to repeat it. Moreover, when I asked him when I could expect it to be fixed he told me he could not give me an estimate as he was not part of the dedicated server people and they were NOT IN yet!

So bottom line is I've just spent £200 on a server that is broken, only has a working hours SLA and I haven't a clue when it will be fixed.

And please, no comments about how I would pay 4 times the price anywhere else etc. I'm only expecting what they are telling me I will get, nothing more, nothing less.