[Andy]

Yes it's H/W RAID but they probably use non-hot swap which is why it had to be shut down. I know it was RAID1 as they removed a drive and replaced it and the RAID rebuilt itself. I have access to it through the web browser, it's an LSI 3ware card.

[Gomjaba]

Seems fine to be honest, I haven't had any issues so far but support has been superior in comparison to OVH for my HDD failure (1.5hrs and it was replaced, server downtime was 8 minutes thanks to H/W RAID1).

You sure you got hardware raid ? Why would the server need to be shutdown ? Sounds more like "fake" raid (bios based software raid).

Not that it makes any difference anyway (plus for decent hardware raid you need their Flexi Pack - still worth every penny).

[Andy]

They've been like this for years and they're only getting worse in my opinion, which is sad because they have so much to offer and have expanded so much because of their great prices and specs. They just have to focus more on actually supporting their products.

[AndyJ]

Hilarious thread. The longer we wait for OVH to fix things, the longer we have to find a new host.

[Andy]

Seems fine to be honest, I haven't had any issues so far but support has been superior in comparison to OVH for my HDD failure (1.5hrs and it was replaced, server downtime was 8 minutes thanks to H/W RAID1).

[Myatu]

How is Hetzner's network, Andy? Tried them ages ago, and it was abysmal then with constant drop-outs and crappy throughput.

[Andy]

And that there is exactly why I ditched OVH for Hetzner.

[glidewave]

Very sad to say hardware failures, especially drive failures, are painful to deal with (at OVH-EU (Not OVH-Canada). Even with downtime for days with drive failed, they will mess around and then give excuses as to why they are not responsible under the SLA. Of course, randomly, you'll get a tech who gets the problem and initiates a replacement based on your wishes immediately; which I wish was the norm.

It is a fact of life that using OVH requires a greater amount of patience and ability to dismiss most forms of common sense when dealing with hardware failures (among other things).

I still wonder how no one has brought up in a "Management Meeting": Ubiquitous Customer Experience

J.

[Andy]

It's like turning off your modem when your internet isn't working properly to get an engineer out. The people on the phone don't understand the issue, but the engineer will I did this a few times and the engineer solved it every time. TC just kept saying "reboot it. Oh it's working now, goodbye" even though it wasn't working properly.

[bart613]

If you've seen output from smartmontools before that indicate broken HDD just reuse the numbers if you want support to change the drive for you. It's not ethical and in fact, it's absolutely ridiculous, but it works. It's just a shame you've to resort to this kind of dodgy techniques to work around the system.

[ctype_alnum]

The only things putting me off Hetzner is the lack of anti-ddos and the fact that if you get even a small ddos attack against you they will just null route your machine.

Well they aren't going to let traffic flood into their network and affect other customers hoping the ddos will end soon when a simple null route does the job.

[Telejeesus]

9Hours.. you are lucky man..last summer it took week for me.

Thos this week I had superfast HD change.. But normally I have all the same problem that you have had.

[Kode]

The only things putting me off Hetzner is the lack of anti-ddos and the fact that if you get even a small ddos attack against you they will just null route your machine.

[Andy]

This is exactly what I said and everyone I told said too

[macole111]

Dare I say that it is German efficiency?

-macole111

[Andy]

Take a note of this OVH, this is why I left you for another company.

My new server just got a HDD failure (it wasn't a new disk as I bought the server from an auction range). Today at around 1am one of the disks reported timeouts and ECC errors and then promptly went offline (the RAID controller kicked it off the array). Let me run down what happened next:

8.35am - I checked my e-mails and noticed the RAID controller had e-mailed drive errors to me.
8.37am - I contacted Hetzner support to report the failure.
9.15am - Support replied asking for the drive serial number, which I replied with.
9.33am - The server goes offline for drive replacement.
9.46am - The server is back online with a new drive.
9.48am - I login and check the RAID to find it is rebuilding itself.

That is the sort of support I expected from OVH. If Hetzner can do it, why can't OVH?

[Andy]

Hardware RAID1 means the OS has no part in the mirror process so it's not aware it exists. No there is no parity but it has to rebuild and maintain.

[NeddySeagoon]

Andy,

I have 2x1.5TB drives sorry, typo in my first post. I have H/W RAID1 on those. ...

With two drives, you get at best, Raid1. Why do you need HW raid?
There is no parity data to calculate, it saves a DMA setup and some memory bandwidth on writes only. On the down side it adds a new single point of failure.

What am I missing?

[LinuxGam]

With the price of their SSD range and them being 2x 240GB , unless you need loads of space, then they are awesome. Saying that, my Xeon is SSD and got the cheapest 4770 i7 SATA for backup and non critical VM's. However, they are both CPU wise, pretty high end machines compared with most low cost offers from anywhere else. Even going to the higher E5 range there is not a massive difference.

I have an E5 hybrid CacheCade at OVH that costs me £160 or something and its nice. However an E3 1270v3 and i7 4470 are 4/5 as fast, but I can get two of them for redundancy and way way way more IO and CPU combined than that server for less cash. When they are connected at 1GB/s in the same rack.... that's a much better idea, for redundancy, CPU speed, hard disk IO not being mixed between SQL and web server and mail. The only reason I can think in hindsight is when one process / app needs 32GB plus and all that CPU. If you can split it into mupltiple VM's or servers, its far better value to get sep servers and if you security wary, install Tinc recommended by Myatu (Thanks, it is awesome)

[Andy]

It was an auction offer.

[LinuxGam]

I have 2x1.5TB drives sorry, typo in my first post. I have H/W RAID1 on those. I don't know if they charged yet as they haven't invoiced me for the server yet either. I need to e-mail their support to find out when it will be invoiced. I actually had it for more than 2 hours but I wasn't using it at that point. I saw no way to say I was finished with it. You have to give them an ISO of the OS you want, they then put it on a bootable USB stick which is plugged into the LARA system.

Fair enough if they charge you for that, as it requires manpower and they are a "low cost" provider with decent hardware, a bit like OVH are/were. Did you get a new offer (please post link) or is that an auction offer?

[Andy]

I have 2x1.5TB drives sorry, typo in my first post. I have H/W RAID1 on those. I don't know if they charged yet as they haven't invoiced me for the server yet either. I need to e-mail their support to find out when it will be invoiced. I actually had it for more than 2 hours but I wasn't using it at that point. I saw no way to say I was finished with it. You have to give them an ISO of the OS you want, they then put it on a bootable USB stick which is plugged into the LARA system.

[LinuxGam]

A Couple of things, you have 1 x 1.5TB and bonus hardware RAID? Also, did they charge you for the LARA use? I just checked my control panel and you can definitely use it for free on your installed server for 2 hours at least on my setup.. Do you have to get them to put a CD or image in the machine for you?

[Andy]

Well I got my server with similar specs to the one I have with OVH. Core i7 920 (OVH's is a Xeon), 12GB RAM, 2x1.5TB and H/W RAID as an additional bonus. I also got 1Gbps with 20TB bandwidth which is more than enough. I barely use 3TB as it stands. But as I say I had to use LARA for installing the OS, but that's no bad thing as it allows me to configure it exactly as I wanted it. They even supplied the RAID drivers on the USB stick they use to boot your ISO! Perfect. I did have to ask support about that one but <5 mins and I had an answer back by e-mail (OVH take note, this is the support level I expect).

[LinuxGam]

My only gripe with them so far is that reinstalls (for me anyway) will cost me €25 because I have to use their LARA KVM system. But due to the cost savings over my current OVH server this doesn't bother me. It still works out cheaper.

Hi Andy, I too decided to try them as was worried about the uncertainty of things here, I have no confidence anymore in any timelines sticking or even company policies staying for more than a month or so. I want somewhere stable from year to year and a reasonable price for newish hardware. My account and first server at Hetzner including checking my scanned Id etc happened by 9am Monday and was logged at the weekend when they don't do new accounts. My second server from when I ordered to when it went live and emailed me details was about 6 minutes!

I too read somewhere that LARA charged for use and then I logged a support call as I was "convinced" a certain setting wasn't enabled in BIOS. I received a personal response in about 3 minutes! Telling me to use LARA, which annoyed me as I had read about the charges somewhere. Anyway.. turns out that you can use LARA for the first two hours for free! it's only after that you get charged if you need it for longer.

The servers all say guaranteed 200mbs, I can consistently get 800mb/s both ways as long as it's in Germany, but it proves they are not the bottleneck. So my experience so far, great hardware i7 4770 Haswell and Xeon X3-v1270v3 Haswell, 240Gb RAIDed SSD's disk and very fast support and setup.

I still also have a server at OVH for now, but so far my tests are going well.

[Andy]

Was I surprised? Yes and no. I knew OVH support was slow, they have been for as long as I can remember but you usually get things done after a bit of a wait and a phone call. After all those of you on here who follow my topics would know I've had a few HDD failures over the years. They've always been replaced within 24 hours (which is a little slow but otherwise fine). But this time they couldn't even do that. No amount of e-mails and phone calls to support made it any quicker. In fact I suspect my constant badgering may have made it slower, not to mention my attitude (can you blame me for that?).

So no, overall I'm not surprised, and that's why I have chosen to make my move away from OVH for dedicated servers, at least for the time being. OVH have been a great company to me over the years but my expectations are clearly too high now. I've decided to go to a competing company, Hetzner, after many recommendations from others both here and from a few friends who haven't dealt with OVH.

So while I may be moving away from OVH's servers on a sour tone, that's not to say I won't hang around here and help out and it won't stop me from coming back if I hear improvements in support have been made. For the time being however, Hetzner is my choice. I have already had to use their support today after ordering and using an e-mail that was hosted on my OVH server that was down at the time. I e-mailed to get it changed and it was done in less than 10 minutes. What more could I ask for?

My only gripe with them so far is that reinstalls (for me anyway) will cost me €25 because I have to use their LARA KVM system. But due to the cost savings over my current OVH server this doesn't bother me. It still works out cheaper.

I apologise to my attitude towards Neil as it's not his fault, he's only following the policies outlined for him. However I'm sure you can understand my frustration.

[LawsHosting]

when something fails, you wait hours, sometimes longer, to get any form of "support".

With the new "raised" price structure, I'm sure support will be dire as usual.

[AdamD]

Terrible service you received Andy. But, is it any surprise? Really?
That's why I moved away from hosting anything important at OVH, when something fails, you wait hours, sometimes longer, to get any form of "support".

[Andy]

Yes it was. I will e-mail from my secondary account as my primary is still offline while I rebuild the server.

[Neil]

I can see the reinstall was successful this time, for the compensation send us an email and we will take a look.

[Andy]

I am trying now.

[Neil]

I launched it 3 times, it failed each time. The technician launched it, it failed, then he put it in rescue mode like it would help me solve the problem.

Please sort out my compensation request.

Please can you try again and let us know the error you get when you relaunch the reinstall?

[Andy]

I launched it 3 times, it failed each time. The technician launched it, it failed, then he put it in rescue mode like it would help me solve the problem.

Please sort out my compensation request.

[Neil]

Hi

You need to relaunch the reinstall from the OVH Manager, it has failed:

ERROR: Win 2008 setup.exe failed (c:\Windows was not created)

[Andy]

Still waiting... And now I'm going to bed.

To Neil (or anyone else from UK support), I request the maximum compensation for a LEVEL 1 incident please. Yes, LEVEL 1, not level 2. My server is UNAVAILABLE due to the technicians incompetence and as you have broken the 4 hour SLA by say, 6 hours now I count (the server was classed as unavailable from the moment I agreed to replace the disk at 2:20pm), I believe that entitles me to 10 hours * 5% of the server cost in refund, which is the 50% maximum.

[Andy]

The intervention was closed with "BIOS check" as the details. It's still not reinstalled. Honestly, what the hell is wrong with the tech people that they can't follow instructions I left in the ticket? I even explained over the phone and they still can't get it right.

That's it, I'm done with OVH. You've just lost my loyal business because you can't seem to get anything right. Maybe one day if you reconsider your support levels I'll come back, but not if they're going to continue like this.

I'll be e-mailing about compensation for your incompetence. This is now 2 days the server has been offline because of the disk fault which took you around 36 hours to complete.

[Andy]

The tech claimed he changed the boot order to the 2nd disk but it didn't boot. Either I didn't do the RAID properly or he screwed up. In any case when it booted into WinRescue I couldn't get diskpart to rebuild the array. I gave up and decided to just use my backups instead but Windows wouldn't install. The tech is intervening now according to the ticket. It says he's intervening with a "serial connection", whatever that means.

[NeddySeagoon]

Bummer, I've not heard of a raid1 that would not support a failed disk replacement. Its quite common that the box won't boot because its trying to boot from the new (blank) drive. That needs BIOS access to fix though

[Andy]

8pm the new HDD was finally installed. My RAID didn't hold out and couldn't be rebuilt so I started a reinstall of Windows only to be confronted with an install error, now the system has locked me out so I can't enter rescue mode either. Here's to another 24 hour wait for that to be fixed as well...

[macole111]

Why would anyone get anything more than a Kimsufi for this level of support OVH?

-macole111

[Andy]

Thanks Neil. This still took longer than it should have in my opinion.

[Neil]

4 hours now since they said they would do the disk... still waiting...

It was just processed by the incident team:

An operation was added for ****** to the list of
interventions at 2013-11-28 18:30:00.

Here are the details of this operation: Replacement of a
disk in the RAID

[Andy]

4 hours now since they said they would do the disk... still waiting...

[Andy]

Well I'm still waiting for the technician to replace the disk. This is getting incredibly stupid. How many people is this technician managing?

[dayvo]

All these goings on, as well as the delay on being able to order new servers really has put a dent in my confidence with OVH. I only hope that things are better once they are selling stuff normally again.

There is a saying which I think OVH should adapt:

Under promise and over deliver

Unfortunately I think that's been swapped over in translation

[Andy]

I have plenty of backups so I don't worry about that (uploading them afterwards is a slow challenge if I don't have a fast connection to do it on). That doesn't worry me and isn't OVH's concern anyway, keeping the hardware running is. I'm certainly looking at the competition for my next server. OVH won't even sell me or anyone a server right now so that puts them in a bad position from the off.

[crowarth]

3 1/2 hours. Wow, this is some definition of "shortly".

These last few months have forced me to look at what the competition have on offer - as well as being more concerned with backups. I guess you could say that the issue you're having now has made me sort out my own backups as a pre-emptive measure. I'd hate to lose any databases I have (I did have mysql replication going, haven't set it up again since getting the mSP) or any other data.

All these goings on, as well as the delay on being able to order new servers really has put a dent in my confidence with OVH. I only hope that things are better once they are selling stuff normally again.

It makes me wonder, all these manual orders ... people specified what they wanted, but my guess is the "token" they'll receive just allows them to order from the new ranges. A kind of gated ordering process with the token being the key. After that everyone else can order (or wait for the new stuff to be in stock ... -_-)

[Andy]

I have just had the reply from the technician.

[Andy]

3 1/2 hours. Wow, this is some definition of "shortly".

[Andy]

So let me ask, how long is "shortly"? I've been waiting over 2 hours since you said they would post in the ticket and it's still not there. This is the sort of thing I'm talking about with this topic.

[Neil]

Hi

I can see you will get the reply shortly so when you are ready just reply to the ticket once the response is there.

[Andy]

Hi Neil,

Thanks for your reply. This is all I wanted in the first place so why it couldn't have been offered in the beginning I really don't know. My server has been online since I was told it would not be replaced so I'll have to do some backups again before I agree to it being replaced.

Edit: I can't see the reply in my ticket. Is it under ticket 1570118?

[Neil]

Hi

The incident team have replied to you asking whether you want the hard drive to be replaced or not, although it has not failed yet they are still giving you the option, just reply if you want it changed.

[Myatu]

The "RAW VALUE" field is vendor specific. The few things that are generally the same across vendors is the start-stop count, powered on hours and the temperatures. You'll have to look at the "VALUE" and "WORST" column instead, and check if these values are lower (not higher) than "THRESH" (or on the Windows screenshot "Warn"). This is also why the "WORST" column will show a lower or equal number than the "VALUE" column (like with your raw read errors).

[Andy]

All of the reallocated sectors appeared within few weeks, that's not normal on a healthy drive. Also you're missing the fact that the other two attributes are increasing ALL OF THE TIME. Every time I load the SMART data, they've increased. Look at those values now compared to what they were before:




Like I said, I know what a failing hard drive looks like.

1. Increasing Reallocated Sectors
2. Increasing Raw Read Error Rate
3. Increasing Seek Error Rate
4. Disk performance reduced
5. Disk Active Time pegged at 100% on light loads

I've seen this all before and eventually that drive is going to get so unresponsive the server will barely function. This is why SMART data exists, why can't you understand that? We're talking about a few identifications of a failing drive here, not just one. Unfortunately sometimes diagnostics don't give enough information to determine a failing drive which is exactly what has happened here. The OVH tech has looked at it and dismissed it because of his own ignorance, lack of knowledge and experience.

This is what it looks like after I finished writing this post:



Look at the difference in the values.

[NeddySeagoon]

Andy,

The SMART data under VALUE WORST THRESH is normalised, so that a failure is indicated by VALUE or WORST being <= THRESH.
Reallocated sectors hurt drive performance but as yet, your 97 Current_Pending_Sector count is still zero, so no data has been lost yet. Thtat's a count of sectors the drive wants to allocate. The raw values often mean very little and are vendor or even drive model specific. There are often several raw values packed into into the reported RAW_VALUE field, so you see huge (but harmless) numbers.

Its worth running the long self test on the drive, which does a full disk read, and see how the SMART data changes. More reallocated sectors or a changed Current_Pending_Sector count would be a worry.

The drive will have had a non zero Reallocated_Sector_Ct wen it was new. Its how drives appear to be perfect. Small increases with time is part of normal wear and tear. I'm a Linux user and I know that its software raid works, I've had three raid set members fail. I would be surprised if Windows software raid didn't work.

[Andy]

The problem is I always have to fight with OVH to get the support I need. Every time I contact them it's the same. Even the last HDD replacement took over 9 hours to be completed and that disk really had failed, and even that was only after a phone call to the technicians in France once UK support closed.

OVH is a great company when your server is working, I won't deny that for a second. When things go wrong you get to see how sour things can really go. I'm paying a premium price when you look at all the cheapo Kimsufi servers that have been sold and I expect to get the level of support to match it.

Clearly my loyalty to OVH for 5, almost 6 years is worth nothing and I find that incredibly sad.

[LinuxGam]

I feel for you Andy, I wasted two weeks of my time, proving to OVH that there was network routing traffic/issue that was marked as closed by I think 3 lower level technicians as it must be a VM Guest ISSUE, even with clear proof from me it wasn't in the call. I explained that it was routing different with a fail over than the local native host IP which they were testing in rescue so there tests were comparing apples with oranges.

Eventually after a lot of shouting and copy/pasting proving it, it was escalated to a REAL network engineer who was very helpful and quickly worked out it was an issue with some QOS limit setting on a route in RBX affecting that IP range. They did refund my server costs for the month, but if I had not done all my own testing, re-imaging the whole server etc.. it would have just stayed closed 3 times by first level.

We could all go somewhere else and pay a another company prob 2x the price for older hardware and they have a guy on it fixing it for you on the phone in 30 secs. I moved to OVH from a place like that. I had had my server with them 3 years and they had always been excellent, but the price was going up and the hardware was still not being refreshed. So greed/ambition took over and I came to OVH :-)

My first two servers had issues, first was hardware, second was Support/the routing

[Andy]

This may very well be the last topic I make here before I leave OVH for good and find a company worth my time and money.

I again this week suffered a failing HDD in my server. I don't think it's even been 6 months since the last time it happened. It's to be expected though, it's a Seagate drive after all. I contacted support in the usual manner (through e-mail) asking if they would replace it and advise on software RAID1 setups. I got some helpful replies from Neil and I was happy that the RAID1 had a good chance of working properly after doing my own testing in a virtual machine, etc.

Neil put the ticket in for me at around 12pm today. I didn't get a reply to it until after 7pm by one of the technicians. They asked for more information even though Neil supplied the screenshot of SMART errors I gave him. I did as the technician asked and posted the details on the ticket. Over an hour went by and the tech hadn't been back to reply to it, so I called up the 24hr hotline. I got through to someone who spoke decent English so I thought it would go well. It didn't.

He looked at my ticket and said nothing was wrong with the disk even though it shows all of these SMART errors. Let me show you the errors I am talking about. Having experienced many disk failures in my life I know what a failing disk looks like. This is it:

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH
TYPE UPDATED WHEN_ FAILED
RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 115 096 006 Pre-fail Always - 85248529
3 Spin_Up_Time 0x0003 100 100 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 6
5 Reallocated_Sector_Ct 0x0033 099 099 036 Pre-fail Always - 70
7 Seek_Error_Rate 0x000f 083 060 030 Pre-fail Always - 223831287
9 Power_On_Hours 0x0032 096 096 000 Old_age Always - 4344
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 6
184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0
187 Reported_Uncorrect 0x0032 033 033 000 Old_age Always - 67
188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0
189 High_Fly_Writes 0x003a 001 001 000 Old_age Always - 694
190 Airflow_Temperature_Cel 0x0022 063 050 045 Old_age Always - 37 (Min/Max 37/37)
194 Temperature_Celsius 0x0022 037 050 000 Old_age Always - 37 (0 18 0 0)
195 Hardware_ECC_Recovered 0x001a 040 038 000 Old_age Always - 85248529
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 187153199927544
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 3841613719
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 2009020410

Add to that the fact that this disk is hitting 100% disk active time a hell of a lot when the other disk it is in RAID1 with is barely touching 20% and you have even more evidence to show that the disk is failing.

Why can the technicians not see this? I am paying £84 a month for this server. The server is no longer worth this price yet I continue to pay it because I am a loyal customer, however loyalty appears to be something OVH don't care for. I have been with OVH since 2008, how many others here can say that? In that time I have never been more annoyed with OVH's support than I am right now.

Now, SLA's, that's a whole new story. I bring you to the following table posted on the OVH.co.uk website.

Level 1
(Server unavailable, faulty component)
- Intervention (GTI) 1 hour
- Repair (GTR) 4 hours

Level 2
(Unusual functioning of the service except level 3)
- Intervention (GTI) 12 hours
- Repair (GTR)

In this instance, my server is currently under support level 1. Why? I have a faulty component. A HDD failing is a faulty component, right? Sure it is. Well not according to OVH support it isn't. I'm told that apparently it's a level 2 because my disk has not failed yet. Absurd right? Damn right it is. You don't wait until a disk has completely failed before you replace it. Where is the good in that? It degrades server performance and puts your data at risk. By all means replace it for me, test the drive and put it back into use if it's found OK but don't tell me the drive is OK when it's clearly not. SMART errors aren't there for fun, they're there to advise you of a failing disk before it's too late.

So anyway here I was sat waiting for a disk to be replaced, my server offline because I've made my backups and don't want to risk data being changed, and they tell me it's under a level 2 support level and then tell me the disk is just fine.

Well I'm sorry OVH, but your support has just proven how crap they are. I'll say this for the UK staff, you're great when you get going (Neil for example, you're a star when you work your magic like you did investigating RAID1 for me), but as for the French staff there is a lot to be desired, and a LOT that needs to change.

You market your servers as professional, yet where is the professional in any of what I have just outlined? Where is the support for loyal customers such as myself? It's non-existent, and for that I am sorry. I can't see myself continuing to have servers from OVH for much longer. You have no care or regard for customers or their loyalties. My £84 can buy me a lot better else where, hell, it can buy me 2 servers at half the price so I have a backup, something OVH won't let me because for some reason you don't sell servers anymore.

My server is now sat there limping along without any idea when the disk will fail. I'm likely going to lose some of my data as a result unless the RAID1 holds up, but since it's only software RAID1 I have no idea how it will fare through a full disk failure. Wish me luck, I'm going to need it.

Andy