OVH Community, your new community space.

This is NOT professional OVH


Thelen
24-08-2012, 03:16
Hetzner do similar things... all the budget providers that allow warez (unofficially) do things like this because they have to protect thier networks. They just want the money from that market as well :P

This whole thread shouldn't even be, the OP should be paying at least twice as much for what he wants.

Andy
23-08-2012, 23:04
I'm pretty sure you can't remove professional usage once you have it on by the way.

I agree, the protection detecting floods as a hacked server is wrong. OVH is the only company I know that threatens to shut off servers and force you to reinstall when they "suspect" a hacked server. They never prove it. If their system says it's hacked then they stand by that without a manual check.

Arran
23-08-2012, 21:43
Quote Originally Posted by Neil
Hi

From what I can see at the moment there is an incoming attack on your server and the protection is in place so you may not be able to access the server, even in WinRescue.
Luckily no more "hack" was detected and we got into winRescue to disable firewall but something wasn't working so I had to get professional usage for 1 month and used the included KVM to disable the faulty rule. Then we spent some hours recovering a partition than had disappeared which I guess was due to the amount of mess ups we were making, lol.

This hack detection thing shouldn't be considering flood attacks as a "hack" though, what if it would have detected it again? According to the message we'd have had to spend 2 days installing everything again.

Andy
23-08-2012, 19:38
To be honest I've never thought about those, but they have so little market share that the amount you will lose as a result is probably negligible.

chmac
23-08-2012, 15:18
Interesting, thanks for the info Andy, it's not my area of expertise, but I'll run that past our SEO folks and see what comes back. What about the plethora of other search engines like Yahoo / that Chinese one that's always spidering us, etc? :-)

Andy
23-08-2012, 15:16
You don't need a UK IP for search engines (I work in SEO by the way). Just sign up with Google and Bing webmaster tools and associate your domain geographically.

If you have a .co.uk domain, it's automatically UK targeted regardless of your IP.

For non geographical domains (.com/.info/.org) then whatever the IP is will be your target unless you set it in webmaster tools and override it.

chmac
23-08-2012, 15:14
Quote Originally Posted by Andy
What do you need UK IP's for?
Something to do with search engine and being identified as a UK business I think. In fairness though, OVH's prices on SSD machines cannot be beaten, we're now all SSD, and we wouldn't be with Hetzner, so I'm not 100% sure we'd switch immediately if they offered a UK IP.

Andy
23-08-2012, 15:12
Quote Originally Posted by chmac
We maintain a failover with an alternate provider (in this case hetzner.de, very reasonable pricing) because of this very issue. Given OVH's price point, the service has to be on par with the cost, and so if we had some kind of outage that lasted longer than an hour, I want to know I have an alternate machine standing by. As an aside, if we could get UK IPs from Hetzner, we'd probably be with them, the support is rumoured to be excellent.
What do you need UK IP's for?

Neil
23-08-2012, 14:08
Quote Originally Posted by Arran
Whatever detects these "hacks" is rubbish, been a victim of its false positive detections too.

- Accidentally blocked everything in Windows firewall.
- Email saying technicians will check out the unresponsiveness.
- Got messages saying that the servers IPs are blocked for 1 hour due to some attack. (There was a little SYN flooding so fair enough)
- Now says its been hacked and was deactivated and placed into rescue mode.

"However, if despite your intervention, new unusual activity is detected, your server will be deactivated again and leaving no option but that of a full re-installation."

If I start the server again and this silly thing detects another "hack" (how was it being hacked when everything was blocked...) I'll have to re-install the server just because of some SYN flooding?

As long as your server responds to pings they don't seem to care because we've been getting SYN flooded and even a 900mbit HTTP DOS not long ago but as soon as something happens when the server is unresponsive they say the server was hacked.

Had this accident before and OVH techs were nice and they just try "last known good configuration" which fixes firewall screw up, but they've made it a lot, lot more complicated for me. Gonna try winRescue and disable firewall then try normal boot up. Gotta hope that stupid detection thing won't detect it again and make me re-install just because of some hack that doesn't even exist.
Hi

From what I can see at the moment there is an incoming attack on your server and the protection is in place so you may not be able to access the server, even in WinRescue.

chmac
23-08-2012, 13:59
We maintain a failover with an alternate provider (in this case hetzner.de, very reasonable pricing) because of this very issue. Given OVH's price point, the service has to be on par with the cost, and so if we had some kind of outage that lasted longer than an hour, I want to know I have an alternate machine standing by. As an aside, if we could get UK IPs from Hetzner, we'd probably be with them, the support is rumoured to be excellent.

Arran
22-08-2012, 23:34
Whatever detects these "hacks" is rubbish, been a victim of its false positive detections too.

- Accidentally blocked everything in Windows firewall.
- Email saying technicians will check out the unresponsiveness.
- Got messages saying that the servers IPs are blocked for 1 hour due to some attack. (There was a little SYN flooding so fair enough)
- Now says its been hacked and was deactivated and placed into rescue mode.

"However, if despite your intervention, new unusual activity is detected, your server will be deactivated again and leaving no option but that of a full re-installation."

If I start the server again and this silly thing detects another "hack" (how was it being hacked when everything was blocked...) I'll have to re-install the server just because of some SYN flooding?

As long as your server responds to pings they don't seem to care because we've been getting SYN flooded and even a 900mbit HTTP DOS not long ago but as soon as something happens when the server is unresponsive they say the server was hacked.

Had this accident before and OVH techs were nice and they just try "last known good configuration" which fixes firewall screw up, but they've made it a lot, lot more complicated for me. Gonna try winRescue and disable firewall then try normal boot up. Gotta hope that stupid detection thing won't detect it again and make me re-install just because of some hack that doesn't even exist.

chmac
10-04-2012, 13:41
I read this and was intrigued, I found lowendbox.com which lists lots of VPS providers, and quite a few in the $15-$30/yr price range. Makes an interesting backup proposition.

freshwire
23-03-2012, 11:32
Even for my personal mail server I have a backup machine ($20/yr VPS).

Thelen
23-03-2012, 05:37
I'm not saying don't host business services, but if they truly are critical, ie 5 minute downtime is a problem, then as Neil says you have to recognise the different SLAs that OVH goes by.

TBH in many parts of the world 4 hour SLA is still pretty damn good for infrastructure, so if it is that critical you need to look at backup systems rather than increasing the SLA of that single piece.

Just look at Amazon, Google, and Microsoft. They've had massive downtimes already this year..

Neil
21-03-2012, 10:39
Quote Originally Posted by William_GL
Would be interesting to hear OVH's response to your comments.

Are we foolish to host business critical services here? It would seem to me that some of OVH's packages are geared directly towards such services, or am I missing something?
We have plenty of businesses that host with us, you can see the SLA we offer on HGs, MGs and we have a VIP Service if you want peace of mind as well https://www.ovh.co.uk/support/vip_su...nformation.xml

Just make sure you have the right support level for what you are hosting, Kimsufi servers do not have an SLA other than hardware failure which is 4 hour.

Quote Originally Posted by Andy
Good to know. Now wheres my compensation for the 6 hours down time?
Quote Originally Posted by maybars
And 12 hours for me?
You need to contact us for compensation it is not applied automatically.

William_GL
21-03-2012, 00:19
Would be interesting to hear OVH's response to your comments.

Are we foolish to host business critical services here? It would seem to me that some of OVH's packages are geared directly towards such services, or am I missing something?

Thelen
20-03-2012, 23:54
You shouldn't be using OVH for business critical services. They only promise 99.95 % on the network, and only refund for level 1 and network incidents and only up to 100%, which face it, OVH is dirt cheap so the refund really makes no difference.

Sorry to keep harping on about it, but you need to align your expectations with OVH, not the other way around. OVH is hardly the only company to cut off servers, and not even only budget companies do it. You'll find the same issue with even companies like Softlayer...

Mark1978
20-03-2012, 10:58
We need a more detailed response from OVH on how they are going to resolve this problem. As others here we run servers which are absolutely critical to the running of our business - if the servers go down we have no business!

We are actively putting in place a backup system hosted by ourselves to allow failover for precisely this problem. We had considered a disaster recovery system using the Strasbourg data centre, but presumably OVH could cut off servers there too, and at the same time?

Trapper
11-03-2012, 17:45
Hi All,

Thought I should add my 2 cents...

Two of my servers have been affected. (I have 4 Kim's and 2 OVH's). Both of the affected ones were Kimsufi running Server 2008 R2.

One was down from Friday night to Sunday Morning - Almost 36 hours!

The other was down for about 10 hours late on Saturday.

I did some checks on the IP's which were listed along side mine, mentioned in the "attack log". I checked to see if they had also been taken offline and they had not!

To test I just entered the IP address into my browser, which, in most cases gave me a response. From this I could see that not all of the servers were 2008 R2's, or if they were, they were running apache (suspect not!).

I think we should all take this as a chance to look at what we do with these servers. I run website and mail services from mine, and I have one specifically for backups. I am going to drop my Kimsufi backup server, and buy this elsewhere, so that it is not taken down just when it is needed!

maybars
11-03-2012, 14:07
Quote Originally Posted by Andy
Good to know. Now wheres my compensation for the 6 hours down time?
And 12 hours for me?

Andy
11-03-2012, 11:42
Good to know. Now wheres my compensation for the 6 hours down time?

aozm48
11-03-2012, 11:17
I just got mailed this from the OVH datacentre tech, seems to indicate they changed something in their hack detection methods and had a load of false positives
http://translate.google.com/translat...3D6476&act=url

Andy
11-03-2012, 10:42
Perhaps so, but you have to remember the number of customers and servers OVH have to cater for. It would be a little unreasonable to expect them to filter all of these things out.

However we've determined it could be an operating system issue as there are a lot of people reporting having the same issues. 2 of my friends so far have had the same thing happen and both run the same OS as me. 2 other friends don't run Windows and nothing so far. That can't be coincidence surely? Note that none of these friends are in any way linked with me or my server so there would not be the same security problem software wise in that sense.

freshwire
11-03-2012, 08:10
This is a very specific thing they disabled server for. Surely they can filter it out themselves and notify you of it?

Andy
11-03-2012, 01:23
I just got hit again on my IP failover. Thankfully they're just disabling the IP when it's an IP failover rather than the whole server. I just happened to check my email before going to bed when I noticed. I've also blocked port 1 outgoing now as well.

maybars
10-03-2012, 21:47
They should. It is KimFS-10T, we are talking about 9TB data :S Disks are not reachable via rootftp which is very ridiculous. Just like yours, my IP is mentioned just once:

--------------------------- LOGS OF SCAN --------------------------

Unallowed traffic

startime endtime scrort dstort
----------------------------------------------------------- -----------------------------------
2012-03-10 07:00:36 2012-03-10 07:00:36 176.31.252.134:1 96.241.212.128:0
2012-03-10 07:00:36 2012-03-10 07:00:36 188.165.122.228:1 96.241.212.128:0
2012-03-10 07:00:36 2012-03-10 07:00:36 176.31.130.2:1 96.241.212.128:0
2012-03-10 07:00:37 2012-03-10 07:00:37 178.33.111.120:1 96.241.212.128:0
2012-03-10 07:00:37 2012-03-10 07:00:37 188.165.206.11:1 96.241.212.128:0
2012-03-10 07:00:40 2012-03-10 07:00:40 188.165.227.200:1 96.241.212.128:0
2012-03-10 07:00:41 2012-03-10 07:00:41 94.23.8.37:1 96.241.212.128:0
2012-03-10 07:00:41 2012-03-10 07:00:41 94.23.89.181:1 96.241.212.128:0
2012-03-10 07:00:41 2012-03-10 07:00:41 94.23.251.163:1 96.241.212.128:0
2012-03-10 07:00:41 2012-03-10 07:00:41 188.165.195.124:1 96.241.212.128:0
2012-03-10 07:00:40 2012-03-10 07:00:40 87.98.246.102:1 96.241.212.128:0

--------------------------- END OF LOGS ---------------------------

Andy
10-03-2012, 21:18
If you have your server back on, go into the firewall and use the firewall rules for outgoing and block port 0 on UDP and TCP. Simple. You can also block the destination IP from the log if you like.

maybars
10-03-2012, 15:19
It happened to me right now. Andy what should I do? Please guide me.

Andy
10-03-2012, 10:04
Actually I'm happy with how this turned out. OVH replied pretty damn quick considering I posted at 8am. I had a reply by 10am, 1 hour after they opened and by 10.30 they'd allowed it back on the network. I've never had a response that quick, so they ARE getting better. I can't deny that.

Anyway it would see that it is almost definitely a 2008 R2 SP1 issue as a friend of mine also had his go down yesterday evening for the same reason and he doesn't use anywhere near the configuration I do, thus it couldn't be a software issue outside of the OS.

Thelen
10-03-2012, 05:44
Mission critical, on OVH? Are you handicapped??

OVH knows perfectly well what market they are in, and it isn't mission critical on 100GBP/month servers. They only offer 99.95% on the network, and level2 incidents are 12 hours. Plus they only have 5 day phone access typically.

Anyway, I know that isn't what you want to hear, and I sympathise as I've had it happen to me (actually I made a post on here, everyone replied saying I was a moron and the server was hacked and it wasn't OVH.. funny how times change..), but I see no fault from OVH. Pay peanuts get monkeys.

Andy
09-03-2012, 14:24
Cheers. I've blocked outgoing port 0 TCP and UDP, and also the IP they said I was attacking.

aozm48
09-03-2012, 14:23
just added a new rule in windows firewall with advanced security (actually 2 rules) that block all traffic to destination port 0 for TCP and UDP

Andy
09-03-2012, 14:19
How did you block outbound traffic to port 0? Not really played with blocking things before so not entirely sure how to do it.

EDIT: Never mind, I think it's done. That wasn't so hard actually.

aozm48
09-03-2012, 14:18
ok, can't find anything, i've now blocked all outbound traffic to destination port 0, both TCP and UDP, and I've unblocked the IP address that OVH blocked.

Something Neil has said is that all the servers that were detected doing this "attack" were all running windows 2008 r2 sp1, which i find a bit of a coincidence to say the least

Andy
09-03-2012, 13:44
Only people I know have hosting on my server. I don't sell hosting to the general public.

The fact that OVH confirmed it wasn't just my server that was doing this means it wasn't someone I host anyway, that much is clear.

3r1c
09-03-2012, 13:42
It looks like it was sending a packet to port 0 which triggered it.
Port 0 is not a valid port, so just block it on your firewall.

You probably have some vulnerable app or you sold hosting to a bad customer.
Its unlikely your malware scans will find anything.
Its very common that they run their code only in memory to leave no traces on the disk, or that the file is deleted from the disk when its run.

It would be more useful if OVH leave the server running but blocked so only the owner could access it.
Rebooting into FTP only mode makes it very difficult to find out what was going on in the server before it was shut down.

Andy
09-03-2012, 13:11
I have some strange incoming issues according to malware bytes but nothing outgoing.

They all do this though...

C:\Program Files (x86)\copSSH>tracert 46.17.100.243

Tracing route to sr5-1262.hostkey.ru [46.17.100.243]
over a maximum of 30 hops:

1 General failure.

Trace complete.

C:\Program Files (x86)\copSSH>tracert 213.186.127.3

Tracing route to 213.186.127.3.utel.net.ua [213.186.127.3]
over a maximum of 30 hops:

1 General failure.

Trace complete.

C:\Program Files (x86)\copSSH>tracert 46.17.96.43

Tracing route to sr5-42.hostkey.ru [46.17.96.43]
over a maximum of 30 hops:

1 General failure.

Andy
09-03-2012, 12:55
Malware Bytes has also just finished with nothing found.

So I can't see where the issue could possibly be coming from. Any ideas anyone?

aozm48
09-03-2012, 12:22
I've also even run a rootkit scan on the box, and can't find anything amiss.

Andy
09-03-2012, 12:21
Exactly. And every scan I've done so far of my server shows nothing out of the ordinary.

I've run:
- Symantec Antivirus,
- Spybot Search and Destroy
- MS Security Essentials (not finished yet)
- Malware Bytes

And they've returned nothing so far.

aozm48
09-03-2012, 12:19
the alledged attack in my case shows the same type of log entries as you posted, source is port 1 on my server, to port 0 on a remote server.
This is certainly looking like someone spoofing the IPs, given the sheer amount of IPs that are showing hitting the same destination IP and port.

Andy
09-03-2012, 12:16
Is it possible this is a "spoofed" attack? As in the IP's have been spoofed?

I also wonder if it's specifically a Windows issue and it's something in Windows causing this, not an actual malicious piece of software, as it was just confirmed to me it has affected Windows servers only.

aozm48
09-03-2012, 12:10
You aren't the only one Andy, I had the exact same thing happen around an hour ago to my ESX host, thankfully they only disabled 1 IP and not the entire server.
I've mailed in asking for full details on the alledged hack

Andy
09-03-2012, 11:54
So far nothing has been found - as I suspected.

Andy
09-03-2012, 11:31
Thank you for your response. I will look into the problem as soon as it's back online.

Abdurrahman
09-03-2012, 11:29
Hi Andy,

You should now be able to put the server back online, we've enabled the ability for you to switch back to booting from the HD. While our logs that we provide in the ticket aren't fully exhaustive, I can tell you that your server was used in multiple attacks (I will include more detailed information in a reply to your email).

In any case, we have to take these precautions, especially in cases of large attacks, and for the most part it's entirely automated. It does look like something was compromised, though I have no doubt you'll be able to source it out quickly.

Andy
09-03-2012, 10:58
I keep HTTP logs etc, but not firewall. They just get too large too quickly and I'm already close to the maximum HDD capacity as it is.

I can almost guarantee that there is no problem with my server and that's what I'm trying to get OVH to see. They still haven't replied here, by e-mail or on the ticket open in the manager.

yonatan
09-03-2012, 10:55
Quote Originally Posted by Andy
I don't keep firewall logs, I get too much traffic to do that. The logs would be tens of GB per day. The logs OVH sent don't show any issue with my server, it's a false positive so I'd like my server back up ASAP.

As I said this isn't the first time this has happened and turned out not to be my problem.
Not having logs is quite harsh when you get into bad situations like this one,
sounds like it's not from your end after-all , but maybe someone spoofed traffic using your IP.


As for logs, a good idea would be to host a syslogd on a remote box and transmit logs over on the fly.
( quite fun to install that if you up for experimenting some server configurations )

hope you get your server back as soon as possible.

Andy
09-03-2012, 10:50
I don't keep firewall logs, I get too much traffic to do that. The logs would be tens of GB per day. The logs OVH sent don't show any issue with my server, it's a false positive so I'd like my server back up ASAP.

As I said this isn't the first time this has happened and turned out not to be my problem.

yonatan
09-03-2012, 10:43
Quote Originally Posted by Andy
The point is I shouldn't have to do this. It was a single scan, which is not a sign of any hack problem.

I'm not actually scanning anything with my server, I know not to do that.
Yeah,generally scanning is not considered a legitimate activity at OVH, they give you access from the manager to their scanning service and SMS, but if you want to host your own probe & sms gateway, it should be done offsite.


I moved all my cacti operation to another location which keeps us informed in real time , without no robots to wake up to.

In your case Andy it looks like the same server was getting hit outside the network, i wonder how many more servers got suspended together with yours at the same time, this looks like a bug without more logs to shed some light.

during this time you can still access your data, so it could be a good time to check your logs, see if you have that remote IP is showing up in any recent log.

Andy
09-03-2012, 09:58
The point is I shouldn't have to do this. It was a single scan, which is not a sign of any hack problem.

I'm not actually scanning anything with my server, I know not to do that.

yonatan
09-03-2012, 09:42
Hate it when they got false positives on this darn robot..

I had this happen when i run a service test script which probes my all my servers every 5 minutes for ports 80 22 .

actually i was told that the way to get over this robot and allow legitimate port probes is to create a PTR to the main server hostname.
I guess your PTR is not set up, you might want to give that a try.

the quick way around this is just to format, which is quite stupid.

Andy
09-03-2012, 09:12
OK so it's happened again, my server is down due to a "hack" as OVH call it.

As usual this happened with no warning between 5am-6am this morning, and the server was immediately taken offline. I didn't know this until just 20 minutes ago and here I am now trying to find out what is going on.

This is not professional. This is an OVH.co.uk server and I would expect a lot more support and leeway that I have been given with this. What's more is looking at the log I've been sent, my servers IP is listed ONCE in the entire log.

We had to urgently disable your dedicated server
ns202921.ovh.net to block an attack. It seems that your
server has a security hole or a user
attacker has gained access.

Your server is currently in a mode that allows you
to access your data via FTP (access codes are
to be sent by email).
You can request the complete reinstallation of the system
through the manager.

If you want to take control of your server to
remove the source of SCAN, you must make contact
with our technicians.


--------------------------- LOGS OF SCAN
---------------------------

Unallowed traffic

startime endtime
scrort dstort
-----------------------------------------------------------
-----------------------------------
2012-03-09 05:14:11 2012-03-09 05:14:42
91.121.167.122:1 95.211.136.211:0
2012-03-09 05:14:07 2012-03-09 05:14:40
91.121.28.129:1 95.211.136.211:0
2012-03-09 05:14:07 2012-03-09 05:14:39
91.121.133.176:1 95.211.136.211:0
2012-03-09 05:14:03 2012-03-09 05:14:39
91.121.123.177:0 95.211.136.211:0
2012-03-09 05:14:14 2012-03-09 05:14:39
91.121.79.208:1 95.211.136.211:0
2012-03-09 05:14:15 2012-03-09 05:14:46
91.121.165.174:1 95.211.136.211:0
2012-03-09 05:14:08 2012-03-09 05:14:27
91.121.133.48:1 95.211.136.211:0
2012-03-09 05:14:11 2012-03-09 05:14:24
91.121.2.42:1 95.211.136.211:0
2012-03-09 05:14:08 2012-03-09 05:14:51
94.23.89.113:1 95.211.136.211:0
2012-03-09 05:14:20 2012-03-09 05:14:50
94.23.206.77:1 95.211.136.211:0
2012-03-09 05:14:07 2012-03-09 05:14:49
176.31.235.131:1 95.211.136.211:0
So explain to me, how is this a "hack"? This could have been someone running a script or pinging another server from a hosting account on my server. This is not proof of a hack at all, its a single scan.

Had this been a list of hundreds if not thousands I would understand, but it's not, so don't claim that this is proof of a hack. I can say with almost 100% certainty that there is nothing wrong with my server.

If you've got this far, thank you for actually reading. Now I'd like my server back online please. Once it's back I will run every possible search I can for the cause of the problem but I can't do that until it's back online.

OVH has to understand that we run MISSION CRITICAL services on these servers, you can't just take them down willy nilly because your automatic scans say it's got a problem. At least check it out yourself as well if you suspect there is a problem.

Better still contact the owner and give them a 12 hour window to do something about it. It was the middle of the night, I'm not going to be awake to know it's gone down. How fair is that?

It's not on OVH, please get this problem sorted once and for all. I'm not the only one to suffer this issue and it's not the first time for me either.