OVH Community, your new community space.

The attacks


oles@ovh.net
19-05-2011, 16:26
http://status.ovh.net/?do=details&id=1449

Following an ongoing attack on an IP, we
fine tuning the rules and we decreased the burst
authorized during an attack from 10000 to 8000.
The attack's passed from 70Mbps to 10Mbps. It goes on
but no longer has any impact on the server.

#sh inter f0/15 | i 30 sec
30 second input rate 2822000 bits/sec, 303 packets/sec
30 second output rate 62419000 bits/sec, 121785 packets/sec
[...]
#sh inter f0/15 | i 30 sec
30 second input rate 5422000 bits/sec, 585 packets/sec
30 second output rate 10334000 bits/sec, 20076 packets/sec

Do not hesitate if we trace the problems exist.

LawsHosting
18-05-2011, 18:38
I vote to make a rule for everyone to use Chrome on here, it translates text automatically :P (i.e solves double-posts)

#justsaying (with my tongue in my cheek!)

jonlewi5
18-05-2011, 11:06
ello,
Protections against attacks give very
good results. We had to intervene once
for several days, while we usually
must manage multiple attacks per day.

Example of an attack that started yesterday and 22H
continues. 4Gbps UDP to an IP at OVH.
http://demo.ovh.net/fr/ba3c2a2c8e7d3...c6dcf88ab240d/

Protections filter this attack and the only
evidence that there is an ongoing attack this graph.
Which is not bad

We have been other attacks on accommodation
shared this time. The infrastructure has not
held and there were 2 crashes 2 days ago
about. We removed temporarily AX
production (the traffic passing by the stage
done with the lowest ACE). Then we improved
these settings to avoid crashes.

In short, the attacks is the daily life of a
host and it is part of the trade. It is not
a war we win. It's just
repel the attack without the customers are
impacted. That's the challenge ...

Thank you for the information back if you see
fewer attacks, fewer problems, fewer things
"Weird" that can no longer, or if it's the same
and nothing has changed, or is it the worst and cata
total outrage and we want the skin ovh? Thank you
advance for the feedback!

Regards
Octave

oles@ovh.net
18-05-2011, 10:36
Hello,

Protections against attacks gives very good results. We had to intervene once in several days, while we usually must manage multiple attacks per day.

Example of an attack that started yesterday and 22 hours continues. 4Gbps UDP to an IP at OVH. http://demo.ovh.net/fr/ba3c2a2c8e7d3...6dcf88ab240d//

The protections filter this attack and the only evidence that there is an ongoing attack this graph. Which is not bad.

We have had other attacks on shared hosting this time. The infrastructure has not held and there were 2 crashes in two days. We removed temporarily AX production (the traffic passing by the stage done with the lowest ACE). Then we improved these settings to avoid crashes.

In short, the attacks is the daily life of a host and it is part of the trade. It is not a war we win. It's just repel the attack without the customers impacted. That's the challenge ...

Thank you for the feedback if you see fewer attacks, fewer problems, fewer things "weird" that can no longer, or if it's the same and nothing has changed, or is it the worst and total outrage and we want the skin ovh? Thank you in advance for the feedback!

Regards
Octave

yonatan
17-05-2011, 18:54
Quote Originally Posted by marks
Not sure I understand how/why you want us to provide this info. Could you explain it?

In any case, that's traffic analysis, getting into IP and TCP/UDP headers. I'm afraid that we don't have this info for each of our servers.

The best way to know this info is to analyse the traffic yourself with tcpdump-like applications.


The 50Mbps is per server.

On a server with an external firewall ,you can log into the firewall and see on which ports people are probing the server from what src and to what dst.

in case of an attack on a cretin port, it will be possible to see what service was the one who knocked down the machine , therefore refine the iptables rate limit/restrictions on that port alone.

currently i rate limit all ports equally and if someone knocks more then i would consider allowed - he is blocked automatically.
but sometimes this can limit a legit connection ...

to see on which ports this actions takes place can help better understand and configure the firewall on the server.

last time we had a DDOS it came from an amazon cloud server and we couldn't tell who was the src from the manager, only after asking the incident team on the phone , they gave us the src IP for the attack.

so if we are attacked, we need to wait for a person to lookup the src before we can file an abuse complaint to the src ISP .

if we had this information beforehand we could stop the attack faster.

marks
17-05-2011, 14:52
Quote Originally Posted by Myatu
So the mC's are able to use UDP again?
yeap. those are the protections in place now

Quote Originally Posted by Thelen
Marks, so you're saying each server can do maximum 50Mbps UDP traffic, regardless of source/destination or anything??
yes, that's exactly what it says. 50Mbps UDP per IP source (protection on the incoming traffic) and 50Mbps UDP per IP destination (protection on the outgoing traffic, to reduce the attacks originating at OVH)

Thelen
17-05-2011, 13:29
Marks, so you're saying each server can do maximum 50Mbps UDP traffic, regardless of source/destination or anything??

Myatu
16-05-2011, 19:07
Quote Originally Posted by oles@ovh.net
The VPS and mC have the following protections:
- 100Mbps per IP over TCP
- 5Mbps per IP over UDP
- 32Kbps per IP over ICMP
So the mC's are able to use UDP again?

marks
16-05-2011, 15:37
Quote Originally Posted by yonatan
This is great news!!
..., is it possible to get info about which ports/protocols are used and which ports are eating the bandwidth ( from the network level , not with software on the server ).

I am sure you guys at the OVH office can see this information ,so why not pass it to the client?
Not sure I understand how/why you want us to provide this info. Could you explain it?

In any case, that's traffic analysis, getting into IP and TCP/UDP headers. I'm afraid that we don't have this info for each of our servers.

The best way to know this info is to analyse the traffic yourself with tcpdump-like applications.

Quote Originally Posted by Thelen
So is this 50Mbps per server pair or what?
The 50Mbps is per server.

Thelen
16-05-2011, 14:24
So is this 50Mbps per server pair or what?

yonatan
15-05-2011, 10:26
This is great news!!

Note on this , .. this might be helpful :

Find a way to get an indication of which kind of traffic is going thought the server other then the MRTG in the manager, is it possible to get info about which ports/protocols are used and which ports are eating the bandwidth ( from the network level , not with software on the server ).

I am sure you guys at the OVH office can see this information ,so why not pass it to the client?

this information might be very valuable for some applications.

BoxSlots
15-05-2011, 00:18
Quote Originally Posted by marks
all servers should have now this protection. this should help to at least mitigate attacks like the one you had some weeks ago
Great to see the company trying to over come the bad days +kudos.

Andy
14-05-2011, 21:32
Quote Originally Posted by marks
all servers should have now this protection. this should help to at least mitigate attacks like the one you had some weeks ago
Thanks marks.

ExW
13-05-2011, 17:15
I have sent an email :P, hope i don't get charged

marks
13-05-2011, 16:59
Quote Originally Posted by Andy
Are EG servers protected in the same way for UDP?
all servers should have now this protection. this should help to at least mitigate attacks like the one you had some weeks ago

Quote Originally Posted by ExW
I have an EG server and everything on port 80 is dead slow since last night
do you want to drop us an email to the support? we'll have a look.

ExW
13-05-2011, 16:49
I have an EG server and everything on port 80 is dead slow since last night

Andy
13-05-2011, 16:29
Are EG servers protected in the same way for UDP?

LawsHosting
13-05-2011, 10:42
From 12am to 3am GMT, a server's connection in RBX3 was laggy, port 80 was fine, but other ports were timing out, slow or just dead - even SSH. Was confusing on my part.

Not sure if this was the reason (I saw vss-3-6k in the red, still is), but I've re-opened a ticket just in case.

oles@ovh.net
13-05-2011, 08:56
Hello,

Following the introduction of the protections against attacks
on the UDP layer, after 24h we haven't had to intervene
to protect the infrastructure. We received a 10th of the
usual attacks that did not have any affect on our
customers.

We can estimate that the settings are correct
and sufficient. Done fast, done well.

Yes! Let's hope it lasts.

The summary:

-we've set up protection on the entrance of
our network: we limit UDP traffic to 50Mbps by
IP source. i.e. a specific IP on the Internet
can not send to the OVH network more than 50Mbps
UDP.

-we have put in place protection on the data center
routers: we limit UDP traffic to 50Mbps to
IP destination. i.e. a specific IP at OVH
can not get more than 50Mbps UDP traffic from the Internet.

The summary of protections already in place (for the past 1-2 years):
- we have a restriction by IP source to 32Kbps
towards OVH on ICMP layer and TCP/SYN (with some exceptions).

The VPS and mC have the following protections:
- 100Mbps per IP over TCP
- 5Mbps per IP over UDP
- 32Kbps per IP over ICMP

There are no other limitations and we don't foresee
any more new ones.

We had a good welcome for putting in place these
protections. 1 client was not happy and we've received
plenty of feedback with a "uufff". I think these
protections create a good added value to our offers
because they strengthen the security services that
we offer to our customers. Whether it's a game server,
a website or a DSL connection, to receive a competitor's
DoS attack is very unpleasant. At OVH, you're
now protected against the mood of your competitors.

Regards
Octave



Amicalement
Octave

unclebob
12-05-2011, 18:52
Wow. That's pretty appalling. Is there definitely no other solution?

oles@ovh.net
12-05-2011, 15:12
http://status.ovh.net/?do=details&id=1449

we're going to activate the protections on the
routers in the datacentre:

vrack: done
HG 2010/2011: already done
pCC: done

oles@ovh.net
12-05-2011, 12:02
Hello,

At the gateway to the backbone, we have just changed the
configuration. We remove the filter on the whole
IP layer and we only keep the UDP.

Thus, any IP on the Internet is limited to 50Mbps UDP
towards the entire OVH network.

If you have problems, do let us know.
It's not because we have to manage an emergency that
can not refine it right away. It's always the same
email as usual if it's a matter of life or death oles@ovh.net

Early afternoon, we'll continue to refine
them to reach the final 3 new rules:

-limitation on UDP on source IP to OVH
currently limited to 50Mbps and we will try
to go down to 20Mbps around 14:00

-limitation on UDP on destination IP to OVH
currently implemented on the HG network
to 50Mbps. We do not yet know whether it's useful and
whether to configure it on all routers

-limitation on UDP on OVH source IP to the Internet
is not yet in place. The goal is to prevent an
OVH server sending an attack towards the Internet.

Regards
Octave

jonlewi5
12-05-2011, 10:17
Good evening,

Considering the amount of attacks that is being
be taken every day, we decided
dig up the hatchet It's possible.
Today alone, there is more than 30 attacks
and it makes 5 customer networks impacted with
temporary degradation of service.

Then:

A source IP (the Internet) can not send
OVH to the network, more than 50Mbps over the entire
IP layer. Ultimately we believe that applying the
UDP layer.

We've also added a limitation on the
network of HG on the destination IP in UDP
OVH out all IP to 50Mbps.

If you have problems please us
back on oles@ovh.net , noc@ovh.net

More:
http://travaux.ovh.net/?do=details&id=5443

Regards
Octave

oles@ovh.net
12-05-2011, 01:06
Good evening,

Considering the amount of attacks that we
are receiving every day, we decided to
unearth the battle axe We cannot allow it anymore.
Today alone, there are more than 30 attacks
and they've impacted 5 networks for our clients with
temporary degradation of the service.

Then:

A source IP (the Internet) cannot send
towards OVH network more than 50Mbps over the entire
IP layer. Ultimately we think to apply it only to the
UDP layer.

We've also added a limitation on the
HG network on the destination IP on UDP
from all the IPs to 50Mbps.

If you have problems please send an email
to oles@ovh.net noc@ovh.net

More:
http://status.ovh.net/?do=details&id=1449

Regards
Octave