We are in the process of migrating this forum. A new space will be available soon. We are sorry for the inconvenience.

Poor man's Proxmox cluster with NAT


Masterjoa
14-07-2016, 19:52
Hello,
Thanks for the tutorial. I followed all the steps and I can ping my node/containers between each other.
But can't connect to the internet from the containers (no problem from the node).
How can I fix this ?

EDIT:
fixed : the problem came from the server I was trying to connect to...

GregorX
26-05-2016, 11:56
Okay thx, but when I use for example drbd to copy the vmimages what do you do against fencing?

alvaroag
18-05-2016, 00:10
Quote Originally Posted by GregorX
Hi Myatu,
I'm not sure where in this tutorial the ha cluster syncs the files and how he start the vm on 2nd node when node 1 goes down?
or did I misunderstand something.
Greetings
Greg
Hi. Yo can check the information on the Proxmox Wiki: https://pve.proxmox.com/wiki/High_Av...ty_Cluster_4.x

GregorX
17-05-2016, 23:09
Hi Myatu,
I'm not sure where in this tutorial the ha cluster syncs the files and how he start the vm on 2nd node when node 1 goes down?
or did I misunderstand something.
Greetings
Greg

trapak
08-08-2014, 00:39
Hi! Thanks for the tutorial!!!

I followed all the instructions and I was able to create the cluster. I am able to ping B and C from node A, A from B and C, etc. (private IP addresses)

I am also able to migrate containers from a node to another one using the web GUI.

I have two problems, though:
1) I have created a container on A, and I can ping it using its private IP address, but not from B nor C.
2) nodes go offline for (apparently) no reason.

Any idea what the problem might be? I haven't done anything to enable/make fancing work. Might that be the cause?

Thanks!

Myatu
01-06-2014, 20:44
Quote Originally Posted by DigitalDaz
A long time ago and I may have documented it on here somewhere, I had a cloud vps with dediserve in London and I was allocating additional IPs there and they were then attached to virtual machines on a Proxmox Kimsufi box. I was using a vpn obviously but I had switched off encryption etc to achieve it. Everything was done hypervisor side, the vms were just configured with the dediserve public IP
You can get it setup automatically, indeed. IP forwarding and ARP proxy are two of the requirements for that, but would have to ensure it is combined with a correctly configured firewall, as you don't want the possibility of that leaking to the outside of your private network - one reason why OVH is so adamant about people fixing their ARP / netmasks.

DigitalDaz
22-05-2014, 18:22
Quote Originally Posted by Myatu
You can, but only for a specific port (or a range of ports).

On the remote server, simply setup Tinc (as described in the first post) and ensure it connects to the Proxmox node. You could then link 1.1.1.1 port 80 to say, 192.168.14.1 port 80 (or another port) - a small sample on how to do so is also given in the first post, and Google will likely give you more when you search for iptables port forwarding.

If you are talking about long distances, say from France to the USA, you may need to tweak Tinc a little - ie., try TCP instead of UDP - or increase its priority. Use mtr to check the quality of the link.
A long time ago and I may have documented it on here somewhere, I had a cloud vps with dediserve in London and I was allocating additional IPs there and they were then attached to virtual machines on a Proxmox Kimsufi box. I was using a vpn obviously but I had switched off encryption etc to achieve it. Everything was done hypervisor side, the vms were just configured with the dediserve public IP

Myatu
12-05-2014, 22:09
Quote Originally Posted by zhuanyi
Thanks so much. Another question, say if I have a spare VPS somewhere (in another DC) with like 4 external/public IPv4 address, say they are:

1.1.1.1
1.1.1.2
1.1.1.3
1.1.1.4

Would it be possible to forward all incoming traffic from one particular IP to an internal IP in my Proxmox box?

Say for example all traffic through 1.1.1.1 goes to 192.168.140.1,all traffic through 1.1.1.2 goes to 192.168.140.2, and so on.
You can, but only for a specific port (or a range of ports).

On the remote server, simply setup Tinc (as described in the first post) and ensure it connects to the Proxmox node. You could then link 1.1.1.1 port 80 to say, 192.168.14.1 port 80 (or another port) - a small sample on how to do so is also given in the first post, and Google will likely give you more when you search for iptables port forwarding.

If you are talking about long distances, say from France to the USA, you may need to tweak Tinc a little - ie., try TCP instead of UDP - or increase its priority. Use mtr to check the quality of the link.

zhuanyi
12-05-2014, 14:34
Quote Originally Posted by Myatu
You can, but running them simultaneously would result in unexpected results, as you've noticed.
Thanks so much. Another question, say if I have a spare VPS somewhere (in another DC) with like 4 external/public IPv4 address, say they are:

1.1.1.1
1.1.1.2
1.1.1.3
1.1.1.4

Would it be possible to forward all incoming traffic from one particular IP to an internal IP in my Proxmox box?

Say for example all traffic through 1.1.1.1 goes to 192.168.140.1,all traffic through 1.1.1.2 goes to 192.168.140.2, and so on.

Myatu
11-05-2014, 16:12
Quote Originally Posted by zhuanyi
Is it possible to over-deploy the resources in Proxmox? I tried to over-assign the RAM and CPU resources and Proxmox just shut down one of my VMs whenever I have the other one booted up.
You can, but running them simultaneously would result in unexpected results, as you've noticed.

zhuanyi
11-05-2014, 15:57
Quote Originally Posted by Myatu
Post #36 gives the details on how to ensure a VM can connect to the internet from a private IP address.



This is because of the netmask (/23). A handy netmask calculator can be found at:

http://jodies.de/ipcalc?host=192.168...ask1=23&mask2=

It'll show the min (192.168.14.1) / max (192.168.15.254) IP addresses, broadcast address and other stuff that can be useful with configuring your network.

Thanks so much, that makes a lot more sense now. I think I just got my RDP sessions to work! Now I have 2 Windows VPS running, thanks so much!

Is it possible to over-deploy the resources in Proxmox? I tried to over-assign the RAM and CPU resources and Proxmox just shut down one of my VMs whenever I have the other one booted up.

Myatu
11-05-2014, 15:09
Quote Originally Posted by zhuanyi
Small question though: if I only have one physical node but would like to deploy several VMs on it (each one runs independently, could be Linux or Windows) , how should I configure the gateway so that my VM instance would be connected to internet (outbound)?
Post #36 gives the details on how to ensure a VM can connect to the internet from a private IP address.

Right now I am using address 192.168.15.20/23 as my vmbr1 address, skipped steps 3-6 I am not too sure how I could use 192.168.14.x as my the IP of my VM, does the 14.x somehow connected to 15.x? And what should be the gateway of the server? 192.168.14.1 or 192.168.15.1 or 192.168.15.20?
This is because of the netmask (/23). A handy netmask calculator can be found at:

http://jodies.de/ipcalc?host=192.168...ask1=23&mask2=

It'll show the min (192.168.14.1) / max (192.168.15.254) IP addresses, broadcast address and other stuff that can be useful with configuring your network.

zhuanyi
09-05-2014, 20:25
@Myatu Thanks for the super-useful tutorial, really appreciate it.

Small question though: if I only have one physical node but would like to deploy several VMs on it (each one runs independently, could be Linux or Windows) , how should I configure the gateway so that my VM instance would be connected to internet (outbound)?

Right now I am using address 192.168.15.20/23 as my vmbr1 address, skipped steps 3-6 I am not too sure how I could use 192.168.14.x as my the IP of my VM, does the 14.x somehow connected to 15.x? And what should be the gateway of the server? 192.168.14.1 or 192.168.15.1 or 192.168.15.20?

Thanks so much!

benarcher
08-09-2013, 22:09
Post error.

Myatu
08-09-2013, 18:42
If you followed the instructions to a tee with regards to the IPs, then you can use an IP in 192.168.14.* or 192.168.15.* and use a netmask of "255.255.254.0" for a VM. The gateway is the IP address you have assigned to vmbr1.

You can use OVH's DNS servers, your own local one (preferred) or any other public DNS.

And I'm glad you brought this up actually, because if you do not setup a VPN/Cluster (steps 3-6), then you will need to do two things still on the host, which is:

Code:
/sbin/iptables -t nat -A POSTROUTING -o vmbr0 -j MASQUERADE
echo 1 > /proc/sys/net/ipv4/ip_forward
This bit allows a VM to use the Internet.

Note that the above two commands won't survive a reboot, so you'll need to edit two files if you want this done automatically for you. The first is /etc/sysctl.conf and uncomment the line (by removing the # sign):

Code:
net.ipv4.ip_forward=1
And edit the vmbr0 stanza in /etc/network/interfaces by adding a line to the existing stuff:

Code:
# vmbr0: Bridging. Make sure to use only MAC adresses that were assigned to you.
auto vmbr0
iface vmbr0 inet static
   (...existing stuff that you do NOT change...)
   post-up /sbin/iptables -t nat -A POSTROUTING -o vmbr0 -j MASQUERADE
Its important you do not change anything else in the vmbr0 stanza, and keep the spacing/tab in front of the last line that starts with "post-up".

Delvey
08-09-2013, 18:05
Thanks.
So select vmbr1, then in Windows assign gateway as 192.168.15.*, assign IP address of VM as say 192.168.15.8 for example and then use OVH dns of 213.186.33.99 and net mask of 255.255.0.0

Myatu
08-09-2013, 17:51
Quote Originally Posted by Delvey
Excellent! Still learning, read earlier that steps 3-7 do not apply if your only using one Vm, still correct?
Currently setting up now so might be back with more questions
Steps 3-6 actually, the 7th step explains a bit more about assigning the IP address.

Quote Originally Posted by Delvey
Setting up VM now. Should I use NAT mode under network or bridged mode with vmbr1?
If you are using a VM (full virtualization), you probably want to stick to using bridged with vmbr1. You then need to give the OS inside the VM an IP address manually (you use the IP of vmbr1 as the "gateway").

Delvey
08-09-2013, 17:41
Setting up VM now. Should I use NAT mode under network or bridged mode with vmbr1?

Delvey
08-09-2013, 16:55
Excellent! Still learning, read earlier that steps 3-7 do not apply if your only using one Vm, still correct?
Currently setting up now so might be back with more questions

Myatu
08-09-2013, 16:49
You need to edit that file, not call it as an executable (which it isn't, hence the error).

Use:

Code:
nano /etc/network/interfaces
and exit using CTRL+X (and confirming you wish to save the changes).

Delvey
08-09-2013, 16:45
Little problem if anyone can help
Lock into server as root via Putty, but when entering /etc/network/interfaces i get permission denied. Any ideas why?

DigitalDaz
02-09-2013, 09:24
Quote Originally Posted by rickyday
Oh I know that, I am the biggest Hyper-V fan on these forums I suspect (that reminds me I really should post on the thread about remote management problems, know the answer to that one) but its great just to test these things

Hardware Virtualisation most definitely is the way to go of course
You just have to much time on your hands to play!

rickyday
01-09-2013, 23:04
Quote Originally Posted by LinuxGam
Virtualization without VTx is really not the way to go.
Oh I know that, I am the biggest Hyper-V fan on these forums I suspect (that reminds me I really should post on the thread about remote management problems, know the answer to that one) but its great just to test these things

Hardware Virtualisation most definitely is the way to go of course

DigitalDaz
01-09-2013, 21:50
I just used to notice a much higher CPU load when running the two together.

Myatu
01-09-2013, 15:34
Quote Originally Posted by DigitalDaz
Myatu: Have you experience of mixing KVM and OVZ on the same box? I'm sure I had pretty bad performance.
I've had no issues with that. Anything in particular that didn't perform well?

LinuxGam
01-09-2013, 15:15
Quote Originally Posted by rickyday
I have 4G and being the geek I am its great to tinker and test things!

Generally interested to see how a VM would run without no hardware virtualisation! Its great the option even exists!
I know what you mean, I pay for stuff often just to try things out or play. My i7 laptop makes a great server for that and after 12 months or if I need to travel I still have the laptop.

Virtualization without VTx is really not the way to go.

xreyuk
01-09-2013, 12:59
Quote Originally Posted by Myatu
I'd say to keep 1 GB of RAM aside on the 16G server. You can probably get away with less than that, particularly if you don't use Corosync for a cluster (which, incidentally, has a memory leak), but I'd err on the safe side especially on a server that has gobs of memory. The node should already have its own partition for disk storage in the default setup, but if you use chose to use your own partitioning, give it at least 20GB of disk space to use.
Thanks, I didn't see this before, really appreciate it.

rickyday
01-09-2013, 11:30
Quote Originally Posted by LinuxGam
I don't understand why you would be running VM's with a server that doesn't support VTx. If the server you are getting is that low power, just run Linux or Windows on it.

If you want to experiment, just use/buy a old laptop. I can see that the 2G is obv cheaper than buying most old laptops, but is seriously under powered to be running a Windows KVM on.
I have 4G and being the geek I am its great to tinker and test things!

Generally interested to see how a VM would run without no hardware virtualisation! Its great the option even exists!

LinuxGam
01-09-2013, 08:09
Quote Originally Posted by rickyday
Is there any Linux virtualisation distro (one that doesn't require VTx) that will support Windows operating systems?

I believe with no VTx the host requires a Para virtualisation guest OS, is this correct?
I don't understand why you would be running VM's with a server that doesn't support VTx. If the server you are getting is that low power, just run Linux or Windows on it.

If you want to experiment, just use/buy a old laptop. I can see that the 2G is obv cheaper than buying most old laptops, but is seriously under powered to be running a Windows KVM on.

DigitalDaz
31-08-2013, 22:52
Myatu: Have you experience of mixing KVM and OVZ on the same box? I'm sure I had pretty bad performance.

Myatu
24-08-2013, 20:57
Quote Originally Posted by rickyday
Is there any Linux virtualisation distro (one that doesn't require VTx) that will support Windows operating systems?

I believe with no VTx the host requires a Para virtualisation guest OS, is this correct?
KVM/Qemu can be run without VTx, but that will obviously be slower. As Proxmox uses KVM, you can do this with Proxmox, but you'll need to adjust a setting first:

After creating the "VM" (not "CT"), select the VM from the left column, select "Options", double-click on the line that sayd "KVM hardware virtualization" and remove the tick from the "Enabled" box and save. You can now start the KVM-based VM without VTx.

rickyday
24-08-2013, 20:22
Quote Originally Posted by Myatu
Though Proxmox will run on a KS 4G, it doesn't have nearly the CPU power as the next level up, nor does it have VTx. But you could still run 2, 3 OpenVZ VMs that don't require much CPU power.
Is there any Linux virtualisation distro (one that doesn't require VTx) that will support Windows operating systems?

I believe with no VTx the host requires a Para virtualisation guest OS, is this correct?

Myatu
24-08-2013, 18:57
I'd say to keep 1 GB of RAM aside on the 16G server. You can probably get away with less than that, particularly if you don't use Corosync for a cluster (which, incidentally, has a memory leak), but I'd err on the safe side especially on a server that has gobs of memory. The node should already have its own partition for disk storage in the default setup, but if you use chose to use your own partitioning, give it at least 20GB of disk space to use.

xreyuk
24-08-2013, 18:06
Thanks for that, really appreciate your reply.

I'll be getting the 16G when they become available anyway so that's no problem.

On a 16G, how much resource should I keep for the main node?

Thanks again

Myatu
24-08-2013, 15:51
Quote Originally Posted by xreyuk
How many VMs will this enable me to have?

How do I setup this if I just have one KS server, by which I mean, which steps should I miss out?
If you just want it on a single server, you can leave out step 3 through 6 really. It's fine if you still do them, and then you'll be ready for any future additions, but won't be necessary otherwise.

The amount of VMs is limited by the amount of resources you have available. Just keep in mind that the host node will need some RAM and CPU for itself, so you can't allocate the full 16 GB of memory to VMs (in case of a KS 16G).

Though Proxmox will run on a KS 4G, it doesn't have nearly the CPU power as the next level up, nor does it have VTx. But you could still run 2, 3 OpenVZ VMs that don't require much CPU power.

One more thing to keep in mind, is that the VMs run on a private IP address. The final step simply forwards a port on the public IP address to one of the VMs. There are ways of sharing the same port across multiple VMs on a private IP as well. But you can't assign a public IP address to a VM directly anymore (which requires a failover IP or RIPE IP address block, neither of which are available on the Kimsufi range now). The VMs will have unhindered access to the Internet though. Think of the node acting as a router, whilst the VMs are are just computers on a private network connected to that router.

xreyuk
24-08-2013, 01:31
Big thank you for this.

I was nearly thinking of not buying a server from Kimsufi, but this will be a big help!

Just a couple of questions if you wouldn't mind though?

How many VMs will this enable me to have?

How do I setup this if I just have one KS server, by which I mean, which steps should I miss out?

szisti
23-08-2013, 22:07
Quote Originally Posted by Myatu
D'oh! Yes it is. Or else it won't do much! Thanks for pointing that out.
Took me half an hour on google to work out why it's not working

Myatu
23-08-2013, 20:03
Quote Originally Posted by szisti
I think this is missing from the tinc.conf
D'oh! Yes it is. Or else it won't do much! Thanks for pointing that out.

szisti
23-08-2013, 18:56
Myatu:

I think this is missing from the tinc.conf

ConnectTo = name
Specifies which other tinc daemon to connect to on startup. Multiple ConnectTo variables may be specified, in which case outgoing connections to each specified tinc daemon are made. The names should be known to this tinc daemon (i.e., there should be a host configuration file for the name on the ConnectTo line).

If you don't specify a host with ConnectTo, tinc won't try to connect to other daemons at all, and will instead just listen for incoming connections.

cartwright118
22-08-2013, 00:33
Thank you for such an in depth tutorial, as usual!

This is certainly going to be of use!


Myatu
21-08-2013, 22:40
No problem

DigitalDaz
21-08-2013, 12:36
Myatu, thanks, it was a no brainer really that you would produce something slick like this

Jasgriff
20-08-2013, 20:54
Myatu,

Thanks a lot for taking the time to write this

Myatu
20-08-2013, 20:24
Quote Originally Posted by szisti
Myatu, when you have some time would you mind helping me with something bit more expensive?
My goal would be to have 2 webservers always on with UK IP addresses
2xmSP will be ordered in October when this all clears up

what I don't fully understand (and was mentioned in other topic also) how to use the Failover IP addresses for this, and how to get the network settings automatically updated when the VM is migrated from one host to another
It can be done in various ways. The thing with a failover IP is that it needs to be told to point to a particular server - it won't happen automatically. This can either manually through the OVH manager, or through a script using OVH's SOAPi (see http://www.ovh.com/soapi/en/?method=...FailoverUpdate).

You could use Proxmox in a cluster, and let it manage a HA VM/CT, and the VM itself will then have to manage the IP. Or you could do this "bare metal", without the help of something like Proxmox, through the use of, say hearbeat and DRBD. For the latter I've written a simple OCF resource (script) for use with OVH failover IPs, which I'll probably chuck onto here someday.

But if you need some help, I'll be happy to do so where I can. You might want to re-raise the question in the forum once you're ready, as surely there are many others here that are knowledgeable in this area and are willing to lend a hand, too.

szisti
20-08-2013, 18:57
Myatu, when you have some time would you mind helping me with something bit more expensive?
My goal would be to have 2 webservers always on with UK IP addresses
2xmSP will be ordered in October when this all clears up

what I don't fully understand (and was mentioned in other topic also) how to use the Failover IP addresses for this, and how to get the network settings automatically updated when the VM is migrated from one host to another

Myatu
20-08-2013, 16:28
Quote Originally Posted by rickyday
You are an early bird aren't you? or do you never sleep? 05:17
I was up all night. Long weekend

wii89
20-08-2013, 16:14
Thanks for this, Very useful

szisti
20-08-2013, 12:36
Thank you for this

rickyday
20-08-2013, 11:27
Excellent post Myatu.

You are an early bird aren't you? or do you never sleep? 05:17

Myatu
20-08-2013, 06:17
I thought I'd share a quick how-to on how to cobble together a Proxmox cluster, geared towards use with the new, inexpensive Kimsufi's.

Since you can no longer route failover/RIPE IPs to a Kimsufi, the idea here is to use NAT to share a public IP across multiple VMs (or simply run a VM without a public IP). NAT simply forwards a port, let's say 80 for HTTP traffic, to a specific VM.

As this little how-to uses a VPN to link the servers into a cluster, if you have more than one server, you can move VM's between servers and even use the public IP on server "A" for a VM that's on server "B".

Some notes: A "node" is a "server", and I may use these interchangeable. I also recommend you do this on fresh installations, so if something doesn't go to plan, it's not a disaster (ie., live VMs). Once you get the hang of it, then you can be more adventurous And the usual disclaimer: you're doing this at your own risk.

Step 1

Install Proxmox. That's a given, of course. You can select Proxmox from the turnkey Linux distributions from your OVH manager. The default partition will work fine for most. One thing to keep in mind, is that if you do choose your own partition layout, is that you want a separate mount point for "/var/lib/vz", preferably on LV and using ext3 (not ext4).

Step 2

The first thing you need to do is change the "vmbr1" bridge a little. The other entries in the /etc/network/interfaces file can be left as-is. Particularly: DON'T EDIT vmbr0!

You also need to think of what private IP block you would like to use, and assign each node an IP from within that private IP block. For example, I use the IP range 192.168.14.0/23 (which is 192.168.14.1-192.168.15.254 and a netmask of 255.255.254.0). 192.168.15.x I assign to the nodes (servers), whereas 192.168.14.x I assign to VMs. Using that IP range, you would change /etc/network/interfaces as following:

Code:
# for Routing
auto vmbr1
iface vmbr1 inet static
    address 192.168.15.20/23
    bridge_ports dummy0
    bridge_stp off
    bridge_fd 0
So the main changes are to change "manual" from "static" in the "iface ..." line, and to add an IP address to the interface. You may have noticed that I've removed a line "post-up /etc/pve/kvm-networking.sh" -- for one, we don't need it, and two, it doesn't even exist.

You can force the changes using:

Code:
ifdown vmbr1 && ifup vmbr1
You will need to do this on each node/server, taking care to select a different IP address. Keep it simple, start at 192.168.15.1, and increment the last number for each new server.

Note: If you do not want a VPN / Cluster setup, but just want to run this on a single server, you can skip steps 3-6. Instead, make the two changes in the files outlined here: http://forum.ovh.co.uk/showpost.php?...0&postcount=36

Step 3

In this how-to, I use Tinc to create the VPN. You could use something else, like OpenVPN or IPSEC. The former is a bit on the heavy side for the task, whilst the latter may not have all the features we need. Specifically, Tinc allows us to create an auto-meshing network, packet switching and use multicast. Multicast will be needed to create a Proxmox cluster, whilst the virtual switching ensures packets will eventually be routed to the right node/VM.

So on the node, install Tinc:

Code:
apt-get install tinc -y
Next, create a directory where the configuration for the VPN will reside (you can have multiple configurations as such):

Code:
mkdir -p /etc/tinc/vpn/hosts
Next, we create a basic configuration, which tells Tinc to use a "switch" mode and what this node's "name" is. For sake of simplicity, use the hostname for the "name":

Code:
cat > /etc/tinc/vpn/tinc.conf <
Then we create a server-specific configuration. Note that the filename is the same as specified in "Name =" above.

Code:
cat > /etc/tinc/vpn/hosts/ks12345 <
The above configuration turns of compression, as traffic on a Kimsufi is unlimited (and so is inter-OVH traffic for others). For the "Address =" field, ensure you specify the public IP address of the server.

Now we need to create a public/private key. The private key will remain exactly that: private. The public key will be appended to the file we just created (/etc/tinc/vpn/hosts/ks12345), which will eventually be distributed to the other nodes.

Code:
tincd -n vpn -K4096
It will ask you to confirm two file locations. The default should be correct (with the last/2nd one the file as mentioned above).

Now we need an up/down script, to do some post configuration of the network when the VPN comes up (or goes away). This is a simple copy & paste:

Code:
cat > /etc/tinc/vpn/tinc-up < /proc/sys/net/ipv4/ip_forward

# To limit the chance of Corosync Totem re-transmission issues:
echo 0 > /sys/devices/virtual/net/vmbr1/bridge/multicast_snooping
EOF

cat > /etc/tinc/vpn/tinc-down < /proc/sys/net/ipv4/ip_forward
EOF

chmod +x /etc/tinc/vpn/tinc-up
chmod +x /etc/tinc/vpn/tinc-down
What the above does, is add the VPN tunnel to the vmbr1 bridge. Furthermore, it allows multicast messages over vmbr1. It also sets the use of masquerading, to allow a VM on a private IP to communicate successfully with the outside world - it will use the IP address of vmbr0 to do so.

Finally, you need to tell Tinc that the contents in the "vpn" sub-directory should be started whenever it starts:

Code:
echo "vpn" >> /etc/tinc/nets.boot
You will need to do this on each node for the VPN. In addition, the files within the directory /etc/tinc/vpn/hosts/ needs to be distributed to all nodes (so that all nodes have the files from the other nodes). Its simple enough to script this, if you want to go that route, but that's beyond the scope here.

Note: As pointed out by szisti, when you will also need to add a ConnectTo= line to the configuration, as outlined here: http://tinc-vpn.org/documentation/ti...nnections-work -- This will tell the node to connect another node.

Restart Tinc:

Code:
service tinc restart
Test your network by pinging another node on its private IP, ie:

Code:
ping -c3 192.168.15.32
Note I use the "-c3" here, to limit the amount of pings. If the VPN was not configured correctly, or a firewall is interfering, you may otherwise end up with a large number of "Host or destination is unreachable" errors.

Step 4

We need to force Proxmox, or more specifically, Corosync to use the private IP addresses rather than the public IP address, as we cannot use multicast over OVH's network. This will be different if you have your own vRack, but given the focus is on Kimsufi servers here...

The easiest, but also the "dirtiest" method is to simply change the /etc/hosts, which I will outline here.

The first step is to ensure that the /etc/hosts file is read before attempting to do a full DNS lookup by Bind or DNSMasq (preferred). By default, this is not the case on OVH Proxmox installations, so we fix that with (copy & paste):

Code:
cat > /etc/host.conf <
Next edit the /etc/hosts file, by commenting out the original line, and adding our own:

Code:
...
# Original:
#123.4.5.6 ks12345.kimsufi.com ks12345

# Ours:
192.168.15.20 ks12345.kimsufi.com ks12345
Make sure that the private IP address matches the one you assigned to vmbr1 (double check with ifconfig vmbr1).

You can double check this with "nslookup", which should return the private IP address, not a public IP address. Ie.:

Code:
# nslookup ks12345
Server:		127.0.0.1
Address:	127.0.0.1#53

Name:	ks12345.kimsufi.com
Address: 192.168.15.20
Or, if you do not use Bind / DNSMasq, you should use "getent hosts" to lookup up the IP, ie.:

Code:
# getent hosts ks12345
192.168.15.20   ks12345.kimsufi.com ks12345
And can be verified with a ping:

Code:
# ping -c1 ks12345
PING ks12345.kimsufi.com (192.168.15.20) 56(84) bytes of data.
...
If this is not the case, restart Bind or DNSMasq if it is installed (ie., service bind9 restart), and double check you've correctly edited the /etc/host.conf as well as /etc/hosts file.

At this stage, reboot the server to ensure the changes get picked up and everything works as expected (that is, your server comes back up online )

Step 5

If you do not yet have a cluster configured, you need to create one first. So pick a particular server that you would like to consider as a "main node" and perform the following:

Code:
pvecm create 
Where is something of your own choosing. Keep the name short and simple, without spaces or other funny characters.

The "main node" is a loose term really, as any node within the cluster can manage other nodes. But use it as a consistent starting point for adding nodes to the cluster.

You can check if things are working correctly with:

Code:
# pvecm status
...
Node name: ks12345
Node ID: 1
...
Node addresses: 192.168.15.20
In particular, you'd want to make sure that the "Node addresses:" portion is the private IP address as on vmbr1.

Step 6

Adding a node to the cluster will need a little preparation. Specifically, because we use private IP addresses for the cluster, we need to force other nodes to do the same when trying to contact another node. In other words, if ks12345 wants to contact ks6789, it should use the 192.x range instead of the public IP address.

So, based on the above example, on ks12345 we need to add a line to the /etc/hosts like this:

Code:
cat >> /etc/hosts <
Note the double ">>" brackets. If you use a single ">" one, you overwrite the entire file with just that line. You've been warned.

And on ks6789, we need to make sure ks12345 is contacted using the private IP as well, so on that node, we perform:

Code:
cat >> /etc/hosts <
All of this can be made much fancier with your own DNS server and bindings, but again, this is beyond the scope and goes on the assumption you don't mind doing this for the 2, 5 or 10 servers or so you may have. If you have a few hundred, well...

On the node that you will be adding to the cluster, make sure that "nslookup" gives the correct internal IP address for the "main node", and that you can successfully ping that private IP address.

If tested OK, then still on that node (thus the one that isn't yet part of the cluster), use:

Code:
pvecm add ks12345
Where "ks12345" is the "main node" (the one on which you first created the cluster). It will ask you for the root password for SSH for ks12345, and then does its thing with configuration. So on that note, if you have disabled password-based root logins using SSH, you may have to temporarily enable it.

After this has been done, the node should automatically on your web-based GUI and can be verified from the CLI using:

Code:
pvecm nodes
If the nodes show up in the "pvecm nodes" command and GUI, then you have successfully created the cluster.

NOTE: A note about a 2-node cluster and quorum: http://pve.proxmox.com/wiki/Proxmox_..._quorum_issues

Step 7

You can now create VMs that can be migrated between the nodes.

You can either assign the private IP address directly (venet, only on OpenVZ containers) or as a network device (veth) attached to "vmbr1". An important note regarding the latter, is to NOT use "vmbr0". Doing so will cause your server to be kicked from the OVH network (in other words, render it inaccessible).

The private IP address should be within the range of your specified netmask on vmbr1. So going by the above example of using 192.168.14.0/23, that's anything between 192.168.14.1 and 192.168.15.254. Make sure the IP isn't already used by another VM or a node (see initial notes, re 192.168.14.x for VMs).

If you fire up the VM, its private IP address should be ping-able from any node, and from within the VM, you can ping any private or public IP address. If this is not the case, the network configuration was not done correctly.

Final Step

You should now have at least one VM with a private IP address. Its good and well if this VM doesn't need to be accessed from the outside world, but if you want to give it such access, you will need to use NAT on the node. This will instruct the node that incoming traffic on a particular port will need to be forwarded to a particular VM.

For example, TCP port 25 on 123.4.5.6 is forwarded to VM on IP 192.168.14.1:

Code:
iptables -A FORWARD -p tcp -i vmbr0 -d 192.168.14.1 --dport 25 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 25 -j DNAT --to-destination 192.168.14.1:25
The first line allows us to forward to 192.168.14.1, whereas the 2nd line gives the specific instruction on how to forward. This can easily be changed for for UDP by replacing "-p tcp" with "-p udp" on both lines.

Final notes:

Again, this is just a simple setup to help you get started. More importantly, it doesn't include any basic security measures such as a firewall.

And as said before: you do this at your own risk. This was aimed for Kimsufi servers, something to play with, and may contain some errors (which I'll gladly fix if pointed out to them).