I thought I'd share a quick how-to on how to cobble together a Proxmox cluster, geared towards use with the new, inexpensive Kimsufi's.
Since you can no longer route failover/RIPE IPs to a Kimsufi, the idea here is to use NAT to share a public IP across multiple VMs (or simply run a VM without a public IP). NAT simply forwards a port, let's say 80 for HTTP traffic, to a specific VM.
As this little how-to uses a VPN to link the servers into a cluster, if you have more than one server, you can move VM's between servers and even use the public IP on server "A" for a VM that's on server "B".
Some notes: A "node" is a "server", and I may use these interchangeable. I also recommend you do this on fresh installations, so if something doesn't go to plan, it's not a disaster (ie., live VMs). Once you get the hang of it, then you can be more adventurous And the usual disclaimer: you're doing this at your own risk.
Step 1
Install Proxmox. That's a given, of course. You can select Proxmox from the turnkey Linux distributions from your OVH manager. The default partition will work fine for most. One thing to keep in mind, is that if you do choose your own partition layout, is that you want a separate mount point for "/var/lib/vz", preferably on LV and using ext3 (not ext4).
Step 2
The first thing you need to do is change the "vmbr1" bridge a little. The other entries in the
/etc/network/interfaces file can be left as-is. Particularly: DON'T EDIT
vmbr0!
You also need to think of what private IP block you would like to use, and assign each node an IP from within that private IP block. For example, I use the IP range 192.168.14.0/23 (which is 192.168.14.1-192.168.15.254 and a netmask of 255.255.254.0). 192.168.15.x I assign to the nodes (servers), whereas 192.168.14.x I assign to VMs. Using that IP range, you would change
/etc/network/interfaces as following:
Code:
# for Routing
auto vmbr1
iface vmbr1 inet static
address 192.168.15.20/23
bridge_ports dummy0
bridge_stp off
bridge_fd 0
So the main changes are to change "manual" from "static" in the "iface ..." line, and to add an IP address to the interface. You may have noticed that I've removed a line "post-up /etc/pve/kvm-networking.sh" -- for one, we don't need it, and two, it doesn't even exist.
You can force the changes using:
Code:
ifdown vmbr1 && ifup vmbr1
You will need to do this on each node/server, taking care to select a different IP address. Keep it simple, start at 192.168.15.1, and increment the last number for each new server.
Note: If you do not want a VPN / Cluster setup, but just want to run this on a single server, you can skip steps 3-6. Instead, make the two changes in the files outlined here: http://forum.ovh.co.uk/showpost.php?...0&postcount=36
Step 3
In this how-to, I use
Tinc to create the VPN. You could use something else, like OpenVPN or IPSEC. The former is a bit on the heavy side for the task, whilst the latter may not have all the features we need. Specifically, Tinc allows us to create an auto-meshing network, packet switching and use multicast. Multicast will be needed to create a Proxmox cluster, whilst the virtual switching ensures packets will eventually be routed to the right node/VM.
So on the node, install Tinc:
Code:
apt-get install tinc -y
Next, create a directory where the configuration for the VPN will reside (you can have multiple configurations as such):
Code:
mkdir -p /etc/tinc/vpn/hosts
Next, we create a basic configuration, which tells Tinc to use a "switch" mode and what this node's "name" is. For sake of simplicity, use the hostname for the "name":
Code:
cat > /etc/tinc/vpn/tinc.conf <
Then we create a server-specific configuration. Note that the filename is the same as specified in "Name =" above.
Code:
cat > /etc/tinc/vpn/hosts/ks12345 <
The above configuration turns of compression, as traffic on a Kimsufi is unlimited (and so is inter-OVH traffic for others). For the "Address =" field, ensure you specify the public IP address of the server.
Now we need to create a public/private key. The private key will remain exactly that: private. The public key will be appended to the file we just created (
/etc/tinc/vpn/hosts/ks12345), which will eventually be distributed to the other nodes.
Code:
tincd -n vpn -K4096
It will ask you to confirm two file locations. The default should be correct (with the last/2nd one the file as mentioned above).
Now we need an up/down script, to do some post configuration of the network when the VPN comes up (or goes away). This is a simple copy & paste:
Code:
cat > /etc/tinc/vpn/tinc-up < /proc/sys/net/ipv4/ip_forward
# To limit the chance of Corosync Totem re-transmission issues:
echo 0 > /sys/devices/virtual/net/vmbr1/bridge/multicast_snooping
EOF
cat > /etc/tinc/vpn/tinc-down < /proc/sys/net/ipv4/ip_forward
EOF
chmod +x /etc/tinc/vpn/tinc-up
chmod +x /etc/tinc/vpn/tinc-down
What the above does, is add the VPN tunnel to the vmbr1 bridge. Furthermore, it allows multicast messages over vmbr1. It also sets the use of masquerading, to allow a VM on a private IP to communicate successfully with the outside world - it will use the IP address of vmbr0 to do so.
Finally, you need to tell Tinc that the contents in the "vpn" sub-directory should be started whenever it starts:
Code:
echo "vpn" >> /etc/tinc/nets.boot
You will need to do this on each node for the VPN. In addition, the files within the directory
/etc/tinc/vpn/hosts/ needs to be distributed to all nodes (so that all nodes have the files from the other nodes). Its simple enough to script this, if you want to go that route, but that's beyond the scope here.
Note: As pointed out by szisti, when you will also need to add a
ConnectTo= line to the configuration, as outlined here:
http://tinc-vpn.org/documentation/ti...nnections-work -- This will tell the node to connect another node.
Restart Tinc:
Code:
service tinc restart
Test your network by pinging another node on its private IP, ie:
Code:
ping -c3 192.168.15.32
Note I use the "-c3" here, to limit the amount of pings. If the VPN was not configured correctly, or a firewall is interfering, you may otherwise end up with a large number of "Host or destination is unreachable" errors.
Step 4
We need to force Proxmox, or more specifically, Corosync to use the private IP addresses rather than the public IP address, as we cannot use multicast over OVH's network. This will be different if you have your own vRack, but given the focus is on Kimsufi servers here...
The easiest, but also the "dirtiest" method is to simply change the
/etc/hosts, which I will outline here.
The first step is to ensure that the
/etc/hosts file is read before attempting to do a full DNS lookup by Bind or DNSMasq (preferred). By default, this is not the case on OVH Proxmox installations, so we fix that with (copy & paste):
Code:
cat > /etc/host.conf <
Next edit the
/etc/hosts file, by commenting out the original line, and adding our own:
Code:
...
# Original:
#123.4.5.6 ks12345.kimsufi.com ks12345
# Ours:
192.168.15.20 ks12345.kimsufi.com ks12345
Make sure that the private IP address matches the one you assigned to vmbr1 (double check with
ifconfig vmbr1).
You can double check this with "nslookup", which should return the private IP address, not a public IP address. Ie.:
Code:
# nslookup ks12345
Server: 127.0.0.1
Address: 127.0.0.1#53
Name: ks12345.kimsufi.com
Address: 192.168.15.20
Or, if you do not use Bind / DNSMasq, you should use "getent hosts" to lookup up the IP, ie.:
Code:
# getent hosts ks12345
192.168.15.20 ks12345.kimsufi.com ks12345
And can be verified with a ping:
Code:
# ping -c1 ks12345
PING ks12345.kimsufi.com (192.168.15.20) 56(84) bytes of data.
...
If this is not the case, restart Bind or DNSMasq if it is installed (ie.,
service bind9 restart), and double check you've correctly edited the
/etc/host.conf as well as
/etc/hosts file.
At this stage, reboot the server to ensure the changes get picked up and everything works as expected (that is, your server comes back up online )
Step 5
If you do not yet have a cluster configured, you need to create one first. So pick a particular server that you would like to consider as a "main node" and perform the following:
Where
is something of your own choosing. Keep the name short and simple, without spaces or other funny characters.
The "main node" is a loose term really, as any node within the cluster can manage other nodes. But use it as a consistent starting point for adding nodes to the cluster.
You can check if things are working correctly with:
Code:
# pvecm status
...
Node name: ks12345
Node ID: 1
...
Node addresses: 192.168.15.20
In particular, you'd want to make sure that the "Node addresses:" portion is the private IP address as on vmbr1.
Step 6
Adding a node to the cluster will need a little preparation. Specifically, because we use private IP addresses for the cluster, we need to force other nodes to do the same when trying to contact another node. In other words, if ks12345 wants to contact ks6789, it should use the 192.x range instead of the public IP address.
So, based on the above example, on ks12345 we need to add a line to the
/etc/hosts like this:
Code:
cat >> /etc/hosts <
Note the double ">>" brackets. If you use a single ">" one, you overwrite the entire file with just that line. You've been warned.
And on ks6789, we need to make sure ks12345 is contacted using the private IP as well, so on that node, we perform:
Code:
cat >> /etc/hosts <
All of this can be made much fancier with your own DNS server and bindings, but again, this is beyond the scope and goes on the assumption you don't mind doing this for the 2, 5 or 10 servers or so you may have. If you have a few hundred, well...
On the node that you will be adding to the cluster, make sure that "nslookup" gives the correct internal IP address for the "main node", and that you can successfully ping that private IP address.
If tested OK, then still on that node (thus the one that isn't yet part of the cluster), use:
Where "ks12345" is the "main node" (the one on which you first created the cluster). It will ask you for the root password for SSH for ks12345, and then does its thing with configuration. So on that note, if you have disabled password-based root logins using SSH, you may have to temporarily enable it.
After this has been done, the node should automatically on your web-based GUI and can be verified from the CLI using:
If the nodes show up in the "pvecm nodes" command and GUI, then you have successfully created the cluster.
NOTE: A note about a 2-node cluster and quorum:
http://pve.proxmox.com/wiki/Proxmox_..._quorum_issues
Step 7
You can now create VMs that can be migrated between the nodes.
You can either assign the private IP address directly (venet, only on OpenVZ containers) or as a network device (veth) attached to "vmbr1". An important note regarding the latter, is to NOT use "vmbr0". Doing so will cause your server to be kicked from the OVH network (in other words, render it inaccessible).
The private IP address should be within the range of your specified netmask on vmbr1. So going by the above example of using 192.168.14.0/23, that's anything between 192.168.14.1 and 192.168.15.254. Make sure the IP isn't already used by another VM or a node (see initial notes, re 192.168.14.x for VMs).
If you fire up the VM, its private IP address should be ping-able from any node, and from within the VM, you can ping any private or public IP address. If this is not the case, the network configuration was not done correctly.
Final Step
You should now have at least one VM with a private IP address. Its good and well if this VM doesn't need to be accessed from the outside world, but if you want to give it such access, you will need to use NAT on the node. This will instruct the node that incoming traffic on a particular port will need to be forwarded to a particular VM.
For example, TCP port 25 on 123.4.5.6 is forwarded to VM on IP 192.168.14.1:
Code:
iptables -A FORWARD -p tcp -i vmbr0 -d 192.168.14.1 --dport 25 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 25 -j DNAT --to-destination 192.168.14.1:25
The first line allows us to forward to 192.168.14.1, whereas the 2nd line gives the specific instruction on how to forward. This can easily be changed for for UDP by replacing "-p tcp" with "-p udp" on both lines.
Final notes:
Again, this is just a simple setup to help you get started. More importantly, it doesn't include any basic security measures such as a firewall.
And as said before: you do this at your own risk. This was aimed for Kimsufi servers, something to play with, and may contain some errors (which I'll gladly fix if pointed out to them).
Originally Posted by GregorX