How to set up Proxmox network in hybrid mode?

In this post I will be talking about how to configure some virtual machines behind Nat, and some using public failover IP. So in the end, you will have two subnetwork at proxmox. All the instructions took place at Ovh provider.

When you buy dedicated server at Ovh, You will get one public Ip address. From this Ip you can configure forwarding rules using iptables, so those devices will be available from Internet. You can also configure postrouting rules. They can make your machines will access internet. But network in Proxmox can be configured more complex, I mean we can add next failover Ip, we can create net subnet behind Nat etc.

I had the following problem. My 3 VM (database, Apache server and Windows Server) were behind Nat and they used local Ips from this class 10.0.3.0/24. They could seen each other, they could access Internet. Gateway to the Internet was Ovh's server Ip. It was working till time, when I got another task. I should have moved vm from other server to this server and the vm uses public ip too. So it was problem, because 2 services in Proxmox cannot use the same port (80, www) and be visible in Internet simultaneously.

There are a few steps, how to make it working.

1. Proxmox network configuration of dedicated server
2. Proxmox network configuration of VMs on hardware level
3. Network configuration of Vms inside Vm

First step is to configure network of the dedicated server and we will do this through editing /etc/network/interfaces file. We won't do this through Proxmox interfaces, because it's very poor and you won't achieve all the options we need. In this file we create to local interfaces vmbr0 (with public failover ip) and vmbr1 (local network). Then we connect them to gateway and we will configure iptables rules, so all of the machines will have internet access and some of them will be visible from Internet.

Below is network configuration of the dedicated server. To protect my server address, I put xxx in place of Ip octet. It's an address as you bought from Ovh hosting provider.

iface eno1 inet manual

 

auto vmbr0

iface vmbr0 inet static

    address 51.xx.xx.59

    broadcast 51.xx.xx.255

    netmask 255.255.255.0

    pointopoint 51.xx.xx.254

    gateway 51.xx.xx.254

    bridge_ports eno1

    bridge_stp off

    bridge_fd 0

 

auto vmbr1

iface vmbr1 inet static

    address 10.0.3.1

    netmask 255.255.255.0

    bridge_ports none

    bridge_stp off -s '10.0.3.0/24'

    bridge_fd 0

    post-up echo 1 > /proc/sys/net/ipv4/ip_forward

    post-up iptables -t nat -A POSTROUTING -s '10.0.3.0/24' -o vmbr0 -j MASQUERADE

    post-up iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 80 -j DNAT --to 10.0.3.8:80    

    post-up iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 443 -j DNAT --to 10.0.3.8:443

    post-up iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 3389 -j DNAT --to 10.0.3.16:3389    

Short explanation:  eno1 is a physical interface but must be set to manual, not static. Otherwise nat subnet or failover ip subnet won't work. vmbr0 interface has all the addresses as physical interface and it bridges eno1 interface. vmbr1 doesn't bridge any interface. It uses ip forward (you must enable it, because by default in unix systems it's disabled) ans masquerade. All the machines within vmbr1 network can see each other. If you want to grant access to the Internet to the machines, you must add prerouting rule for iptables. If you want to have vm's services available on Internet, you must add prerouting rules for iptables.

Now I will show you my Proxmox network interfaces and how Vms are configured.

 

Let's assume, Vm 101 is a Debian or Ubuntu system and it uses public failover ip. The net in this machine is configured via yaml file. So edit /etc/netplan/50-cloud-init.yaml file. 

network:
    ethernets:
        ens18:
            addresses:
            - 51.xx.yy.yy/32
            nameservers:
                addresses:
                - 8.8.8.8
                - 208.67.222.222
                search: []
            optional: true
            routes:
                - to: 0.0.0.0/0
                  via: 51.xx.xx.254
                  on-link: true
    version: 2

Be careful during configuraton, because gateway of the interface is main server ip with last octet 254. It's not bought failover ip with 254 ending! Note: 'yy' is bought failover ip.

When you edit the file, then type in terminal:

sudo netplan try - if there are some errors, you will get notices about it

sudo netplan apply - it applies changes

Next important thing. In order to use failover ip, you must set up its mac address. It can't be a random address! So, when you buy failover ip in Ovh panel, then click on 'Add virtual mac' and copy the mac to the virtual machine network interface.

Last, but not least, I will show you how network is configured on Ubuntu machine (behind nat, Vm 207) and Windows Server machine Vm 202.

network:
    ethernets:
        ens18:
            addresses:
            - 10.0.3.8/24
            gateway4: 10.0.3.1
            nameservers:
                addresses:
                - 208.67.222.222
                - 8.8.8.8
                search: []
                #            optional: true
                #            routes:
                #            - to: 0.0.0.0/0
                #              via: 10.0.3.1
                #              on-link: true
    version: 2
Note: 10.0.3.1 is a local gateway.

If you have any questions-post. I spent a few days on it because of false and uncomplete information on the web.