I recently bought a used HP Z840 workstation to use as a server for a Proxmox [1] virtualization environment. The first virtual machine (VM) I added was an Ubuntu Server 22.04 LTS instance with nothing on it but the Cockpit [2] management tool and the WireGuard [3] VPN solution. I planned to use WireGuard to connect to my home network from anywhere, so that I can back up and retrieve files as needed and manage the other devices in my home lab. WireGuard also gives me the ability to use those sketchy WiFi networks that you find at cafes and in malls with less worry about someone snooping on my traffic.

The Z840 has a total of seven network interface cards (NICs) installed: two on the motherboard and five more on two separate add-in cards. My second server with a backup WireGuard instance has 4 gigabit NICs in total. Figure 1 is a screenshot from NetBox that shows how everything is connected to my two switches and the ISP-supplied router for as much redundancy as I can get from a single home network connection.

Figure 1: Topology of my home network.

The Problem

On my B250m-based server, I had previously used one connection directly to the ISP's router and the other three to the single no-name switch, which is connected to the ISP router from one of its ports. All four of these connections are bonded with the balance-alb mode, as you can see in the netplan config file (Listing 1).

network:
  version: 2
  renderer: networkd
  ethernets:
    enp6s0:
      dhcp4: false
      dhcp6: false
    enp7s0:
      dhcp4: false
      dhcp6: false
    enp2s4f0:
      dhcp4: false
      dhcp6: false
    enp2s4f1:
      dhcp4: false
      dhcp6: false
  bonds:
    bond0:
      dhcp4: false
      dhcp6: false
      interfaces:
        - enp6s0
        - enp7s0
        - enp2s4f0
        - enp2s4f1
      addresses: [192.168.0.20/24]
      routes:
        - to: default
          via: 192.168.0.1
      nameservers:
        addresses: [8.8.8.8, 1.1.1.1, 8.8.4.4]
      parameters:
        mode: balance-alb
        mii-monitor-interval: 100

For those who are not familiar with the term, bonding (or teaming) is using multiple NIC interfaces to create one connection. The config file in Listing 1 is all that is needed to create a bond in Ubuntu. Since 2018 in version 18.04, Canonical has included netplan as the standard utility for configuring networks. Netplan is included in both server and desktop versions, and the nice thing about it is that it only requires editing a single YAML file for your entire configuration. Netplan was designed to be human-readable and easy to use, so (as shown in Listing 1) it makes sense when you look at it and can be directly modified and applied while running.

To change your network configuration, go to /etc/netplan, where you will see any yaml config file for your system. If you are running a typical Ubuntu Server 22.04 install, it will likely be named 00-installer-config.yaml. To change your config, you just need to edit this file using nano (Ubuntu Server) or gnome-text-editor (Ubuntu Desktop), save it, and run sudo netplan to apply the changes. If there are errors in your config, netplan will notify you upon running the apply command. Note that you will need to use spaces in this files (not tabs), and you will need to be consistent with the spacing.

In Listing 1, you can see that I have four NICs and all of them are set to false for DHCP4 and DHCP6. This ensures that the bond gets the IP address, not an individual NIC. Under the bonds section, I have made one interface called bond0 using all four NICs. I used a static IP address, and so I kept DHCP set to false for the bond also. Since I configured a static IP address, I also need to define the default gateway under the routes section, and I always define DHCP servers as a personal preference, though that part wouldn't be required for this config. The last section is where you define what type of bonding you would like to use, and I always choose to go with balance-alb or adaptive load balancing for transmit and receive, as it fits the homelab use case in my experience very well. See the box entitled "Bonding" for a summary of the available bonding options.

The best schema for bonding in your case might not be the best for me. With that in mind, I would recommend researching your particular use case to see what others have done. For most homelab use where utilization isn't constantly maxed out, I believe you will typically find that balance-alb is the best option.

Findings

What I discovered was that setting up Proxmox with a dedicated port for WireGuard and the remaining ports bonded for all other VMs actually resulted in slower and less consistent speeds for WireGuard than what I had been getting on my previous B250m-based machine with bonded NICs. This is something which I didn't expect, but in retrospect, perhaps I should have.

The initial plan for my new gear was to use one NIC for management only, one for WireGuard only, and the remaining 5 NICs for all of my other VMs on the Proxmox server. My expectation was that having a dedicated NIC used only for the WireGuard VPN would help me to realize faster speeds but also more consistent speeds because the VPN would be independent of my other VMs' network performance. Although that would mean no redundancy for WireGuard on that individual machine, I didn't care, because I now had two servers running. If my new server went down, I could simply connect to the old one.

After experimenting with the configuration, I eventually discovered it was better not to put the VPN on a separate NIC but to use a single port for management only and to team the other 6 NICs in my Proxmox server as that resulted in the best speed and consistency running WireGuard, regardless of the fact that all of my other VMs are using that same bond. Figure 2 shows the configuration. You will see 10 NICs in Figure 2, but three of them are not running. This is an oddity of some quad-port cards in Proxmox. Run the following command to reload the network interface configuration on an hourly basis:

ifreload -a

Figure 2: Proxmox network configuration.

This command ensures I get all six up and running, albeit with a "failure" each time I ifreload. (Note that it isn't actually a failure since those NICs don't actually exist. You might encounter this problem if you decide to use Proxmox with a bonded quad-port card.)

Results

Figures 3 and 4 show network speeds and ping times. You can see that by bonding the single NIC that was previously dedicated to WireGuard into a team with the other 5 NICs I was able to achieve better ping times and also better speeds. More importantly, the WireGuard speeds were very consistent. Across five runs, I only saw a variation of 0.05Mbps maximum with the six bonded NICs in Proxmox versus a variation of up to 0.45Mbps max in speed variance when using the dedicated NIC. With my previous four NIC B250M setup, the consistency was in the middle at about 0.34Mbps variance, but the speeds were about 0.2Mbps slower on average.

Figure 3: Comparing network speeds.

Figure 4: Comparing ping times.