Install RHEL 9 on Hetzner Cloud

Introduction

Hetzner Cloud is a really good "small" Cloud with particularly good value for money. While it does not have hundreds of services like AWS or Azure, but it has most of the basic stuff. Hetzner cloud has built-in support for Red Hat Enterprise Linux (RHEL) clones such as Rocky Linux 9. However, you cannot spin up real RHEL 9 instances without spending some time on it. This article will give you all the details you need to install RHEL 9 on Hetzner Cloud. We do that in a way that you can reuse your RHEL 9 installation to spin up multiple servers from one snapshot. This way you don't have to run the RHEL installer for every RHEL server you want to spin up.

Step 1: Get the RHEL installation ISO image in Hetzner Cloud

The first step is to get the RHEL installation image to Hetzner Cloud. This consists of three steps:

1. Download the RHEL installation ISO
2. Make the ISO image publicly available with http(s) and curl somewhere
3. Test that it actually works with curl
4. Create a Hetzner support request, ask them to add the ISO image and specify the URL.
5. Wait for the ISO image to appear in Hetzner Cloud

Step 2: Install RHEL 9 on Hetzner Cloud (with some trickery)

The sacrificial lamb

Installing RHEL 9 on Hetzner Cloud is a manual process. The first thing to do is to launch what I call a "sacrificial lamb virtual machine". It has only a single purpose in life: to allow you to run the RHEL installation and to get killed in the process. You could probably use any base image you want, but I chose Rocky Linux 9 to be as close to RHEL 9 as possible.

NOTE: Please check the "Multiple network interfaces in Hetzner Cloud with RHEL" section if you wish to set up multiple network interfaces on the server.

Start install of RHEL 9 on Hetzner Cloud

Once you've launched the sacrificial lamb VM, attach the RHEL 9 installation ISO to it and reboot it. Then open the virtual machine console in Hetzner Cloud and you should see the RHEL 9 installation screen. Use the default non-GUI mode to start the RHEL installation process. When you get to the actual graphical installer switch on the GUI mode.

Installation tips

The RHEL 9 installation process does not require any special tweaks for Hetzner. A couple things worth noting, though:

• Register the VM with Red Hat during installation. This will save you some effort later. NOTE: this may not always be possible, see the Multiple network interfaces in Hetzner Cloud with RHEL section.
• You need to delete the sacrificial lamb VMs main partition to free up space for RHEL.
• Create an admin user. This is necessary because default RHEL installation does not include cloud-init, which would normally set up the default system user and configure sudo for it.
• Do not use "cloud-user" as the name of your admin user. This is to avoid overlap with cloud-init's default username on RHEL.

In general the installation of RHEL 9 on Hetzner Cloud should go smoothly. After installation has finished unmount the ISO and reboot the VM.

Step 3: Post-install steps

You can't use SSH to access your RHEL system after install. Instead, you must use the Hetzner Cloud server console. Login with your credentials and sudo up proceed with the post-install steps:

Fix networking

If your VM has multiple network interfaces then your network is still in a broken state. Fix it as described in the "Multiple network interfaces in Hetzner Cloud with RHEL" section.

Register the system with Red Hat

Registering a RHEL system with Red Hat is a requirement for installing software packages. You use subscription-manager to login to your Red Hat account:

$ subscription-manager register

If have Simple Content Access Mode enabled then you should not need to repeat this process. This should hold true even if you spin up new VMs from a snapshot based on your RHEL installation.

Enable SSH access

One of the annoyances with Hetzner Cloud virtual machine console is that it does not (seem to) support copy-and-paste. That makes adding things like SSH authorized keys very difficult. So, it is best to make SSH work as soon as possible. You should enable password authentication and possibly root logins in /etc/ssh/sshd_config:

# If you wish to login as root with SSH at this stage you need this 
#PermitRootLogin yes

# You need to enable 
PasswordAuthentication yes

Restart sshd after the configuration change:

$ systemctl restart sshd

Now you should be able to login to the node with SSH

Add SSH authorized keys

You can easily add SSH authorized keys once you get initial SSH access by simple copy-and-paste, scp or rsync. If you do not intend to use cloud-init then you can add keys for the root user as well and they will stick.

Install cloud-init?

With your new SSH capabilities you could, at this point, set up and configure cloud-init ease. Check "Should you use cloud-init?" section to determine if that is wise or not.

Disable password logins to SSH

You should not allow password authentication for SSH for publicly facing system. In partiular you should avoid allowing password logins for the root user. So, edit /etc/ssh/sshd_config accordingly:

PermitRootLogin without-password
PasswordAuthentication no

Then restart sshd:

$ systemctl restart sshd

Install hcloud-utils

This highly recommented if you want to create new servers from a snapshot of your RHEL system. See "Network interface names are not consistent across instance types" for details.

Remove SSH host keys

You may want to use your RHEL 9 system as a basis for other RHEL 9 server in Hetzner Cloud. In that case you should remove SSH host keys as the last step before creating a snapshot:

$ rm -rf /etc/ssh/ssh_host_*

Shut down the server and create a snapshot. You can then use this snapshot to spin up new RHEL 9 server.

Should you use cloud-init?

Taming cloud-init?

Theoretically cloud-init is a good tool for doing post-install configuration of RHEL 9, or any other Linux system for that matter. It is essentially a limited, state-based configuration management language similar to Terraform or Puppet. The main issue with cloud-init is that it seems to be unpredictable and/or indeterministic. Typical pattern with cloud-init is this:

• You find a solution that works several times in a row.
• You try the solution the Nth time and it no longer works.

Maybe you made a small (seemingly unrelated) change to your cloud-init config which breaks all your other cloud-init configs in a spectacular fashion. In the end you have no idea what happened. Perhaps some hints can be found from /var/log/cloud*.log, or not.

Making cloud-init fast by avoiding user-data

You should not use Hetzner Cloud's user data to pass cloud-init configurations. Doing so means each (failed) experiment takes 15-20 minutes. Fortnately you can test cloud-init configs on a running system to cut that time down to seconds:

$ ( cd /var/lib/cloud && rm -rf * ) && cloud-init -d init 2>&1

Create, update or delete cloud-init configurations under /etc/cloud/cloud.cfg.d/ and run the above command to apply them immediately.

In theory you should be able to pass your "proven to work" cloud-init configuration as user data in Hetzner Cloud console, Terraform or elsewhere. That said, user-data apparently runs in a later stage than cloud-init configs found from the filesystem. That may affect things, or not.

Example cloud-init configuration for Hetzner Cloud

I used cloud-init to force use of root user as the "default user" in the Hetzner Cloud style. The term "default user" is actually misleading: it seems that in cloud-init context it carries a special meaning, and root can never be a true default user. This distinction has certain implications. For example, the "ssh" module's very convenient allow_public_ssh_keys parameter has no effect on root. What it would do is get the SSH public keys exposed by Hetzner Cloud and put them to the default user's authorized_keys.

My cloud-init configuration also forces recreation of SSH host keys, which is something you really, really want when building new servers from snapshots. It also installs Hetzner Cloud utils ("hc-utils") package to ensure that things like networking does not break if network interface names change (see "Network interface names are not consistent across instance types").

The cloud-init configuration, /etc/cloud/cloud.cfg.d/90_custom.cfg, looks like this:

#cloud-config
users:
  - name: root
    ssh_authorized-keys:
      - <some-ssh-public-key>
      - <another-ssh-public-key>
ssh:
  disable_root: false
  allow_public_ssh_keys: true
  ssh_deletekeys: true
runcmd:
  - [ curl, https://packages.hetzner.com/hcloud/rpm/hc-utils-0.0.4-1.el8.noarch.rpm, -o, /tmp/hc-utils-0.0.4-1.el8.noarch.rpm, -s ]
  - [ dnf, -y, install, /tmp/hc-utils-0.0.4-1.el8.noarch.rpm ]

Documentation for cloud-init is "not terrible, not great". Nevertheless here are good starting points:

• Cloud-init documentation (cloud-init website)
• Chapter 2. Introduction to cloud-init (RHEL 8 documentation).
• Chapter 4. Configuring cloud-init (RHEL 8 documentation).
• Basic Cloud Config (cloud-init tutorial from Hetzner)

Caveats

Network interface names are not consistent across instance types

This means that a VM created from snapshot may have different network device names than the VM snapshot was created from. This results in mismatch between NetworkManager .nmconnection files and reality. And the result is a completely broken network.

In Hetzner Cloud the network interface names are consistent, but only within these two groups of server types:

• CX, CCX*1
• CPX, CAX, CCX2, CCX3

If you install RHEL on, say, a CX* system, snapshot it and then spin up a CPX* system from the snapshot your networking will be completely broken. The fix is to install the hc-utils package as described in Hetzner Cloud documentation:

$ curl https://packages.hetzner.com/hcloud/rpm/hc-utils-0.0.4-1.el8.noarch.rpm -o /tmp/hc-utils-0.0.4-1.el8.noarch.rpm -s
$ dnf install /tmp/hc-utils-0.0.4-1.el8.noarch.rpm

Multiple network interfaces in Hetzner Cloud with RHEL

At this point you should be aware of a caveat in Hetzner Cloud if you wish to attach multiple network interfaces to the server. If you launch server with more than one network interfaces the RHEL installer's network may be broken. If that is the case, the network will also be broken on installed RHEL system. The underlying cause of this is related to NetworkManager and the way it creates default routes.

You can find and explanation with a workaround about that in here, but the crux is this: by default NetworkManager adds a default route for every network interface it configures. In Hetzner Cloud only public interfaces have a default gateway, so if you attach you server to a private network, that will also get a default route. Moreover, the network interface attached to the private network will end up having a lower metric than the one attached to the public network. Here's an example with ens10 being the private interface and ens3 being the public interface:

$ ip route show                                             
default via 10.100.5.1 dev ens10 proto dhcp src 10.100.5.22 metric 100                                                                                      
default via 172.31.1.1 dev ens3 proto dhcp src 92.211.133.51 metric 101                                                                                       
10.100.5.0/23 via 10.100.5.1 dev ens10 proto dhcp src 10.100.5.22 metric 100 
10.100.5.1 dev ens10 proto dhcp scope link src 10.100.5.22 metric 100                                                                                       
172.31.1.1 dev ens3 proto dhcp scope link src 92.211.133.51 metric 101 

The end result is that outbound Internet traffic is completely broken. The fix trivial though:

$ nmcli connection modify ens10 ipv4.never-default yes
$ systemctl reboot

Rebooting is recommended to ensure that the bad default route does not reappear at a later time.