Enabling Azure Backup on Linux VMs with Terraform
This article shows you how to enable Azure Backup on Linux VMs. It is recommended to read the Understanding Azure Backup for Linux VMs article first before trying to enable backups with Terraform. Terraform AzureRM provider has three relevant resources:
- azurerm_linux_virtual_machine: parameters provision_vm_agent and allow_extension_operations should be true or enabling backups will fail (with or without Terraform)
- azurerm_virtual_machine_extension: this can be used to run/enable a VM extension on a VM
- azurerm_backup_protected_vm: this resource enables backups on a VM
- azurerm_recovery_services_vault: the recovery services vault where backups will be stored
- azurerm_backup_policy_vm: backup schedule and retainment rules
I’ll go through the basic process first, which may work for some people. I’ll then take a deep dive into the timeout issues you may or may not encounter in your environment.
Understanding Azure Backup for Linux VMs
Azure Backup is an Azure service that allows, among other things, backing up Windows and Linux VMs in Azure. The backups are essentially virtual machine snapshots, but backing up and/or restoring individual files is also possible. This article tries to explain how Azure Backup and Linux VMs interact and what is required for them to work. If you’re interested in using Terraform to configure Azure Backup, please have a look at the Enabling Azure Backup on Linux VMs with Terraform article.
Terraform: converting an aws_instance network interface into an explicitly managed interface
The aws_instance resource in Terraform can automatically create the default network interface for you. There are cases, however, when you notice that the default network interface is not enough anymore, and modifying it via the limited aws_instance parameters is not sufficient. In these cases you can convert the interface into an aws_network_interface resource, but the process is far from straightforward.
The first step is to create a Terraform resource for the network interface:
Terraform, Azure and MissingSubscriptionRegistration
When deploying with Terraform to Azure you may sometimes encounter errors such as this:
╷
│ Error: creating Automation Account (Subscription: "83fa140a-2bea-175a-395c-914cae2902ce"
│ Resource Group Name: "production"
│ Automation Account Name: "backup-automation"): automationaccount.AutomationAccountClient#CreateOrUpdate: Failure responding to request: StatusCode=409 -- Or
iginal Error: autorest/azure: Service returned an error. Status=409 Code="MissingSubscriptionRegistration" Message="The subscription is not registered to use
namespace 'Microsoft.Automation'. See https://aka.ms/rps-not-found for how to register subscriptions." Details=[{"code":"MissingSubscriptionRegistration","mes
sage":"The subscription is not registered to use namespace 'Microsoft.Automation'. See https://aka.ms/rps-not-found for how to register subscriptions.","targe
t":"Microsoft.Automation"}]
The problem is that in Azure you may need to register the provider for the service you intend to manage with Terraform. If you add resources from Azure Portal this registration part is handled automation. In the above case the Azure Automation provider was missing.
Finding which MySQL/MariaDB tables are modified by a GUI
I was working with Keycloak realm private/public key automation and it was not immediately obvious where Keycloak stores the keys. Figuring it out was actually easy, and this method applies to any web application that uses MySQL/MariaDB, not just Keycloak.
Anyhow, on Ubuntu, you’d navigate to /var/lib/mysql/name-of-database. For example:
cd /var/lib/mysql/keycloak
Make sure that no changes have been made recently (within one minute):
$ find -mmin -1
Then go to the web application and change whatever you’re trying to locate from the database. Here I changed the Keycloak RSA key priority. Then run the find command again:
Managing custom realm keys in Keycloak programmatically
What are Keycloak realm keys?
Keycloak’s authentication protocols make use of private and public keys for signing and encrypting, as described in the official documentation. These keys are realm-specific, and by default managed internally in Keycloak. So, when you create a realm using the Keycloak Admin API, kcadm.sh or manually using the Web UI, new keypair(s) get generated automatically. These are called “Managed keys”. However, you can also use custom realm keys in Keycloak, which is what this article is about. Using custom realm keys allow you, for example, to use a single keypair in multiple realms. When you import a custom private key, Keycloak uses it to automatically create a certificate. If you want, you can pass both a private key and a certificate signed with that private key instead. That way you can bypass Keycloak’s automatic certificate generation, should you need to.
AWS: creating AMIs from EBS snapshots shared with you
In AWS EBS (“Elastic Block Storage”) is the underlying technology that (virtual) hard disks of your instances (virtual machines) use. You can take snapshots of those virtual hard disks and use those snapshots to, for example:
- Debugging issues with unbootable virtual machines: attach and then mount the snapshot on another virtual machine and investigate what is wrong.
- Creating AMI images from which you can launch new instances
- Sharing EBS volumes (between regions and AWS accounts), e.g. for purposes of migrating VMs
Here we’ll focus on the last use-case: being able to create copies of virtual machines on another AWS account. The reason why I even bothered writing this blog post is that most of the articles on the Internet do not cover this use-case: they assume you’re working within one AWS account and/or one region. The use-case covered here requires a few extra steps:
Modern cronjob part 1: Azure Automation with Terraform
Microsoft Azure has a nice service for scheduling tasks called Azure Automation. While Azure Automation is able to other things as well, such as being able to act as a Powershell DSC pull server, we’ll focus on the runbooks and scheduling. Runbooks are scripts that do things, e.g. run maintenance and reporting tasks. Runbooks often, but not necessarily manipulate objects in Azure. Runbooks are serverless like Azure Functions and the pricing is therefore similar - you pay for the amount of computing resources you use and don’t have to pay for a server that’s mostly idle.
Joining nodes to FreeIPA using a non-admin user
Typically Linux nodes are joined to FreeIPA using admin credentials. While this works, it exposes fully privileged credentials unnecessarily, for example when used within a configuration management system (see for example puppet-ipa).
Fortunately joining nodes to FreeIPA is possible with more limited privileges. The first step is to create a new FreeIPA role, e.g. “Enrollment administrator” with three privileges:
- DNS Administrators
- Host Administrators
- Host Enrollment
Then you create a new user, e.g. “enrollment”, and join it to the “Enrollment administrator” role. After that you should be able to join nodes using that “enrollment” user.
Using .sync.yml in Puppet Development Kit (PDK)
Puppet Development Kit is probably the best thing since sliced bread if you work a lot with Puppet. It makes adding basic validation and unit tests trivial with help from rspec-puppet. It also makes it very easy to build module packages for the Puppet Forge.
That said, there is a minor annoyance with it: whenever you run “pdk update” to update PDK to the latest version, all your local changes to files such as .gitignore get destroyed, because PDK creates those files from PDK templates and any local changes are mercilessly wiped. Fortunately there is a way to sync your local changes to the files generated from the templates with .sync.yml. While .sync.yml is is pretty well documented, documentation is lots and lots of words with very few examples. So, here’s a fairly trivial .sync.yml example that ensures local changes to .pdkignore and .gitignore persist across PDK updates: