Terraform on Puppeteers Oy

Debugging Terraform AWS UnauthorizedOperation errors

Wed, 11 Jan 2023 00:00:00 +0000

Causes for the Terraform AWS UnauthorizedOperation errors

Terraform is an infrastructure as code tool you can use to configure Cloud resources in AWS. When using Terraform AWS provider you frequently run into various UnauthorizedOperation errors when creating, modifying or deleting resources. That happens unless you do what you should not do and let Terraform use an AWS API key tied to an IAM user with full admin privileges. In almost all cases Terraform UnauthorizedOperation errors are caused by missing IAM permissions. It can sometimes be a bit tedious to get to the root cause, because Terraform may not show it in its normal output.

Terraform downloads the deprecated azure provider

Tue, 20 Dec 2022 00:00:00 +0000

You are 100% sure that all your Terraform resources are using terraform-provider-azurerm, yet Terraform attempts to download the deprecated “azure” provider:

$ terraform init
Initializing modules...

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/local versions matching "2.2.3"...
- Finding latest version of hashicorp/azure...
- Finding hashicorp/azurerm versions matching "3.17.0"...
- Installing hashicorp/local v2.2.3...
- Installed hashicorp/local v2.2.3 (signed by HashiCorp)
- Installing hashicorp/azurerm v3.17.0...
- Installed hashicorp/azurerm v3.17.0 (signed by HashiCorp)
╷
│ Error: Failed to query available provider packages
│ 
│ Could not retrieve the list of available versions for provider hashicorp/azure: provider registry registry.terraform.io does not have a provider named
│ registry.terraform.io/hashicorp/azure
│ 
│ Did you intend to use terraform-providers/azure? If so, you must specify that source address in each module which requires that provider. To see which
│ modules are currently depending on hashicorp/azure, run the following command:
│     terraform providers
╵

You grep the state file and find no references to the “azure” provider. You assume that the cause is some nested module that depends on it, but no, that’s not it. You run “terraform providers” and see that indeed, the “azure” provider is required:

Importing DNS records from Cloudflare to Terraform

Fri, 30 Sep 2022 00:00:00 +0000

Computers were supposed to relieve us humans from boring and repetitive jobs. Here we turn this upside down and do the boring and repetitive job of a computer by importing Cloudflare DNS records to Terraform ourselves. Not fun, but someone’s gotta do it sometimes. If you’re reading this, that someone is probably you. Condolences. My hope is that someone lazy enough gets bored enough to write a program to do this. I didn’t. Yet.

Terraform, AWS Cloudfront and CNAMEAlreadyExists error

Sat, 17 Sep 2022 00:00:00 +0000

When you create a distribution, AWS creates several DNS A records with the same name (e.g. d25gma2ea3ckma.cloudfront.net) which point to IPs the distribution is using. Then, typically, you would define CNAME(s) pointing to that cloudfront.net address in your own DNS. Each Cloudfront distribution has a list of aliases, similar to Subject Alternative Names (“SAN”) in SSL certificates. The aliases should match the CNAME(s) you’ve set in DNS.

Now, this is all good until you need to migrate those Cloudfront distributions to, say, a new AWS account. If you attempt to create a new distribution (here with Terraform) with the same alias you will get this error:

Enabling Azure Backup on Linux VMs with Terraform

Mon, 05 Sep 2022 00:00:00 +0000

This article shows you how to enable Azure Backup on Linux VMs. It is recommended to read the Understanding Azure Backup for Linux VMs article first before trying to enable backups with Terraform. Terraform AzureRM provider has three relevant resources:

azurerm_linux_virtual_machine: parameters provision_vm_agent and allow_extension_operations should be true or enabling backups will fail (with or without Terraform)
azurerm_virtual_machine_extension: this can be used to run/enable a VM extension on a VM
azurerm_backup_protected_vm: this resource enables backups on a VM
azurerm_recovery_services_vault: the recovery services vault where backups will be stored
azurerm_backup_policy_vm: backup schedule and retainment rules

I’ll go through the basic process first, which may work for some people. I’ll then take a deep dive into the timeout issues you may or may not encounter in your environment.

Terraform: converting an aws_instance network interface into an explicitly managed interface

Tue, 23 Aug 2022 00:00:00 +0000

The aws_instance resource in Terraform can automatically create the default network interface for you. There are cases, however, when you notice that the default network interface is not enough anymore, and modifying it via the limited aws_instance parameters is not sufficient. In these cases you can convert the interface into an aws_network_interface resource, but the process is far from straightforward.

The first step is to create a Terraform resource for the network interface:

Terraform, Azure and MissingSubscriptionRegistration

Sat, 20 Aug 2022 00:00:00 +0000

When deploying with Terraform to Azure you may sometimes encounter errors such as this:

╷                                                                                                                                                             
│ Error: creating Automation Account (Subscription: "83fa140a-2bea-175a-395c-914cae2902ce"                                                                    
│ Resource Group Name: "production"                                             
│ Automation Account Name: "backup-automation"): automationaccount.AutomationAccountClient#CreateOrUpdate: Failure responding to request: StatusCode=409 -- Or
iginal Error: autorest/azure: Service returned an error. Status=409 Code="MissingSubscriptionRegistration" Message="The subscription is not registered to use 
namespace 'Microsoft.Automation'. See https://aka.ms/rps-not-found for how to register subscriptions." Details=[{"code":"MissingSubscriptionRegistration","mes
sage":"The subscription is not registered to use namespace 'Microsoft.Automation'. See https://aka.ms/rps-not-found for how to register subscriptions.","targe
t":"Microsoft.Automation"}]

The problem is that in Azure you may need to register the provider for the service you intend to manage with Terraform. If you add resources from Azure Portal this registration part is handled automation. In the above case the Azure Automation provider was missing.

Modern cronjob part 1: Azure Automation with Terraform

Thu, 11 Aug 2022 00:00:00 +0000

Microsoft Azure has a nice service for scheduling tasks called Azure Automation. While Azure Automation is able to other things as well, such as being able to act as a Powershell DSC pull server, we’ll focus on the runbooks and scheduling. Runbooks are scripts that do things, e.g. run maintenance and reporting tasks. Runbooks often, but not necessarily manipulate objects in Azure. Runbooks are serverless like Azure Functions and the pricing is therefore similar - you pay for the amount of computing resources you use and don’t have to pay for a server that’s mostly idle.

Multi-part cloud-init provisioning with Terraform

Thu, 28 Apr 2022 00:00:00 +0000

Cloud-Init is “a standard for customizing” cloud instances, typically on their first boot. It is allows mixing state-based configuration management with imperative provisioning commands (details in our IaC article). By using cloud-init most of the annoyances of SSH-based provisioning can be avoided:

Having to use (possibly shared) SSH keys for provisioning
Having to have direct network access to the VM being provisioned (may require a VPN connection)

That said, neither cloud-init itself, nor its use within Terraform is particularly well documented. Therefore it can be an effort to create cloud-init-based provisioning that works and adapts easily to different use-cases. This article attempts to fill that gap to some extent.

Data-driven Terraform: Kubernetes cluster in Hetzner Cloud

Fri, 15 Apr 2022 00:00:00 +0000

Terraform does not have a particularly strong decoupling between data and code, at least not from a best practices perspective. It is possible and useful, however, to use data to define Terraform resources - if not for any other reason but to reduce code repetition for common resources that require defining lots of parameters. Here’s an example of setting up a Kubernetes cluster in Hetzner Cloud using Terraform with a “data-driven” approach:

Terraform Enum data type

Tue, 12 Apr 2022 00:00:00 +0000

In Terraform you have access to basic data types like bool or string. Defining the data type is a good start for starting to improve the quality of your modules. However, you may want to validate that a certain string matches a list of pre-defined options, and if not, fail validation early.

Terraform, unlike Puppet, does not have a built-in Enum data type. However, you can emulate an Enum like this:

Conditional provisioner blocks in Terraform

Wed, 05 Jan 2022 00:00:00 +0000

I’ll start with a spoiler: what the title suggests is not possible. It is, however, possible to accomplish this with cloud-init and Terraform templates as described in the Multi-part cloud-init provisioning with Terraform blog post.

If you need to use SSH/WinRM provisioning, then there are various workarounds you can apply, and this article explains some of them and goes into depth into one in particular.

The first workaround is relatively clean, but is not usable in all cases, especially when working with complex modules. The idea is to have two resources, here of type aws_instance. Those resources are identical except for the provisioner blocks:

Terraform Azure resource provider registration fails

Tue, 26 Oct 2021 00:00:00 +0000

Terraform has good support for Microsoft Azure through the Terraform Azure provider and the AzureRM backend. However, you may hit a glitch when adding or importing resources if you lack permissions to register Azure resource providers:

│ Error: Error ensuring Resource Providers are registered.
|
│ Terraform automatically attempts to register the Resource Providers it supports
| to ensure it's able to provision resources.
│                                                                                                                                                                                   
│ If you don't have permission to register Resource Providers you may wish to use
| the "skip_provider_registration" flag in the Provider block to disable this
| functionality.
│                                                                                                                                                                                   
│ Please note that if you opt out of Resource Provider Registration and Terraform
| tries to provision a resource from a Resource Provider which is unregistered,
| then the error may appear misleading - for example
|                                                                                                                                                                                   
│ > API version 2019-XX-XX was not found for Microsoft.Foo
|                                                                                                                                                                                   
│ Could indicate either that the Resource Provider "Microsoft.Foo" requires
| registration, but this could also indicate that this Azure Region doesn't support | this API version.
|
│ More information on the "skip_provider_registration" flag can be found here:
|
│ https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs#skip_provider_registration
│
│ Original Error: Cannnot register providers: Microsoft.DataLakeAnalytics, Microsoft.DBforPostgreSQL, Microsoft.AVS, Microsoft.Media, Microsoft.ApiManagement, Microsoft.Search, Mic
rosoft.Maintenance, Microsoft.ServiceBus, Microsoft.HealthcareApis, Microsoft.Relay, Microsoft.DesktopVirtualization, Microsoft.NotificationHubs, Microsoft.SecurityInsights, Micros
oft.DBforMariaDB, Microsoft.Kusto, Microsoft.DataLakeStore, Microsoft.Automation, Microsoft.CognitiveServices, Microsoft.ManagedServices, Microsoft.EventGrid, Microsoft.BotService,

--- snip ---

At first glance this error look cryptic, but it is quite clear once you understand what Azure resource providers are - I like to think of them as “enabled Azure services” or “enabled Azure features”.

Managing self-hosted GitLab with Terraform

Sun, 16 May 2021 00:00:00 +0000

GitLab has an extensive API which allows managing its internal configuration such as users, groups and projects. The official GitLab Terraform provider builds on top of this API.

It assumed here that the Terraform controller (e.g. your laptop) is able to reach your GitLab instance’s TCP port 443 - if it can’t there’s no way Terraform’s API calls will work. The same approach should work with gitlab.com - you can probably (=untested) skip the base_url parameter in that case.

Product launch: 'Puppeteers Cloud Modeling'

Tue, 13 Apr 2021 00:00:00 +0000

Puppeteers has launched a new service product which allows you to gain control of your Cloud resources by having them imported to Terraform. This allows you to manage changes to your Cloud and radically reduces the need for the pesky “who changes this and why?”, “who set this up and is it still needed?” and “what does it do?” discussions. As a bonus you get much better visibility into your cloud.

Terraform stalls when recreating security groups

Tue, 23 Mar 2021 00:00:00 +0000

Sometimes Terraform stalls when trying to remove AWS EC2 security groups and Terraform does no give any hint as to what is wrong. The problem is caused by that security group being attached to an EC2 instance or network interface.

Interestingly Terraform messes up the order of the AWS API calls even when it (attempts to) remove the security group and detach it from all EC2 instances that use it. In other words, Terraform tries to remove the security group before it has been detached from the EC2 instances, which results in the stall:

Terraform resources with dynamic provider values

Thu, 11 Feb 2021 00:00:00 +0000

Sometimes you’d like to pass a dynamic value to a Terraform resource’s provider parameter. This can be done, but some background first.

Terraform allows you to define multiple providers of the same type using provider aliases. This is useful when you’re working with a provider that is tied to a region, the AWS provider being a prime example. The moment you want to create resources in multiple regions you will have to learn provider aliases.

AWS to Azure resource translation table

Sun, 17 Jan 2021 00:00:00 +0000

We use Terraform for managing our Cloud infrastructure. Our customers typically use AWS and that’s what we’re most familiar with. Each public Cloud has its own terminology, so this page is a translation table between Terraform resource names in the AWS and Azure providers:

AWS	Azure
aws_instance	azurerm_linux_virtual_machine
aws_network_interface	azurerm_network_interface
aws_security_group	azurerm_network_security_group
aws_security_group_rule	azurerm_network_security_rule
aws_vpc	azurerm_virtual_network
aws_subnet	azurerm_subnet
inapplicable	azurerm_resource_group
aws_s3_bucket	azurerm_storage_account
region	location

Mapping of Terraform AWS resource names to Azure resource names

Sed spell to ease moving to Terraform 0.13

Mon, 28 Dec 2020 00:00:00 +0000

When moving from older versions of Terraform 0.12.x to latest 0.12.x (now: 0.12.29) you may notice the following warnings:

Warning: Interpolation-only expressions are deprecated                                                                                                                              

  on backup.tf line 11, in resource "hcloud_server_network" "backup_example_org":                                                                                                 
  11:   server_id = "${module.backup_example_org.id}"

These make sense, but may come as a surprise as earlier 0.12.x versions were perfectly fine with them.

You may have tons of these, so fixing them manually gets tiresome. The fix is not too difficult, but is quite unreadable due to escaping:

Terraform 0.11->0.12 migration: stabs at the state file

Fri, 30 Oct 2020 00:00:00 +0000

We maintain a rather complex AWS environment built with Terraform that uses nested modules and lots of per-resource provider configurations. Recently the pain of staying with Terraform 0.11 became too much, so we had to start the migration to Terraform 0.12, with 0.13 going to follow soon after. In general the official upgrade instructions were pretty good. Of course there was a lot of manual fixing that had to be done, and as I did not keep detailed notes I can’t recall all the steps I went through. But I think all or most of those can be found from other sources.

Terraform: Importing resources from a non-default AWS region

Tue, 04 Feb 2020 00:00:00 +0000

I recently - again - had to import resources to Terraform (0.11.14) from AWS that were outside of the default AWS region and were managed in a nested (non-root) module. The root module contained the provider configuration, including multiple aliases to allow creation of resources in different regions, as well as the nested module call, here in a very simple version:

provider "aws" {
  region = "us-west-2"
}

provider "aws" {
  region = "us-east-1"
  alias  = "us-east-1"
}

provider "aws" {
  region = "us-west-2"
  alias  = "us-west-2"
}

module "cloud" {
  param = value
}

The goal was to import some aws_ses_email_identity resources into the “cloud” module. So, originally the cloud module had this:

Terraform S3 access denied problems

Wed, 11 Sep 2019 00:00:00 +0000

I’ve stumbled upon interesting access denied problems with S3-based Terraform state files recently. Suppose you have two or more Terraform root modules which use the same bucket for storing the state and just use a different key (=state file):

terraform {   backend "s3" {     bucket = "terraform-state"     key    = "root-module-1"     region = "eu-central-1"   } }

and

terraform {   backend "s3" {     bucket = "terraform-state"     key    = "root-module-2"     region = "eu-central-1"   } }

However, trying to access the state file in one root module works, but on the other you get a generic S3 Access Denied error:

Lack of default connection settings in Terraform 0.12.x

Thu, 29 Aug 2019 00:00:00 +0000

I was recently bit by a feature (one might call it a bug) in Terraform. We wanted to try out the “Networks” feature in Hetzner Cloud but that required upgrading terraform-provider-hcloud, which in turn required upgrading Terraform to 0.12.x. For the most part migration from 0.11.x code to 0.12.x code was straightforward with some help from the automated migration tool. I even went as far as read some of the migration documentation, but missed one important part: connection blocks no longer have default values. To make the matter worse Terraform 0.12 does not error out if values are unset: instead if just runs the provisioner and then times out. This seems to be related to Terraform bug #21690.