Aws on Puppeteers Oy

How to reduce RDS storage size in an automated way

Thu, 10 Aug 2023 00:00:00 +0000

Introduction

Amazon RDS is a relational database service available on Amazon Web Services. It is essentially a managed database server with a volume for the data. Both cost money. Having a large dataset is not the only reason have big volume: the volume size determines the number of I/O operations (IOPS) the volume can do. If you have high load on your RDS instance you may very well run out of IOPS which can result in a disaster. Scaling up the volume size is easy to avoid such disasters, but it is much more difficult to reduce RDS storage size. This is because downsizing RDS storage size is not officially supported by AWS. If the load on your RDS instance goes down (e.g. due to refactoring), you may end up having a database with an oversized volume that costs lots of money for no reason.

Downsize AWS RDS Volume

Mon, 08 May 2023 00:00:00 +0000

It is trivial to downsize an AWS RDS instance. It is equally trivial to upsize AWS RDS volumes. However, to downsize AWS RDS volume you need to jump through lots of hoops.

So, why would you ever want to downsize an AWS RDS volume? One reason might be that you increased the volume size simply to get more burst credits for the volume. Then you did refactoring that reduced the need to have that many credits and want to save money. Or maybe you genuinely had a huge amount of data on your RDS instance volume, removed some of it but got stuck with a huge volume.

Let's Encrypt on AWS: when, where and why

Wed, 15 Mar 2023 00:00:00 +0000

Let’s Encrypt on AWS can be a can of worms if you don’t know what you’re doing. Photo credit.

Introduction

Let’s Encrypt and the ACME protocol enable getting free (as in beer) TLS/SSL certificates. The main use for Let’s Encrypt is to enable HTTPS on web servers. Let’s Encrypt TLS certificates expire pretty quickly, in 3 months, but the idea is that renewal is automated so that you don’t ever really have to worry about expiration, unlike with commercial TLS certificates, which you typically need to manually renew every 12 months. Using Let’s Encrypt on AWS is in some cases a very viable option, but not so much in others. In many cases trying to use Let’s Encrypt certificates leads you down the path of Lovecraftian horrors and madness. Because “it all depends” I’ll try to explain when using Let’s Encrypt is viable in the AWS context and when you really should just give up an use certificates generated by the Amazon Certificate Manager (“ACM”) instead.

Wrong Route53 domain contact details

Thu, 02 Mar 2023 00:00:00 +0000

The symptoms

I recently received and email from Amazon Route53 asking me to verify the contact details for one of our registered DNS domains (replaced here with example.org). And sure enough, I saw wrong Route53 domain contact details in the email:

Subject: Confirm that contact information for example.org is correct.
Message body:

Dear AWS customer,

This message is a reminder to help you keep the contact information for your domain registration up to date. WHOIS records include the following information:

  Domains:
  example.org

  Registrant:
  Name: On behalf of example.org owner
  Organization Name: Identity Protection Service
  Address: PO Box 786
  City: Hayes
  State/Province: Middlesex
  Country: GB
  Postal Code: UB3 9TR

  Administrative Contact:
  Name: On behalf of example.org owner
  Organization Name: Identity Protection Service
  Address: PO Box 786
  City: Hayes
  State/Province: Middlesex
  Country: GB
  Postal Code: UB3 9TR
  Phone: +44.1483307527
  Fax: +44.1483304031
  Email: e793d865-97ab-42ab-805c-50cc36fd9202@identity-protect.org

  Technical Contact:
  Name: On behalf of example.org owner
  Organization Name: Identity Protection Service
  Address: PO Box 786
  City: Hayes
  State/Province: Middlesex
  Country: GB
  Postal Code: UB3 9TR
  Phone: +44.1483307527
  Fax: +44.1483304031
  Email: e793d865-97ab-42ab-805c-50cc36fd9202@identity-protect.org

The contact information provided above is what is shown in the public WHOIS. If your domain is privacy protected, the information will be different than the contact data you have submitted. To verify your contact information, please open the Amazon Route 53 console to view your current contact information and make any necessary corrections:
https://console.aws.amazon.com/route53/home#DomainDetail:example.org

If your information is accurate, you do not need to take any action.

Important
Under the terms of your registration agreement, providing false contact data can be grounds for the cancellation of your domain name registration.

Regards,
Amazon Route 53

These wrong Route53 domain contact details originate from whois data for the registered domain. One might reasonably expect the whois data to match the contact details for AWS Route53 domain. However, in this case the Route 53 contact details were correct, but whois data seemed to be out of sync with those.

Debugging Terraform AWS UnauthorizedOperation errors

Wed, 11 Jan 2023 00:00:00 +0000

Causes for the Terraform AWS UnauthorizedOperation errors

Terraform is an infrastructure as code tool you can use to configure Cloud resources in AWS. When using Terraform AWS provider you frequently run into various UnauthorizedOperation errors when creating, modifying or deleting resources. That happens unless you do what you should not do and let Terraform use an AWS API key tied to an IAM user with full admin privileges. In almost all cases Terraform UnauthorizedOperation errors are caused by missing IAM permissions. It can sometimes be a bit tedious to get to the root cause, because Terraform may not show it in its normal output.

How to disable S3 bucket ACLs

Tue, 10 Jan 2023 00:00:00 +0000

Introduction

AWS recommends that you disable S3 bucket ACLs for all new buckets. To understand why some background information is needed.

AWS S3 providers two ways to manage access to S3 buckets and objects:

Access policies: these are standard IAM policies that you attach to users (“user policies”) or resources (“resource-based policies”). IAM policies are the recommended method for managing access to Amazon S3 nowadays.
Access Control Lists (ACLs): ACLs are an older access control method for AWS S3. ACL map to IAM permissions. ACLs works on the bucket and object (file) file.

Terraform, AWS Cloudfront and CNAMEAlreadyExists error

Sat, 17 Sep 2022 00:00:00 +0000

When you create a distribution, AWS creates several DNS A records with the same name (e.g. d25gma2ea3ckma.cloudfront.net) which point to IPs the distribution is using. Then, typically, you would define CNAME(s) pointing to that cloudfront.net address in your own DNS. Each Cloudfront distribution has a list of aliases, similar to Subject Alternative Names (“SAN”) in SSL certificates. The aliases should match the CNAME(s) you’ve set in DNS.

Now, this is all good until you need to migrate those Cloudfront distributions to, say, a new AWS account. If you attempt to create a new distribution (here with Terraform) with the same alias you will get this error:

Terraform: converting an aws_instance network interface into an explicitly managed interface

Tue, 23 Aug 2022 00:00:00 +0000

The aws_instance resource in Terraform can automatically create the default network interface for you. There are cases, however, when you notice that the default network interface is not enough anymore, and modifying it via the limited aws_instance parameters is not sufficient. In these cases you can convert the interface into an aws_network_interface resource, but the process is far from straightforward.

The first step is to create a Terraform resource for the network interface:

AWS: creating AMIs from EBS snapshots shared with you

Mon, 15 Aug 2022 00:00:00 +0000

In AWS EBS (“Elastic Block Storage”) is the underlying technology that (virtual) hard disks of your instances (virtual machines) use. You can take snapshots of those virtual hard disks and use those snapshots to, for example:

Debugging issues with unbootable virtual machines: attach and then mount the snapshot on another virtual machine and investigate what is wrong.
Creating AMI images from which you can launch new instances
Sharing EBS volumes (between regions and AWS accounts), e.g. for purposes of migrating VMs

Here we’ll focus on the last use-case: being able to create copies of virtual machines on another AWS account. The reason why I even bothered writing this blog post is that most of the articles on the Internet do not cover this use-case: they assume you’re working within one AWS account and/or one region. The use-case covered here requires a few extra steps:

Dealing with multiple AWS accounts with one Keycloak client for Single-Sign On

Thu, 18 Nov 2021 00:00:00 +0000

This article assumes that the user backend for Keycloak is FreeIPA. Regardless of that the instructions will apply to any other setup with minor modifications. Here we use two different AWS accounts renamed to 123412341234 and 567856785678 to protect the personal information of the innocent. The Keycloak staging cluster on which this integration was done is called id-staging.example.org and the production cluster is called id.example.org. The Keycloak realm is EXAMPLE.ORG. Replace those with the values correct for your environment.

Enabling AWS EC2 instance automatic recovery with Terraform

Thu, 28 Oct 2021 00:00:00 +0000

AWS EC2 instances are subject to two types of status checks (AWS docs):

System status check (issues with the underlying hardware/networking: “the AWS side”)
Instance status check (issues with the OS, e.g. OOM, file system corruption, broken networking, etc: “our side”)

The official AWS EC2 instance recovery documentation claims that automatically recovering from an EC2 instance failure is only possible for system status checks, but the AWS documentation is outdated: Cloudwatch metrics are available for both types of check failures and people have set up recovery for both types of failures using AWS CLI.

Creating Puppet Bolt groups based on AWS tags

Thu, 01 Jul 2021 00:00:00 +0000

The Using tags in Puppet Bolt aws_inventory target_mapping showed how to use AWS “Name” tag as the target name for Puppet Bolt. Use of tags can be extended to creating Bolt target groups:

groups:
  - name: webservers
    targets:
     - _plugin: aws_inventory
       region:
         _plugin: env_var
         var: AWS_DEFAULT_REGION
       filters:
         - name: tag:Type
           values: [webserver]
       target_mapping:
         name: tags.Name
         uri: public_ip_address

All you need to do is add a “filter” section with one filter. The “name” parameter tells Bolt that the filter is based on tags and what tag to look for. The “Type” that follows “tag” is the name of the tag you want to filter by. The “values” array lists acceptable values for that tag. Any nodes that don’t have the tag (here: “Type”) or where tag value is not acceptable (e.g. “db”) will not be shown in the group.

To containerize in AWS or not: the cost perspective

Tue, 15 Jun 2021 00:00:00 +0000

I recently checked the pricing model for Amazon Fargate to see if migrating a fair number of EC2 instance-based workloads to containers would save money. In theory this should have been the case, as a container has less “fat” compared to a full virtual machine. In this case the workload itself was perfectly suited for a container: a single process with single configuration file that was mostly network-bound running on t3.micro instances.

Using tags in Puppet Bolt aws_inventory target_mapping

Sat, 05 Jun 2021 00:00:00 +0000

We’re migrating away from Ansible to Puppet Bolt and the fact that Ansible updates broke the old ec2.py inventory script expedited that process. While that inventory script was quite rudimentary, it was able to automatically add human-readable names to the EC2 instance names. So, for example, you could target a node using “server_example_org” if the Name tag was set to “server.example.org”.

Puppet Bolt’s aws_inventory seemed to lack that feature, but after taking a good look at its code this was just a case of missing documentation, which I fixed in this PR. Here’s an example of inventory.yaml for Bolt that maps the value of the “Name” tag to the name of the Bolt target:

Delegating external parties access to AWS

Fri, 14 May 2021 00:00:00 +0000

This blog post shows how to grant access to an AWS account for some external party. For simplicity we will call this external party a “contractor”. If you want to do the same for Microsoft Azure look at this blog post instead.

This method requires the contractor to have its own AWS account, but it has several benefits:

No need to create AWS user accounts for every single person working for the contractor
You can force the contractor’s personnel to use two-factor authentication (2FA)
You can force the contractor to use an additional password (shared secret)
You can disable contractor’s access completely by just removing the IAM role

This approach works equally well for AWS Console access, AWS CLI and tools like Terraform. Some tools may not like the extra security measures required here or be able to assume a role out of the box.

Terraform stalls when recreating security groups

Tue, 23 Mar 2021 00:00:00 +0000

Sometimes Terraform stalls when trying to remove AWS EC2 security groups and Terraform does no give any hint as to what is wrong. The problem is caused by that security group being attached to an EC2 instance or network interface.

Interestingly Terraform messes up the order of the AWS API calls even when it (attempts to) remove the security group and detach it from all EC2 instances that use it. In other words, Terraform tries to remove the security group before it has been detached from the EC2 instances, which results in the stall:

AWS to Azure resource translation table

Sun, 17 Jan 2021 00:00:00 +0000

We use Terraform for managing our Cloud infrastructure. Our customers typically use AWS and that’s what we’re most familiar with. Each public Cloud has its own terminology, so this page is a translation table between Terraform resource names in the AWS and Azure providers:

AWS	Azure
aws_instance	azurerm_linux_virtual_machine
aws_network_interface	azurerm_network_interface
aws_security_group	azurerm_network_security_group
aws_security_group_rule	azurerm_network_security_rule
aws_vpc	azurerm_virtual_network
aws_subnet	azurerm_subnet
inapplicable	azurerm_resource_group
aws_s3_bucket	azurerm_storage_account
region	location

Mapping of Terraform AWS resource names to Azure resource names