AWS EC2 instances are subject to two types of status checks (AWS docs):

1. System status check (issues with the underlying hardware/networking: “the AWS side”)
2. Instance status check (issues with the OS, e.g. OOM, file system corruption, broken networking, etc: “our side”)

The official AWS EC2 instance recovery documentation claims that automatically recovering from an EC2 instance failure is only possible for system status checks, but the AWS documentation is outdated: Cloudwatch metrics are available for both types of check failures and people have set up recovery for both types of failures using AWS CLI.

The Using tags in Puppet Bolt aws_inventory target_mapping showed how to use AWS “Name” tag as the target name for Puppet Bolt. Use of tags can be extended to creating Bolt target groups:

groups:
  - name: webservers
    targets:
     - _plugin: aws_inventory
       region:
         _plugin: env_var
         var: AWS_DEFAULT_REGION
       filters:
         - name: tag:Type
           values: [webserver]
       target_mapping:
         name: tags.Name
         uri: public_ip_address

All you need to do is add a “filter” section with one filter. The “name” parameter tells Bolt that the filter is based on tags and what tag to look for. The “Type” that follows “tag” is the name of the tag you want to filter by. The “values” array lists acceptable values for that tag. Any nodes that don’t have the tag (here: “Type”) or where tag value is not acceptable (e.g. “db”) will not be shown in the group.

I recently checked the pricing model for Amazon Fargate to see if migrating a fair number of EC2 instance-based workloads to containers would save money. In theory this should have been the case, as a container has less “fat” compared to a full virtual machine. In this case the workload itself was perfectly suited for a container: a single process with single configuration file that was mostly network-bound running on t3.micro instances.

We’re migrating away from Ansible to Puppet Bolt and the fact that Ansible updates broke the old ec2.py inventory script expedited that process. While that inventory script was quite rudimentary, it was able to automatically add human-readable names to the EC2 instance names. So, for example, you could target a node using “server_example_org” if the Name tag was set to “server.example.org”.

Puppet Bolt’s aws_inventory seemed to lack that feature, but after taking a good look at its code this was just a case of missing documentation, which I fixed in this PR. Here’s an example of inventory.yaml for Bolt that maps the value of the “Name” tag to the name of the Bolt target:

This blog post shows how to grant access to an AWS account for some external party. For simplicity we will call this external party a “contractor”. If you want to do the same for Microsoft Azure look at this blog post instead.

This method requires the contractor to have its own AWS account, but it has several benefits:

• No need to create AWS user accounts for every single person working for the contractor
• You can force the contractor’s personnel to use two-factor authentication (2FA)
• You can force the contractor to use an additional password (shared secret)
• You can disable contractor’s access completely by just removing the IAM role

This approach works equally well for AWS Console access, AWS CLI and tools like Terraform. Some tools may not like the extra security measures required here or be able to assume a role out of the box.

Sometimes Terraform stalls when trying to remove AWS EC2 security groups and Terraform does no give any hint as to what is wrong. The problem is caused by that security group being attached to an EC2 instance or network interface.

Interestingly Terraform messes up the order of the AWS API calls even when it (attempts to) remove the security group and detach it from all EC2 instances that use it. In other words, Terraform tries to remove the security group before it has been detached from the EC2 instances, which results in the stall:

We use Terraform for managing our Cloud infrastructure. Our customers typically use AWS and that’s what we’re most familiar with. Each public Cloud has its own terminology, so this page is a translation table between Terraform resource names in the AWS and Azure providers:

Mapping of Terraform AWS resource names to Azure resource names