Let's Encrypt on AWS: when, where and why

Let’s Encrypt on AWS can be a can of worms if you don’t know what you’re doing. Photo credit.

Introduction

Let’s Encrypt and the ACME protocol enable getting free (as in beer) TLS/SSL certificates. The main use for Let’s Encrypt is to enable HTTPS on web servers. Let’s Encrypt TLS certificates expire pretty quickly, in 3 months, but the idea is that renewal is automated so that you don’t ever really have to worry about expiration, unlike with commercial TLS certificates, which you typically need to manually renew every 12 months. Using Let’s Encrypt on AWS is in some cases a very viable option, but not so much in others. In many cases trying to use Let’s Encrypt certificates leads you down the path of Lovecraftian horrors and madness. Because “it all depends” I’ll try to explain when using Let’s Encrypt is viable in the AWS context and when you really should just give up an use certificates generated by the Amazon Certificate Manager (“ACM”) instead.

Cloud resources versus virtual machines

Cloud resources such as Cloudfront distributions, Elastic Load Balancers and API Gateways are by their nature very inflexible, because they’re essentially standardized commodities. Because of this it is often difficult or impossible to bend them to your will. For this reason is is a good general practice to decouple certificate renewals and the certificate consumers. In fact, this is far more important than the choice between Let’s Encrypt certificates and ACM certificates.

Things that you have full control over, most notably virtual machines, are different beasts: you’re not artificially limited in what you can do. For this reason the easiest option is typically to let the virtual machine handle its own certificate renewal with certbot.

Basics of Amazon Certificate Manager

Before I start talking about Let’s Encrypt on AWS it have to talk a bit about Amazon Certificate Manager, or ACM for short. ACM is a service for provisioning and managing certificates. All AWS Cloud resources that use TLS/SSL must use certificates that are present in ACM - who or what created the certificates in the first place does not matter.

There are two ways to get a certificate into ACM:

  1. Import an existing certificate. The certificate can be a commercial certificate, a Let’s Encrypt certificate or something else.

  2. Request a certificate from within ACM. Just like certbot, ACM creates a challenge for you and if it passes, you get a certiticate in return.

ACM supports two challenge types: email and DNS. However, in practice you should use DNS validation only. DNS validation makes you set up a special CNAME record for each domain name you want to have in the certificate.

Note that ACM certificates can only be used with a subset of AWS services. This limitation seems to affect any certificates in ACM, whether they are “Amazon-issued” (=ACM-generated) or imported.

Let’s Encrypt challenge types: individual domain names vs. wildcards

Let’s Encrypt supports multiple challenge types. The most common challenge is probably the HTTP-01 challenge, where certbot (or another ACME client) sets up special challenge file on a web server in well-known place which ACME servers can then verify. If the certificate verification succeed, certbot gets a signed certificate in return. The certificate fields include a domain name and one or more subject alternative names (SANs). All of the SANs are verified separately. For example, you can request a certificate that is valid for foo.example.com and bar.acme.inc at the same time, but the challenge file has to be reachable from both locations or challenge verification will fail.

Another common challenge type is the DNS-01 challenge. In this case certbot gets a certificate by adding a challenge entry in DNS. ACME servers then verify the DNS entry and give certbot a signed certificate in return. Unlike HTTP-01 challenge you can get a wildcard certificate, e.g. *.example.org, using the DNS-01 challenge. The main benefit of the DNS-01 challenge in the Cloud context is that certificate renewal is decoupled from the Cloud resources that use that certificate.

Getting certificates for AWS EC2 instances

Let’s Encrypt is likely to be the best option for AWS EC2 instance or other virtual machines. Setting up certbot with the HTTP-01 challenge type is probably the most convenient approach for web servers. For other server types such as mail servers you should consider getting a wildcard certificate with the DNS-01 challenge instead. The reason is simple: ACME servers can’t talk to servers that are supposed to be accessible from the VPN

The DNS-01 challenge is also convenient if you want certificates for a domain that is internally facing, but is still registered in the public DNS. For example if you have a VPC with an associated private DNS zone (e.g. *.example.internal) you can get a wildcard certificate easily with certbot and one of the certbot DNS plugins.

The only caveat with certificates generated with the DNS-01 challenge is that you need to somehow handle distributing renewed certificates to the servers that need them. Tools such as Puppet or Ansible can help a lot here.

Let’s Encrypt on AWS: challenges with Cloud resources

Cloudfront distributions: certbot and the HTTP-01 challenge

Amazon Cloudfront speeds up distribution of content by serving it from edge locations that are geographically close to the client that is requesting them. Without Cloudfront content would be served directly from the origin which might be on the other side of the globe, depending on the client.

Here I assume that you use Amazon S3 as the origin for the Cloudfront distribution. For that particular use-case there is a certbot plugin called certbot-s3front that is designed for managing HTTP-01 challenges with Cloudfront and S3. There are two limitations, though:

  1. You need to generate the initial Let’s Encrypt certificate on a real webserver first

  2. You cannot add any Subject Alternative Names once your certificate is live on Cloudfront

Both of the above limitations are a direct consequence of Cloudfront’s features / limitations:

  • A Cloudfront distribution will not serve content for URLs for which it does not have a matching alias

  • You cannot add an alias which is not already present in the certificate the Cloudfront distribution is using

  • Aliases don’t work unless you have HTTPS enabled for the Cloudfront distribution, so cheating by temporarily disabling HTTPS while adding aliases does not work.

In other words you have a huge chicken-and-egg problem on your hands if you ever need to add SANs to your certificate. If you can live with static list of SANs you’re probably going to be ok certbot-s3front.

Elastic Load Balancers: certbot and the HTTP-01 challenge

Elastic Load Balancers can be made to work with certbot and the HTTP-01 challenge type. However, there’s no certbot plugin you could use for this job. Without going into excessive detail what you need to do is:

  • Request the certificate using certbot-s3front plugin. This ensure that the challenge file gets added to an S3 bucket.

  • Set up a listener on the ELB’s port 80. Any traffic going to port 80 should be directed to a Lambda function that handles the ACME challenge. The Lambda function of course needs read access to the S3 bucket with the challenge to function.

Overall, this is definitely not a job for the faint of heart. That said, if you really really really want to go this route, please contact us and we may be able to publish the Lambda function we were using for this purpose.

API Gateways: certbot and the HTTP-01 challenge

Short answer here is: no, this s not possible. The long answer is that it is theoretically possible, but is an exercise in madness, and here’s why:

  1. AWS API Gateway only supports HTTPS

    • There is nothing listening on port 80 (HTTP)

  2. Letsencrypt HTTP-01 challenge always uses HTTP first

    • This will fail when the domain name is pointing to an API Gateway, because there is nothing listening on port 80

  3. Letencrypt HTTP-01 challenge will only switch to HTTPS when it encounters a redirect from HTTP to HTTPS

    • AWS API Gateway does not provide such a redirect

  4. There is no way to configure the challenge request to go through HTTPS at the client (certbot) side, because the challenge request originates from the ACME servers

There seems to be only one realistic option for solving this problem: have a Cloudfront distribution do the HTTP->HTTPS direct. This is described in this blog post, but I’ve never tried this myself.

What about certificate rotation, then?

When you configure a Cloud resource (e.g. a load balancer) use a certificate you point it the certificate’s ARN (=unique identifier). What this means is that when you create or import a new certificate it will have a different ARN from the old one. So, you need to make your Cloud resource use the new certificate instead of the old one: this is called certificate rotation. If you use certbot - whether with HTTP-01 or DNS-01 challenge type, you need to handle certificate rotation on renewal yourself using a custom-made script.

The case for Amazon Certificate Manager certificates

There are several major benefits of letting ACM generate certificates for you:

  1. Certificates are valid for 12 months

  2. Certificate renewal is handled automatically by AWS

  3. Adding Subject Alternative names is supported and only requires passing the challenge for new SANs, not for any of the old ones

  4. Certificate renewal does not change the certificate arn. In other words, you don’t need to worry about rotating the certificate in all the resources that use it

  5. It supports wildcards (e.g. *.example.org) as well as multiple subject alternative names (SANs) in the certificate

All of this makes ACM a superior choice for AWS Cloud resources.

Summary: what am I supposed to do then?

The conclusion we can draw from all this is as follows:

  • Use Amazon Certificate Manager certificates for Cloud resources

  • If you insist on using Let’s Encrypt on AWS for Cloud resources, use the DNS-01 validation type

  • Use Let’s Encrypt with HTTP-01 validation for normal web servers running on virtual machines

  • Use Let’s Encrypt with DNS-01 validation for non-web servers such as mail servers running on virtual machines

  • Use Let’s Encrypt with DNS-01 validation for internal servers that are unreachable for the public ACME servers.

If you do not follow this advise you may be ok, but “it all depends” on the nitty gritty details.