Debugging Terraform AWS UnauthorizedOperation errors

January 11, 2023

Causes for the Terraform AWS UnauthorizedOperation errors

Terraform is an infrastructure as code tool you can use to configure Cloud resources in AWS. When using Terraform AWS provider you frequently run into various UnauthorizedOperation errors when creating, modifying or deleting resources. That happens unless you do what you should not do and let Terraform use an AWS API key tied to an IAM user with full admin privileges. In almost all cases Terraform UnauthorizedOperation errors are caused by missing IAM permissions. It can sometimes be a bit tedious to get to the root cause, because Terraform may not show it in its normal output.

Visible errors

You can resolve most access denied easily because they’re not trying to hide. Typically Terraform prints out a message like this:

│ Error: reading inline policies for IAM role foobar, error: AccessDenied:
| User: arn:aws:iam::012345678901:user/terraform is not authorized to
| perform: iam:ListRolePolicies on resource: role foobar because no
| identity-based policy allows the iam:ListRolePolicies action
│       status code: 403, request id: 3bf72b74-74e5-4f54-883b-cac962f6e21a

In this case it is enough to grant Terraform the iam:ListRolePolicies permission.

Hidden errors

Sometimes you get an UnauthorizedOperation error with no hint as to what permission is missing:

│ Error: reading EC2 Instance (i-0123456789abcdef0): 
| UnauthorizedOperation: You are not authorized to perform this operation.
│       status code: 403, request id: 307459e3-156d-4d1b-b8a8-7a30b52acb93

To figure out what IAM permission is missing you need to run Terraform in debug mode:

$ TF_LOG=debug terraform apply

2023-01-11T14:12:28.597+0200 [DEBUG] provider.terraform-provider-aws_v4.21.0_x5: [aws-sdk-go] DEBUG: Response ec2/DescribeInstances Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 403 Forbidden
Transfer-Encoding: chunked
Cache-Control: no-cache, no-store
Content-Type: text/xml;charset=UTF-8
Date: Wed, 11 Jan 2023 12:12:27 GMT                                                                                            
Server: AmazonEC2                                   
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: accept-encoding                                                                                                          
X-Amzn-Requestid: 41f54e26-be21-4d8e-9918-58461e8f5642
                                                               
                                                                                                                               
-----------------------------------------------------: timestamp=2023-01-11T14:12:28.596+0200
2023-01-11T14:12:28.597+0200 [DEBUG] provider.terraform-provider-aws_v4.21.0_x5: [aws-sdk-go] <?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>UnauthorizedOperation</Code><Message>You are not authorized to perform this operation.</Message></Error></Errors><RequestID>41f54e26-be21-4d8e-9918-58461e8f5642</RequestID></Response>: timestamp=2023-01-11T14:12:28.596+0200
2023-01-11T14:12:28.597+0200 [DEBUG] provider.terraform-provider-aws_v4.21.0_x5: [aws-sdk-go] DEBUG: Validate Response ec2/DescribeInstances failed, attempt 0/25, error UnauthorizedOperation: You are not authorized to perform this operation.
        status code: 403, request id: 41f54e26-be21-4d8e-9918-58461e8f5642: timestamp=2023-01-11T14:12:28.596+0200

From the output it is clear that the missing IAM permission is ec2:DescribeInstances. Because TF_LOG=debug produces tons of output you may want to use the -target parameter to reduce the scope of your terraform apply (or plan).

Encoded errors

Sometimes UnauthorizedOperation errors come in encoded form. Why, I can only fathom. In any case the output looks like this:

│ Error: attaching EBS Volume (vol-abcdef0123456789a) to EC2 Instance
| (i-0123456789abcdef0): UnauthorizedOperation: You are not authorized to
| perform this operation. Encoded authorization failure message:
| <really-long-random-looking-string>

│       status code: 403, request id: 9fe9c991-21ed-4b27-9277-771e3fc1318c

This type of error messages can decoded with AWS CLI:


$ aws sts decode-authorization-message --encoded-message '<really-long-random-looking-string>' --query DecodedMessage --output text
--- snip ---
"action": "ec2:AttachVolume"
--- snip ---

In this case the problem is a missing ec2:AttachVolume IAM permission.

Summary of Terraform AWS UnauthorizedOperation errors

IAM permissions are the main cause for Terraform UnauthorizedOperation errors in AWS. Terraform shows some errors immediately, but AWS hides or encodes others. You can, in almost all cases, find out the cause for it with the simple tricks shown in this article.

Deep-Dive