Terraform stalls when recreating security groups
Sometimes Terraform stalls when trying to remove AWS EC2 security groups and Terraform does no give any hint as to what is wrong. The problem is caused by that security group being attached to an EC2 instance or network interface.
Interestingly Terraform messes up the order of the AWS API calls even when it (attempts to) remove the security group and detach it from all EC2 instances that use it. In other words, Terraform tries to remove the security group before it has been detached from the EC2 instances, which results in the stall:
aws_security_group.foo: Destroying... [id=sg-0123456789abcdef0]
aws_security_group.foo: Still destroying... [id=sg-0123456789abcdef0, 10s elapsed]
aws_security_group.foo: Still destroying... [id=sg-0123456789abcdef0, 20s elapsed]
--- snip ---If the security group is manually detached from the EC2 instance during the Terraform run then the removal of the security group works:
aws_security_group.foo: Destruction complete after 1m21s
aws_instance.bar: Modifying... [id=i-abcdef0123456789a]
aws_instance.bar: Modifications complete after 4s [id=i-abcdef0123456789a]As can be seen from the above, aws_instance.bar was “modified” without any actual changes. That’s because Terraform’s plan included detaching aws_security_group.foo from the instance, but when it was time to do it, the security group attachment was already gone.
So, disassociate the to-be-deleted security group from any AWS EC2 instances and/or network interfaces first, then destroy the security group with Terraform. This issue also affects rebuilding of security groups, as that requires destroying them before recreating them.