Solving an obscure AWS S3 access denied error in a cross account scenario

At Macuject, we’re building in the AWS cloud, and one of the critical drivers of our architecture is the AWS Well-Architected Framework. The framework comprises a set of six pillars that describe key concepts, design principles, and architectural best practices for designing and running workloads in the cloud; these include:

• Operational Excellence
• Security
• Reliability
• Performance Efficiency
• Cost Optimization
• Sustainability

Within the security pillar, you’ll find the foundational concept of SEC01-BP01, “account separation”, which is the basis of the problem described here. If this is new to you, AWS has a whitepaper that might interest you; you’ll also want to invest time in AWS Organizations, AWS Control Tower, and AWS Identity Center (IDC). For this post, we’ll define three AWS accounts:

• AccountA has:
  • Users defined in IAM
  • AWS Organizations
  • AWS Identity Center, along with policies and roles that make everyone’s access secure and adhere to the Principle of Least Privilege (PoLP)
• AccountB has an S3 bucket called our-shared-widgets
• AccountC has various pieces of compute used for development

All is well

For some time, our users have been able to access the our-shared-widgets bucket in AccountB while logged in using IDC policies associated with AccountC without issue. Some of these policies grant some of our users GetObject, some PutObject, and others with both. Policies providing this access can be tricky to set up, so this walk-through is an excellent place to start if you want more information.

Then it wasn’t

I recently needed to provide a new user access to our-shared-widgets while logged in using IDC policies associated with AccountB. If you missed it, the critical difference is the IDC policies and their account association, AccountC vs. AccountB. The authentication process was successful; IDC functioned as expected, but on attempting to get an S3 object, a 403 Access Denied error occurred.

Investigation

S3 is simple until it isn’t. As a starting point, I reviewed the following aspects of our IAM policies and S3 bucket policies to rule out any obvious issues:

• Access is denied if the IAM policy grants the necessary permissions but the bucket policy does not.
• Access is denied if the bucket policy grants the necessary permissions, but the IAM policy does not.
• If both the IAM and bucket policies grant the necessary permissions, access is allowed unless there is an explicit Deny statement in either policy. Explicit Deny statements always take precedence over Allow statements.

None of the above was the culprit, our new user still couldn’t get objects, and our pre-existing users kept operating happily.

Next I logged in with full administrative rights in AccountB and pushed a test file to our-shared-widgets; our new user could get that object. Now I am intrigued, cross-context puts are at fault.

My next avenue of investigation was objects somehow being associated with the wrong server-side key when encrypting files which meant the IDC policies associated with AccountB would not be able to decrypt them, hence the 403. On review, this was not the case; the key used to encrypt existing objects was the KMS CMK for our-shared-widgets, as expected:

❯ aws s3api head-object --bucket our-shared-widgets --key file.ext

{
    "AcceptRanges": "bytes",
    "LastModified": "2023-04-17T01:14:05+00:00",
    "ContentLength": 15,
    "ETag": "\"52d22dba61ed2f00b1fcd41\"",
    "VersionId": "NPWsAIu_oO04xAeL4rE_me0Z.bEScteI",
    "ContentType": "text/plain",
    "ServerSideEncryption": "aws:kms",
    "Metadata": {},
    "SSEKMSKeyId": "arn:aws:kms:...:...:key/677379e1...",
    "BucketKeyEnabled": true
}

Now, I’m racking my brain and making little progress on diagnosis. Some fresh air and an espresso later, I decide to consider object ownership:

❯ aws s3api get-object-acl --bucket our-shared-widgets --key file.ext

{
    "Owner": {
        "DisplayName": "AccountC",
        "ID": "63a4d1..."
    },
    "Grants": [
        {
            "Grantee": {
                "DisplayName": "AccountC",
                "ID": "63a4d1...",
                "Type": "CanonicalUser"
            },
            "Permission": "FULL_CONTROL"
        }
    ]
}

RESULT!

S3 ACLs

AWS S3 Access Control Lists (ACLs) are a legacy access control mechanism that allows you to manage the permissions of individual objects within an S3 bucket. With ACLs, you can define which AWS accounts, or groups are granted access to an object and their specific permissions, such as read, write, or delete access. They’re complex, and as such, I’ve always used IAM policies and bucket policies.

That said, by default, when an object is uploaded to a bucket, it is owned by the AWS account that uploaded it; this means that the object’s permissions are determined by the ACL settings specified by the uploading account. In a cross-account scenario like ours, this can lead to a situation where the bucket owner does not have full access to the objects uploaded by other AWS accounts, even though the objects are stored in their bucket.

I’d forgotten this key aspect when investigating this fault, so it is worth repeating; by default, when an object is uploaded to an S3 bucket, it is owned by the AWS account that uploaded it.

Root cause identified. As we saw, the objects in the bucket are owned by AccountC, so regardless of appropriate IAM and S3 Bucket policies, any user associated via IDC policies with AccountB would be denied access by ACLs.

Solution

Amazon S3 introduced the bucket owner-enforced feature in late 2021. By enabling the s3.ObjectOwnership.BUCKET_OWNER_ENFORCED setting on the bucket, the owner can ensure complete control over all objects, regardless of which account uploaded them. It is worth noting this setting entirely disables ACLs, good riddance!

You can make this change via the console; however, I recommend doing this via the CDK (you created the bucket via CDK, right?):

const bucket = new s3.Bucket(this, 'Bucket', {
  bucketName: 'our-shared-widgets',
  ...
  objectOwnership: s3.ObjectOwnership.BUCKET_OWNER_ENFORCED,
});

Changes to object ownership for existing buckets defined in a stack are applied by CloudFormation; there is no need to define a custom lambda function to update the bucket. With these changes in place, all is well again; our new user can get all objects, and our pre-existing users can still get and put objects. Going forward ownership of objects will always be, AccountB regardless of how they’re uploaded. IAM and S3 bucket policies will function as expected without ACL interference. 🎉

Conclusion

This default is archaic, much like allowing public access to buckets. No doubt I’m not the first person to have suffered this, but likely not the last.

AWS, to their credit, have been working on these issues for a while now and plan to change the defaults per https://aws.amazon.com/about-aws/whats-new/2022/12/amazon-s3-automatically-enable-block-public-access-disable-access-control-lists-buckets-april-2023/. This change is probably rolling out around the same time I publish this blog post. That said, the changes won’t be retroactive, so if you have existing buckets, you might trip over this and still need to take action.