s3 backend role chaining permission failure

[Folling]

Terraform Version

Terraform v1.10.5
on darwin_arm64

Terraform Configuration Files

This works:

backend "s3" {
  bucket  = "redacted"
  region  = "redacted"
  key     = "redacted.tfstate"
  encrypt = true

  assume_role = {
    role_arn = "arn:aws:iam::1234567890:role/redacted"
  }
}

This fails:

backend "s3" {
  bucket  = "redacted"
  region  = "redacted"
  key     = "redacted.tfstate"
  encrypt = true

  assume_role = {
    role_arn = "arn:aws:iam::0987654321:role/redacted"
  }

  assume_role = {
    role_arn = "arn:aws:iam::1234567890:role/redacted"
  }
}

Debug Output

The first example initialises fine, the second fails with:

│ Error: Error loading state:
│     Unable to access object "redacted.tfstate" in S3 bucket "redacted": operation error S3: HeadObject, https response error StatusCode: 403, RequestID: redacted, HostID: redacted, api error Forbidden: Forbidden
│
│ Terraform failed to load the default state from the "s3" backend.
│ State migration cannot occur unless the state can be loaded. Backend
│ modification and state migration has been aborted. The state in both the
│ source and the destination remain unmodified. Please resolve the
│ above error and try again.

Expected Behavior

The request should work with chained role assumptions.

Actual Behavior

It fails with a role inbetween.

Steps to Reproduce

1. terraform init

Additional Context

I have manually verified that

1. The role has access (by assuming it directly)
2. The chain is intact (assuming works on the CLI and the final role has access to S3 if role-chained)

I am also suspect that the roles are actually chained. The following configuration:

backend "s3" {
  bucket  = "redacted"
  region  = "redacted"
  key     = "redacted.tfstate"
  encrypt = true

  assume_role = {
    role_arn = "arn:aws:iam::0987654321:role/redacted"
  }

  assume_role = {
    role_arn = "gibberish_arn"
  }
}

Still yields the "403" error, instead of a "invalid ARN" error, as I would have expected.

The documentation is a bit iffy as well. It just says "multiple values can be specified", but isn't clear about whether that means

1. passing an array of objects
2. passing multiple role_arn properties in the object
3. passing multiple blocks, like in the provider config.

I tried all of these and more, but looking at the source code I believe my above code is what would be intended.

References

• Duplicate property error when chaining roles using s3 backend with backend config file #36506

Generative AI / LLM assisted development?

No response

[jbardin]

Hi @Folling,

This looks like an error in the configuration, you can't assign different things to the same attribute and the above configuration returns an Attribute redefined error during decoding so you shouldn't be able to get past initialization. Can you show how you are getting to the point of receiving a response from AWS with the given configuration?

[Folling]

The attribute explicitly is supposed to allow for multiple values to allow for AWS role chaining.

This is supposed to provide parity with the AWS provider, which supports this feature here.

#36454 also talks about this feature, where they were able to get it running somehow. Which is confusing to me because:
The feature was implemented in #35720 but was reverted in #35827.

Looking at

terraform/internal/backend/remote-state/s3/backend.go

Line 297 in eb02c65

var assumeRoleSchema = singleNestedAttribute{

it doesn't seem like it was ever re-renabled again.

But that conflicts with the documentation here and with the discussion in #36454.

[jbardin]

Yeah, I also don't understand what was going on in #36454, but you're correct about the configuration, the schema is explicitly a single attribute. I suspect it's a discrepancy in the documentation, but I'll wait for the AWS developers to chime in on the actual intent here.