[403] {\"message\":\"The security token included in the request is expired\"} #24
Comments
|
This could be handled by custom assume roles. ref.) |
|
OP's Fluentd config doesn't mention passing any static credentials, and his screenshot shows the use of an IAM role associated with an EKS worker node. That makes me think he's already using IAM role(s) for Fluentd to assume. This is how I'm passing these to the plugin: I encounter this same error. I am using IRSA so that the Fluentd pods assume the IAM role, not the node like OP. Same overall approach, though. I was using This looks to me like the plugin successfully retrieves the credentials for the role at startup, but does not renew them any time after that. Same behavior with Here's a graph of my buffers. The long lead up is when I was using the older container/plugin versions. Then I deployed updated containers with this plugin, I see logs flowing in fine in Kibana, no errors from Fluentd. Then it starts throwing this 403 error. I restart them, it's fine for a while, then buffers begin filling again as it gets the 403 since the credentials it's trying to pass are no longer valid.
|
|
@DaemonDude23 Could you please help in setting up IRSA so that the Fluentd pods assume the IAM role, facking this issue is My Dockerfile My role in IAM dashboard
OpenSearch Cluster security configuration
|
|
@vishalmamidi It's fairly complicated, but I'll try to help get you started. https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html What you don't need:
You need:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Action": "es:ESHttpPost",
"Resource": "arn:aws:es:us-east-1:REDACTED:domain/REDACTED"
}
]
}
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::REDACTED:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/REDACTED"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringLike": {
"oidc.eks.us-east-1.amazonaws.com/id/REDACTED:sub": "system:serviceaccount:NAMESPACE_OF_POD_HERE:EXACT_NAME_OF_K8S_SERVICE_ACCOUNT"
}
}
}
]
}
|
|
@DaemonDude23 Thank you soo much |
|
Hi, I've encountered the same issue but in a slightly different environment. Note that I, too, have a working environment with the old plugin, so I know the IAM role is wired correctly.
I don't see anything missing between what's written above and what I have, that would explain this "outage" after 1 hour and the fact that it worked with the older plugin. Any ideas? |
|
Still, I am facing the "The security token included in the request is expired" issue and in my fluentd config I am not passing any and issue stays like that for some hrs and suddenly it starts working again below are the updated configuration Any help is appreciated. Thanks in advance! |
|
I have the same issue here with bare metal td-agent on an AWS Ubuntu EC2.
I was observing the process of this issue all the day today, I feel it's a bug now. I have to turn back to the old aws-elasticsearch plugin, and block my td-agent version at 4.2.0, because there is some other compatibility issue. |
|
I feel I found a workaround, see #46 (comment) |
Issue
Initially Fluentd is working file for some days,
but after some times facing this issue and Fluentd doesn't sends logs to opensearch
and this issue gets resolves after restarting POD or deleting and re-creating Fluentd POD
Issue log
Steps to replicate
DockeFile file used for deploying fluentd to Kubernetes
Config file used to connect to AWS OpenSearch
https://docs.aws.amazon.com/opensearch-service/latest/developerguide/fgac.html
have added backend role to connect to OpenSearch trough fine grade access control
Expected Behavior or What you need to ask
how to solve this issue without restarting POD
...
Using Fluentd and OpenSearch plugin versions
2022-02-23 02:32:28 +0000 [info]: gem 'fluent-plugin-avro' version '1.1.1'
2022-02-23 02:32:28 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2022-02-23 02:32:28 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.14'
2022-02-23 02:32:28 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'
2022-02-23 02:32:28 +0000 [info]: gem 'fluent-plugin-json-in-json-2' version '1.0.2'
2022-02-23 02:32:28 +0000 [info]: gem 'fluent-plugin-kafka' version '0.17.3'
2022-02-23 02:32:28 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '2.9.2'
2022-02-23 02:32:28 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2022-02-23 02:32:28 +0000 [info]: gem 'fluent-plugin-opensearch' version '1.0.1'
2022-02-23 02:32:28 +0000 [info]: gem 'fluent-plugin-parser-avro' version '0.3.1'
2022-02-23 02:32:28 +0000 [info]: gem 'fluent-plugin-parser-cri' version '0.1.1'
2022-02-23 02:32:28 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.2'
2022-02-23 02:32:28 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0'
2022-02-23 02:32:28 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2022-02-23 02:32:28 +0000 [info]: gem 'fluent-plugin-stdout-pp' version '0.2.0'
2022-02-23 02:32:28 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.5'
2022-02-23 02:32:28 +0000 [info]: gem 'fluentd' version '1.14.3'
2022-02-21 13:50:38 +0000 [info]: starting fluentd-1.14.3 pid=7 ruby="2.6.9"
The text was updated successfully, but these errors were encountered: