Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add elsif statement for region && c.sts_http_proxy && c.sts_endpoint_url #197

Merged
merged 1 commit into from
May 19, 2020
Merged

add elsif statement for region && c.sts_http_proxy && c.sts_endpoint_url #197

merged 1 commit into from
May 19, 2020

Conversation

JiHyunSong
Copy link
Contributor

caused by below exception

2020-03-16 23:14:09 +0900 [warn]: #1 got unrecoverable error in primary and no secondary error_class=Aws::Errors::MissingRegionError error="missing region; use :region option or export region name to ENV['AWS_REGION']"
  2020-03-16 23:14:09 +0900 [warn]: #1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.91.0/lib/aws-sdk-core/plugins/regional_endpoint.rb:51:in `after_initialize'
  2020-03-16 23:14:09 +0900 [warn]: #1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.91.0/lib/seahorse/client/base.rb:78:in `block in after_initialize'
  2020-03-16 23:14:09 +0900 [warn]: #1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.91.0/lib/seahorse/client/base.rb:77:in `each'
  2020-03-16 23:14:09 +0900 [warn]: #1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.91.0/lib/seahorse/client/base.rb:77:in `after_initialize'
  2020-03-16 23:14:09 +0900 [warn]: #1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.91.0/lib/seahorse/client/base.rb:21:in `initialize'
  2020-03-16 23:14:09 +0900 [warn]: #1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.91.0/lib/aws-sdk-sts/client.rb:310:in `initialize'
  2020-03-16 23:14:09 +0900 [warn]: #1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.91.0/lib/seahorse/client/base.rb:99:in `new'
  2020-03-16 23:14:09 +0900 [warn]: #1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-kinesis-3.2.1/lib/fluent/plugin/kinesis_helper/client.rb:137:in `setup_credentials'
  2020-03-16 23:14:09 +0900 [warn]: #1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-kinesis-3.2.1/lib/fluent/plugin/kinesis_helper/client.rb:102:in `client_options'
  2020-03-16 23:14:09 +0900 [warn]: #1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-kinesis-3.2.1/lib/fluent/plugin/kinesis_helper/client.rb:85:in `client'
  2020-03-16 23:14:09 +0900 [warn]: #1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-kinesis-3.2.1/lib/fluent/plugin/out_kinesis_firehose.rb:51:in `block in write'
  2020-03-16 23:14:09 +0900 [warn]: #1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-kinesis-3.2.1/lib/fluent/plugin/kinesis_helper/api.rb:91:in `batch_request_with_retry'
  2020-03-16 23:14:09 +0900 [warn]: #1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-kinesis-3.2.1/lib/fluent/plugin/kinesis.rb:151:in `block (2 levels) in write_records_batch'
  2020-03-16 23:14:09 +0900 [warn]: #1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-kinesis-3.2.1/lib/fluent/plugin/kinesis_helper/api.rb:86:in `split_to_batches'
  2020-03-16 23:14:09 +0900 [warn]: #1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-kinesis-3.2.1/lib/fluent/plugin/kinesis.rb:149:in `block in write_records_batch'
  2020-03-16 23:14:09 +0900 [warn]: #1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/buffer/memory_chunk.rb:81:in `open'
  2020-03-16 23:14:09 +0900 [warn]: #1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/buffer/memory_chunk.rb:81:in `open'
  2020-03-16 23:14:09 +0900 [warn]: #1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-kinesis-3.2.1/lib/fluent/plugin/kinesis.rb:147:in `write_records_batch'
  2020-03-16 23:14:09 +0900 [warn]: #1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-kinesis-3.2.1/lib/fluent/plugin/out_kinesis_firehose.rb:47:in `write'
  2020-03-16 23:14:09 +0900 [warn]: #1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/output.rb:1122:in `try_flush'
  2020-03-16 23:14:09 +0900 [warn]: #1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/output.rb:1428:in `flush_thread_run'
  2020-03-16 23:14:09 +0900 [warn]: #1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/output.rb:458:in `block (2 levels) in start'
  2020-03-16 23:14:09 +0900 [warn]: #1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'

with fluentd config

      @type "kinesis_firehose"
      region "ap-northeast-2"
      http_proxy xxxxxx
      delivery_stream_name "xxxxxx"

Signed-off-by: JiHyunSong [email protected]

Issue #, if available:

Description of changes:

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@JiHyunSong
Copy link
Contributor Author

@simukappu
hi~
check this pr, please. If this pr is not on your plan, reject this please.
thanks :)

@simukappu simukappu self-requested a review March 21, 2020 04:08
simukappu
simukappu requested changes Mar 21, 2020
View reviewed changes
Copy link
Contributor

@simukappu simukappu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added simple comment for you. I would appreciate if you could confirm that. Thank you for your contribution!

lib/fluent/plugin/kinesis_helper/client.rb Outdated
@@ -139,6 +139,8 @@ def setup_credentials
credentials_options[:client] = Aws::STS::Client.new(region: @region, http_proxy: c.sts_http_proxy)
elsif @region and c.sts_endpoint_url
credentials_options[:client] = Aws::STS::Client.new(region: @region, endpoint: c.sts_endpoint_url)
elsif @region and c.sts_http_proxy and c.sts_endpoint_url
Copy link
Contributor

@simukappu simukappu Mar 21, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This condition (elsif @region and c.sts_http_proxy and c.sts_endpoint_url) will not work because compatible condition in L136 (if c.sts_http_proxy and c.sts_endpoint_url) and L138 (elsif @region and c.sts_http_proxy) are defined before this line. Can you confirm that?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your feedback, :)

@JiHyunSong
add elsif statement for region && c.sts_http_proxy && c.sts_endpoint_url
99754e0 
caused by below exception

```
2020-03-16 23:14:09 +0900 [warn]: awslabs#1 got unrecoverable error in primary and no secondary error_class=Aws::Errors::MissingRegionError error="missing region; use :region option or export region name to ENV['AWS_REGION']"
  2020-03-16 23:14:09 +0900 [warn]: awslabs#1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.91.0/lib/aws-sdk-core/plugins/regional_endpoint.rb:51:in `after_initialize'
  2020-03-16 23:14:09 +0900 [warn]: awslabs#1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.91.0/lib/seahorse/client/base.rb:78:in `block in after_initialize'
  2020-03-16 23:14:09 +0900 [warn]: awslabs#1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.91.0/lib/seahorse/client/base.rb:77:in `each'
  2020-03-16 23:14:09 +0900 [warn]: awslabs#1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.91.0/lib/seahorse/client/base.rb:77:in `after_initialize'
  2020-03-16 23:14:09 +0900 [warn]: awslabs#1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.91.0/lib/seahorse/client/base.rb:21:in `initialize'
  2020-03-16 23:14:09 +0900 [warn]: awslabs#1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.91.0/lib/aws-sdk-sts/client.rb:310:in `initialize'
  2020-03-16 23:14:09 +0900 [warn]: awslabs#1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.91.0/lib/seahorse/client/base.rb:99:in `new'
  2020-03-16 23:14:09 +0900 [warn]: awslabs#1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-kinesis-3.2.1/lib/fluent/plugin/kinesis_helper/client.rb:137:in `setup_credentials'
  2020-03-16 23:14:09 +0900 [warn]: awslabs#1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-kinesis-3.2.1/lib/fluent/plugin/kinesis_helper/client.rb:102:in `client_options'
  2020-03-16 23:14:09 +0900 [warn]: awslabs#1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-kinesis-3.2.1/lib/fluent/plugin/kinesis_helper/client.rb:85:in `client'
  2020-03-16 23:14:09 +0900 [warn]: awslabs#1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-kinesis-3.2.1/lib/fluent/plugin/out_kinesis_firehose.rb:51:in `block in write'
  2020-03-16 23:14:09 +0900 [warn]: awslabs#1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-kinesis-3.2.1/lib/fluent/plugin/kinesis_helper/api.rb:91:in `batch_request_with_retry'
  2020-03-16 23:14:09 +0900 [warn]: awslabs#1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-kinesis-3.2.1/lib/fluent/plugin/kinesis.rb:151:in `block (2 levels) in write_records_batch'
  2020-03-16 23:14:09 +0900 [warn]: awslabs#1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-kinesis-3.2.1/lib/fluent/plugin/kinesis_helper/api.rb:86:in `split_to_batches'
  2020-03-16 23:14:09 +0900 [warn]: awslabs#1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-kinesis-3.2.1/lib/fluent/plugin/kinesis.rb:149:in `block in write_records_batch'
  2020-03-16 23:14:09 +0900 [warn]: awslabs#1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/buffer/memory_chunk.rb:81:in `open'
  2020-03-16 23:14:09 +0900 [warn]: awslabs#1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/buffer/memory_chunk.rb:81:in `open'
  2020-03-16 23:14:09 +0900 [warn]: awslabs#1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-kinesis-3.2.1/lib/fluent/plugin/kinesis.rb:147:in `write_records_batch'
  2020-03-16 23:14:09 +0900 [warn]: awslabs#1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-kinesis-3.2.1/lib/fluent/plugin/out_kinesis_firehose.rb:47:in `write'
  2020-03-16 23:14:09 +0900 [warn]: awslabs#1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/output.rb:1122:in `try_flush'
  2020-03-16 23:14:09 +0900 [warn]: awslabs#1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/output.rb:1428:in `flush_thread_run'
  2020-03-16 23:14:09 +0900 [warn]: awslabs#1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/output.rb:458:in `block (2 levels) in start'
  2020-03-16 23:14:09 +0900 [warn]: awslabs#1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
```

with fluentd config

```
      @type "kinesis_firehose"
      region "ap-northeast-2"
      http_proxy xxxxxx
      delivery_stream_name "xxxxxx"
```

Signed-off-by: JiHyunSong <[email protected]>
@JiHyunSong
Copy link
Contributor Author

@simukappu excuse me, sir?

simukappu
simukappu approved these changes Apr 12, 2020
View reviewed changes
Copy link
Contributor

@simukappu simukappu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thank you!

@simukappu simukappu merged commit 9d20515 into awslabs:master May 19, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@simukappu simukappu simukappu approved these changes

Assignees
No one assigned
Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@JiHyunSong @simukappu