Send data to different Kinesis stream

[jenishshah671993]

Hi,
I have JSON data. On the basis of JSON key value I need to send data to different kinesis stream. So can you suggest me if it's possible

[cosmo0920]

It is possible to implement with built-in placeholders, which is the part of brand new Fluentd v1 API, for https://github.com/awslabs/aws-fluent-plugin-kinesis/blob/master/lib/fluent/plugin/out_kinesis_streams.rb#L49
Currently, it is not implemented.

[cosmo0920]

I've registered a PR to fix this issue.
Does anyone want to try out this?

[simukappu]

Thank you for your contributing.
I think we need to separate buffer for each destination streams for plugin durability. What do you think?

[cosmo0920]

@simukappu Fluentd v1 separates buffer for each destination streams:
https://docs.fluentd.org/v1.0/articles/buffer-section#other-keys

With the testing configuration, Fluentd handles buffer per $.key.nested.

[simukappu]

Sounds good. Does anybody have any feedback for this issue or PR?

[amitdhawan]

Is this done?
I want to use this feature.

[amitdhawan]

Sounds good. Does anybody have any feedback for this issue or PR?

@simukappu Lets merge it.

[cosmo0920]

@amitdhawan Can #174 patch work under your created AWS kinesis environment?
Testing on a few users are required.

[amitdhawan]

Should i install this plugin from this branch on my fluentd config? How can i do that, could you please share the steps.I can test that on my AWs environment then.

[amitdhawan]

Let me know the steps how can install this plugin on Fluentd. I can test it on my environment then.

[cosmo0920]

$ git clone [email protected]:awslabs/aws-fluent-plugin-kinesis.git
$ git fetch origin pull/174/head:pr-174
$ git checkout pr-174
$ gem build fluent-plugin-kinesis.gemspec

And then,

$ td-agent-gem install fluent-plugin-kinesis-*.gem
# Or
$ fluent-gem install fluent-plugin-kinesis-*.gem

[amitdhawan]

ok..let me give it a shot.

[simukappu]

Released as v3.2.0.
https://rubygems.org/gems/fluent-plugin-kinesis/versions/3.2.0

If you have any feedback, feel free to share with us.