Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Derive stream name from fluentd tag #67

Closed
cbroglie opened this issue May 12, 2016 · 2 comments · Fixed by #68
Closed

Derive stream name from fluentd tag #67

cbroglie opened this issue May 12, 2016 · 2 comments · Fixed by #68
Labels
feature requests
Milestone
v1.1.0

Comments

@cbroglie
Copy link
Contributor

cbroglie commented May 12, 2016 •
edited
Loading

We have an application where we would like to route data to different Kinesis streams based on the fluentd tag. This is doable by specifying a different <match></match> configuration for each tag, but it means that there will be a separate KPL process for each configuration. And if we have 20 different streams, that means 20 instances of the KPL process, which wastes resources.

Since the KPL requires the stream name to be specified with each record, the fluentd plugin can dynamically determine the stream name for each record it processes. One possible way to do this would be to allow users to specify a stream name prefix, and a regex to extract other data from the tag. A simpler (and more efficient) alternative would be a simple concatenation.

I'm happy to provide a PR for this functionality, if there is interest in accepting it.
cbroglie pushed a commit to cbroglie/aws-fluent-plugin-kinesis that referenced this issue May 14, 2016
Support deriving stream name from tag
4bb945d 
Add support for kinesis_producer to derive the stream name by
concatenating stream_name_prefix (new configuration option) and the
fluentd tag. Fixes awslabs#67.
This was referenced May 14, 2016
Merged
Merged
cbroglie pushed a commit to cbroglie/aws-fluent-plugin-kinesis that referenced this issue May 17, 2016
Support deriving stream name from tag
5018f6d 
Add support for kinesis_producer to derive the stream name by
concatenating stream_name_prefix (new configuration option) and the
fluentd tag. Fixes awslabs#67.
cbroglie pushed a commit to cbroglie/aws-fluent-plugin-kinesis that referenced this issue May 19, 2016
Support deriving stream name from tag
34a9ca7 
Add support for kinesis_producer to derive the stream name by
concatenating stream_name_prefix (new configuration option) and the
fluentd tag. Fixes awslabs#67.
@riywo riywo added this to the v1.1.0 milestone Jun 2, 2016
@riywo riywo closed this as completed in #68 Jun 3, 2016
@riywo
Copy link
Contributor

riywo commented Jun 13, 2016

Hi @cbroglie ,

I've released v1.1.0 and it contains this contribution. Thank you for your PR!
https://rubygems.org/gems/fluent-plugin-kinesis/versions/1.1.0

@cbroglie
Copy link
Contributor Author

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature requests
Projects
None yet
Milestone
v1.1.0
Development

Successfully merging a pull request may close this issue.

Support deriving stream name from tag cbroglie/aws-fluent-plugin-kinesis
2 participants
@riywo @cbroglie