Plugin putting one record at a time to Kinesis

[uskudnik]

I am experimenting with putting our data to Kinesis with fluentd but as it seems this plug-in is putting one record at a time to the kinesis and waiting for the response, basically making it useless for high traffic data volumes.

Nothing too fancy in my fluentd.conf:

<source>
    type tail
    path var/spool/tracking-logs/%Y-%m-%d_%H-%M.events
    pos_file var/spool/tracking-logs/events.log.pos
    tag kinesis.eventtracker
    read_from_head true
    format json
    refresh_interval 5
</source>

# Kinesis
<match kinesis.eventtracker>
type kinesis

stream_name eventStream

aws_key_id ID
aws_sec_key KEY

region us-east-1
debug true
#partition_key sessionId
partition_key timestamp
</match>

Basically, in a 5 minute interval roughly 800 records are put into stream, making throughput something like 3 records per second, way below 1000 RpS that a single Kinesis shard is able to process. Switching debug mode on/off doesn't affect performance. Is there a way to fire up more events to the kinesis/make it perform async calls?

Any hints of how to improve performance or is this an underlying problem related to AWS SDK for Ruby?

[jasonmcmunn]

Have you tried passing in num_threads > 1? The library itself isn't multithreaded, but it looks like the buffered output plugin is designed to be concurrent.

[yuta-imai]

Hi, we have released Version0.2.0 which supports DetachMultiProcessMixin. Also you can use num_threads. You can find how we leverage it on our README.

0e50b32