Consider removing kinesis connection from 'start'

[wryun]

If kinesis is for some reason inaccessible when this plugin starts, the whole fluentd process crashes (exceptions thrown in start aren't handled, it seems).

In my opinion this is not ideal behaviour, both because there might be other output sources that should keep functioning in the absence of kinesis, AND if kinesis returns fluentd should have been buffering in the meanwhile.

Consider, for instance, the fluentd elasticsearch plugin's approach: https://github.com/uken/fluent-plugin-elasticsearch/blob/master/lib/fluent/plugin/out_elasticsearch.rb
(nothing is done in 'start')

What do you think?

[yuta-imai]

@wryun

Thanks for your idea. I understand your situation. In my opinion, I think we should leave it as is, because if we cannot ensure connection with Kinesis Stream at initialization, we cannot guarantee even if Kinesis Stream exists or not. Please refer to S3 plugin

I know in some case, Kinesis does not respond to initial connection ensuring, however this is not often.

What do you think?

[wryun]

But say kinesis is really unreachable (this happened to us because of a bad network configuration). Don't we still want fluentd to buffer and send logs to other places?

(maybe this is more a problem with fluentd's handling of exception in 'start'...)

[wryun]

(it's also hard to detect failed starts in this situation in upstart, because it takes a long time to crash)

[yuta-imai]

OK, so how about disabling it as configuration option?

[wryun]

That sounds like a great idea. Or even just document this as the failure behaviour (it surprised me).

[yuta-imai]

@wryun

OK! I will do this in next week.

[yuta-imai]

I will do this tonight.

[yuta-imai]

I have left Amazon last week, so I need PR from this time.

[riywo]

Added a new option by #35. Thank you for the PR, @imaifactory !

[wryun]

Thanks very much, @imaifactory and @riywo :)

[yuta-imai]

@riywo thanks for merging :)