time field seems not supporting miliseconds on Elasticsearch

[davidgacc]

I am following this pipeline : kinesis-lambda-ES . The time field seems not supporting miliseconds when I see on kibana:

fluentd : 0.14.19
elasticsearch : 5.3

time : May 17th 2017, 00:00:04.000

timestamp: 2017-05-17 00:00:04,244

This is my conf:

<source>
        @type tail
        path /path/to/log/test_900.log
        pos_file /var/log/td-agent/td-agent-test-117.log
        read_from_head true
        format multiline
        format_firstline /[0-9]+-[0-9]+-[0-9]+/
        format1 /^(?<timestamp>[0-9]+-[0-9]+-[0-9]+\s+[0-9]+:[0-9]+:[0-9]+,[0-9]+)\s+(?<log_level>[a-zA-Z]+)\s+\[(?<class>.*)\]\s+\((?<thread_pool>.*)\)\s+(?<log_message>.*)/
        keep_time_key true
        time_key timestamp
        #read_lines_limit 10000
        time_format %Y-%m-%d %H:%M:%S,%L # %L-> miliseconds ref : http://ruby-doc.org/stdlib-1.9.3/libdoc/date/rdoc/DateTime.html
        tag kinesis-test.log
</source>

## Send Data to Kinesis Stream
<match *.**>
        @type copy
        <store>
                @type kinesis_producer
                region us-east-1
                stream_name stream-activation
                include_time_key true
        </store>
</match>

Any ideas? I would appreciate your help.

Thank you,
David

[riywo]

Hi @davidgacc ,

Thank you for reporting this. I have to dig into this issue. It seems that sub-second information is dropped somewhere inside plugin. Seems related to #87 , but not sure.

[riywo]

I've released v2.0.0.rc.2.0 and it supports nanosecond format time_key with proper format for Elasticsearch. The config could be like below:

<match *>
  @type kinesis_streams

  <inject>
    time_key time
  </inject>
</match>