You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
147 lines
5.5 KiB
147 lines
5.5 KiB
= dummy_service
|
|
|
|
|
|
|
|
|
|
|
|
== fluent.conf
|
|
|
|
config file: WriteToMongo/fluent/fluent.conf
|
|
|
|
== edit
|
|
|
|
[source,yaml]
|
|
----
|
|
<match debug.*>
|
|
@type stdout
|
|
</match>
|
|
|
|
<source> // <1>
|
|
@type tail
|
|
|
|
path /var/log/*.log
|
|
path_key tailed_path
|
|
pos_file /tmp/fluentd--1605454018.pos
|
|
pos_file_compaction_interval 30s
|
|
refresh_interval 30s
|
|
skip_refresh_on_startup
|
|
|
|
tag stats.node
|
|
|
|
enable_stat_watcher false
|
|
enable_watch_timer true
|
|
read_from_head true
|
|
follow_inodes true # Without this parameter, file rotation causes log duplication
|
|
|
|
<parse>
|
|
@type none
|
|
|
|
keep_time_key true
|
|
</parse>
|
|
</source>
|
|
|
|
# define the source which will provide log events
|
|
<source> // <1>
|
|
@type tail // <2>
|
|
|
|
path /var/log-in/*/* // <3>
|
|
path_key tailed_path
|
|
pos_file /tmp/fluentd--1605454014.pos // <4>
|
|
pos_file_compaction_interval 10s
|
|
refresh_interval 30s
|
|
skip_refresh_on_startup
|
|
|
|
tag log.node // <5>
|
|
|
|
enable_stat_watcher false // <6>
|
|
enable_watch_timer true // <7>
|
|
read_from_head true
|
|
follow_inodes true # Without this parameter, file rotation causes log duplication
|
|
<parse>
|
|
@type none // <8>
|
|
|
|
keep_time_key true
|
|
</parse>
|
|
</source>
|
|
|
|
## match tag=log.* and write to mongo
|
|
<match log.*> // <9>
|
|
@type copy
|
|
copy_mode deep // <10>
|
|
<store ignore_error> // <11>
|
|
@type mongo // <12>
|
|
|
|
connection_string mongodb://mongo.poc-datacollector_datacollector-net:27017/fluentdb // <13>
|
|
|
|
#database fluentdb
|
|
collection test
|
|
#host mongo.poc-datacollector_datacollector-net
|
|
#port 27017
|
|
|
|
num_retries 60
|
|
capped // <14>
|
|
capped_size 100m
|
|
<inject>
|
|
# key name of timestamp
|
|
time_key time
|
|
</inject>
|
|
|
|
<buffer>
|
|
retry_wait 50s
|
|
flush_mode immediate // <15>
|
|
#flush_interval 10s
|
|
</buffer>
|
|
</store>
|
|
|
|
<store ignore_error>
|
|
@type stdout
|
|
</store>
|
|
|
|
<store ignore_error>
|
|
@type file
|
|
path /tmp/mylog
|
|
<buffer>
|
|
timekey 1d
|
|
timekey_use_utc true
|
|
timekey_wait 10s
|
|
</buffer>
|
|
</store>
|
|
|
|
</match>
|
|
----
|
|
<1> *<source>* directives determine the input sources. The source submits events to the Fluentd routing engine. An event consists of three entities: *tag*, *time* and *record*.
|
|
<2> The *tail Input plugin* allows Fluentd to read events from the tail of text files. Its behavior is similar to the tail -F command. xref:doc-dummy_service.adoc#tailmongo[see image: type tai]
|
|
<3> The *path(s) to read*. Multiple paths can be specified, separated by comma ','. '*' format can be included to add/remove the watch file dynamically. At the interval of refresh_interval, Fluentd refreshes the list of watch files.
|
|
<4> *pos_file* handles multiple positions in one file so no need to have multiple pos_file parameters per source. Don't share *pos_file* between *tail* configurations. It causes unexpected behavior e.g. corrupt pos_file content.
|
|
<5> The *tag* of the event.
|
|
<6> Enables the additional inotify-based watcher. *Either of enable_watch_timer or enable_stat_watcher must be true*
|
|
<7> Enables the additional watch timer. *Either of enable_watch_timer or enable_stat_watcher must be true*
|
|
<8> The *none parser plugin* parses the line as-is with the single field. This format is to defer the parsing/structuring of the data.
|
|
<9> *<match>* directives determine the output destinations. The match directive looks for events with matching tags and processes them. The most common use of the match directive is to *output events to other systems*.
|
|
<10> Chooses how to pass the events to <store> plugins. *deep* copied events to each store plugin. This mode is useful when you modify the nested field after out_copy, e.g. Docker Swarm/Kubernetes related fields.
|
|
<11> Specifies the *storage destinations*. The format is the same as the <match> directive. This section is required at least once.
|
|
<12> The *mongo Output plugin* writes records into , the document-oriented database system.
|
|
<13> The https://docs.mongodb.com/manual/reference/connection-string/#std-label-connections-standard-connection-string-format[MongoDB connection string^] for URI.
|
|
<14> This option enables the *capped collection*. This is always recommended. https://docs.mongodb.com/manual/core/capped-collections/[Capped collections^] are fixed-size collections that support high-throughput operations that insert and retrieve documents based on insertion order. Capped collections work in a way similar to circular buffers: once a collection fills its allocated space, it makes room for new documents by overwriting the oldest documents in the collection.
|
|
<15> *Flushing* Parameters:
|
|
|
|
[#tailmongo]
|
|
.type tail
|
|
image::ROOT:fluent-mongo.png[float=right]
|
|
|
|
|
|
== restart
|
|
|
|
|
|
To make changes become effective a restart is required
|
|
|
|
[source,bash]
|
|
----
|
|
./tools/poc-service-restart-readmongo
|
|
----
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|