Fluentd hide password or encrypt in configuration - ruby-on-rails

Because of security reasons, we can't keep SQL authentication in plain text, is there a way to hide or encrypt passwords?
<source>
#type sql
#id output_sql
host "sqlserverhost.aws_region.rds.amazonaws.com"
database db_name
adapter sqlserver
username user
password pwd
tag_prefix myrdb # optional, but recommended
select_interval 60s # optional
select_limit 500 # optional
state_file /var/run/fluentd/sql_state
<table>
table tbl_name
update_column insert_timestamp
</table>
</source>
<match **>
#type stdout
</match>

Related

Fluentd - Not able to connect to SQL server using windows authentication

I'm able to successfully connect to SQL server using SQL SERVER AUTHENTICATION however it does not work with WINDOWS AUTHENTICATION, is it a bug, or I'm missing something in the configuration?
<source>
#type sql
host HOSTNAME
database db_name
adapter sqlserver
username WindowsUser
password WindowsPwd
<table>
table tbl_name
update_column insert_timestamp
</table>
</source>
<match **>
#type stdout
</match>
I get below error:
[warn]: #0 failed to flush the buffer. retry_time=1 next_retry_seconds=2021-09-01 22:12:40 238620126384680326147/703687441776640000000 +0530 chunk="5caf1c0f1dfbb6d0ca989ce4ffd28fa3" error_class=TinyTds::Error error="Adaptive Server connection failed (localhost)
The issue is resolved, make sure to add a schema name with the table name.

Fluentd configuration for ORACLE clob datatype

I'm using fluentd configuration to read data from a text file and it is pushed to the Oracle database, I have clob & nclob type of datatype when fluentd pushed data, the column is always null and I don't see any errors. I'm not sure how to resolve this issue in fluentd, below is the configuration that I have done.
I 'm using oracle enhanced adapter & sql plugin
https://github.com/rsim/oracle-enhanced
https://github.com/fluent/fluent-plugin-sql/issues
#
# Fluentd configuration file
#
# Config input
<source>
#type forward
port 24224
</source>
# Config output
<match cpu_*>
#type stdout
</match>
<match foo_*>
#type stdout
</match>
<match memory_*>
#type sql
host {DATABASE_HOSTNAME}
port 1521
database {DATABASE_NAME}
adapter oracle_enhanced
username {DATABASE_USERNAME}
password {DATABASE_PASSWORD}
<table>
table fluentd_log
column_mapping 'timestamp:created_at,Mem.text:mem_text,Mem.used:mem_used'
# This is the default table because it has no "pattern" argument in <table>
# The logic is such that if all non-default <table> blocks
# do not match, the default one is chosen.
# The default table is required.
</table>
</match>
CREATE TABLE FLUENTD_LOG
(
ID NUMBER(8),
CREATED_AT VARCHAR2(50 BYTE),
MEM_TEXT CLOB,
MEM_USED VARCHAR2(50 BYTE)
)
ID CREATED_AT MEM_TEXT MEMUSED
1 29-08-99 null test

Fluentd sql output plugin configuration for auto incremented column

I have a fluentd configuration that pulls data from the file and pushes to the SQL server however there is a primary key with the auto-incremented column, so, in my fluentd configuration if I don't mention that column it throws an error saying that the field is missing and if I include the column in the configuration it gives identity error, in below configuration "Id" is the primary and auto-incremented column, also let me know if adapter "sqlserver" is the right thing to use.
<filter record.**>
#type record_transformer
enable_ruby true
<record>
Id ${id}
</record>
<record>
timestamp ${time}
</record>
</filter>
<filter record.**>
#type stdout
</filter>
<match record.**>
#type sql
host myhost
username myuser
password mypassword
database mydb
adapter sqlserver
<table>
table simple_table
column_mapping 'Id:Id,timestamp:timestamp'
</table>
flush_interval 1s
# disable_retry_limit
# num_threads 8
# slow_flush_log_threshold 40.0
</match>
Well, I figured this out, it's mandatory to send the column name in the column_mapping even though if its primary key and auto-incremented, if you login with some other SQL credential it will give you an error, however, if you login with the same details used at the time of table creation it works.

How to inject `time` attribute based on certain json key value?

I am still new on fluentd, I've tried various configuration, but I am stuck.
Suppose I have this record pushed to fluend that has _epoch to tell the epoch time the record is created.
{"data":"dummy", "_epoch": <epochtime_in_second>}
Instead of using time attribute being processed by fluentd, I want to override the time with this _epoch field. How to produce fluentd output with time overriden?
I've tried this
# TCP input to receive logs from the forwarders
<source>
#type forward
bind 0.0.0.0
port 24224
</source>
# HTTP input for the liveness and readiness probes
<source>
#type http
bind 0.0.0.0
port 9880
</source>
# rds2fluentd_test
<filter rds2fluentd_test.*>
#type parser
key_name _epoch
reserve_data true
<parse>
#type regexp
expression /^(?<time>.*)$/
time_type unixtime
utc true
</parse>
</filter>
<filter rds2fluentd_test.*>
#type stdout
</filter>
<match rds2fluentd_test.*>
#type s3
#log_level debug
aws_key_id "#{ENV['AWS_ACCESS_KEY']}"
aws_sec_key "#{ENV['AWS_SECRET_KEY']}"
s3_bucket foo-bucket
s3_region ap-southeast-1
path ingestion-test-01/${_db}/${_table}/%Y-%m-%d-%H-%M/
#s3_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}
# if you want to use ${tag} or %Y/%m/%d/ like syntax in path / s3_object_key_format,
# need to specify tag for ${tag} and time for %Y/%m/%d in <buffer> argument.
<buffer time,_db,_table>
#type file
path /var/log/fluent/s3
timekey 1m # 5 minutes partition
timekey_wait 10s
timekey_use_utc true # use utc
chunk_limit_size 256m
</buffer>
time_slice_format %Y%m%d%H
store_as json
</match>
But upon receiving data like above, it shows warning error like this:
#0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="parse failed no implicit conversion of Integer into Hash" location="/usr/local/bundle/gems/fluentd-1.10.4/lib/fluent/plugin/filter_parser.rb:110:in `rescue in filter_with_time'" tag="rds2fluentd_test." time=1590578507 record={....
was getting the same warning message, setting hash_value_field parsed under filter section solved the issue.

FluentD configuration to index on all the fields in the log in elastic

Hi I have the below log from springboot microservice. What to create a index on all the below fields like timestamp, level, logger etc in elastic. How to achieve this in fluentd configuration? Tried the below and it didnt work
Log
timestamp:2020-04-27 09:37:56.996 level:INFO level_value:20000 thread:http-nio-8080-exec-2 logger:com.scb.nexus.service.phoenix.components.ApplicationEventListener context:default message:org.springframework.web.context.support.ServletRequestHandledEvent traceId:a122e51aa3d24d4a spanId:a122e51aa3d24d4a spanExportable:false X-Span-Export:false X-B3-SpanId:a122e51aa3d24d4a X-B3-TraceId:a122e51aa3d24d4a
fluentd conf
<match **>
#type elasticsearch
time_as_integer true
include_timestamp true
host host
port 9200
user userName
password password
scheme https
ssl_verify false
ssl_version TLSv1_2
index_name testIndex
</match>
<filter **>
#type parser
key_name log
reserve_data true
<parse>
#type json
</parse>
</filter>
Logs are not in JSON format, therefore you cant use the Json parser. You have the following options to solve this issue
1- use regex parser as described here https://docs.fluentd.org/parser/regexp
2- use record_reformer plugin and extract items manually
example:
<match **>
#type record_reformer
tag parsed.${tag_suffix[2]}
renew_record false
enable_ruby true
<record>
timestamp ${record['log'].scan(/timestamp:(?<param>[^ ]+ [^ ]+)/).flatten.compact.sort.first}
log_level ${record['log'].scan(/level:(?<param>[^ ]+)/).flatten.compact.sort.first}
level_value ${record['log'].scan(/level_value:(?<param>[^ ]+)/).flatten.compact.sort.first}
</record>
</match>
<match parsed.**>
#type elasticsearch
time_as_integer true
include_timestamp true
host host
port 9200
user userName
password password
scheme https
ssl_verify false
ssl_version TLSv1_2
index_name testIndex
</match>

Resources