fluent-bit when trying to use OUTPUT gelf getting connection timeout after 10 seconds - graylog

Question

We are trying to get EKS logs to Graylog.
Deployed, Graylog using Helm Charts.
We used MongoDB, Elasticsearch, and Graylog to deploy Graylog. Graylog works fine.
After Graylog was created.
To get EKS logs, we deployed Fluent-bit.
To send logs to Graylog in Fluent-bit configuration
inputs: |
[INPUT]
Name tail
Tag kube.*
Path /var/log/containers/*.log
DB /var/log/flb_graylog.db
Parser docker
Docker_Mode On
Mem_Buf_Limit 50MB
Skip_Long_Lines On
Refresh_Interval 10
Key log
[INPUT]
Name systemd
Tag host.*
Systemd_Filter _SYSTEMD_UNIT=kubelet.service
Read_From_Tail On
## https://docs.fluentbit.io/manual/pipeline/filters
filters: |
[FILTER]
Name kubernetes
Match kube.*
Merge_Log On
Merge_Log_Key log_processed
Keep_Log Off
K8S-Logging.Parser On
K8S-Logging.Exclude Off
Annotations Off
Labels On
[FILTER]
Name nest
Match *
Operation lift
Nested_under kubernetes
## https://docs.fluentbit.io/manual/pipeline/outputs
outputs: |
[OUTPUT]
Name es
Match kube.*
Host elasticsearch-master
Port 9200
Logstash_Format On
Retry_Limit Off
Replace_Dots On
[OUTPUT]
Name es
Match host.*
Host elasticsearch-master
Port 9200
Logstash_Format On
Logstash_Prefix node
Retry_Limit Off
Replace_Dots On
[OUTPUT]
Name gelf
Match *
Host graylog.example.com
Port 12201
Mode tcp
Gelf_Short_Message_Key short_message
[OUTPUT]
Name syslog
Match *
Host graylog.example.com
Port 541
Mode udp
Syslog_Format rfc5424
Syslog_Maxsize 2048
Syslog_Severity_Key severity
Syslog_Facility_Key facility
Syslog_Sd_Key sd
Syslog_Message_key message
## https://docs.fluentbit.io/manual/pipeline/parsers
customParsers: |
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
# Command | Decoder | Field | Optional Action
# =============|==================|=================
Decode_Field_As escaped log
[PARSER]
Name syslog
Format regex
Regex ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
Time_Key time
Time_Format %b %d %H:%M:%S
In fluent-bit logs, I am getting the error
[2022/08/02 06:50:18] [error] [upstream] connection #141 to graylog.example.com:12201 timed out after 10 seconds
[2022/08/02 06:50:18] [error] [upstream] connection #126 to graylog.example.com:12201 timed out after 10 seconds
[2022/08/02 06:50:18] [error] [upstream] connection #143 to graylog.example.com:12201 timed out after 10 seconds
[2022/08/02 06:50:18] [error] [upstream] connection #125 to graylog.example.com:12201 timed out after 10 seconds
[2022/08/02 06:50:18] [error] [upstream] connection #144 to graylog.example.com:12201 timed out after 10 seconds
[2022/08/02 06:50:18] [error] [upstream] connection #139 to graylog.example.com:12201 timed out after 10 seconds
[2022/08/02 06:50:18] [error] [output:gelf:gelf.2] no upstream connections available
[2022/08/02 06:50:18] [error] [output:gelf:gelf.2] no upstream connections available
[2022/08/02 06:50:18] [error] [output:gelf:gelf.2] no upstream connections available
[2022/08/02 06:50:18] [error] [output:gelf:gelf.2] no upstream connections available
[2022/08/02 06:50:18] [error] [output:gelf:gelf.2] no upstream connections available
[2022/08/02 06:50:18] [error] [output:gelf:gelf.2] no upstream connections available
[2022/08/02 06:50:18] [ warn] [engine] chunk '1-1659422985.239025920.flb' cannot be retried: task_id=108, input=tail.0 > output=gelf.2
[2022/08/02 06:50:18] [ warn] [engine] chunk '1-1659422988.238308295.flb' cannot be retried: task_id=15, input=tail.0 > output=gelf.2
[2022/08/02 06:50:18] [ warn] [engine] chunk '1-1659422988.801295849.flb' cannot be retried: task_id=116, input=tail.0 > output=gelf.2
[2022/08/02 06:50:18] [ warn] [engine] failed to flush chunk '1-1659423006.238302940.flb', retry in 11 seconds: task_id=46, input=tail.0 > output=gelf.2 (out_id=2)
[2022/08/02 06:50:18] [ warn] [engine] chunk '1-1659422989.738179384.flb' cannot be retried: task_id=105, input=tail.0 > output=gelf.2
[2022/08/02 06:50:18] [ warn] [engine] failed to flush chunk '1-1659423007.739931411.flb', retry in 10 seconds: task_id=56, input=tail.0 > output=gelf.2 (out_id=2)