I have a docker-compose.yml in which I have mention logging driver as fluentd.
But I want logging driver to be enabled or disabled with a flag.(That means whenever I need, I can enable the logging driver or can disable completly)
Also I want to make IP address and port configurable. How can I change them whenever I need?
srvc-name:
image: <DOCKER-REISTRY-PATH:IMAGE-NAME>
environment:
- "JAVA_OPTS=-XX:MaxRAM=4g"
logging:
driver: "fluentd"
options:
fluentd-address: localhost:24224
fluentd-async-connect: "true"
tag: tag.name
volumes:
- app-name-config:/usr/local/tomcat/conf/app1
- app-name-logs:/usr/local/tomcat/logs
Related
Tracing makes finding parts in code, worthwhile a developers time and attention, much easier. For that reason, I attached Jaeger as tracer to a set of microservices inside Docker containers. I use Traefik as ingress controller/ service-mesh to route and proxy requests.
The problem I am facing is, that something's wrong with the tracing config in Traefik. Jaeger can not find the span context to connect the single/ service-dependend spans to a whole trace.
The following line appears in the logs:
{
"level":"debug",
"middlewareName":"tracing",
"middlewareType":"TracingEntryPoint",
"msg":"Failed to extract the context: opentracing: SpanContext not found in Extract carrier",
"time":"2021-02-02T23:16:51+01:00"
}
What I tried/ searched/ confirmed so far:
I already checked ports (they are open inside the Docker host network) and everything's reachable. So interconnectivity is not the problem here.
The forwarding of headers is set via Docker Compose labels: loadbalancer.passhostheader=true.
The following snippets describe the Docker Compose setup.
Traefik: Ingress Controller
This is a stripped down version of the traefik Container.
# Network
ROOT_DOMAIN=example.test
DEFAULT_NETWORK=traefik
---
version: '3'
services:
image: "traefik:2.4.2"
hostname: "controller"
restart: on-failure
security_opt:
- no-new-privileges:true
ports:
- "443:443"
- "80:80"
# The Web UI (enabled by --api.insecure=true)
- "8080:8080"
- "8082:8082"
- "8083:8083"
networks:
- default
working_dir: /etc/traefik
volumes:
- /private/etc/localtime:/etc/localtime:ro
- ${PWD}/controller/static.yml:/etc/traefik/traefik.yml:ro
- ${PWD}/controller/dynamic.yml:/etc/traefik/dynamic.yml:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
- cert-storage:/usr/local/share/ca-certificates:ro
- ${PWD}/logs/traefik:/var/log/traefik
volumes:
cert-storage:
driver_opts:
type: none
o: bind
device: ${PWD}/certs/certs
networks:
default:
external: true
name: ${DEFAULT_NETWORK}
Traefik is set up using the file provider as base and Docker Compose labels on top of it:
# static.yaml (Traefik conf)
debug: true
log:
level: DEBUG
filePath: /var/log/traefik/error.log
format: json
serversTransport:
insecureSkipVerify: true
api:
dashboard: true
insecure: true
debug: true
providers:
docker:
exposedByDefault: false
swarmMode: false
watch: true
defaultRule: "Host(`{{ normalize .Name }}.example.test`)"
endpoint: "unix:///var/run/docker.sock"
network: traefik
file:
filename: /etc/traefik/dynamic.yml
watch: true
tracing:
serviceName: "controller"
spanNameLimit: 250
jaeger:
samplingType: const
samplingParam: 1.0
samplingServerURL: http://tracer:5778/sampling
localAgentHostPort: 127.0.0.1:6831
gen128Bit: true
propagation: jaeger
traceContextHeaderName: "traefik-trace-id"
collector:
endpoint: http://tracer:14268/api/traces?format=jaeger.thrift
Jaeger: Open Tracing/ Open Telemetry
---
version: '3'
services:
tracer:
image: "jaegertracing/all-in-one:1.21.0"
hostname: "tracer"
command:
- "--log-level=info"
- "--admin.http.host-port=:14269"
- "--query.ui-config=/usr/local/share/jaeger/ui/conf.json"
environment:
SPAN_STORAGE_TYPE: memory
restart: on-failure
security_opt:
- no-new-privileges:true
expose:
- 5775/udp
- 6831/udp
- 6832/udp
- 5778
- 14250
- 14268
- 14269
- 14271
- 16686
- 16687
volumes:
- /private/etc/localtime:/etc/localtime:ro
- ${PWD}/tracer/conf:/usr/local/share/jaeger
- ${PWD}/logs/jaeger:/var/log/#TODO
- cert-storage:/usr/local/share/ca-certificates
networks:
- default
labels:
- "traefik.enable=true"
- "traefik.docker.network=${DEFAULT_NETWORK}"
# Admin UI router
- "traefik.http.routers.tracer-router.rule=Host(`tracer.$ROOT_DOMAIN`)"
- "traefik.http.routers.tracer-router.entrypoints=https"
- "traefik.http.routers.tracer-router.tls=true"
- "traefik.http.routers.tracer-router.tls.options=default"
- "traefik.http.routers.tracer-router.service=tracer"
# Service/ Load Balancer
- "traefik.http.services.tracer.loadbalancer.passhostheader=true"
- "traefik.http.services.tracer.loadbalancer.server.port=16686"
- "traefik.http.services.tracer.loadbalancer.server.scheme=http"
I'm not 100% sure what the problem is you're experiencing, but here's some things to consider.
According to this post on the Traefik forums, that message you're seeing is debug level because it's not something you should be worried about. It's just logging that no trace context was found, so a new one will be created. That second part is not in the message, but apparently that's what happens.
You should check to see if you're getting data appearing in Jaeger. If you are, that message is probably nothing to worry.
If you are getting data in Jaeger, but it's not connected, that will be because Traefik can only only work with trace context that is already in inbound requests, but it can't add trace context to outbound requests. Within your application, you'll need to implement trace propagation so that your outbound requests include the trace context that was received as part of the incoming request. Without doing that, every request will be sent without trace context and will start a new trace when it is received at the next Traefik ingress point.
The problem actually was with the traceContextHeaderName. Sadly I can not tell exactly what the problem was as the git diff only shows that nothing changed around traefik and jaeger at the point where I fixed it. I assume config got "stuck" somehow. I tracked down the related lines in source, but as I am no Go-Dev, I can only guess if there's a bug.
What I did was to switch back to uber-trace-id, which magically "fixed" it. After I ran some traces and connected another service (node, npm jaeger-client with process.env.TRACER_STATE_HEADER_NAME set to an equal value), I switched back to traefik-trace-id and things worked.
I'm using a K3S Cluster in a docker(-compose) container in my CI/CD pipeline, to test my application code. However I have problem with the certificate of the cluster. I need to communicate on the cluster using the external addres. My docker-compose script looks as follows
version: '3'
services:
server:
image: rancher/k3s:v0.8.1
command: server --disable-agent
environment:
- K3S_CLUSTER_SECRET=somethingtotallyrandom
- K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml
- K3S_KUBECONFIG_MODE=666
volumes:
- k3s-server:/var/lib/rancher/k3s
# get the kubeconfig file
- .:/output
ports:
# - 6443:6443
- 6080:6080
- 192.168.2.110:6443:6443
node:
image: rancher/k3s:v0.8.1
tmpfs:
- /run
- /var/run
privileged: true
environment:
- K3S_URL=https://server:6443
- K3S_CLUSTER_SECRET=somethingtotallyrandom
ports:
- 31000-32000:31000-32000
volumes:
k3s-server: {}
accessing the cluster from python gives me
MaxRetryError: HTTPSConnectionPool(host='192.168.2.110', port=6443): Max retries exceeded with url: /apis/batch/v1/namespaces/mlflow/jobs?pretty=True (Caused by SSLError(SSLCertVerificationError("hostname '192.168.2.110' doesn't match either of 'localhost', '172.19.0.2', '10.43.0.1', '172.23.0.2', '172.18.0.2', '172.23.0.3', '127.0.0.1', '0.0.0.0', '172.18.0.3', '172.20.0.2'")))
Here are my two (three) question
how can I add additional IP adresses to the cert generation? I was hoping the --bind-address in the server command triggers taht
how can I fall back on http providing an --http-listen-port didn't achieve the expected result
any other suggestion how I can enable communication with the cluster
changing the python code is not really an option as I would like o keep the code unaltered for testing. (Fallback on http works via kubeconfig.
The solution is to use the parameter tls-san
server --disable-agent --tls-san 192.168.2.110
I am trying to capture syslog messages sent over the network using rsyslog, and then have rsyslog capture, transform and send these messages to elasticsearch.
I found a nice article on the configuration on https://www.reddit.com/r/devops/comments/9g1nts/rsyslog_elasticsearch_logging/
Problem is that rsyslog keeps popping up an error at startup that it cannot connect to Elasticsearch on the same machine on port 9200. Error I get is
Failed to connect to localhost port 9200: Connection refused
2020-03-20T12:57:51.610444+00:00 53fd9e2560d9 rsyslogd: [origin software="rsyslogd" swVersion="8.36.0" x-pid="1" x-info="http://www.rsyslog.com"] start
rsyslogd: omelasticsearch: we are suspending ourselfs due to server failure 7: Failed to connect to localhost port 9200: Connection refused [v8.36.0 try http://www.rsyslog.com/e/2007 ]
Anyone can help on this?
Everything is running in docker on a single machine. I use below docker compose file to start the stack.
version: "3"
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:7.6.1
environment:
- discovery.type=single-node
- xpack.security.enabled=false
ports:
- 9200:9200
networks:
- logging-network
kibana:
image: docker.elastic.co/kibana/kibana:7.6.1
depends_on:
- logstash
ports:
- 5601:5601
networks:
- logging-network
rsyslog:
image: rsyslog/syslog_appliance_alpine:8.36.0-3.7
environment:
- TZ=UTC
- xpack.security.enabled=false
ports:
- 514:514/tcp
- 514:514/udp
volumes:
- ./rsyslog.conf:/etc/rsyslog.conf:ro
- rsyslog-work:/work
- rsyslog-logs:/logs
volumes:
rsyslog-work:
rsyslog-logs:
networks:
logging-network:
driver: bridge
rsyslog.conf file below:
global(processInternalMessages="on")
#module(load="imtcp" StreamDriver.AuthMode="anon" StreamDriver.Mode="1")
module(load="impstats") # config.enabled=`echo $ENABLE_STATISTICS`)
module(load="imrelp")
module(load="imptcp")
module(load="imudp" TimeRequery="500")
module(load="omstdout")
module(load="omelasticsearch")
module(load="mmjsonparse")
module(load="mmutf8fix")
input(type="imptcp" port="514")
input(type="imudp" port="514")
input(type="imrelp" port="1601")
# includes done explicitely
include(file="/etc/rsyslog.conf.d/log_to_logsene.conf" config.enabled=`echo $ENABLE_LOGSENE`)
include(file="/etc/rsyslog.conf.d/log_to_files.conf" config.enabled=`echo $ENABLE_LOGFILES`)
#try to parse a structured log
action(type="mmjsonparse")
# this is for index names to be like: rsyslog-YYYY.MM.DD
template(name="rsyslog-index" type="string" string="rsyslog-%$YEAR%.%$MONTH%.%$DAY%")
# this is for formatting our syslog in JSON with #timestamp
template(name="json-syslog" type="list") {
constant(value="{")
constant(value="\"#timestamp\":\"") property(name="timereported" dateFormat="rfc3339")
constant(value="\",\"host\":\"") property(name="hostname")
constant(value="\",\"severity\":\"") property(name="syslogseverity-text")
constant(value="\",\"facility\":\"") property(name="syslogfacility-text")
constant(value="\",\"program\":\"") property(name="programname")
constant(value="\",\"tag\":\"") property(name="syslogtag" format="json")
constant(value="\",") property(name="$!all-json" position.from="2")
# closing brace is in all-json
}
# this is where we actually send the logs to Elasticsearch (localhost:9200 by default)
action(type="omelasticsearch" template="json-syslog" searchIndex="rsyslog-index" dynSearchIndex="on")
#################### default ruleset begins ####################
# we emit our own messages to docker console:
syslog.* :omstdout:
include(file="/config/droprules.conf" mode="optional") # this permits the user to easily drop unwanted messages
action(name="main_utf8fix" type="mmutf8fix" replacementChar="?")
include(text=`echo $CNF_CALL_LOG_TO_LOGFILES`)
include(text=`echo $CNF_CALL_LOG_TO_LOGSENE`)
First of all you need to run all the containers on the same docker network which in this case are not. Second , after running the containers on the same network , login to rsyslog container and check if 9200 is available.
I am using EFK stack to build a monitoring system. According to the Docker Logging Driver, I can add customized tag to enrich the metadata of the container log.
Here is my docker-compose file:
version: "3.3"
services:
watcher:
image: image_name
deploy:
replicas: 1
placement:
constraints: [node.role == manager]
logging:
driver: fluentd
options:
tag: "docker/{{.ImageName}}"
networks:
- elastic
Here is my Fluent-bit configuration:
[SERVICE]
Flush 5
Daemon Off
Log_Level debug
Parsers_File /conf/parsers.conf
[INPUT]
Name Forward
Port 24224
[OUTPUT]
Name es
Match *
Host elasticsearch
Port 9200
Index fluent_bit
Type json
As you can see, I have already add the tag: "docker/{{.ImageName}}" to the docker-compose file. And the container was restarted as well. The log I got in Kibana should include such a tag. But here is the log I got:
#timestamp:Mar 18, 2020 # 15:35:23.000 container_id:06dde90cb998c78962e321c8396c1f992119450a6868eecb7fa14c5b348670b1 container_name:/test_container source:stderr log:2020-03-18 14:35:23 - INFO - module: __main__ - action: Watcher is started - Watcher Start _id:RmcS7nABi-qh6YwdCII3 _type:json _index:fluent_bit _score: -
There are still only the container name and container id in the metadata and nothing more. Can anybody tell me what could be the reason for this?
I just found where does the problem come from.
When you add a tag option to the log-driver, it will not be automatically included into the fluent-bit/fluentd output. The Include_tag_key should also be set to true in the output section.
[OUTPUT]
Name es
Match *
Host elasticsearch
Port 9200
Index fluent_bit
Type json
Include_Tag_Key true
I am setting up a docker-compose with several services, all of them writing to a common syslog container / service... which actually is a logstash service (a complete elk image as a matter of fact) with the logstash-input-syslog plugin enabled..
kind of as follows (using custom 5151 port since default 514 was giving me a hard time due to permission issues):
services:
elk-custom:
image: some_elk_image
ports:
- 5601:5601
- 9200:9200
- 5044:5044
- 5151:5151
service1:
image: myservice1_image
logging:
driver: syslog
options:
syslog-address: "tcp://127.0.0.1:5151"
service2:
image: myservice2_image
logging:
driver: syslog
options:
syslog-address: "tcp://127.0.0.1:5151"
My question is how can I add a field (an option rather under logging) so that each log entry in logstash ends up with a field, whose value will help determine whether the log came from service1 or service2.
I kind of managed to do this using the tag field, but the information ends up being part of the message, and not a separate field which I can use for queries in elasticsearch.
For the time being, kibana displays log entries as follows:
#timestamp:September 26th 2017, 12:00:47.684 syslog_severity_code:5
port:53,422 syslog_facility:user-level #version:1 host:172.18.0.1
syslog_facility_code:1 message:<27>Sep 26 12:00:47 7705a2f9b22a[2128]:
[pid: 94|app: 0|req: 4/7] 172.18.0.1 () {40 vars in 461 bytes} [Tue
Sep 26 09:00:47 2017] GET /api/v1/apikeys => generated 74 bytes in 5
msecs (HTTP/1.1 401) 2 headers in 81 bytes (1 switches on core 0)
type:syslog syslog_severity:notice tags:_grokparsefailure
_id:AV69atD4zBS_tKzDPfyh _type:syslog _index:logstash-2017.09.26 _score: -
From what I also know, we cannot define custom syslog-facilities since they are predefined by the syslog RFC.
Thx.
Ended up using port multiplexing and adding custom field based on this condition:
docker-compose.yml
elk-custom:
image: some_elk_image
ports:
- 5601:5601
- 9200:9200
- 5044:5044
- 5151:5151
- 5152:5152
service1:
image: myservice2_image
logging:
driver: syslog
options:
syslog-address: "tcp://127.0.0.1:5151"
service2:
image: myservice2_image
logging:
driver: syslog
options:
syslog-address: "tcp://127.0.0.1:5152"
logstash-conf
input {
tcp {
port => 5151
type => syslog
add_field => {'received_from' => 'service1'}
}
tcp {
port => 5152
type => syslog
add_field => {'received_from' => 'service2'}
}
}