Monitoring Elassandra with Prometheus using Docker - docker

I'm trying to run jmx-exporter container with special configuration following this instruction.
I did all the instructions step-by-step. Here is my docker-compose file and conf/config.yml.
docker-compose:
version: '2'
services:
prometheus:
image: prom/prometheus
container_name: prometheus
volumes:
- /prometheus:/prometheus
- ./conf/prome.yml:/etc/prometheus/prometheus.yml
command:
- --config.file=/etc/prometheus/prometheus.yml
ports:
- '6009:9090'
jmx-exporter:
image: sscaling/jmx-prometheus-exporter
container_name: jmx-exporter
ports:
- 6011:5556
depends_on:
- elassandra
elassandra:
image: strapdata/elassandra
container_name: elassandra
volumes:
- /var/lib/cassandra:/var/lib/cassandra
environment:
- CASSANDRA_LISTEN_ADDRESS=localhost
- CASSANDRA_RPC_ADDRESS=localhost
- CASSANDRA_START_RPC=false
- CASSANDRA_CLUSTER_NAME='DockerTest Cluster'
- CASSANDRA_NUM_TOKENS=8
- CASSANDRA#
- LOCAL_JMX="no"
- Dcassandra.jmx.remote.port="7199"
- Dcom.sun.management.jmxremote.port="7199"
- Dcom.sun.management.jmxremote.rmi.port="7199"
- Dcom.sun.management.jmxremote.authenticate=false
- JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=127.0.0.1"
- JMX_PORT="7199"
- JVM_OPTS="$JVM_OPTS -Dcassandra.jmx.remote.port=5556"
- JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.rmi.port=5556"
- JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=false"
volumes:
prometheus_data: {}
elassandra_data: {}
conf/config.yml
startDelaySeconds: 0
hostPort: elassandra:7199
#username:
#password:
#jmxUrl: service:jmx:rmi:///jndi/rmi://127.0.0.1:60/jmxrmi
ssl: false
lowercaseOutputName: false
lowercaseOutputLabelNames: false
whitelistObjectNames: ["org.apache.cassandra.metrics:*"]
blacklistObjectNames: ["org.apache.cassandra.metrics:type=ColumnFamily,*"]
rules:
- pattern: 'org.apache.cassandra.metrics<type=(\w+), name=(\w+)><>Value: (\d+)'
name: cassandra_$1_$2
value: $3
valueFactor: 0.001
labels: {}
help: "Cassandra metric $1 $2"
type: GAUGE
attrNameSnakeCase: false
whenever I run docker-compose -f docker-compose.yml up, I face with the following page on localhost:6011:
The problem is that I don't know how to monitor elassandra and its metrics using jmx-exporter, I mainly get the java metrics!

In your config.yml you're using hostPort: localhost:5556 but you should try to connect to the host elassandra (the name of your docker container in your docker-compose.yml file).
Make sure Elassandra is configured properly to allow remote JMX connections. See How to enable remote JMX connections in Elassandra? for details how to set that up.

Related

Docker uses an undefined network

Hi Stackoverflow fellows,
I am facing an issue while running docker-compose up. Whereas docker-compose runs the jenkins locally. This complete docker file is as follows.
version: '2.3'
services:
jenkins:
container_name: jenkins
build: ./master
image: jenkins_casc
environment:
- CASC_JENKINS_CONFIG=/var/jenkins_casc/jenkins.yaml
- SECRETS=/var/jenkins_casc/secrets
ports:
- "8080:8080"
volumes:
- jenkins_master_home:/var/jenkins_home
jenkins_slave_docker:
container_name: jenkins_agent_docker
build: ./agent
image: jenkins_agent_docker
init: true
environment:
- JENKINS_AGENT_SSH_PUBKEY=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0xJ5n9MY0PFBR/aCHSb8JBQgbIUo0C/bPlaxM9v0uCT2CQJvNyrHUfJKaM9wJsdT7wdKBUIvhODdfoE7kc59j0WpO5TQ5Q2MeG7fpQAalM0ATwv/o7hCTvWev5gpJPSsIg9N/+VusO2R4V1H7LpZm65hHL/0lt9SmvtZzQBR+lt5IhrliEMZpo1UdNql/ueR6Em3mFW/tJvprBD445xTa0kxACGXdMI3nF2+SF49oXhTPjNFKSJilWDsoWzf9swyIf1vbH6zr3slMm7jUvOSCC3gGcqNrSG9Y3wkBzqUDe20CjbeAHMq490xlkGQeg9BAByTvn9uOU7ym3mMUnkKR
- DOCKER_CERT_PATH=/certs/client
- DOCKER_HOST=tcp://docker:2376
- DOCKER_TLS_VERIFY=1
restart: on-failure
depends_on:
- jenkins
volumes:
- jenkins-docker-certs:/certs/client:ro
- jenkins_slave_docker_workdir:/home/jenkins:z
- jenkins_slave_docker:/home/jenkins/.jenkins
docker:
container_name: docker
networks:
- harbor
image: docker:dind
command: ["--insecure-registry=proxy:8080"]
environment:
- DOCKER_TLS_CERTDIR=/certs
volumes:
- jenkins-docker-certs:/certs/client
- jenkins_slave_docker_workdir:/home/jenkins:z
privileged: true
volumes:
jenkins_master_home:
jenkins_slave_docker:
jenkins-docker-certs:
jenkins_slave_docker_workdir:
Whereas the error is as follows:
ERROR: Service "docker" uses an undefined network "harbor"
Everything is correct!
You need to define harbor network in your docker-compose file. It may be just "simple" bridge and docker-compose will create this network automatically on your behalf or you can define it as "external" network in case it already exists.
networks:
harbor:
external:
name: harbor

multiple docker compose files with traefik (v2.1) and database networks

I would like to build a docker landscape. I use a container with a traefik (v2. 1) image and a mysql container for multiple databases.
traefik/docker-compose.yml
version: "3.3"
services:
traefik:
image: "traefik:v2.1"
container_name: "traefik"
restart: always
command:
- "--log.level=DEBUG"
- "--api=true"
- "--api.dashboard=true"
- "--providers.docker=true"
- "--providers.docker.exposedbydefault=false"
- "--providers.docker.network=proxy"
- "--entrypoints.web.address=:80"
- "--entrypoints.websecure.address=:443"
- "--entrypoints.traefik-dashboard.address=:8080"
- "--certificatesresolvers.devnik-resolver.acme.httpchallenge=true"
- "--certificatesresolvers.devnik-resolver.acme.httpchallenge.entrypoint=web"
#- "--certificatesresolvers.devnik-resolver.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
- "--certificatesresolvers.devnik-resolver.acme.email=####"
- "--certificatesresolvers.devnik-resolver.acme.storage=/letsencrypt/acme.json"
ports:
- "80:80"
- "443:443"
- "8080:8080"
volumes:
- "./letsencrypt:/letsencrypt"
- "./data:/etc/traefik"
- "/var/run/docker.sock:/var/run/docker.sock:ro"
networks:
- "proxy"
labels:
- "traefik.enable=true"
- "traefik.http.routers.traefik.rule=Host(`devnik.dev`)"
- "traefik.http.routers.traefik.entrypoints=traefik-dashboard"
- "traefik.http.routers.traefik.tls.certresolver=devnik-resolver"
#basic auth
- "traefik.http.routers.traefik.service=api#internal"
- "traefik.http.routers.traefik.middlewares=auth"
- "traefik.http.middlewares.auth.basicauth.usersfile=/etc/traefik/.htpasswd"
#Docker Networks
networks:
proxy:
database/docker-compose.yml
version: "3.3"
services:
#MySQL Service
mysql:
image: mysql:5.7
container_name: mysql
restart: always
ports:
- "3306:3306"
volumes:
#persist data
- ./mysqldata/:/var/lib/mysql/
- ./init:/docker-entrypoint-initdb.d
networks:
- "mysql"
environment:
MYSQL_ROOT_PASSWORD: ####
TZ: Europe/Berlin
#Docker Networks
networks:
mysql:
driver: bridge
For the structure I want to control all projects via multiple docker-compose files. These containers should run on the same network as the traefik container and some with the mysql container.
This also works for the following case (but only sometimes)
dev-releases/docker-compose.yml
version: "3.3"
services:
backend:
image: "registry.gitlab.com/devnik/dev-releases-backend/master:latest"
container_name: "dev-releases-backend"
restart: always
volumes:
#laravel logs
- "./logs/backend:/app/storage/logs"
#cron logs
- "./logs/backend/cron.log:/var/log/cron.log"
labels:
- "traefik.enable=true"
- "traefik.http.routers.dev-releases-backend.rule=Host(`dev-releases.backend.devnik.dev`)"
- "traefik.http.routers.dev-releases-backend.entrypoints=websecure"
- "traefik.http.routers.dev-releases-backend.tls.certresolver=devnik-resolver"
networks:
- proxy
- mysql
environment:
TZ: Europe/Berlin
#Docker Networks
networks:
proxy:
external:
name: "traefik_proxy"
mysql:
external:
name: "database_mysql"
As soon as I restart the containers in dev-releases/ via docker-compose up -d I get the typical error "Gateway timeout" when calling them in the browser.
As soon as I comment the network networks: #- mysql and restart the docker-compose in dev-releases/ it works again.
My guess is that I have not configured the external networks correctly. Is it not possible to use 2 external networks?
I'd like some container have access to the 'mysql' network but it should not be accessible for the whole traefik network.
Let me know if you need more information
EDIT (26.03.2020)
I make it running.
I put all my containers into one network "proxy". It seems mysql also have to be in the proxy network.
So I add following to database/docker-compose.yml
networks:
proxy:
external:
name: "traefik_proxy"
And removed the database_mysql network out of dev-releases/docker-compose.yml
based on the names of the files, your mysql network should be mysql_mysql.
you can verify this by executing
$> docker network ls
You are also missing a couple of labels for your services such as
traefik command line
- '--providers.docker.watch=true'
- '--providers.docker.swarmMode=true'
labels
- traefik.docker.network=proxy
- traefik.http.services.dev-releases-backend.loadbalancer.server.port=yourport
- traefik.http.routers.dev-releases-backend.service=mailcatcher
You can check this for more info

how to setup kibana user credentials with docker elk stack

How to setup login credentials for kibana gui with docker elk stack containers.
What arguments and environmental variables must be passed in docker-compose.yaml file to get this working.
For setting kibana user credentials for docker elk stack, we have to set xpack.security.enabled: true either in elasticsearch.yml or pass this as a environment variable in docker-compose.yml file.
Pass username & password as environment variable in docker-compose.yml like below:
version: '3.3'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:6.6.1
ports:
- "9200:9200"
- "9300:9300"
configs:
- source: elastic_config
target: /usr/share/elasticsearch/config/elasticsearch.yml
environment:
ES_JAVA_OPTS: "-Xmx256m -Xms256m"
ELASTIC_USERNAME: "elastic"
ELASTIC_PASSWORD: "MyPw123"
http.cors.enabled: "true"
http.cors.allow-origin: "*"
xpack.security.enabled: "true"
networks:
- elk
deploy:
mode: replicated
replicas: 1
logstash:
image: docker.elastic.co/logstash/logstash:6.6.1
ports:
- "5044:5044"
- "9600:9600"
configs:
- source: logstash_config
target: /usr/share/logstash/config/logstash.yml:rw
- source: logstash_pipeline
target: /usr/share/logstash/pipeline/logstash.conf
environment:
LS_JAVA_OPTS: "-Xmx256m -Xms256m"
xpack.monitoring.elasticsearch.url: "elasticsearch:9200"
xpack.monitoring.elasticsearch.username: "elastic"
xpack.monitoring.elasticsearch.password: "MyPw123"
networks:
- elk
deploy:
mode: replicated
replicas: 1
kibana:
image: docker.elastic.co/kibana/kibana:6.6.1
ports:
- "5601:5601"
configs:
- source: kibana_config
target: /usr/share/kibana/config/kibana.yml
networks:
- elk
deploy:
mode: replicated
replicas: 1
configs:
elastic_config:
file: ./elasticsearch/config/elasticsearch.yml
logstash_config:
file: ./logstash/config/logstash.yml
logstash_pipeline:
file: ./logstash/pipeline/logstash.conf
kibana_config:
file: ./kibana/config/kibana.yml
networks:
elk:
driver: overlay
Then add this following lines to kibana.yml:
elasticsearch.username: "elastic"
elasticsearch.password: "MyPw123"
Did not managed to get it working without adding XPACK_MONITORING & SECURITY flags to kibana's container and there was no need for a config file
However I was not able to use kibana user, even after logging in with elastic user and changing kibana's password through the UI.
NOTE: looks like you can't setup default built-in users other than elastic superuser in docker-compose through it's environment. I've tried several times with kibana and kibana_system to no success.
version: "3.7"
services:
elasticsearch:
image: elasticsearch:7.4.0
restart: always
ports:
- 9200:9200
environment:
- discovery.type=single-node
- xpack.security.enabled=true
- ELASTIC_PASSWORD=123456
kibana:
image: kibana:7.4.0
restart: always
ports:
- 5601:5601
environment:
- ELASTICSEARCH_HOSTS=http://elasticsearch:9200
- XPACK_MONITORING_ENABLED=true
- XPACK_MONITORING_COLLECTION_ENABLED=true
- XPACK_SECURITY_ENABLED=true
- ELASTICSEARCH_USERNAME=elastic
- ELASTICSEARCH_PASSWORD="123456"
depends_on:
- elasticsearch
SOURCE
NOTE: looks like this won't work with 8.5.3, Kibana won't accept superuser elastic.
Update
I was able to setup 8.5.3 but with a couple twists. I would build the whole environment, then in elastic's container run the setup-passwords auto
bin/elasticsearch-setup-passwords auto
Grab the auto generated password for kibana_system user and replace it in docker-compose then restart only kibana's container
Kibana 8.5.3 with environment variables:
kibana:
image: kibana:8.5.3
restart: always
ports:
- 5601:5601
environment:
- ELASTICSEARCH_USERNAME="kibana_system"
- ELASTICSEARCH_PASSWORD="sVUurmsWYEwnliUxp3pX"
Restart kibana's container:
docker-compose up -d --build --force-recreate --no-deps kibana
NOTE: make sure to use --no-deps flag otherwise it will restart elastic container if tagged to kibana's

Collect tomcat logs from tomcat docker container to Filebeat docker container

I have a Tomcat docker container and Filebeat docker container both are up and running.
My objective: I need to collect tomcat logs from running Tomcat container to Filebeat container.
Issue: I have no idea how to get collected log files from Tomcat container.
What I have tried so far: I have tried to create a docker volume and add tomcat logs to that volume and access that volume from filebeat container, but ended with no success.
Structure: I have wrote docker-compose.yml file under project Logstash(root directory of the project) with following project structure.(Here I want to up and run Elasticsearch, Logstash, Filebeat and Kibana docker containers from one configuration file). docker-containers(root directory of the project) with following structure (here I want to up and run Tomcat, Nginx and Postgres containers from one configuration file).
Logstash: contain 4 main sub directories (Filebeat, Logstash, Elasticsearch and Kibana), ENV file and docker-compose.yml file. Both sub directories contain Dockerfiles to pull images and build the containers.
docker-containers: contains 3 main sub directories (Tomcat, Nginx and Postgres). ENV file and docker-compose.yml file. Both sub directories contain separate Dockerfiles to pull docker image and build the container.
Note: I think this basic structure my helpful to understand my requirements.
docker-compose.yml files
Logstash.docker-compose.yml file
version: '2'
services:
elasticsearch:
container_name: OTP-Elasticsearch
build:
context: ./elasticsearch
args:
- ELK_VERSION=${ELK_VERSION}
volumes:
- ./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro
ports:
- "9200:9200"
- "9300:9300"
environment:
ES_JAVA_OPTS: "-Xmx256m -Xms256m"
networks:
- elk
filebeat:
container_name: OTP-Filebeat
command:
- "-e"
- "--strict.perms=false"
user: root
build:
context: ./filebeat
args:
- ELK_VERSION=${ELK_VERSION}
volumes:
- ./filebeat/config/filebeat.yml:/usr/share/filebeat/filebeat.yml
environment:
LS_JAVA_OPTS: "-Xmx256m -Xms256m"
networks:
- elk
depends_on:
- elasticsearch
- logstash
logstash:
container_name: OTP-Logstash
build:
context: ./logstash
args:
- ELK_VERSION=${ELK_VERSION}
volumes:
- ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro
- ./logstash/pipeline:/usr/share/logstash/pipeline:ro
expose:
- 5044/tcp
ports:
- "9600:9600"
- "5044:5044"
environment:
LS_JAVA_OPTS: "-Xmx256m -Xms256m"
networks:
- elk
links:
- elasticsearch
depends_on:
- elasticsearch
kibana:
container_name: OTP-Kibana
build:
context: ./kibana
args:
- ELK_VERSION=${ELK_VERSION}
volumes:
- ./kibana/config/:/usr/share/kibana/config:ro
ports:
- "5601:5601"
networks:
- elk
links:
- elasticsearch
depends_on:
- elasticsearch
- logstash
- filebeat
networks:
elk:
driver: bridge
docker-containers.docker-compose.yml file
version: '2'
services:
# Nginx
nginx:
container_name: OTP-Nginx
restart: always
build:
context: ./nginx
args:
- comapanycode=${COMPANY_CODE}
- dbtype=${DB_TYPE}
- dbip=${DB_IP}
- dbname=${DB_NAME}
- dbuser=${DB_USER}
- dbpassword=${DB_PASSWORD}
- webdirectory=${WEB_DIRECTORY}
ports:
- "80:80"
links:
- db:db
volumes:
- ./log/nginx:/var/log/nginx
depends_on:
- db
# Postgres
db:
container_name: OTP-Postgres
restart: always
ports:
- "5430:5430"
build:
context: ./postgres
args:
- food_db_version=${FOOD_DB_VERSION}
- dbtype=${DB_TYPE}
- retail_db_version=${RETAIL_DB_VERSION}
- dbname=${DB_NAME}
- dbuser=${DB_USER}
- dbpassword=${DB_PASSWORD}
volumes:
- .data/db:/octopus_docker/postgresql/data
# Tomcat
tomcat:
container_name: OTP-Tomcat
restart: always
build:
context: ./tomcat
args:
- dbuser=${DB_USER}
- dbpassword=${DB_PASSWORD}
links:
- db:db
volumes:
- ./tomcat/${WARNAME}.war:/usr/local/tomcat/webapps/${WARNAME}.war
ports:
- "8080:8080"
depends_on:
- db
- nginx
Additional files:
filebeat.yml (configuration file inside Logstash/Filbeat/config/)
filebeat.inputs:
- type: log
enabled: true
paths:
- /usr/local/tomcat/logs/.*log
output.logstash:
hosts: ["logstash:5044"]
Additional Info:
System I am using is Ubuntu 18.04
My goal is to collect tomcat logs from running tomcat container and forward them to Logstash and filter logs and forward that logs to Elasticsearch and finally to Kibana for Visualization purpose.
For now I can collect local machine(host) logs and visualize them in Kibana.(/var/log/)
My Problem:
I need to know proper way to get collected tomcat logs from tomcat container and forward them to logstash container via filebeat container.
Any discussion, answer or any help to understand a way to do this is highly expected.
Thanks.
So loooong... Create shared volume among all containers and setup your tomcat to save log files into that folder. If you can put all services into one docker-compose.yml, just setup volume internally:
docker-compose.yml
version: '3'
services:
one:
...
volumes:
- logs:/var/log/shared
two:
...
volumes:
- logs:/var/log/shared
volumes:
logs:
If you need several docker-compose.yml files, create volume globally in advance with docker volume create logs and map it into both compose files:
version: '3'
services:
one:
...
volumes:
- logs:/var/log/shared
two:
...
volumes:
- logs:/var/log/shared
volumes:
logs:
external: true

Traefik with Docker Networks

I really don't get how to use traefik with docker networks.
I try to run "wekan" kanban. If I bind ports to host, it works perfectly, so it really is about adressing it through traefik. Here is my docker-config:
version: '2'
services:
wekandb:
image: mongo:3.2.14
container_name: wekan-db
command: mongod --smallfiles --oplogSize 128
networks:
- wekan-tier
expose:
- 27017
volumes:
- wekan-db:/data/db
- wekan-db-dump:/dump
wekan:
image: wekanteam/wekan:latest
container_name: wekan-app
networks:
- wekan-tier
# ports:
# - 8081:80
environment:
- MONGO_URL=mongodb://wekandb:27017/wekan
- ROOT_URL=https://wekan.domain.com
depends_on:
- wekandb
labels:
- "traefik.port=80"
- "traefik.backend=wekan"
- "traefik.frontend.rule=Host:wekan.domain.com"
- "traefik.docker.network=wekan_wekan-tier"
volumes:
wekan-db:
driver: local
wekan-db-dump:
driver: local
networks:
wekan-tier:
driver: bridge
I can't seem to find a way to access the damn thing... Your answer will be greatly appreciated, not only will it allow me to run Wekan, but also to update my older services where I used linking:linking instead of Docker Networks - Linking being now deprecated.
I believe you have more than one issue here.
First, in your compose you don't have Traefik service, it is OK, Traefik will be able to see containers from the services here, but Traefik will not be able to send the request to it, because Traefik service and wekan service does not share the same network.
So to fix that you need to create an specific network to Traefik and set it in your compose file also.
Example:
$ docker network create traefik-net
$ docker service --name traefik --network traefik-net .... traefik ....
Second, you need to define the network Traefik will use to connect with your service, this network must be one shared with Traefik service.
So your wekan service needs to be like this:
wekan:
image: wekanteam/wekan:latest
container_name: wekan-app
networks:
- wekan-tier
- traefik-net
environment:
- MONGO_URL=mongodb://wekandb:27017/wekan
- ROOT_URL=https://wekan.domain.com
depends_on:
- wekandb
labels:
- "traefik.port=80"
- "traefik.backend=wekan"
- "traefik.frontend.rule=Host:wekan.domain.com"
- "traefik.docker.network=traefik-net"
I have modified your docker-compose file to make it work:
version: '3'
services:
web:
image: wekanteam/wekan:latest
networks:
- wekan-tier
environment:
- MONGO_URL=mongodb://wekandb:27017/wekan
- ROOT_URL=https://wekan.domain.com
labels:
- "traefik.port=80"
- "traefik.docker.network=wekan_wekan-tier"
wekandb:
image: mongo:3.2
command: mongod --smallfiles --oplogSize 128
networks:
- wekan-tier
expose:
- 27017
volumes:
- wekan-db:/data/db
- wekan-db-dump:/dump
traefik:
image: 'traefik:1.6'
command: --web --docker --docker.watch --docker.domain=local --logLevel=DEBUG
labels:
- traefik.docker.network=wekan-tier
- traefik.port=8080
ports:
- '80:80'
- '8080:8080'
volumes:
- '/var/run/docker.sock:/var/run/docker.sock'
networks:
- wekan-tier
volumes:
wekan-db:
driver: local
wekan-db-dump:
driver: local
networks:
wekan-tier:
driver: bridge
Now start containers with the following command:
$ docker-compose -p wekan up -d
To check that traefik is working go to http://localhost:8080/, if you have problems stop your apache server using $ service apache2 stop. On the other hand, if you can see the traefik interface then add the following line to your /etc/hosts file:
127.0.0.1 web.wekan.local
Now go to http://web.wekan.local and you should see Wekan login page :)

Resources