I develop a docker swarm of 6 nodes and I trying to add services to them. I developed my swarm and I tried to add services with docker-complose.yml file. The file looks like this.
version: "3"
services:
frontend:
build: .
image: hotel_app_frontend_single_node_memdb
entrypoint: frontend
container_name: 'hotel_app_frontend'
ports:
- "8080:8080"
restart: always
profile:
build: .
image: hotel_app_profile_single_node_memdb
entrypoint: profile
container_name: 'hotel_app_profile'
ports:
- "8081:8081"
restart: always
search:
build: .
image: hotel_app_search_single_node_memdb
entrypoint: search
container_name: 'hotel_app_search'
ports:
- "8082:8082"
restart: always
geo:
build: .
image: hotel_app_geo_single_node_memdb
container_name: 'hotel_app_geo'
entrypoint: geo
ports:
- "8083:8083"
restart: always
rate:
build: .
image: hotel_app_rate_single_node_memdb
container_name: 'hotel_app_rate'
entrypoint: rate
ports:
- "8084:8084"
restart: always
jaeger:
image: jaegertracing/all-in-one:latest
container_name: 'hotel_app_jaeger'
ports:
- "14269"
- "5778:5778"
- "14268:14268"
- "14267"
- "16686:16686"
- "5775:5775/udp"
- "6831:6831/udp"
- "6832:6832/udp"
restart: always
I used this command. docker stack deploy --compose-file docker-compose.yml mydep
When I am trying to see services this appears. Only the jaeger which I pull form web is replicated.
ID NAME MODE REPLICAS IMAGE PORTS
xm3csgvlli9h mydep_frontend replicated 0/1 hotel_app_frontend_single_node_memdb:latest *:8080->8080/tcp
ypaviqz5rrm6 mydep_geo replicated 0/1 hotel_app_geo_single_node_memdb:latest *:8083->8083/tcp
owp2x9p1xltl mydep_jaeger replicated 1/1 jaegertracing/all-in-one:latest *:5778->5778/tcp, *:14268->14268/tcp, *:16686->16686/tcp, *:30000->14269/tcp, *:30001->14267/tcp, *:5775->5775/udp, *:6831-6832->6831-6832/udp
381u6hr6eh5b mydep_profile replicated 0/1 hotel_app_profile_single_node_memdb:latest *:8081->8081/tcp
mbeb8w2r1hq4 mydep_rate replicated 0/1 hotel_app_rate_single_node_memdb:latest *:8084->8084/tcp
hrwyj1v55ao1 mydep_search replicated 0/1 hotel_app_search_single_node_memdb:latest *:8082->8082/tcp
I also try this command journalctl -f -n10
Nov 14 07:35:28 node0.vasilis-139127.ucy-cs499-dc-pg0.wisc.cloudlab.us systemd[6167]: Listening on GnuPG network certificate management daemon.
Nov 14 07:35:28 node0.vasilis-139127.ucy-cs499-dc-pg0.wisc.cloudlab.us systemd[6167]: Listening on GnuPG cryptographic agent (ssh-agent emulation).
Nov 14 07:35:28 node0.vasilis-139127.ucy-cs499-dc-pg0.wisc.cloudlab.us systemd[6167]: Reached target Sockets.
Nov 14 07:35:28 node0.vasilis-139127.ucy-cs499-dc-pg0.wisc.cloudlab.us systemd[6167]: Reached target Paths.
Nov 14 07:35:28 node0.vasilis-139127.ucy-cs499-dc-pg0.wisc.cloudlab.us systemd[6167]: Reached target Timers.
Nov 14 07:35:28 node0.vasilis-139127.ucy-cs499-dc-pg0.wisc.cloudlab.us systemd[6167]: Reached target Basic System.
Nov 14 07:35:28 node0.vasilis-139127.ucy-cs499-dc-pg0.wisc.cloudlab.us systemd[6167]: Reached target Default.
Nov 14 07:35:28 node0.vasilis-139127.ucy-cs499-dc-pg0.wisc.cloudlab.us systemd[6167]: Startup finished in 61ms.
Nov 14 07:39:34 node0.vasilis-139127.ucy-cs499-dc-pg0.wisc.cloudlab.us sshd[6327]: Received disconnect from 194.42.11.22 port 62111:11: disconnected by user
Nov 14 07:39:34 node0.vasilis-139127.ucy-cs499-dc-pg0.wisc.cloudlab.us sshd[6327]: Disconnected from user vasilis 194.42.11.22 port 62111
Related
I am trying to setup one master one slave and one sentinel on docker, for that I wrote this docker compose file.
version: '3'
services:
redis-master:
container_name: "redis-master"
image: redis
ports:
- "6379:6379"
command: "redis-server /etc/redis.conf"
volumes:
- "./data/master:/data/"
- "./master.conf:/etc/redis.conf"
redis-slave:
container_name: "redis-slave"
image: redis
ports:
- "6380:6379"
command: "redis-server /etc/redis.conf"
volumes:
- "./data/slave:/data/"
- "./slave.conf:/etc/redis.conf"
depends_on:
- redis-master
redis-sentinel:
container_name: 'redis-sentinel'
image: redis
ports:
- "26379:26379"
command: >
bash -c "chmod 777 /etc/sentinel.conf
&& redis-server /etc/sentinel.conf --sentinel"
volumes:
- "./sentinel.conf:/etc/sentinel.conf"
depends_on:
- redis-master
- redis-slave
But when I try to build it using sudo docker-compose up --build --force all the services are runnning fine except the redis-sentinel. I got this error in logs
redis-sentinel | 1:X 16 Dec 2021 19:15:21.486 # +sdown master mymaster 172.23.0.2 6379
redis-sentinel | 1:X 16 Dec 2021 19:15:21.486 # +odown master mymaster 172.23.0.2 6379 #quorum 1/1
redis-sentinel | 1:X 16 Dec 2021 19:15:21.486 # +new-epoch 8
redis-sentinel | 1:X 16 Dec 2021 19:15:21.487 # +try-failover master mymaster 172.23.0.2 6379
redis-sentinel | 1:X 16 Dec 2021 19:15:22.955 # Could not rename tmp config file (Device or resource busy)
redis-sentinel | 1:X 16 Dec 2021 19:15:22.955 # WARNING: Sentinel was not able to save the new configuration on disk!!!: Device or resource busy
I understand this is some file permission and I have to make sentinel.conf executable but I am not able to think of any possible solutions in docker.
First of all, the sentinel command should be only like this (we will sort out the permissions later):
redis-server /etc/sentinel.conf --sentinel
In order to resolve that warning, you have to do the following (for all the redis nodes - master, slave, sentinel):
In the folder where the docker-compose.yml resides, create a "config" folder, with the following structure:
/config
/redis-master
redis.conf
/redis-slave
redis.conf
/redis-sentinel
redis.conf
Inside config/redis-master folder you copy your current master.conf file as redis.conf (for the sake of simplicity).
Inside config/redis-slave you copy your current slave.conf file as redis.conf
Inside config/redis-sentinel you will copy your current sentinel.conf as redis.conf
Then, execute this command, in order to give full rights to all the content of your "config" folder:
chmod -R 0777 config/
Now, change your service definitions in the docker-compose, like this:
(notice my changes in the "command" and "volumes" sections)
redis-master:
container_name: "redis-master"
image: redis
ports:
- "6379:6379"
command: "redis-server /etc/redis-config/redis.conf"
volumes:
- "./data/master:/data/"
- "./config/redis-master:/etc/redis-config"
redis-slave:
container_name: "redis-slave"
image: redis
ports:
- "6380:6379"
command: "redis-server /etc/redis-config/redis.conf"
volumes:
- "./data/slave:/data/"
- "./config/redis-slave:/etc/redis-config"
depends_on:
- redis-master
redis-sentinel:
container_name: 'redis-sentinel'
image: redis
ports:
- "26379:26379"
command: "redis-server /etc/redis-config/redis.conf --sentinel"
volumes:
- "./config/redis-sentinel:/etc/redis-config"
depends_on:
- redis-master
- redis-slave
Conclusions:
In order for redis to have rights to modify the configurations, you have mount the entire configuration folder, not the file itself (and the folder contents should be writable).
Back-up all the original master.conf, slave.conf, sentinel.conf files. Be aware that redis will alter your local configuration files (redis.conf), because eventually you will execute failovers, so the slave config will be turned into a master one and the opposite for the master config. It's up to you how you avoid committing the changes to source control. For example, you can keep the originals in a separate folder, and copy them to the mounted folder by a deploy script, before running "docker-compose up" command.
All the changes that redis will apply to these config files will be appended at the end of the config and will be preceded by this comment:
# Generated by CONFIG REWRITE
This solution is tested by me and it works. I came out with this solution after reading #yossigo 's answer regarding a similar warning:
https://github.com/redis/redis/issues/8172
I´ve written the following docker-compose.yml to build a running Docker-Instance of Hazelcast with Mancenter.
version: "3"
services:
hazelcast:
image: hazelcast/hazelcast:3.12.9
container_name: hazelcast
restart: unless-stopped
environment:
MAX_HEAP_SIZE: "512m"
MIN_HEAP_SIZE: "512m"
JAVA_OPTS: "-Dhazelcast.rest.enabled=true
-Dhazelcast.mancenter.enabled=true
-Dhazelcast.mancenter.url=http://localhost:8080/hazelcast-mancenter"
ports:
- "5701:5701"
networks:
- default
hazelcast-management:
image: hazelcast/management-center:3.12.9
container_name: hazelcast-management
restart: unless-stopped
ports:
- "8080:8080"
networks:
- default
networks:
default:
driver: bridge
The log is always showing the following error, even if I use "127.0.0.1" or my IP instead of localhost. I´m using the same version for both: hc and hc-mancenter.
hazelcast | Sep 23, 2020 11:38:35 AM com.hazelcast.internal.management.ManagementCenterService
hazelcast | INFO: [192.168.160.3]:5701 [dev] [3.12.9] Failed to pull tasks from Management Center
hazelcast | Sep 23, 2020 11:38:35 AM com.hazelcast.internal.management.ManagementCenterService
hazelcast | INFO: [192.168.160.3]:5701 [dev] [3.12.9] Failed to connect to: http://localhost:8080/hazelcast-mancenter/collector.do
Regards, Dom
Services in docker-compose are on the same "docker network" and reachable via the service name. When you use localhost or 127.0.0.1 the container tries to communicate with its "own" localhost. So instead of Dhazelcast.mancenter.url=http://localhost:8080 you should connect to Dhazelcast.mancenter.url=http://hazelcast-management:8080. The container of the hazelcast service should have a host entry that points the name hazelcast-management to the correct container ip.
I am using docker-compose with four postgres containers a redis_master container and a redis_slave container. The redis server boots normally but about every 8 hours after launch the slave cannot reach the master for replication. The docker-compose logs show the following error logs from redis_master repeating:
redis_master | 1:S 15 Jul 2020 11:39:05.338 * Connecting to MASTER UNKNOWN.IP:58270
redis_master | 1:S 15 Jul 2020 11:39:05.338 * MASTER REPLICA sync started
redis_master | 1:S 15 Jul 2020 11:39:05.497 # Error condition on socket for SYNC: Connection refused
redis_master | 1:S 15 Jul 2020 11:39:06.341 * Connecting to MASTER UNKNOWN.IP:58270
redis_master | 1:S 15 Jul 2020 11:39:06.341 * MASTER REPLICA sync started
redis_master | 1:S 15 Jul 2020 11:39:06.506 # Error condition on socket for SYNC: Connection refused
The UNKNOWN.IP is not a private IP address, not my server IP nor the IP of any clients accessing the server. That IP address is first used in the logs here:
redis_master | 1:S 15 Jul 2020 11:37:50.127 * REPLICAOF UNKNOWN.IP:58270 enabled (user request from 'id=525 addr=UNKNOWN.IP:35762 fd=13 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=48 qbuf-free=32720 obl=0 oll=0 omem=0 events=r cmd=slaveof')
This seems to trigger an error when trying to interact with the redis server.
WARN [2020-07-15 13:58:42,036] org.eclipse.jetty.server.HttpChannel: /v1/websocket/
! redis.clients.jedis.exceptions.JedisDataException: READONLY You can't write against a read only replica.
The server has also been hit by a crypto mining malware "kdevtmpfsi" which seems to have gotten into the redis server, possibly causing some of these issues. The malware is using about 400% of CPU and 1GB of memory. I have not been able to get rid of it completely and I am trying the process on a new server with more ports closed to try and stop the malware from getting in again. Any advice for stopping the malware from getting in again? Or what might be causing the redis replication issue. My docker-compose config file is taken from this GitHub repo for cloning the Signal app's server. I start the docker environment with
sudo docker-compose up
and I am using all default configurations for redis.
docker-compose.yml
version: '2.2'
services:
signal_account_database:
image: postgres:11
container_name: postgres_account_database
restart: always
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: signal
PGDATA: /var/lib/postgresql/data/pgdata
ports:
- '5431:5432'
volumes:
- ./postgres_database:/var/lib/postgresql/data
signal_keys_database:
image: postgres:11
container_name: postgres_keys_database
restart: always
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: signal
PGDATA: /var/lib/postgresql/data/pgdata
ports:
- '5432:5432'
volumes:
- ./postgres_keys_database:/var/lib/postgresql/data
signal_message_database:
image: postgres:11
container_name: postgres_message_database
restart: always
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: signal
PGDATA: /var/lib/postgresql/data/pgdata
ports:
- '5433:5432'
volumes:
- ./postgres_message_store:/var/lib/postgresql/data
signal_abuse_database:
image: postgres:11
container_name: postgres_abuse_database
restart: always
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: signal
PGDATA: /var/lib/postgresql/data/pgdata
ports:
- '5434:5432'
volumes:
- ./postgres_abuse_database:/var/lib/postgresql/data
redis_main:
image: redis:5
container_name: redis_master
ports:
- '6379:6379'
volumes:
- ./redis_main:/data
redis_replication:
image: redis:5
container_name: redis_slave
command: redis-server --port 6380
ports:
- '6380:6380'
volumes:
- ./redis_replication:/data
Has anyone else had the same replication problem? Looking for advice on fixing this issue.
redis_main:
image: redis:5
container_name: redis_master
restart: always
command: redis-server --port 6379
ports:
- '6379:6379'
volumes:
- ./redis_main:/data
redis_replication:
image: redis:5
container_name: redis_slave
command: redis-server --slaveof 127.0.0.1 6379 --port 6380 --slave-announce-ip 127.0.0.1
ports:
- '6380:6380'
volumes:
- ./redis_replication:/data
sentinel:
build: ./sentinel
container_name: redis_sentinel
restart: always
command: redis-sentinel --port 26379
ports:
- '26379:26379'
environment:
- SENTINEL_NAME=mysentinel
- HOST_IP=127.0.0.1
volumes:
- ./redis_slave:/data
I have a Golang app, and it is supposed to connect to a FTP Server.
Now, both Golang app and FTP Server is dockerized, but I don't know how to connect to FTP server from Golang app
Here is my docker-compose.yml
version: '2'
services:
myappgo:
image: myappgo:exp
volumes:
- ./volume:/go
networks:
myappgo_network:
env_file:
- test.env
ftpd-server:
container_name: ftpd-server
image: stilliard/pure-ftpd:hardened
ports:
- "21:21"
- "30000-30009:30000-30000"
environment:
PUBLICHOST: "localhost"
FTP_USER_NAME: "test"
FTP_USER_PASS: "test"
FTP_USER_HOME: "/home/test"
restart: on-failure
networks:
myappgo_network:
networks:
myappgo_network:
When I run docker compose, all services are up.
I could get IP of ftp container with:
docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' ftpd-server
And then, I installed a ftp client for alpine in my golang container, lftp:
docker exec -it my_app_go sh
apk add lftp
lftp -d ftp://test:test#172.19.0.2 # -d for debug
lftp test#172.19.0.2:~> ls
---- Connecting to 172.19.0.2 (172.19.0.2) port 21
`ls' at 0 [Connecting...]
What am I missing ?
At least, you need 21/TCP for commands and 20/TCP for data on ftp-server:
ports:
- "21:21"
- "20:20"
- "30000-30009:30000-30009"
I changed your compose-file a little bit:
version: '2'
services:
myappgo:
image: alpine:3.8
tty: true
networks:
swarm_default:
ftpd-server:
container_name: ftpd-server
image: stilliard/pure-ftpd:hardened
ports:
- "21:21"
- "20:20"
- "30000-30009:30000-30009"
environment:
PUBLICHOST: "localhost"
FTP_USER_NAME: "test"
FTP_USER_PASS: "test"
FTP_USER_HOME: "/home/test"
restart: on-failure
networks:
swarm_default:
networks:
swarm_default:
Then I created on ftp-server file /home/test/1 and I can see it from mygoapp-container:
/ # lftp ftp://test:test#172.19.0.2
lftp test#172.19.0.2:/> dir
-rw-r--r-- 1 0 0 0 Jan 22 14:18 1
First simplify your dockerfile
version: '3' # i assume you can migrate to version 3, yes?
services:
myappgo:
image: myappgo:exp
volumes:
- ./volume:/go
env_file:
- test.env
ftpd-server:
image: stilliard/pure-ftpd:hardened
environment:
PUBLICHOST: "0.0.0.0"
FTP_USER_NAME: "test"
FTP_USER_PASS: "test"
FTP_USER_HOME: "/home/test"
restart: on-failure
Second, default network is created by docker-compose; no need to do it explicitly. All services get connected to it under their names, so you access them not by ip but by name like ftpd-server
Third, you dont need to expose your ports if you access them from inside. If you need to access them from outside, then you expose.
Next, launch ftp with binding to 0.0.0.0 - binding any tcp service to localhost or 127.0.0.1 makes it accessable only locally.
Last, use service names to connect. Forget about ip addresses and docker inspect. You connection from myappgo to ftp will look like ftp://ftpd-server/foo/bar
I am trying to deploy my working docker-compose set up to a docker-swarm, everything seems ok, except that the only service that got replicate and generate a running container is the redis one, the 3 others got stuck and never generate any running container, they don't even download their respective images.
I can't find any debug feature, all the logs are empty, I'm completely helpless.
Let me show you the current state of my installation.
docker node ls print =>
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
oapl4et92vjp6mv67h2vw8boq boot2docker Ready Active
x2fal9iwj6aqt1vgobyp1smv1 * manager1 Ready Active Leader
lmtuojednaiqmfmm2izl7npf0 worker1 Ready Active
The docker compose =>
version: '3'
services:
mongo:
image: mongo
container_name: mongo
restart: always
volumes:
- /data/db:/data/db
deploy:
placement:
constraints: [node.role == manager]
ports:
- "27017:27017"
redis:
image: redis
container_name: redis
restart: always
bnbkeeper:
image: registry.example.tld/keepers:0.10
container_name: bnbkeeper
deploy:
replicas: 5
resources:
limits:
cpus: "0.1"
memory: 50M
restart_policy:
condition: on-failure
depends_on:
- mongo
- redis
ports:
- "8080:8080"
links:
- mongo
- redis
environment:
- REDIS_HOST=redis
- MONGO_HOST=mongo
bnbkeeper-ws:
image: registry.example.tld/keepers:0.10
container_name: bnbkeeper-ws
restart: unless-stopped
depends_on:
- mongo
- redis
ports:
- "3800:3800"
links:
- mongo
- redis
environment:
- REDIS_HOST=redis
command: npm run start:subscription
The current state of my services
ID NAME MODE REPLICAS IMAGE PORTS
tbwfswsxx23f stack_bnbkeeper replicated 0/5 registry.example.tld/keepers:0.10
khrqtx28qoia stack_bnbkeeper-ws replicated 0/1 registry.example.tld/keepers:0.10
lipa8nvncpxb stack_mongo replicated 0/1 mongo:latest
xtz2411htcg7 stack_redis replicated 1/1 redis:latest
My redis successful service (docker service ps stack_redis)
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
cqv0njcgsw6f stack_redis.1 redis:latest worker1 Running Running 25 minutes ago
my mongo unsuccessful service (docker service ps stack_mongo)
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
yipokxxiftqq stack_mongo.1 mongo:latest Running New 25 minutes ago
I'm completely new to docker swarm, and probably made a silly mistake here, but I couldn't find much documentation on how to setup such a simple stack.
To monitor, try this:
journalctl -f -n10
Then run the docker stack deploy command in a separate session and see what it shows
try removing port publish and add --endpoint-mode dnsrr to your service.