Redis cluster in docker-swarm mode updated recipe - docker

Question

Anyone has a working recipe of Redis cluster in swarm mode? I tried everything I know and searched the internet but seems like an impossible task.
Here is what I have so far:
version: '3.4'
services:
redis-master:
image: redis
networks:
- redisdb
ports:
- 6379:6379
volumes:
- redis-master:/data
redis-slave:
image: redis
networks:
- redisdb
command: redis-server --slaveof redis-master 6379
volumes:
- redis-slave:/data
sentinel:
image: redis
networks:
- redisdb
ports:
- 26379:26379
command: >
bash -c "echo 'port 26379' > sentinel.conf &&
echo 'dir /tmp' >> sentinel.conf &&
echo 'sentinel monitor redis-master redis-master 6379 2' >> sentinel.conf &&
echo 'sentinel down-after-milliseconds redis-master 5000' >> sentinel.conf &&
echo 'sentinel parallel-syncs redis-master 1' >> sentinel.conf &&
echo 'sentinel failover-timeout redis-master 5000' >> sentinel.conf &&
cat sentinel.conf &&
redis-server sentinel.conf --sentinel"
links:
- redis-master
- redis-slave
volumes:
redis-master:
driver: local
redis-slave:
driver: local
networks:
redisdb:
attachable: true
driver: overlay
I use the following command to deploy as a service:
docker stack deploy --compose-file docker-compose-test.yml redis
The result is deployed services, were redis-master and redis-slave are connecting and I can see the synchronization processes happening as follows:
redis-master log:
1:C 16 Oct 2019 04:19:42.720 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 16 Oct 2019 04:19:42.720 # Redis version=5.0.6, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 16 Oct 2019 04:19:42.720 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
1:M 16 Oct 2019 04:19:42.723 * Running mode=standalone, port=6379.
1:M 16 Oct 2019 04:19:42.723 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:M 16 Oct 2019 04:19:42.723 # Server initialized
1:M 16 Oct 2019 04:19:42.723 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
1:M 16 Oct 2019 04:19:42.723 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
1:M 16 Oct 2019 04:19:42.723 * Ready to accept connections
1:M 16 Oct 2019 04:19:43.976 * Replica 10.0.27.2:6379 asks for synchronization
1:M 16 Oct 2019 04:19:43.976 * Full resync requested by replica 10.0.27.2:6379
1:M 16 Oct 2019 04:19:43.976 * Starting BGSAVE for SYNC with target: disk
1:M 16 Oct 2019 04:19:43.976 * Background saving started by pid 15
15:C 16 Oct 2019 04:19:43.982 * DB saved on disk
15:C 16 Oct 2019 04:19:43.982 * RDB: 0 MB of memory used by copy-on-write
1:M 16 Oct 2019 04:19:44.053 * Background saving terminated with success
1:M 16 Oct 2019 04:19:44.053 * Synchronization with replica 10.0.27.2:6379 succeeded
Redis-slave log:
1:C 16 Oct 2019 04:19:40.776 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 16 Oct 2019 04:19:40.776 # Redis version=5.0.6, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 16 Oct 2019 04:19:40.776 # Configuration loaded
1:S 16 Oct 2019 04:19:40.779 * Running mode=standalone, port=6379.
1:S 16 Oct 2019 04:19:40.779 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:S 16 Oct 2019 04:19:40.779 # Server initialized
1:S 16 Oct 2019 04:19:40.779 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
1:S 16 Oct 2019 04:19:40.779 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
1:S 16 Oct 2019 04:19:40.779 * Ready to accept connections
1:S 16 Oct 2019 04:19:40.779 * Connecting to MASTER redis-master:6379
1:S 16 Oct 2019 04:19:40.817 # Unable to connect to MASTER: Invalid argument
1:S 16 Oct 2019 04:19:41.834 * Connecting to MASTER redis-master:6379
1:S 16 Oct 2019 04:19:41.851 # Unable to connect to MASTER: Invalid argument
1:S 16 Oct 2019 04:19:42.866 * Connecting to MASTER redis-master:6379
1:S 16 Oct 2019 04:19:42.942 # Unable to connect to MASTER: Invalid argument
1:S 16 Oct 2019 04:19:43.970 * Connecting to MASTER redis-master:6379
1:S 16 Oct 2019 04:19:43.975 * MASTER <-> REPLICA sync started
1:S 16 Oct 2019 04:19:43.975 * Non blocking connect for SYNC fired the event.
1:S 16 Oct 2019 04:19:43.975 * Master replied to PING, replication can continue...
1:S 16 Oct 2019 04:19:43.976 * Partial resynchronization not possible (no cached master)
1:S 16 Oct 2019 04:19:43.977 * Full resync from master: 39bb36f74ef0cdefdc08a2dc8d4a86112ea69f12:0
1:S 16 Oct 2019 04:19:44.053 * MASTER <-> REPLICA sync: receiving 175 bytes from master
1:S 16 Oct 2019 04:19:44.054 * MASTER <-> REPLICA sync: Flushing old data
1:S 16 Oct 2019 04:19:44.054 * MASTER <-> REPLICA sync: Loading DB in memory
1:S 16 Oct 2019 04:19:44.054 * MASTER <-> REPLICA sync: Finished with success
Redis-sentinel log:
port 26379
port 26379
dir /tmp
dir /tmp
sentinel monitor redis-master redis-master 6379 2
sentinel monitor redis-master redis-master 6379 2
sentinel down-after-milliseconds redis-master 5000
sentinel down-after-milliseconds redis-master 5000
sentinel parallel-syncs redis-master 1
sentinel parallel-syncs redis-master 1
sentinel failover-timeout redis-master 5000
sentinel failover-timeout redis-master 5000
1:X 16 Oct 2019 04:19:49.506 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
*** FATAL CONFIG FILE ERROR ***
1:X 16 Oct 2019 04:19:49.506 # Redis version=5.0.6, bits=64, commit=00000000, modified=0, pid=1, just started
1:X 16 Oct 2019 04:19:49.506 # Configuration loaded
Reading the configuration file, at line 3
>>> 'sentinel monitor redis-master redis-master 6379 2'
1:X 16 Oct 2019 04:19:49.508 * Running mode=sentinel, port=26379.
1:X 16 Oct 2019 04:19:49.508 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
Can't resolve master instance hostname.
1:X 16 Oct 2019 04:19:49.511 # Sentinel ID is aad9553d4999951f3b37eede5968b4aa262c07a9
1:X 16 Oct 2019 04:19:49.511 # +monitor master redis-master 10.0.27.5 6379 quorum 2
1:X 16 Oct 2019 04:19:49.512 * +slave slave 10.0.27.2:6379 10.0.27.2 6379 # redis-master 10.0.27.5 6379
1:X 16 Oct 2019 04:19:54.514 # +sdown slave 10.0.27.2:6379 10.0.27.2 6379 # redis-master 10.0.27.5 6379
So the slave-announce-ip, replica-announce-ip etc are having issues:
1- because the ips keep changing
2- Overlay networks ips are not always what they show they are
So failover dose not work and if master went down slave dont kick in!