Error attempting to create a Cassandra cluster using Docker Swarm - docker

Question

I am trying to configure a Cassandra cluster using Docker Swarm.
The containers keep on dying and restarting. Based on logs, it seems to be that they cannot connect to each other.
The cluster does not seem healthy:
root#f6d8d694f8d4:/# nodetool describecluster
Cluster Information:
Name: Test Cluster
Snitch: org.apache.cassandra.locator.SimpleSnitch
DynamicEndPointSnitch: enabled
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Schema versions:
UNREACHABLE: [10.0.8.2, 10.0.2.2]
This is a snippet of logs:
io.netty.channel.ConnectTimeoutException: connection timed out: /10.0.2.2:7000
at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$2.run(AbstractEpollChannel.java:576)
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Unknown Source)
WARN [OptionalTasks:1] 2022-10-19 20:29:55,349 CassandraRoleManager.java:344 - CassandraRoleManager skipped default role setup: some nodes were not ready
Based on the logs, it seems to be that some nodes are not reachable.
This is my Dockerfile, which contains just some tools:
FROM cassandra:4.0.6
RUN apt -y update && apt -y install net-tools && apt install iputils-ping
And this is my Docker compose file:
version: '3.2'
services:
cassandra0:
image: my_cassandra:latest
volumes:
- ./cassandra/0:/var/lib/cassandra
networks:
- cassandra
environment:
CASSANDRA_BROADCAST_ADDRESS: "cassandra0"
HEAP_NEWSIZE: "128M"
MAX_HEAP_SIZE: "512M"
ports:
- 7000
- 9072
- 7199
deploy:
restart_policy:
condition: on-failure
max_attempts: 3
window: 120s
cassandra1:
image: my_cassandra:latest
volumes:
- ./cassandra/1:/var/lib/cassandra
networks:
- cassandra
environment:
CASSANDRA_BROADCAST_ADDRESS: "cassandra1"
CASSANDRA_SEEDS: "cassandra0"
HEAP_NEWSIZE: "128M"
MAX_HEAP_SIZE: "512M"
ports:
- 7000
- 9072
- 7199
deploy:
restart_policy:
condition: on-failure
max_attempts: 3
window: 120s
cassandra2:
image: my_cassandra:latest
volumes:
- ./cassandra/2:/var/lib/cassandra
networks:
- cassandra
environment:
CASSANDRA_BROADCAST_ADDRESS: "cassandra2"
CASSANDRA_SEEDS: "cassandra0"
HEAP_NEWSIZE: "128M"
MAX_HEAP_SIZE: "512M"
ports:
- 7000
- 9072
- 7199
deploy:
restart_policy:
condition: on-failure
max_attempts: 3
window: 120s
cassandra3:
image: my_cassandra:latest
volumes:
- ./cassandra/3:/var/lib/cassandra
networks:
- cassandra
environment:
CASSANDRA_BROADCAST_ADDRESS: "cassandra3"
CASSANDRA_SEEDS: "cassandra0"
HEAP_NEWSIZE: "128M"
MAX_HEAP_SIZE: "512M"
ports:
- 7000
- 9072
- 7199
deploy:
restart_policy:
condition: on-failure
max_attempts: 3
window: 120s
cassandra4:
image: my_cassandra:latest
volumes:
- ./cassandra/4:/var/lib/cassandra
networks:
- cassandra
environment:
CASSANDRA_BROADCAST_ADDRESS: "cassandra4"
CASSANDRA_SEEDS: "cassandra0"
HEAP_NEWSIZE: "128M"
MAX_HEAP_SIZE: "512M"
ports:
- 7000
- 9072
- 7199
deploy:
restart_policy:
condition: on-failure
max_attempts: 3
window: 120s
networks:
cassandra:
external: true
This is my overlay network:
...
"Name": "cassandra",
"Id": "w4ooyvfgyuslnlhl0e9tmyfw2",
"Created": "2022-10-19T22:37:14.60262042+02:00",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.0.10.0/24",
"Gateway": "10.0.10.1"
}
]
}
...
The following Docker compose file works, but I cannot see any relevant difference from the previous one. Only I changed were the containers
and the network names.
version: "3.2"
services:
################################################################
# The Casandra cluster
# - cassandra-node1
################################################################
cassandra-001:
image: my_cassandra
environment:
CASSANDRA_BROADCAST_ADDRESS: "cassandra-001"
HEAP_NEWSIZE: "128M"
MAX_HEAP_SIZE: "1024M"
volumes:
- ./volumes/001:/var/lib/cassandra
deploy:
restart_policy:
condition: on-failure
max_attempts: 3
window: 120s
networks:
- cluster_net
################################################################
# The Casandra cluster
# - cassandra-node2
################################################################
cassandra-002:
image: my_cassandra
environment:
CASSANDRA_BROADCAST_ADDRESS: "cassandra-002"
CASSANDRA_SEEDS: "cassandra-001"
HEAP_NEWSIZE: "128M"
MAX_HEAP_SIZE: "1024M"
volumes:
- ./volumes/002:/var/lib/cassandra
deploy:
restart_policy:
condition: on-failure
max_attempts: 3
window: 120s
networks:
- cluster_net
################################################################
# The Casandra cluster
# - cassandra-node3
################################################################
cassandra-003:
image: my_cassandra
environment:
CASSANDRA_BROADCAST_ADDRESS: "cassandra-003"
CASSANDRA_SEEDS: "cassandra-001"
HEAP_NEWSIZE: "128M"
MAX_HEAP_SIZE: "1024M"
volumes:
- ./volumes/003:/var/lib/cassandra
deploy:
restart_policy:
condition: on-failure
max_attempts: 3
window: 120s
networks:
- cluster_net
################################################################
# The Casandra cluster
# - cassandra-node4
################################################################
cassandra-004:
image: my_cassandra
environment:
CASSANDRA_BROADCAST_ADDRESS: "cassandra-004"
CASSANDRA_SEEDS: "cassandra-001"
HEAP_NEWSIZE: "128M"
MAX_HEAP_SIZE: "1024M"
volumes:
- ./volumes/004:/var/lib/cassandra
deploy:
restart_policy:
condition: on-failure
max_attempts: 3
window: 120s
networks:
- cluster_net
################################################################
# The Casandra cluster
# - cassandra-node5
################################################################
cassandra-005:
image: my_cassandra
environment:
CASSANDRA_BROADCAST_ADDRESS: "cassandra-005"
CASSANDRA_SEEDS: "cassandra-001"
HEAP_NEWSIZE: "128M"
MAX_HEAP_SIZE: "1024M"
volumes:
- ./volumes/005:/var/lib/cassandra
deploy:
restart_policy:
condition: on-failure
max_attempts: 3
window: 120s
networks:
- cluster_net
networks:
cluster_net:
external:
name: cassandra

Answer 1

The error stack you posted indicates that the nodes cannot gossip with each other over the private internode port 7000.
You need to make sure there's network connectivity between the containers and that there is bi-directional route on port 7000. Cheers!