I launched a cassandra docker image using the following command,
docker run --name=n1 -d cassandra:3.11.4
344138bb96e5326113af6ba5c44a17d8b40ed710fe36ce063677a56ef0c0117
This works fine,
docker exec -it n1 nodetool status gives,
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 172.17.0.2 103.7 KiB 256 100.0% 0ca2907d-49b9-4ed4-8bf3-6db99d0c8b45 rack1
Now when I try to launch a second node, this what I get,
docker inspect -f '{{ .NetworkSettings.IPAddress}}' n1
172.17.0.2
docker run --name n2 -d cassandra:3.11.4 -seeds 172.17.0.2
1c89b67cb61b049c4a8ae0f21bd0f7cddffad02b7e7964e502cd82e0d2ff2866
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
344138bb96e5 cassandra:3.11.4 "docker-entrypoint.s…" 2 minutes ago Up 2 minutes 7000-7001/tcp, 7199/tcp, 9042/tcp, 9160/tcp n1
docker exec -it n1 nodetool status
Still no second node
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 172.17.0.2 103.7 KiB 256 100.0% 0ca2907d-49b9-4ed4-8bf3-6db99d0c8b45 rack1
docker exec -it n2 nodetool status
Error response from daemon: Container 1c89b67cb61b049c4a8ae0f21bd0f7cddffad02b7e7964e502cd82e0d2ff2866 is not running
docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1c89b67cb61b cassandra:3.11.4 "docker-entrypoint.s…" 2 minutes ago Exited (1) 2 minutes ago n2
344138bb96e5 cassandra:3.11.4 "docker-entrypoint.s…" 4 minutes ago Up 4 minutes 7000-7001/tcp, 7199/tcp, 9042/tcp, 9160/tcp n1
What am I doing wrong here?
I don't know where you get the information which you can use -seeds 172.17.0.2 to specify seed in this docker image, in fact check the log of n2, it said:
getopt: invalid option -- 's'
getopt: invalid option -- 'e'
getopt: invalid option -- 'e'
getopt: invalid option -- 'd'
getopt: invalid option -- 's'
Error parsing arguments! Unknown argument "172.17.0.2"
And from it's doc, CASSANDRA_SEEDS env is the correct way, like next:
docker run --name n2 -d -e CASSANDRA_SEEDS=172.17.0.2 cassandra:3.11.4
Results:
shubuntu1#shubuntu1:~$ docker run --name n1 -d cassandra:3.11.4
1d1d95f0caeff471c409cd8eda2e77aa5d9b3a60831c284937c305e823016885
shubuntu1#shubuntu1:~$ docker exec -it n1 nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 172.17.0.2 112.85 KiB 256 100.0% c6117826-2fff-48a2-964b-ba403e972154 rack1
shubuntu1#shubuntu1:~$ docker run --name n2 -d -e CASSANDRA_SEEDS=172.17.0.2 cassandra:3.11.4
37e5b84ab92b8b9087e6cff37b146b1f7e337ec9499ce907408abeba11f1985c
Wait a few time, then check again, we find the cluster has 2 nodes now:
shubuntu1#shubuntu1:~$ docker exec -it n1 nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 172.17.0.3 112.98 KiB 256 100.0% e61857d3-b924-4fe9-9a68-03d603de7c38 rack1
UN 172.17.0.2 108.62 KiB 256 100.0% c6117826-2fff-48a2-964b-ba403e972154 rack1
Related
I am trying to set up an Apache CassandraDB cluster via docker.
I've managed to create up to and including 3 nodes using docker run --network cassandra -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_SEEDS=node0 --name "node0" -d cassandra.
Whenever I add a 4th node using the same command (just changing the container name), another random node in the cluster crashes and the node that was created also exits shortly after being on the UJ and DJ stages.
Here's what I've tried.
docker network create cassandra
docker run --network cassandra -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_SEEDS=node0 --name "node0" -d cassandra
I then waited until the node was up and nodetool was showing this:
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 172.19.0.2 88.49 KiB 16 100.0% 15573541-fc19-4569-9a43-cb04e49e134f rack1
After that, I added two additional nodes to the cluster.
docker run --network cassandra -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_SEEDS=node0 --name "node1" -d cassandra
docker run --network cassandra -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_SEEDS=node0 --name "node2" -d cassandra
I waited until those two joined the cluster and nodetool showed this:
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 172.19.0.2 74.11 KiB 16 64.7% 15573541-fc19-4569-9a43-cb04e49e134f rack1
UN 172.19.0.4 98.4 KiB 16 76.0% 30afdc85-e863-452c-9031-59803e4b1f11 rack1
UN 172.19.0.3 74.04 KiB 16 59.3% 6d92cf62-65b4-4365-ab28-2d53872605e3 rack1
That seems good! After that, I wanted to add another node to test whether my replication factor was working properly. So, I added another node to the cluster using the same command:
docker run --network cassandra -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_SEEDS=node0 --name "node3" -d cassandra
When I added this node, node1 crashed immediately. node3 (thats's the new one) was briefly on the UJ (UP-Joining) stage and then switched to DJ (DOWN-Joining) and was then removed from the nodelist.
Here are the results from nodetool status, in order:
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 172.19.0.2 74.11 KiB 16 64.7% 15573541-fc19-4569-9a43-cb04e49e134f rack1
UN 172.19.0.4 74.03 KiB 16 76.0% 30afdc85-e863-452c-9031-59803e4b1f11 rack1
DN 172.19.0.3 74.04 KiB 16 59.3% 6d92cf62-65b4-4365-ab28-2d53872605e3 rack1
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UJ 172.19.0.5 20.75 KiB 16 ? 2e4a25e4-3c81-4383-9c9f-6326e4043910 rack1
UN 172.19.0.2 74.11 KiB 16 64.7% 15573541-fc19-4569-9a43-cb04e49e134f rack1
UN 172.19.0.4 74.03 KiB 16 76.0% 30afdc85-e863-452c-9031-59803e4b1f11 rack1
DN 172.19.0.3 74.04 KiB 16 59.3% 6d92cf62-65b4-4365-ab28-2d53872605e3 rack1
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
DJ 172.19.0.5 20.75 KiB 16 ? 2e4a25e4-3c81-4383-9c9f-6326e4043910 rack1
UN 172.19.0.2 74.11 KiB 16 64.7% 15573541-fc19-4569-9a43-cb04e49e134f rack1
UN 172.19.0.4 74.03 KiB 16 76.0% 30afdc85-e863-452c-9031-59803e4b1f11 rack1
DN 172.19.0.3 74.04 KiB 16 59.3% 6d92cf62-65b4-4365-ab28-2d53872605e3 rack1
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 172.19.0.2 74.11 KiB 16 64.7% 15573541-fc19-4569-9a43-cb04e49e134f rack1
UN 172.19.0.4 74.03 KiB 16 76.0% 30afdc85-e863-452c-9031-59803e4b1f11 rack1
DN 172.19.0.3 74.04 KiB 16 59.3% 6d92cf62-65b4-4365-ab28-2d53872605e3 rack1
Here are the logs for node1:
As you can see, the first item in the log was the confirmation that node2 had connected to the cluster.
https://gist.github.com/janic0/7e464e5c819c37e6ed38819fb3c19eff
Here are the logs for node3 (again, that's the new node)
https://gist.github.com/janic0/0968b7136c3beb3ef76a2379f3cd9be5
I've investigated it and found that Docker kills these containers with exit code 137 - or "out of memory".
Yep, I thought something like that was happening.
Each of the nodes used up about 4GB of RAM and the forth node was just enough to force Docker to kill some of the containers. If you do want to host that many nodes on one machine for some reason, you can increase the memory limit:
So I've done something like this before. If you're just going to be doing some local testing and you want a multi-node cluster, I've used Minikube for that before. In fact, I put together a repo which has some resources for doing that: https://github.com/aploetz/cassandra_minikube
But another approach which might be a "quick fix" for you, would be to explicitly adjust the Java heap sizing to something much smaller for each of your nodes. In my Minikube example above, I'd set:
-Xms512M
-Xmx512M
-Xmn256M
This should create a 1/2 GB heap, which is plenty for local dev or some simple testing. You can set these values in your cassandra-env.sh or jvm-server.options file (depending on your Cassandra version).
I am new to docker. I installed docker on Windows 10. Also, I have installed SSH on my win.
I have a pylucene docker container. When I check the status of my docker container, it is Up.
PS C:\Windows\system32> docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d70d6f0ed7ab coady/pylucene "/bin/bash" 4 weeks ago Up 3 hours amazing_dubinsky
Also, I have no problem with executing it and working with its shell.
PS C:\Windows\system32> docker exec -it d70d6f0ed7ab /bin/bash
root#d70d6f0ed7ab:/usr/src# python
Python 3.9.2 (default, Mar 31 2021, 12:13:11)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import lucene
>>> print(lucene.VERSION)
8.8.1
>>>
but when I found its IP and try to ping this docker container from my win, the result is just timeout!
PS C:\Windows\system32> docker inspect -f "{{ .NetworkSettings.IPAddress }}" d70d6f0ed7ab
172.17.0.3
PS C:\Windows\system32> ping 172.17.0.3
Pinging 172.17.0.3 with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Ping statistics for 172.17.0.3:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),
PS C:\Windows\system32>
Also SSH connect to this container fails,
PS C:\Windows\system32> ssh root#172.17.0.3
ssh: connect to host 172.17.0.3 port 22: Connection timed out
PS C:\Windows\system32>
The coady/pylucene image is based on python:latest
python:latest is based on buildpack-deps:buster
buildpack-deps:buster is based on buildpack-deps:scm
buildpack-deps:scm is based on buildpack-deps:buster-curl
buildpack-deps:buster-curl is based on debian:buster
debian:buster is based Debian 10
And... debian 10 does not have openssh-server installed/active by default.
Plus, as mentioned here:
Docker Desktop for Windows can’t route traffic to Linux containers.
However, you can ping the Windows containers.
In other words, those "problems" you are experiencing are perfectly expected.
I am able to start two cassandra node instances up and running through docker.
docker run --name n1 -d tobert/cassandra -dc DC1 -rack RAC1
docker run --name n2 -d tobert/cassandra -seeds 172.17.0.2 -dc DC2 -rack RAC1
When I try to start the new node instance n3, then it doesn't through any error but I do no see any n3 instance came up, I am seeing only 2 nodes.
$ docker run --name n3 -d tobert/cassandra -seeds 172.17.0.2 -dc DC1 -rack RAC2
XXX
$ docker ps (doesnt show the third cassandra node)
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8ab64fa86819 tobert/cassandra "/bin/cassandra-docke" 41 minutes ago Up 41 minutes 7000/tcp, 7199/tcp, 9042/tcp, 9160/tcp, 61621/tcp n2
125fc4ffba4d tobert/cassandra "/bin/cassandra-docke" 42 minutes ago Up 42 minutes 7000/tcp, 7199/tcp, 9042/tcp, 9160/tcp, 61621/tcp n1
$ docker exec -it n1 nodetool status
Datacenter: DC2
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 172.17.0.3 82.43 KB 256 100.0% XXX RAC1
Datacenter: DC1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 172.17.0.2 51.48 KB 256 100.0% XXX RAC1
Will anyone please let me know why is this happening. What conf need to be done to initiate more node instances. It is clear that the node instances more than 2 in my localhost is an issue here. Why?
Looks like sometimes we need to run the docker run command more than once in-order to start-up the new node. No idea why this happens.
I have assigned memory of 4GB to the docker container using command boot2docker --memory 4096 init which gave some space to add a new node which I was expecting to.
Finally here are the nodes that are up and running
$ docker exec -it n1 nodetool status
Datacenter: DC1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 172.17.0.3 98.91 KB 256 64.5% 30156883-aafe-43b8-b8ee-fec2c9225778 RAC2
UN 172.17.0.2 51.51 KB 256 68.3% 486f457c-8be2-4844-9cd0-d5ef37b46cea RAC1
UN 172.17.0.4 98.97 KB 256 67.3% d19ad6a1-8138-4283-815c-3b223a33c987 RAC1
I am watching a kind of old video on PluralSight for Cassandra and linking works with the method below. However, it is not working with the current docker version. This is how I am linking different nodes. For example:
docker run --name=n1 -d tobert/cassandra
docker run --name n2 -d tobert/cassandra -seeds 172.17.0.7
172.17.0.7 is the IP address when I run the command
docker inspect -f '{{ .NetworkSettings.IPAddress}}' n1
However, only one node is allowed to run at a time is seems. When I run the command
docker ps
or
docker exec -it n1 nodetool status
only one of the containers are running. When I try to inspect a container's ip address, it returns empty. Is there another way to link cassandra nodes in docker?
Manage to resolve this.
It appears to have been an issue with the docker machine created by the "Docker quickstart terminal". Perhaps related to default memory of 1024?
Recreating the machine solved it for me
docker-machine rm default
docker-machine create -d virtualbox --virtualbox-memory=4096 default
Yes, Dashu is right, you need more than the default memory to run multiple Cassandra instances. See similar Twitter thread here: https://twitter.com/perksc/status/641159044225466368.
The -seeds parameter is specific to the entrypoint of this Cassandra docker container and facilitates Cassandra's own clustering (see Github project here: https://github.com/tobert/cassandra-docker). With this feature you don't need to use Docker's own container linking to establish a cluster.
(I'm the author of the Pluralsight course in question. Funny that the course is less than a year old, but so much has changed in Docker since then!)
I was having the same problem. It is solved with using the memory parameter -m 4g. So I initialize the nodes with 4 g.
A - Changing The Docker Memory
And it is good to increase the Docker Memory;
B - My Setup
Windows Version: Windows 10 Pro
Memory: 16 gb
Docker version;
docker --version
Docker version 18.09.2, build 6247962
C - Solution Demo
Initializing 1st cassandra node;
docker run --name=n1 -d -m 4g tobert/cassandra
Getting the ip of the first node;
docker inspect -f "{{ .NetworkSettings.IPAddress }}" n1
The output of the above command is;
172.17.0.3
Initializing the second Cassandra node linking with the first one;
docker run --name n2 -d -m 4g tobert/cassandra -seeds 172.17.0.3
Running the nodetools status command at first node;
docker exec -it n1 nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 172.17.0.3 51.5 KB 256 100.0% 002150d0-90b9-4f8f-861f-0d0205e3ea72 rack1
UN 172.17.0.4 82.46 KB 256 100.0% 05c06381-778e-4989-860b-756d05c8cfbc rack1
Running the nodetools status command at second node;
docker exec -it n2 nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 172.17.0.3 51.5 KB 256 100.0% 002150d0-90b9-4f8f-861f-0d0205e3ea72 rack1
UN 172.17.0.4 82.46 KB 256 100.0% 05c06381-778e-4989-860b-756d05c8cfbc rack1
You can link docker containers with --link flag:
docker run --name n1 tobert/cassandra
docker run --name n2 --link n1:cassandra tobert/cassandra -seeds cassandra
Communication across links, from docker documentation, contains detailed explanation how to use --link flag.
I also run your commands and it works on my machine.
Below is output of the command: docker exec -it n1 nodetool status:
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 172.17.0.3 72 KB 256 100.0% f0a34a45-5f7a-46ff-8fe6-1cf5ff36a3e3 rack1
UN 172.17.0.2 51.46 KB 256 100.0% 8036177e-515c-42b1-9263-3b815c25bde6 rack1
Also, when -seed parameter is wrong (not existing ip address) second container is waiting for a while and after ~30s terminates with exeception:
ERROR [main] 2016-02-07 19:12:11,081 CassandraDaemon.java:541 - Exception encountered during startup
java.lang.RuntimeException: Unable to gossip with any seeds
at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1307) ~[apache-cassandra-2.1.7.jar:2.1.7]
at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:533) ~[apache-cassandra-2.1.7.jar:2.1.7]
So I think n2 is not able to access n1 - please double check your n1 ip address (to check your n2 logs run: docker exec xxxx tail -f /data/log/system.log -n 100, where xxxx - n2 container id).
If you do not have any specific requirements on Cassandra version you can also check image from official repository: https://hub.docker.com/_/cassandra/.
When running docker containers a swarm cluster do the containers have access to all the IPs of the cluster nodes via ENV variables or otherwise?
I want to run an Elasticsearch instance on each node in my swarm the cluster. And they will discovery each other in unicast mode. Therefore each Elasticsearch instanc needs to be configured with the list of IPs in the cluster.
If you mean that container of one node can access container's IP of other node , then it is not possible . You have to use weave tool to connect container across different node or other tool .
If you are using latest Docker (1.13+) with a swam overlay network, you should be able to get all the cluster's node IPs through DNS round robin.(--endpoint-mode dnsrr)
1) Create an overlay network.
https://docs.docker.com/engine/swarm/networking/
docker network create \
--driver overlay \
my-network
2) Verify swam nodes:
docker#node1:~$ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
5l07yt2itiee60xfq7g6c01e4 * node1 Ready Active Leader
pckn7qo3xpbxvs89ni6whyql3 node2 Ready Active
3) Create an alpine container on each nodes using "global" mode:
docker service create --mode global --endpoint-mode dnsrr --name testservice --detach=true --network my-network alpine ash -c "apk update;apk add drill; ping docker.com"
4) verify service is running:
docker#node1:~$ docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
lmy5s3flw763 testservice global 2/2 alpine:latest
5) Verify that containers were deployed on individual nodes:
$ docker-machine ssh node1 "docker ps"
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4c7055b01479 alpine:latest "ash -c 'apk updat..." 2 minutes ago Up 2 minutes testservice.5l07yt2itiee60xfq7g6c01e4.atvascigh3rvxvlzttaotkrua
$ docker-machine ssh node2 "docker ps"
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
28da546aa0d5 alpine:latest "ash -c 'apk updat..." 2 minutes ago Up 2 minutes testservice.pckn7qo3xpbxvs89ni6whyql3.ebjz4asni4w1f0srna0p3vj4a
6) Confirm individual virtual IP of each containers on node1 and node2:
| => docker-machine ssh node1 "docker exec 4c7055b01479 ash -c 'ip addr'|grep eth0"
349: eth0#if350: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP
inet 10.0.0.2/24 scope global eth0
| => docker-machine ssh node2 "docker exec 28da546aa0d5 ash -c 'ip addr'|grep eth0"
319: eth0#if320: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP
inet 10.0.0.3/24 scope global eth0
7) Get the container IP addresses for all containers in cluster using Drill dns tool :
| => docker-machine ssh node1 "docker exec 4c7055b01479 ash -c 'drill testservice'"
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 60920
;; flags: qr rd ra ; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;; testservice. IN A
;; ANSWER SECTION:
testservice. 600 IN A 10.0.0.3
testservice. 600 IN A 10.0.0.2
;; AUTHORITY SECTION:
;; ADDITIONAL SECTION:
;; Query time: 0 msec
;; SERVER: 127.0.0.11
;; WHEN: Thu Jul 20 19:20:49 2017
;; MSG SIZE rcvd: 83
8) Verify that containers can ping each other:
docker-machine ssh node1 "docker exec 4c7055b01479 ash -c 'ping -c2 10.0.0.3'"
PING 10.0.0.3 (10.0.0.3): 56 data bytes
64 bytes from 10.0.0.3: seq=0 ttl=64 time=0.539 ms
64 bytes from 10.0.0.3: seq=1 ttl=64 time=0.731 ms
--- 10.0.0.3 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.539/0.635/0.731 ms
docker-machine ssh node2 "docker exec 28da546aa0d5 ash -c 'ping -c2 10.0.0.2'"
PING 10.0.0.2 (10.0.0.2): 56 data bytes
64 bytes from 10.0.0.2: seq=0 ttl=64 time=0.579 ms
64 bytes from 10.0.0.2: seq=1 ttl=64 time=0.736 ms
--- 10.0.0.2 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.579/0.657/0.736 ms