Cassandra Nodes not linking in Docker - docker

I am watching a kind of old video on PluralSight for Cassandra and linking works with the method below. However, it is not working with the current docker version. This is how I am linking different nodes. For example:
docker run --name=n1 -d tobert/cassandra
docker run --name n2 -d tobert/cassandra -seeds 172.17.0.7
172.17.0.7 is the IP address when I run the command
docker inspect -f '{{ .NetworkSettings.IPAddress}}' n1
However, only one node is allowed to run at a time is seems. When I run the command
docker ps
or
docker exec -it n1 nodetool status
only one of the containers are running. When I try to inspect a container's ip address, it returns empty. Is there another way to link cassandra nodes in docker?

Manage to resolve this.
It appears to have been an issue with the docker machine created by the "Docker quickstart terminal". Perhaps related to default memory of 1024?
Recreating the machine solved it for me
docker-machine rm default
docker-machine create -d virtualbox --virtualbox-memory=4096 default

Yes, Dashu is right, you need more than the default memory to run multiple Cassandra instances. See similar Twitter thread here: https://twitter.com/perksc/status/641159044225466368.
The -seeds parameter is specific to the entrypoint of this Cassandra docker container and facilitates Cassandra's own clustering (see Github project here: https://github.com/tobert/cassandra-docker). With this feature you don't need to use Docker's own container linking to establish a cluster.
(I'm the author of the Pluralsight course in question. Funny that the course is less than a year old, but so much has changed in Docker since then!)

I was having the same problem. It is solved with using the memory parameter -m 4g. So I initialize the nodes with 4 g.
A - Changing The Docker Memory
And it is good to increase the Docker Memory;
B - My Setup
Windows Version: Windows 10 Pro
Memory: 16 gb
Docker version;
docker --version
Docker version 18.09.2, build 6247962
C - Solution Demo
Initializing 1st cassandra node;
docker run --name=n1 -d -m 4g tobert/cassandra
Getting the ip of the first node;
docker inspect -f "{{ .NetworkSettings.IPAddress }}" n1
The output of the above command is;
172.17.0.3
Initializing the second Cassandra node linking with the first one;
docker run --name n2 -d -m 4g tobert/cassandra -seeds 172.17.0.3
Running the nodetools status command at first node;
docker exec -it n1 nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 172.17.0.3 51.5 KB 256 100.0% 002150d0-90b9-4f8f-861f-0d0205e3ea72 rack1
UN 172.17.0.4 82.46 KB 256 100.0% 05c06381-778e-4989-860b-756d05c8cfbc rack1
Running the nodetools status command at second node;
docker exec -it n2 nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 172.17.0.3 51.5 KB 256 100.0% 002150d0-90b9-4f8f-861f-0d0205e3ea72 rack1
UN 172.17.0.4 82.46 KB 256 100.0% 05c06381-778e-4989-860b-756d05c8cfbc rack1

You can link docker containers with --link flag:
docker run --name n1 tobert/cassandra
docker run --name n2 --link n1:cassandra tobert/cassandra -seeds cassandra
Communication across links, from docker documentation, contains detailed explanation how to use --link flag.
I also run your commands and it works on my machine.
Below is output of the command: docker exec -it n1 nodetool status:
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 172.17.0.3 72 KB 256 100.0% f0a34a45-5f7a-46ff-8fe6-1cf5ff36a3e3 rack1
UN 172.17.0.2 51.46 KB 256 100.0% 8036177e-515c-42b1-9263-3b815c25bde6 rack1
Also, when -seed parameter is wrong (not existing ip address) second container is waiting for a while and after ~30s terminates with exeception:
ERROR [main] 2016-02-07 19:12:11,081 CassandraDaemon.java:541 - Exception encountered during startup
java.lang.RuntimeException: Unable to gossip with any seeds
at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1307) ~[apache-cassandra-2.1.7.jar:2.1.7]
at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:533) ~[apache-cassandra-2.1.7.jar:2.1.7]
So I think n2 is not able to access n1 - please double check your n1 ip address (to check your n2 logs run: docker exec xxxx tail -f /data/log/system.log -n 100, where xxxx - n2 container id).
If you do not have any specific requirements on Cassandra version you can also check image from official repository: https://hub.docker.com/_/cassandra/.

Related

Error accessing Scylladb cluster outside docker container

I'm running Scylladb locally in a docker container and I want to access the cluster outside the docker container. That's when I'm getting the following error: cassandra.cluster.NoHostAvailable: ('Unable to connect to any servers')
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 172.17.0.2 776 KB 256 ? ad698c75-a465-4deb-a92c-0b667e82a84f rack1
Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless
Cluster Information:
Name: Test Cluster
Snitch: org.apache.cassandra.locator.SimpleSnitch
DynamicEndPointSnitch: disabled
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Schema versions:
443048b2-c1fe-395e-accd-5ae9b6828464: [172.17.0.2]
I have no problem accessing the cluster using cqlsh on port 9042:
Connected to at 172.17.0.2:9042.
[cqlsh 5.0.1 | Cassandra 3.0.8 | CQL spec 3.3.1 | Native protocol v4]
Now I'm trying to access the cluster from my fastapi app that is outside the docker container.
from cassandra.cluster import Cluster
cluster = Cluster(['172.17.0.2'])
session = cluster.connect('Test Cluster')
And here's the Error that I'm getting:
raise NoHostAvailable("Unable to connect to any servers", errors)
cassandra.cluster.NoHostAvailable: ('Unable to connect to any servers', {'172.17.0.2:9042': OSError(51, "Tried connecting to [('172.17.0.2', 9042)]. Last error: Network is unreachable")})
with a little bit of tinkering, it's possible to achieve a connection to the Scylla running in a container outside of the container for local development.
I've tried on M1 Mac with docker desktop:
Run scylla container with couple of new parameters[src]:
--listen-address 0.0.0.0 for simplification as we are spawning Scylla inside the container to allow connection to the container from any network
--broadcast-rpc-address 127.0.0.1 required if --listen-address set to 0.0.0.0. We are going to port forward 9042 from container to host (local) machine, so this is an IP where it will be acessible.
The final command to spawn the container is:
$ docker run --rm -ti \
-p 127.0.0.1:9042:9042 \
scylladb/scylla \
--smp 1 \
--listen-address 0.0.0.0 \
--broadcast-rpc-address 127.0.0.1
The -p 127.0.0.1:9042:9042 is to make port 9042 accessible on host (local) machine.
Install pip3 install scylla-driver as it has support of darwin/arm64 architecture.
Write a simple python script:
# so74265199.py
from cassandra.cluster import Cluster
cluster = Cluster(['127.0.0.1'])
session = cluster.connect()
# Select from a table that is available without keyspace
res = session.execute('SELECT * FROM system.versions')
print(res.one())
Run your script
$ python3 so74265199.py
Row(key='local', build_id='71178cf6db7021896cd8251751b78b3d9e3afa8d', build_mode='release', version='5.0.5-0.20221009.5a97a1060')
Disclaimer: I'm not an expert in Scylla's configuration, so feel free to point out a better approach.

How to connect using Docker published ports? Not working after upgrade

Before upgrading my system, I was able to successfully connect to mongo running in a docker container using published ports. After upgrading, as shown in Case #1 connecting via published ports no longer work for me.
Case #1
~  docker run --rm -d -p 27017:27017 mongo:3.6
2594b7e5cbf481526589d221361c853338ff55ecb32d9e02eae17383960e971a
~  docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2594b7e5cbf4 mongo:3.6 "docker-entrypoint.s…" 4 seconds ago Up 3 seconds 0.0.0.0:27017->27017/tcp dazzling_fermat
Robo3T Logs
Cannot connect to the MongoDB at localhost:27017.
Error:
Network is unreachable. Reason: network error while attempting to run command 'isMaster' on host 'localhost:27017'
~  sudo lsof -i -P -n | grep LISTEN
...
docker-pr 263637 root 4u IPv4 3723123 0t0 TCP *:27017 (LISTEN)
✘  ~  sudo ufw status
Status: inactive
Now I can only connect using the host networking stack.
Case #2
~  docker run --rm -d --network=host mongo:3.6
39929a8d50cc8554d256f7516d039621cd22ed8be86680ac0e1400809464b619
~  docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
39929a8d50cc mongo:3.6 "docker-entrypoint.s…" 5 seconds ago Up 4 seconds admiring_grothendieck
Robo3T Logs
4:13:20 PM Info: Connecting to localhost:27017...
4:13:20 PM Info: Establish connection successful. Connection: localhost
Pre-upgrade:
Linux Mint 19 - Tricia,
Docker version was 19.xx something I believe.
Post Upgrade:
~  lsb_release -a
No LSB modules are available.
Distributor ID: Linuxmint
Description: Linux Mint 20
Release: 20
Codename: ulyana
~  docker --version
Docker version 20.10.7, build 20.10.7-0ubuntu1~20.04.1
I verified there are no running firewalls (UFD, etc), I can connect from container to container when specifying a private docker network for both the server and client. What am I missing? How can I connect using published ports again? Thanks in advance.
Docker on Linux generally uses the host's DNS and modifies your iptables to provide the connectivity between the host and container. If there's a problem with connectivity, in your case the most likely culprits are (in order of likelihood):
DNS entry missing for localhost or wrong IP version target. Try using 127.0.0.1 or ::1 as the hostname instead.
iptables rules are missing. Check the earlier link in my response for remediations and flags that can affect this.
The container might actually have issues starting up. Check the output of docker log <container_id> for errors after you start it. I would say this option is unlikely as things work under host network but don't discount this possibility too quickly.

Problem in Ping or SSH connect to docker container

I am new to docker. I installed docker on Windows 10. Also, I have installed SSH on my win.
I have a pylucene docker container. When I check the status of my docker container, it is Up.
PS C:\Windows\system32> docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d70d6f0ed7ab coady/pylucene "/bin/bash" 4 weeks ago Up 3 hours amazing_dubinsky
Also, I have no problem with executing it and working with its shell.
PS C:\Windows\system32> docker exec -it d70d6f0ed7ab /bin/bash
root#d70d6f0ed7ab:/usr/src# python
Python 3.9.2 (default, Mar 31 2021, 12:13:11)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import lucene
>>> print(lucene.VERSION)
8.8.1
>>>
but when I found its IP and try to ping this docker container from my win, the result is just timeout!
PS C:\Windows\system32> docker inspect -f "{{ .NetworkSettings.IPAddress }}" d70d6f0ed7ab
172.17.0.3
PS C:\Windows\system32> ping 172.17.0.3
Pinging 172.17.0.3 with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Ping statistics for 172.17.0.3:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),
PS C:\Windows\system32>
Also SSH connect to this container fails,
PS C:\Windows\system32> ssh root#172.17.0.3
ssh: connect to host 172.17.0.3 port 22: Connection timed out
PS C:\Windows\system32>
The coady/pylucene image is based on python:latest
python:latest is based on buildpack-deps:buster
buildpack-deps:buster is based on buildpack-deps:scm
buildpack-deps:scm is based on buildpack-deps:buster-curl
buildpack-deps:buster-curl is based on debian:buster
debian:buster is based Debian 10
And... debian 10 does not have openssh-server installed/active by default.
Plus, as mentioned here:
Docker Desktop for Windows can’t route traffic to Linux containers.
However, you can ping the Windows containers.
In other words, those "problems" you are experiencing are perfectly expected.

Can't launch more than 1 cassandra node

I launched a cassandra docker image using the following command,
docker run --name=n1 -d cassandra:3.11.4
344138bb96e5326113af6ba5c44a17d8b40ed710fe36ce063677a56ef0c0117
This works fine,
docker exec -it n1 nodetool status gives,
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 172.17.0.2 103.7 KiB 256 100.0% 0ca2907d-49b9-4ed4-8bf3-6db99d0c8b45 rack1
Now when I try to launch a second node, this what I get,
docker inspect -f '{{ .NetworkSettings.IPAddress}}' n1
172.17.0.2
docker run --name n2 -d cassandra:3.11.4 -seeds 172.17.0.2
1c89b67cb61b049c4a8ae0f21bd0f7cddffad02b7e7964e502cd82e0d2ff2866
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
344138bb96e5 cassandra:3.11.4 "docker-entrypoint.s…" 2 minutes ago Up 2 minutes 7000-7001/tcp, 7199/tcp, 9042/tcp, 9160/tcp n1
docker exec -it n1 nodetool status
Still no second node
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 172.17.0.2 103.7 KiB 256 100.0% 0ca2907d-49b9-4ed4-8bf3-6db99d0c8b45 rack1
docker exec -it n2 nodetool status
Error response from daemon: Container 1c89b67cb61b049c4a8ae0f21bd0f7cddffad02b7e7964e502cd82e0d2ff2866 is not running
docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1c89b67cb61b cassandra:3.11.4 "docker-entrypoint.s…" 2 minutes ago Exited (1) 2 minutes ago n2
344138bb96e5 cassandra:3.11.4 "docker-entrypoint.s…" 4 minutes ago Up 4 minutes 7000-7001/tcp, 7199/tcp, 9042/tcp, 9160/tcp n1
What am I doing wrong here?
I don't know where you get the information which you can use -seeds 172.17.0.2 to specify seed in this docker image, in fact check the log of n2, it said:
getopt: invalid option -- 's'
getopt: invalid option -- 'e'
getopt: invalid option -- 'e'
getopt: invalid option -- 'd'
getopt: invalid option -- 's'
Error parsing arguments! Unknown argument "172.17.0.2"
And from it's doc, CASSANDRA_SEEDS env is the correct way, like next:
docker run --name n2 -d -e CASSANDRA_SEEDS=172.17.0.2 cassandra:3.11.4
Results:
shubuntu1#shubuntu1:~$ docker run --name n1 -d cassandra:3.11.4
1d1d95f0caeff471c409cd8eda2e77aa5d9b3a60831c284937c305e823016885
shubuntu1#shubuntu1:~$ docker exec -it n1 nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 172.17.0.2 112.85 KiB 256 100.0% c6117826-2fff-48a2-964b-ba403e972154 rack1
shubuntu1#shubuntu1:~$ docker run --name n2 -d -e CASSANDRA_SEEDS=172.17.0.2 cassandra:3.11.4
37e5b84ab92b8b9087e6cff37b146b1f7e337ec9499ce907408abeba11f1985c
Wait a few time, then check again, we find the cluster has 2 nodes now:
shubuntu1#shubuntu1:~$ docker exec -it n1 nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 172.17.0.3 112.98 KiB 256 100.0% e61857d3-b924-4fe9-9a68-03d603de7c38 rack1
UN 172.17.0.2 108.62 KiB 256 100.0% c6117826-2fff-48a2-964b-ba403e972154 rack1

Cassandra: How to increase the number of node instances in localhost

I am able to start two cassandra node instances up and running through docker.
docker run --name n1 -d tobert/cassandra -dc DC1 -rack RAC1
docker run --name n2 -d tobert/cassandra -seeds 172.17.0.2 -dc DC2 -rack RAC1
When I try to start the new node instance n3, then it doesn't through any error but I do no see any n3 instance came up, I am seeing only 2 nodes.
$ docker run --name n3 -d tobert/cassandra -seeds 172.17.0.2 -dc DC1 -rack RAC2
XXX
$ docker ps (doesnt show the third cassandra node)
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8ab64fa86819 tobert/cassandra "/bin/cassandra-docke" 41 minutes ago Up 41 minutes 7000/tcp, 7199/tcp, 9042/tcp, 9160/tcp, 61621/tcp n2
125fc4ffba4d tobert/cassandra "/bin/cassandra-docke" 42 minutes ago Up 42 minutes 7000/tcp, 7199/tcp, 9042/tcp, 9160/tcp, 61621/tcp n1
$ docker exec -it n1 nodetool status
Datacenter: DC2
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 172.17.0.3 82.43 KB 256 100.0% XXX RAC1
Datacenter: DC1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 172.17.0.2 51.48 KB 256 100.0% XXX RAC1
Will anyone please let me know why is this happening. What conf need to be done to initiate more node instances. It is clear that the node instances more than 2 in my localhost is an issue here. Why?
Looks like sometimes we need to run the docker run command more than once in-order to start-up the new node. No idea why this happens.
I have assigned memory of 4GB to the docker container using command boot2docker --memory 4096 init which gave some space to add a new node which I was expecting to.
Finally here are the nodes that are up and running
$ docker exec -it n1 nodetool status
Datacenter: DC1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 172.17.0.3 98.91 KB 256 64.5% 30156883-aafe-43b8-b8ee-fec2c9225778 RAC2
UN 172.17.0.2 51.51 KB 256 68.3% 486f457c-8be2-4844-9cd0-d5ef37b46cea RAC1
UN 172.17.0.4 98.97 KB 256 67.3% d19ad6a1-8138-4283-815c-3b223a33c987 RAC1

Resources