I'm trying to copy whole data folder from one instance of CouchDB (single node) into Docker volume and I have issues accessing those databases getting error(s) below
[notice] 2018-12-02T09:46:18.664647Z nonode#nohost <0.313.0> -------- chttpd_auth_cache changes listener died database_does_not_exist at mem3_shards:load_shards_from_db/6(line:395) <= mem3_shards:load_shards_from_disk/1(line:370) <= mem3_shards:load_shards_from_disk/2(line:399) <= mem3_shards:for_docid/3(line:86) <= fabric_doc_open:go/3(line:38) <= chttpd_auth_cache:ensure_auth_ddoc_exists/2(line:187) <= chttpd_auth_cache:listen_for_changes/1(line:134)
[error] 2018-12-02T09:46:18.664764Z nonode#nohost emulator -------- Error in process <0.12460.0> with exit value: {database_does_not_exist,[{mem3_shards,load_shards_from_db,"_users",[{file,"src/mem3_shards.erl"},{line,395}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,370}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,399}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,86}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,38}]},{chttpd_auth_cache,ensure_auth_ddoc_exists,2,[{file,"src/chttpd_auth_cache.erl"},{line,187}]},{chttpd_auth_cache,listen_for_changes,1,[{file,"src/chttpd_auth_cache.erl"},{line,134}]}]}
[error] 2018-12-02T09:51:20.301152Z nonode#nohost <0.17260.0> 45e2192077 req_err(2686395495) internal_server_error : No DB shards could be opened.[<<"fabric_util:get_shard/4 L185">>,<<"fabric:get_security/2 L146">>,<<"chttpd_auth_request:db_authorization_check/1 L98">>,<<"chttpd_auth_request:authorize_request/1 L19">>,<<"chttpd:handle_req_after_auth/2 L315">>,<<"chttpd:process_request/1 L300">>,<<"chttpd:handle_request_int/1 L240">>,<<"mochiweb_http:headers/6 L124">>]
[notice] 2018-12-02T09:51:20.301604Z nonode#nohost <0.17260.0> 45e2192077 127.0.0.1:5984 172.17.0.1 adminis GET /mydatabase 500 ok 4
[error] 2018-12-02T09:51:20.301455Z nonode#nohost <0.17258.0> fe8bc3ea8a req_err(2686395495) internal_server_error : No DB shards could be opened. [<<"fabric_util:get_shard/4 L185">>,<<"fabric:get_security/2 L146">>,<<"chttpd_auth_request:db_authorization_check/1 L98">>,<<"chttpd_auth_request:authorize_request/1 L19">>,<<"chttpd:handle_req_after_auth/2 L315">>,<<"chttpd:process_request/1 L300">>,<<"chttpd:handle_request_int/1 L240">>,<<"mochiweb_http:headers/6 L124">>]
[notice] 2018-12-02T09:51:20.303896Z nonode#nohost <0.17258.0> fe8bc3ea8a 127.0.0.1:5984 172.17.0.1 admin GET /mydatabase-v2 500 ok 6
I'm able to access these data in non-docker couchdb instance so it has to be something within docker that prevents me to properly access the data. Running couchdb v2.2.0 in all instancies, SElinux context and ACL are correct. Only difference I see between docker and non-docker instance is the host definition (eg. docker has nonode#nohost while local instance has couchdb#127.0.0.1).
Any idea what might be wrong?
Related
I'm trying to deploy ejabberd docker image in kubernetes with the following folders are mounted from a persistent volume,
/home/ejabberd/logs
/home/ejabberd/conf
/home/ejabberd/database
populated the database,and conf directory with our configuration files and the database folder
from the docker image using an init container .Upon setting the permissions, we could able to
start the ejabberd service , the logs says that the services (on ports 5222,5269,5280) are ready .
when I check the xmpp server status in the container using " ejabberdctl status " , the output says "node down"
===========ejabberd.log===================================================
2020-12-16 09:18:58.477630+00:00 [info] <0.3406.0>#mod_mqtt:init_topic_cache/2:611 Building MQTT cache for mydomain this may take a while
2020-12-16 09:18:59.087380+00:00 [info] <0.483.0>#ejabberd_mnesia:create/2:267 Creating Mnesia ram table 'bytestream'
2020-12-16 09:19:01.193203+00:00 [info] <0.126.0>#ejabberd_cluster_mnesia:wait_for_sync/1:123 Waiting for Mnesia synchronization to complete
2020-12-16 09:19:02.401537+00:00 [info] <0.126.0>#ejabberd_app:start/2:62 ejabberd 20.4.0 is started in the node 'ejabberd#mydomain' in 49.77s
2020-12-16 09:19:02.403414+00:00 [info] <0.601.0>#ejabberd_listener:init/4:159 Start accepting TCP connections at [::]:5222 for ejabberd_c2s
2020-12-16 09:19:02.403479+00:00 [info] <0.602.0>#ejabberd_listener:init/4:159 Start accepting TCP connections at [::]:5269 for ejabberd_s2s_in
2020-12-16 09:19:02.403956+00:00 [info] <0.603.0>#ejabberd_listener:init/4:159 Start accepting TLS connections at [::]:5443 for ejabberd_http
2020-12-16 09:19:02.403999+00:00 [info] <0.604.0>#ejabberd_listener:init/4:159 Start accepting TCP connections at [::]:5280 for ejabberd_http
2020-12-16 09:19:02.404098+00:00 [info] <0.605.0>#ejabberd_listener:init/4:159 Start accepting TCP connections at [::]:1883 for mod_mqtt
2020-12-16 09:19:02.404345+00:00 [info] <0.3418.0>#ejabberd_listener:init/4:159 Start accepting TCP connections at 10.42.8.15:7777 for mod_proxy65_stream
========================================ejabberdctl status===========================
~ $ ./bin/ejabberdctl status
Failed RPC connection to the node 'ejabberd#mydomain': nodedown
Commands to start an ejabberd node:
start - Start an ejabberd node in server mode
debug - Attach an interactive Erlang shell to a running ejabberd node
iexdebug - Attach an interactive Elixir shell to a running ejabberd node
live - Start an ejabberd node in live (interactive) mode
iexlive - Start an ejabberd node in live (interactive) mode, within an Elixir shell
foreground - Start an ejabberd node in server mode (attached)
Optional parameters when starting an ejabberd node:
--config-dir dir Config ejabberd: /home/ejabberd/conf
--config file Config ejabberd: /home/ejabberd/conf/ejabberd.yml
--ctl-config file Config ejabberdctl: /home/ejabberd/conf/ejabberdctl.cfg
--logs dir Directory for logs: /home/ejabberd/logs
--spool dir Database spool dir: /home/ejabberd/database/ejabberd#mydomain
--node nodename ejabberd node name: ejabberd#mydomain
If anyone has tried ejabberd on kubernetes, Please share your thought on this issue
Thanks in advance
On a Windows 10 system, I am trying to run a Docker containiner running geth which listens to port 8545. This docker-compose.yml has been tested to run perfectly on both Ubuntu and Mac OS X.
docker-compose version 1.21.1, build 7641a569 is being used on the Windows 10 system.
Problem: Docker throws an error after executing docker-compose up.
Fatal: Error starting protocol stack: listen unix /root/.ethereum/geth.ipc: bind: operation not permitted
What might be causing this error, and how can we solve it?
docker-compose.yml
version: '3'
services:
geth:
image: ethereum/client-go:latest
volumes:
- ./nodedata:/root/.ethereum
- ./files/genesis.json:/root/genesis.json:ro
ports:
- 30303:30303
- "30303:30303/udp"
- 8545:8545
- 8546:8546
command: --networkid 1337 --cache 512 --port 30303 --maxpeers 50 --rpc --rpcaddr "0.0.0.0" --rpcapi "eth,personal,web3,net" --bootnodes enode://0b37f58139bef9fef04ff50c1d2d95acade0b6989433ed2148683f294a12e8ca7eb17915864a0dd61d5533e898b7040b75df1a17cca27e90d106f95dea255b45#167.99.55.99:30303
container_name: geth-nosw
Output after running docker-compose up
Starting geth-node ... done
Attaching to geth-node
geth-node | INFO [07-22|20:43:11.482] Maximum peer count ETH=50 LES=0 total=50
geth-node | INFO [07-22|20:43:11.488] Starting peer-to-peer node instance=Geth/v1.8.13-unstable-526abe27/linux-amd64/go1.10.3
geth-node | INFO [07-22|20:43:11.488] Allocated cache and file handles database=/root/.ethereum/geth/chaindata cache=384 handles=1024
geth-node | INFO [07-22|20:43:11.521] Initialised chain configuration config="{ChainID: 1337 Homestead: 1 DAO: <nil> DAOSupport: false EIP150: 2 EIP155: 3 EIP158: 3 Byzantium: 4 Constantinople: <nil> Engine: clique}"
geth-node | INFO [07-22|20:43:11.521] Initialising Ethereum protocol versions="[63 62]" network=1366
geth-node | INFO [07-22|20:43:11.524] Loaded most recent local header number=0 hash=b85de5…3971b4 td=1
geth-node | INFO [07-22|20:43:11.524] Loaded most recent local full block number=0 hash=b85de5…3971b4 td=1
geth-node | INFO [07-22|20:43:11.524] Loaded most recent local fast block number=0 hash=b85de5…3971b4 td=1
geth-node | INFO [07-22|20:43:11.525] Loaded local transaction journal transactions=0 dropped=0
geth-node | INFO [07-22|20:43:11.530] Regenerated local transaction journal transactions=0 accounts=0
geth-node | INFO [07-22|20:43:11.530] Starting P2P networking
geth-node | INFO [07-22|20:43:13.670] UDP listener up self=enode://3e0e8e9a886a347fffb0150e670b45c8ae19f0f87ebb6d3fa0f7f312f17220b426913ac96df9527ae0ca00138c9e50ffe646255d5655e6023c47ef10aabf0224#[::]:30303
geth-node | INFO [07-22|20:43:13.672] Stats daemon started
geth-node | INFO [07-22|20:43:13.674] RLPx listener up self=enode://3e0e8e9a886a347fffb0150e670b45c8ae19f0f87ebb6d3fa0f7f312f17220b426913ac96df9527ae0ca00138c9e50ffe646255d5655e6023c47ef10aabf0224#[::]:30303
geth-node | INFO [07-22|20:43:13.676] Blockchain manager stopped
geth-node | INFO [07-22|20:43:13.677] Stopping Ethereum protocol
geth-node | INFO [07-22|20:43:13.677] Ethereum protocol stopped
geth-node | INFO [07-22|20:43:13.677] Transaction pool stopped
geth-node | INFO [07-22|20:43:13.681] Database closed database=/root/.ethereum/geth/chaindata
geth-node | INFO [07-22|20:43:13.681] Stats daemon stopped
geth-node | Fatal: Error starting protocol stack: listen unix /root/.ethereum/geth.ipc: bind: operation not permitted
geth-node | Fatal: Error starting protocol stack: listen unix /root/.ethereum/geth.ipc: bind: operation not permitted
geth-node exited with code 1
The problem is that you cannot create a unix socket on a volume that is linked to a windows file system.
Here's a link on how to work around that.
I'm facing issues when installing thingsboard using docker-compose on ubuntu
images are correctly pulled , container seems to be up but logs shows :
logs for thingsboard/application:1.2.2 :
thingsboard-db-schema container is still in progress. waiting until it
completed...
thingsboard-db-schema container is still in progress. waiting until it
completed...
thingsboard-db-schema container is still in progress. waiting until it
completed...
thingsboard-db-schema container is still in progress. waiting until it
completed...
thingsboard-db-schema container is still in progress. waiting until it
completed...
thingsboard-db-schema container is still in progress. waiting until it
completed...
logs for thingsboard/thingsboard-db-schema:1.2.2
Wait for Cassandra...
Failed to resolve "db".
WARNING: No targets were specified, so 0 hosts scanned.
Wait for Cassandra...
Failed to resolve "db".
WARNING: No targets were specified, so 0 hosts scanned.
Wait for Cassandra...
seems that the first container waiting cassandra to be up which is not the case
Any suggestions ?
Thanks in advance
Please check output of the DB container using command 'docker-compose logs -f db' and verify that cassandra is ready to accept client on 9042 port:
db_1 | INFO 11:02:07 Waiting for gossip to settle before accepting client requests...
db_1 | INFO 11:02:15 No gossip backlog; proceeding
db_1 | INFO 11:02:15 Netty using native Epoll event loop
db_1 | INFO 11:02:15 Using Netty Version: [netty-buffer=netty-buffer-4.0.39.Final.38bdf86, netty-codec=netty-codec-4.0.39.Final.38bdf86, netty-codec-haproxy=netty-codec-haproxy-4.0.39.Final.38bdf86, netty-codec-http=netty-codec-http-4.0.39.Final.38bdf86, netty-codec-socks=netty-codec-socks-4.0.39.Final.38bdf86, netty-common=netty-common-4.0.39.Final.38bdf86, netty-handler=netty-handler-4.0.39.Final.38bdf86, netty-tcnative=netty-tcnative-1.1.33.Fork19.fe4816e, netty-transport=netty-transport-4.0.39.Final.38bdf86, netty-transport-native-epoll=netty-transport-native-epoll-4.0.39.Final.38bdf86, netty-transport-rxtx=netty-transport-rxtx-4.0.39.Final.38bdf86, netty-transport-sctp=netty-transport-sctp-4.0.39.Final.38bdf86, netty-transport-udt=netty-transport-udt-4.0.39.Final.38bdf86]
db_1 | INFO 11:02:15 Starting listening for CQL clients on /0.0.0.0:9042 (unencrypted)...
Output should be like logs above.
Plus additionally verify that no errors happened during the cassandra start up.
ECS service is exiting after running for a couple of seconds. In case we make a container out of the image manually it is running fine, command: sudo docker run -i -p 9100:9100 -p 9110:9110 -p 9120:9120 -p 9130:9130 847782638323.dkr.ecr.us-east-1.amazonaws.com/bytemark/cap
ECS logs are as follows:
2017-01-23T19:37:00Z [INFO] Created docker container for task bytemark-cap:2 arn:aws:ecs:us-east-1:847782638323:task/5d0c49a2-9591-469e-b05b-ebbaaefac26e, Status: (CREATED->RUNNING) Containers: [bytemark-cap-container (CREATED->RUNNING),]: bytemark-cap-container(847782638323.dkr.ecr.us-east-1.amazonaws.com/bytemark/cap:latest) (CREATED->RUNNING) -> c8ee05c5cd688209939d96b54cb0c74c4122686036362d7bfa19f85bdc2dd56e
2017-01-23T19:37:00Z [INFO] Starting container module="TaskEngine" task="bytemark-cap:2 arn:aws:ecs:us-east-1:847782638323:task/5d0c49a2-9591-469e-b05b-ebbaaefac26e, Status: (CREATED->RUNNING) Containers: [bytemark-cap-container (CREATED->RUNNING),]" container="bytemark-cap-container(847782638323.dkr.ecr.us-east-1.amazonaws.com/bytemark/cap:latest) (CREATED->RUNNING)"
2017-01-23T19:37:01Z [INFO] Task change event module="TaskEngine" event="{TaskArn:arn:aws:ecs:us-east-1:847782638323:task/5d0c49a2-9591-469e-b05b-ebbaaefac26e Status:RUNNING Reason: SentStatus:NONE}"
2017-01-23T19:37:01Z [INFO] Adding event module="eventhandler" change="ContainerChange: arn:aws:ecs:us-east-1:847782638323:task/5d0c49a2-9591-469e-b05b-ebbaaefac26e bytemark-cap-container -> RUNNING, Ports [{9100 9100 0.0.0.0 0} {9110 9110 0.0.0.0 0} {9120 9120 0.0.0.0 0} {9130 9130 0.0.0.0 0}], Known Sent: NONE"
2017-01-23T19:37:01Z [INFO] Adding event module="eventhandler" change="TaskChange: arn:aws:ecs:us-east-1:847782638323:task/5d0c49a2-9591-469e-b05b-ebbaaefac26e -> RUNNING, Known Sent: NONE"
2017-01-23T19:37:01Z [INFO] Sending container change module="eventhandler" event="ContainerChange: arn:aws:ecs:us-east-1:847782638323:task/5d0c49a2-9591-469e-b05b-ebbaaefac26e bytemark-cap-container -> RUNNING, Ports [{9100 9100 0.0.0.0 0} {9110 9110 0.0.0.0 0} {9120 9120 0.0.0.0 0} {9130 9130 0.0.0.0 0}], Known Sent: NONE" change="ContainerChange: arn:aws:ecs:us-east-1:847782638323:task/5d0c49a2-9591-469e-b05b-ebbaaefac26e bytemark-cap-container -> RUNNING, Ports [{9100 9100 0.0.0.0 0} {9110 9110 0.0.0.0 0} {9120 9120 0.0.0.0 0} {9130 9130 0.0.0.0 0}], Known Sent: NONE"
2017-01-23T19:37:01Z [INFO] Redundant container state change for task bytemark-cap:2 arn:aws:ecs:us-east-1:847782638323:task/5d0c49a2-9591-469e-b05b-ebbaaefac26e, Status: (RUNNING->RUNNING) Containers: [bytemark-cap-container (RUNNING->RUNNING),]: bytemark-cap-container(847782638323.dkr.ecr.us-east-1.amazonaws.com/bytemark/cap:latest) (RUNNING->RUNNING) to RUNNING, but already RUNNING
2017-01-23T19:37:01Z [INFO] Sending task change module="eventhandler" event="TaskChange: arn:aws:ecs:us-east-1:847782638323:task/5d0c49a2-9591-469e-b05b-ebbaaefac26e -> RUNNING, Known Sent: NONE" change="TaskChange: arn:aws:ecs:us-east-1:847782638323:task/5d0c49a2-9591-469e-b05b-ebbaaefac26e -> RUNNING, Known Sent: NONE"
2017-01-23T19:37:01Z [INFO] Task change event module="TaskEngine" event="{TaskArn:arn:aws:ecs:us-east-1:847782638323:task/5d0c49a2-9591-469e-b05b-ebbaaefac26e Status:STOPPED Reason: SentStatus:RUNNING}"
2017-01-23T19:37:01Z [INFO] Error retrieving stats for container c8ee05c5cd688209939d96b54cb0c74c4122686036362d7bfa19f85bdc2dd56e: context canceled
2017-01-23T19:37:01Z [INFO] Adding event module="eventhandler" change="ContainerChange: arn:aws:ecs:us-east-1:847782638323:task/5d0c49a2-9591-469e-b05b-ebbaaefac26e bytemark-cap-container -> STOPPED, Exit 0, , Known Sent: RUNNING"
2017-01-23T19:37:01Z [INFO] Adding event module="eventhandler" change="TaskChange: arn:aws:ecs:us-east-1:847782638323:task/5d0c49a2-9591-469e-b05b-ebbaaefac26e -> STOPPED, Known Sent: RUNNING"
2017-01-23T19:37:01Z [INFO] Container c8ee05c5cd688209939d96b54cb0c74c4122686036362d7bfa19f85bdc2dd56e is terminal, stopping stats collection
Is there a way to debug the actual cause of the ECS service exiting.
Thank you.
If you can SSH into the docker instances (if you made it through ECS and specified a key or are using your own instances) you can get the logs from the docker.
I had an issue where my only docker container was dying after a few seconds so you can list the containers with
docker ps -a
Find the container you want to see the logs for and use
docker logs *container id*
I just followed this tutorial step by step for setting up a docker swarm in EC2 -- https://docs.docker.com/swarm/install-manual/
I created 4 Amazon Servers using the Amazon Linux AMI.
manager + consul
manager
node1
node2
I followed the instructions to start the swarm and everything seems to go ok regarding making the docker instances.
Server 1
Running docker ps gives:
The Consul logs show this
2016/07/05 20:18:47 [INFO] serf: EventMemberJoin: 729a440e5d0d 172.17.0.2
2016/07/05 20:18:47 [INFO] serf: EventMemberJoin: 729a440e5d0d.dc1 172.17.0.2
2016/07/05 20:18:48 [INFO] raft: Node at 172.17.0.2:8300 [Follower] entering Follower state
2016/07/05 20:18:48 [INFO] consul: adding server 729a440e5d0d (Addr: 172.17.0.2:8300) (DC: dc1)
2016/07/05 20:18:48 [INFO] consul: adding server 729a440e5d0d.dc1 (Addr: 172.17.0.2:8300) (DC: dc1)
2016/07/05 20:18:48 [ERR] agent: failed to sync remote state: No cluster leader
2016/07/05 20:18:49 [WARN] raft: Heartbeat timeout reached, starting election
2016/07/05 20:18:49 [INFO] raft: Node at 172.17.0.2:8300 [Candidate] entering Candidate state
2016/07/05 20:18:49 [INFO] raft: Election won. Tally: 1
2016/07/05 20:18:49 [INFO] raft: Node at 172.17.0.2:8300 [Leader] entering Leader state
2016/07/05 20:18:49 [INFO] consul: cluster leadership acquired
2016/07/05 20:18:49 [INFO] consul: New leader elected: 729a440e5d0d
2016/07/05 20:18:49 [INFO] raft: Disabling EnableSingleNode (bootstrap)
2016/07/05 20:18:49 [INFO] consul: member '729a440e5d0d' joined, marking health alive
2016/07/05 20:18:50 [INFO] agent: Synced service 'consul'
I registered each node using the following command with appropriate IP's
docker run -d swarm join --advertise=x-x-x-x:2375 consul://x-x-x-x:8500
Each of those created a docker instance
Node1
Running docker ps gives:
With logs that suggest there's a problem:
time="2016-07-05T21:33:50Z" level=info msg="Registering on the discovery service every 1m0s..." addr="172.31.17.35:2375" discovery="consul://172.31.3.233:8500"
time="2016-07-05T21:36:20Z" level=error msg="cannot set or renew session for ttl, unable to operate on sessions"
time="2016-07-05T21:37:20Z" level=info msg="Registering on the discovery service every 1m0s..." addr="172.31.17.35:2375" discovery="consul://172.31.3.233:8500"
time="2016-07-05T21:39:50Z" level=error msg="cannot set or renew session for ttl, unable to operate on sessions"
time="2016-07-05T21:40:50Z" level=info msg="Registering on the discovery service every 1m0s..." addr="172.31.17.35:2375" discovery="consul://172.31.3.233:8500"
...
And lastly when I get to the last step of trying to get host information like so on my Consul machine,
docker -H :4000 info
I see no nodes. Lastly when I try the step of running an app, I get the obvious error:
[ec2-user#ip-172-31-3-233 ~]$ docker -H :4000 run hello-world
docker: Error response from daemon: No healthy node available in the cluster.
See 'docker run --help'.
[ec2-user#ip-172-31-3-233 ~]$
Thanks for any insight on this. I'm still pretty confused by much of the swarm model and not sure where to go from here to diagnose.
It looks like Consul is either not binding to a public IP address, or is not accessible on the public IP due to security group or VPC settings. You are setting the discovery URL to consul://172.31.3.233:8500 on the Docker nodes, so I would sugest trying to connect to that address from an external IP, either in your browser or via curl like this:
% curl http://172.31.3.233:8500/ui/dist/
HTML
If you cannot connect (connection refused or timeout) then add a TCP port 8500 ingress rule to your AWS VMs, and try again.
After investigating your issue, I see that you forgot open port 2375 for Docker Engine in all four nodes.
Before starting Swarm Manager or Swarm Node, you have to open a TCP Port for Docker Engine, so Swarm will work with Docker Engine via that Port.
With Docker on Ubuntu 14.04, you can open the port by change file /etc/default/docker and add -H tcp://0.0.0.0:2375 to DOCKER_OPTS. For example:
DOCKER_OPTS="-H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock"
After that, you restart Docker Engine
service docker restart
If you are using CentOS, the solution is same, you can read my blog article https://sonnguyen.ws/install-docker-docker-swarm-centos7/
And the other thing, I think that you should install and run Consul in all nodes (4 servers). So your Swarm can work with Consul on its node