In shell I typed bin/dev page foo and shell returned Node is not running, I checked my logs and noticed the message epmd: epmd: node name already occupied nitrogen
Then, in shell I typed epmd -names and it returned
epmd: up and running on port 4369 with data:
name nitrogen at port 61109
Running epmd -debug gives
epmd: Thu Jun 27 01:01:52 2013: epmd running - daemon = 0
epmd: Thu Jun 27 01:01:52 2013: there is already a epmd running at port 4369
I cannot stop the node, and when I try apparently it is active in the db
epmd: local epmd responded with <>
Killing not allowed - living nodes in database.
In Eshell, I received the following
=ERROR REPORT==== 27-Jun-2013::00:49:53 ===
** Connection attempt from disallowed node 'nitrogen_maint_19141#127.0.0.1' **
Is there a method to get Eshell to recognize this node, in order to run bin/dev function?
I've noticed you posting on the Nitrogen mailing list, and as I understand it, you've got it straightened out, but in this situation, I'd kill the running node manually with a ps aux | grep nitrogen, then kill the process it finds with a simple kill XYZ.
That, or, I've seen the "Node is not running" thing pop up when the process was launched with a different user, such that you don't have access to the erlang pipe.
Admittedly, my advice isn't terribly scientific (killing a process is pretty nasty), but it's a simple solution if for whatever reason something got hosed during launching and you're unable to attach to the node.
Related
I'm using the test-network from the hyperledger fabric samples at LTS version 2.2.3. I bring up the network with ./network.sh up createChannel -s couchdb followed by the command for adding the third org in the addOrg3 folder: ./addOrg3.sh up -c mychannel -s couchdb. Sometimes I want to have a fresh start when working on a smart contract so I bring down the network with ./network.sh down. Then when I restart the network with the previously mentioned commands sometimes one of the peer nodes will just fail to start. The log just shows this:
2022-02-18 13:10:25.087 UTC [nodeCmd] serve -> INFO 001 Starting peer:
Version: 2.2.3
Commit SHA: 94ace65
Go version: go1.15.7
OS/Arch: linux/amd64
Chaincode:
Base Docker Label: org.hyperledger.fabric
Docker Namespace: hyperledger
2022-02-18 13:10:25.087 UTC [peer] getLocalAddress -> INFO 002 Auto-detected peer address: 172.18.0.9:11051
2022-02-18 13:10:25.088 UTC [peer] getLocalAddress -> INFO 003 Returning peer0.org3.example.com:11051
I tried connecting to the container and attach to the process peer node start which is the process that brings up the container to get some more info on why its hanging. But since it is the init process with pid 1 one neither attach to it nor kill it. Also killing the container is not working as it is just not responding so I need to kill the whole docker engine. I tried the following without success: Purging docker with docker system prune -a --volumes, restarting my computer, re-downloading the fabric folder and binaries. Still the same error occurs. How is this possible, which information is still on my machine that makes it fail? At least I assume there is something on my machine as the same freshly downloaded code works on another machine and after many times repeating the pruring and restarting and redownloading it also works again on my computer.
Using docker-desktop on macOS.
I'm trying to run a node following the instructions on this page.
The database name is node, which is the same as the username: node. The user has access to the database and can log in using psql client.
Connection strings I've tried in the .env file:
postgresql://node#localhost/node
postgresql://node:password#localhost/node
postgresql://node:password#localhost:5432/node
postgresql://node:password#127.0.0.1:5432/node
postgresql://node:password#127.0.0.1/node
When I run the start command: cd ~/.chainlink-kovan && docker run -p 6688:6688 -v ~/.chainlink-kovan:/chainlink -it --env-file=.env smartcontract/chainlink local n , using docker-desktop on macOS, I get the following stack trace:
2020-09-15T14:24:41Z [INFO] Starting Chainlink Node 0.8.15 at commit a904730bd62c7174b80a2c4ccf885de3e78e3971 cmd/local_client.go:50
2020-09-15T14:24:41Z [INFO] SGX enclave *NOT* loaded cmd/enclave.go:11
2020-09-15T14:24:41Z [INFO] This version of chainlink was not built with support for SGX tasks cmd/enclave.go:12
2020-09-15T14:24:41Z [INFO] Locking postgres for exclusive access with 500ms timeout orm/orm.go:69
2020-09-15T14:24:41Z [ERROR] unable to lock ORM: dial tcp 127.0.0.1:5432: connect: connection refused logger/default.go:139 stacktrace=github.com/smartcontractkit/chainlink/core/logger.Error
/chainlink/core/logger/default.go:117
...
Does anyone know how I can resolve this?
The problem probably caused by the fact that your chainlink database has been locked with Exclusive Lock and before stopping node that locks never removed.
What you do in this situation (as what works for me) is use PgAdmin Ui or similar way to find all Locks then find the Exclusive Lock that is held on the chainlink database and note down its Process id or ids (if multiple exclusive locks there are on chainlink DB)
Log in to your pg client and run SELECT pg_terminate_backend(<pid>) or SELECT pg_cancel_backend(<pid>); Enter PID of those locks here without quotes and meanwhile keep refreshing on pg admin URL to see if those processes stopped If stopped then rerun your chainlink node.
The problem is with docker networking.
Add --network host to the docker run command so that it is:
cd ~/.chainlink-kovan && docker run -p 6688:6688 -v ~/.chainlink-kovan:/chainlink -it --env-file=.env smartcontract/chainlink --network host local n
This fixes the issue.
I am attempting to run the following docker container:
https://hub.docker.com/r/bgruening/pubmedportable/
I am doing so using the following command:
sudo docker run -d -v /home/$USER/docker_pubmedportable/:/export/ -p 9999:5432 bgruening/pubmedportable
The only output I get is immediately returned:
9b76caddaddbe262bf30d3edbab30da9fa29b9e5f1ad3a4148e753f4e5e929bd
And that is all that is done. There should be a postgres server that is instantiated/created, filled with data, and then hosted at the port 9999 on localhost.
I tried looking at the logs via:
docker logs -f 9b76caddaddbe262bf30d3edbab30da9fa29b9e5f1ad3a4148e753f4e5e929bd
However, this also returns no information.
Also, running docker ps provides absolutely nothing after the commands are issued.
It is my understanding that docker containers are supposed to "just work" on any platform, with little to no effort required.
However, this docker container has not been able to create and host this database and does not appear to be running at all.
Is there a method to determine which section of the docker container is causing a problem?
The OS is archlinux.
Probably some error is making the container exits.
Run it without the -d option, so you can see the log.
I was able to bring up the container with your command. I adapted the path to my environment.
..[$] <()> docker run -d -v ${pwd}:/export/ -p 9999:5432 bgruening/pubmedportable
1d21b00a5fdd376016bb09aeb472a295b86f74aea385a609ca8b33a0ba87f306
..[$] <()> docker logs 1d21b00a5fdd376016bb09aeb472a295b86f74aea385a609ca8b33a0ba87f306
Starting PostgreSQL 9.1 database server: main.
Initialized with 4 processes
######################
###### Finished ######
######################
programme started - Sat Sep 15 04:47:35 2018
programme ended - Sat Sep 15 04:47:36 2018
/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py:3779: SAWarning: Textual SQL expression '\n SELECT \n ...' should be explicitly declared as text('\n SELECT \n ...') (this warning may be suppressed after 10 occurrences)
{"expr": util.ellipses_string(element)})
-------------
processing files from year 1809 to 2016
-------------
got articles from PostgreSQL database
-------------
now indexing articles in Xapian
-------------
no search of synonyms performed, use "python RunXapian.py -h" for parameter view
2017-06-01 00:50:17 UTC LOG: aborting any active transactions
2017-06-01 00:50:17 UTC LOG: autovacuum launcher shutting down
2017-06-01 00:50:17 UTC LOG: shutting down
2017-06-01 00:50:17 UTC LOG: database system is shut down
2018-09-15 04:47:34 UTC LOG: database system was shut down at 2017-06-01 00:50:17 UTC
2018-09-15 04:47:34 UTC LOG: database system is ready to accept connections
2018-09-15 04:47:34 UTC LOG: autovacuum launcher started
2018-09-15 04:47:34 UTC LOG: incomplete startup packet
2018-09-15 04:47:36 UTC LOG: could not receive data from client: Connection reset by peer
2018-09-15 04:47:36 UTC LOG: unexpected EOF on client connection
..[$] <()> psql -h localhost -p 9999 -U parser pubmed
Password for user parser:
psql (10.5, server 9.1.24)
SSL connection (protocol: TLSv1.2, cipher: DHE-RSA-AES256-GCM-SHA384, bits: 256, compression: on)
Type "help" for help.
pubmed=#
My docker service (epst) fails to start if I'm also running VSCode. The error is:
ERROR: for epst Cannot start service epst: driver failed programming external connectivity on endpoint epst (long-hash): Error starting userland proxy: Bind for 0.0.0.0:5123 failed: port is already allocated
ERROR: Encountered errors while bringing up the project.
If I shut down VSCode and re-launch docker-compose, then everything comes up fine.
So my question is how do I identify what is binding to port 5123 in VSCode?
I believe you might be looking for lsof -i :5123?
See man page for lsof.
This would the return a list of processes running on the port you entered (5123).
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
appName 5123 yourUser -- ---- -------------------------
You could then kill 5123 to free up the desired port.
I have 3 containers running on 3 machines. One is called graphite, one is called back and one is called front. The front container needs both the others to run, so i link them separately like this:
[Unit]
Description=front hystrix
[Service]
TimeoutStartSec=0
ExecStartPre=-/usr/bin/docker kill front
ExecStartPre=-/usr/bin/docker rm -v front
ExecStartPre=/usr/bin/docker pull blurio/hystrixfront
ExecStart=/usr/bin/docker run --name front --link graphite:graphite --link back:back -p 8080:8080 blurio/hystrixfront
ExecStop=/usr/bin/docker stop front
I start both the other containers, wait till they're up and running, then start this one with fleetctl and it just instantly fails with this message:
fleetctl status front.service
? front.service - front hystrix
Loaded: loaded (/run/fleet/units/front.service; linked-runtime; vendor preset: disabled)
Active: failed (Result: exit-code) since Tue 2015-05-12 13:46:08 UTC; 24s ago
Process: 922 ExecStop=/usr/bin/docker stop front (code=exited, status=0/SUCCESS)
Process: 912 ExecStart=/usr/bin/docker run --name front --link graphite:graphite --link back:back -p 8080:8080 blurio/hystrixfront (code=exited, status=1/FAILURE)
Process: 902 ExecStartPre=/usr/bin/docker pull blurio/hystrixfront (code=exited, status=0/SUCCESS)
Process: 892 ExecStartPre=/usr/bin/docker rm -v front (code=exited, status=1/FAILURE)
Process: 885 ExecStartPre=/usr/bin/docker kill front (code=exited, status=1/FAILURE)
Main PID: 912 (code=exited, status=1/FAILURE)
May 12 13:46:08 core-04 docker[902]: 8b9853c10955: Download complete
May 12 13:46:08 core-04 docker[902]: 0dc7a355f916: Download complete
May 12 13:46:08 core-04 docker[902]: 0dc7a355f916: Download complete
May 12 13:46:08 core-04 docker[902]: Status: Image is up to date for blurio/hystrixfront:latest
May 12 13:46:08 core-04 systemd[1]: Started front hystrix.
May 12 13:46:08 core-04 docker[912]: time="2015-05-12T13:46:08Z" level="fatal" msg="Error response from daemon: Could not get container for graphite"
May 12 13:46:08 core-04 systemd[1]: front.service: main process exited, code=exited, status=1/FAILURE
May 12 13:46:08 core-04 docker[922]: front
May 12 13:46:08 core-04 systemd[1]: Unit front.service entered failed state.
May 12 13:46:08 core-04 systemd[1]: front.service failed.
I also want to include the fleetctl list-units output, where you can see that the other two are running without problems.
fleetctl list-units
UNIT MACHINE ACTIVE SUB
back.service 0ff08b11.../172.17.8.103 active running
front.service 69ab2600.../172.17.8.104 failed failed
graphite.service 2886cedd.../172.17.8.101 active running
there are a couple issues here. first, you can't use the --link argument for docker. this is a docker specific instruction for linking one container to another on the same docker engine. in your example, you have multiple engines, so this technique won't work. If you want to use that technique, you will need to employ the ambassador pattern: coreos ambassador, either that, you you can use the X-Fleet directive MachineOf: to make all of the docker containers run on the same machine, however, I think that would defeat your goals.
Often with cloud services one service needs another, like in your case. If the other service is not running (yet), then the services that need it should be well behaved and either exit, or wait for the needed service to be ready. So the needed service must be discovered. There are many techniques for the discovery phase, and the waiting phase. For example, you can write a 'wrapper' script in each of your containers. That wrapper can do these duties. In your case, you could have a script in the back.service and graphite.service which writes information to the etcd database, like:
ExecStartPre=/usr/bin/env etcdctl set /graphite/status ready }'
Then in the startup script in front you can do a etcdctl get /graphite/status to see when the container becomes ready (and not continue until it is). If you like you can store the ip address and port in the graphite script so that the front script can pick up the place to connect to.
Another technique for discovery is to use registrator. This is a super handy docker container that updates a directory structure in etcd everytime a container comes and goes. This makes it easier to use a discovery technique like I listed above without having each container having to announce itself, it becomes automatic. You still need the 'front' container to have a startup script that waits for the service to appear in the etcd database. I usually start registrator on coreos boot. In fact, I start two copies, one for discovering internal addresses (flannel ones) and one for external (services that are available outside my containers). Here is an example of the database registrator manages on my machines:
core#fo1 ~/prs $ etcdctl ls --recursive /skydns
/skydns/net
/skydns/net/tacodata
/skydns/net/tacodata/services
/skydns/net/tacodata/services/cadvisor-4194
/skydns/net/tacodata/services/cadvisor-4194/fo2:cadvisor:4194
/skydns/net/tacodata/services/cadvisor-4194/fo1:cadvisor:4194
/skydns/net/tacodata/services/cadvisor-4194/fo3:cadvisor:4194
/skydns/net/tacodata/services/internal
/skydns/net/tacodata/services/internal/cadvisor-4194
/skydns/net/tacodata/services/internal/cadvisor-4194/fo2:cadvisor:4194
/skydns/net/tacodata/services/internal/cadvisor-4194/fo1:cadvisor:4194
/skydns/net/tacodata/services/internal/cadvisor-4194/fo3:cadvisor:4194
/skydns/net/tacodata/services/internal/cadvisor-8080
/skydns/net/tacodata/services/internal/cadvisor-8080/fo2:cadvisor:8080
/skydns/net/tacodata/services/internal/cadvisor-8080/fo1:cadvisor:8080
/skydns/net/tacodata/services/internal/cadvisor-8080/fo3:cadvisor:8080
You can see the internal and external available ports for cadvisor. If I get one of the records:
etcdctl get /skydns/net/tacodata/services/internal/cadvisor-4194/fo2:cadvisor:4194
{"host":"10.1.88.3","port":4194}
you get everything you need to connect to that container internally. This technique really starts to shine when coupled with skydns. Skydns presents a dns service using the information presented by registrator. So, long story short, I can simply make my application use the hostname (the hostname defaults to be the name of the docker image, but it can be changed). So in this example here my application can connect to cadvisor-8080, and dns will give it one of the 3 ip addresses it has (it is on 3 machines). The dns also supports srv records, so, if you aren't using a well know port the srv record can give you the port number.
Using coreos and fleet it is difficult not to get the containers themselves involved in the publish/discovery/wait game. At least that's been my experience.
-g