Port error when setting up Dev mode of Hyperledger Fabric - docker

I'm setting up the development environment following the instructions on Hyperledger fabric's official website:
https://hyperledger-fabric.readthedocs.io/en/latest/peer-chaincode-devmode.html
I have started the orderer successfully using:
ORDERER_GENERAL_GENESISPROFILE=SampleDevModeSolo orderer
This command didn't work at first but it worked after I cd fabric/sampleconfig
2020-12-21 11:23:15.084 CST [orderer.common.server] Main -> INFO 009 Starting orderer: Version: 2.3.0 Commit SHA: dc2e59b3c Go version: go1.15.6 OS/Arch: darwin/amd64
2020-12-21 11:23:15.084 CST [orderer.common.server] Main -> INFO 00a Beginning to serve requests
but when I start the peer using:
export PATH=$(pwd)/build/bin:$PATH
export FABRIC_CFG_PATH=$(pwd)/sampleconfig
export FABRIC_LOGGING_SPEC=chaincode=debug
export CORE_PEER_CHAINCODELISTENADDRESS=0.0.0.0:7052
peer node start --peer-chaincodedev=true
An error is spotted:
FABRIC_LOGGING_SPEC=chaincode=debug
CORE_PEER_CHAINCODELISTENADDRESS=0.0.0.0:7052
peer node start --peer-chaincodedev=true
2020-12-21 11:25:13.047 CST [nodeCmd] serve -> INFO 001 Starting peer: Version: 2.3.0 Commit SHA: dc2e59b3c Go version: go1.15.6 OS/Arch: darwin/amd64 Chaincode: Base Docker Label: org.hyperledger.fabric Docker Namespace: hyperledger
2020-12-21 11:25:13.048 CST [peer] getLocalAddress -> INFO 002 Auto-detected peer address: 10.200.83.208:7051
2020-12-21 11:25:13.048 CST [peer] getLocalAddress -> INFO 003 Host is 0.0.0.0 , falling back to auto-detected address: 10.200.83.208:7051 Error: failed to initialize operations subsystem: listen tcp 127.0.0.1:9443: bind: address already in use
this is the error:
Error: failed to initialize operations subsystem: listen tcp 127.0.0.1:9443: bind: address already in use
I checked this issue and it seems this happens because the peer node is using the same port 9443 as the orderer node for the same service. How can I get the two nodes running separately? It seems the docker is running as well.

If you see your error, you can easily follow
Error: failed to initialize operations subsystem: listen tcp 127.0.0.1:9443: bind: address already in use
It is said that the 9443 port is already in use.
It seems that you are not running the orderer and peer as separate containers on the docker-based virtual network, but running on the host pc.
This eventually seems to conflict with two servers requesting one port 9443 on your pc.\
Referring to the configuration below of fabric-2.3/sampleconfig, you can see that each port 9443 is assigned to the server. Assigning one of them to the other port solves this.
fabric-2.3/sampleconfig/orderer.yaml
configuration of orderer
# orderer.yaml
...
Admin:
# host and port for the admin server
ListenAddress: 127.0.0.1:9443
...
fabric-2.3/sampleconfig/core.yaml
configuration of peer
# core.yaml
...
operations:
# host and port for the operations server
# listenAddress: 127.0.0.1:9443
listenAddress: 127.0.0.1:10443
...

This is not a direct answer to the port mapping / collision issue, but we've had great success using the new Kubernetes Test Network as a development platform running on a local system with a virtual Kubernetes cluster running in KIND (Kubernetes in Docker).
In this mode, applications can be developed using the Gateway client (exposed via a port forward or ingress), and smart contracts running As a Service can be launched either in the cluster OR run on the local host OS in a container, binary, or launched in a debugger.
The documentation for the development setup is still sparse, but we'd love to hear feedback on the overall approach, as it offers an exponentially better experience for working with a test network in a development context. In general the process of "port juggling" with Compose is no longer relevant when working on a local Kubernetes cluster. In this mode, you can run services on the host network, instructing peers/orderers/etc. to connect to the remote process running on the host OS.

Related

Hyperledger Fabric peer container fails to start after network was shut down

I'm using the test-network from the hyperledger fabric samples at LTS version 2.2.3. I bring up the network with ./network.sh up createChannel -s couchdb followed by the command for adding the third org in the addOrg3 folder: ./addOrg3.sh up -c mychannel -s couchdb. Sometimes I want to have a fresh start when working on a smart contract so I bring down the network with ./network.sh down. Then when I restart the network with the previously mentioned commands sometimes one of the peer nodes will just fail to start. The log just shows this:
2022-02-18 13:10:25.087 UTC [nodeCmd] serve -> INFO 001 Starting peer:
Version: 2.2.3
Commit SHA: 94ace65
Go version: go1.15.7
OS/Arch: linux/amd64
Chaincode:
Base Docker Label: org.hyperledger.fabric
Docker Namespace: hyperledger
2022-02-18 13:10:25.087 UTC [peer] getLocalAddress -> INFO 002 Auto-detected peer address: 172.18.0.9:11051
2022-02-18 13:10:25.088 UTC [peer] getLocalAddress -> INFO 003 Returning peer0.org3.example.com:11051
I tried connecting to the container and attach to the process peer node start which is the process that brings up the container to get some more info on why its hanging. But since it is the init process with pid 1 one neither attach to it nor kill it. Also killing the container is not working as it is just not responding so I need to kill the whole docker engine. I tried the following without success: Purging docker with docker system prune -a --volumes, restarting my computer, re-downloading the fabric folder and binaries. Still the same error occurs. How is this possible, which information is still on my machine that makes it fail? At least I assume there is something on my machine as the same freshly downloaded code works on another machine and after many times repeating the pruring and restarting and redownloading it also works again on my computer.

Error: unable to perform an operation on node 'rabbit#localhost'

So I have an issue with docker-compose and rabbitmq.
I run docker-compose up. Everything spins up. Docker-compose:
services:
rabbitmq3:
image: "rabbitmq:3-management"
hostname: "localhost"
command: rabbitmq-server
ports:
- 5672:5672
- 15672:15672
Then I do sudo rabbitmqctl status to check connection with node. I get this error:
Error: unable to perform an operation on node 'rabbit#localhost'. Please see diagnostics information and suggestions below.
Most common reasons for this are:
* Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
* CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)
* Target node is not running
In addition to the diagnostics info below:
* See the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more
* Consult server logs on node rabbit#localhost
* If target node is configured to use long node names, don't forget to use --longnames with CLI tools
DIAGNOSTICS
===========
attempted to contact: [rabbit#localhost]
rabbit#localhost:
* connected to epmd (port 4369) on localhost
* epmd reports: node 'rabbit' not running at all
no other nodes on localhost
* suggestion: start the node
Current node details:
* node name: 'rabbitmqcli-25456-rabbit#localhost'
* effective user's home directory: /Users/olof.grund
* Erlang cookie hash: d1oONiVA/qogGxkf6vs9Rw==
When I do it in the container docker-compose exec -T rabbitmq3 rabbitmqctl status it works.
Do I need to expose something from docker somehow? Some rabbitmq client or node maybe?
I used all the tips that I have found in other sources. (adding IP to /etc/hosts/, restarts of containers, services). Took me a day to finally get this to work and it boils down to this.
<wait for 60secs since the rabbit container has been started>
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl force_boot
rabbitmqctl start_app
Rabbitmq uses Erlang's distribution protocol, which requires port 4369 open for the EPMD (Erlang Port Mapper Daemon), expose it in the docker-compose and stop the EPMD running in your host.

Cannot connect to GCP Memorystore from GCP Dataflow

I'm attempting to use GCP Memorystore to handle session ids for a event streaming job running on GCP Dataflow. The job fails with a timeout when trying to connect to Memorystore:
redis.clients.jedis.exceptions.JedisConnectionException: Failed connecting to host 10.0.0.4:6379
at redis.clients.jedis.Connection.connect(Connection.java:207)
at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:101)
at redis.clients.jedis.Connection.sendCommand(Connection.java:126)
at redis.clients.jedis.Connection.sendCommand(Connection.java:117)
at redis.clients.jedis.Jedis.get(Jedis.java:155)
My Memorystore instance has these properties:
Version is 4.0
Authorized network is default-auto
Master is in us-central1-b. Replica is in us-central1-a.
Connection properties: IP address: 10.0.0.4, Port number: 6379
> gcloud redis instances list --region us-central1
INSTANCE_NAME VERSION REGION TIER SIZE_GB HOST PORT NETWORK RESERVED_IP STATUS CREATE_TIME
memorystore REDIS_4_0 us-central1 STANDARD_HA 1 10.0.0.4 6379 default-auto 10.0.0.0/29 READY 2019-07-15T11:43:14
My Dataflow job has these properties:
runner: org.apache.beam.runners.dataflow.DataflowRunner
zone: us-central1-b
network: default-auto
> gcloud dataflow jobs list
JOB_ID NAME TYPE CREATION_TIME STATE REGION
2019-06-17_02_01_36-3308621933676080017 eventflow Streaming 2019-06-17 09:01:37 Running us-central1
My "default" network could not be used since it is a legacy network, which Memorystore would not accept. I failed to find a way to upgrade the default network from legacy to auto and did not want to delete the existing default network since this would require messing with production services. Instead I created a new network "default-auto" of type auto, with the same firewall rules as the default network. The one I believe is relevant for my Dataflow job is this:
Name: default-auto-internal
Type: Ingress
Targets: Apply to all
Filters: IP ranges: 10.0.0.0/20
Protocols/ports:
tcp:0-65535
udp:0-65535
icmp
Action: Allow
Priority: 65534
I can connect to Memorystore using "telnet 10.0.0.4 6379" from a Compute Engine instance.
Things I have tried, which did not change anything:
- Switched Redis library, from Jedis 2.9.3 to Lettuce 5.1.7
- Deleted and re-created the Memorystore instance
Is Dataflow not supposed to be able to connect to Memorystore, or am I missing something?
Figured it out. I was trying to connect to Memorystore from code called directly from the main method of my Dataflow job. Connecting from code running in a Dataflow step worked. On second though (well, actually more like 1002nd thought) this makes sense because main() is running on the driver machine (my desktop in this case) whereas the steps of the Dataflow graph will run on GCP. I have confirmed this theory by connecting to Memorystore on localhost:6379 in my main(). This works since I have an SSH tunnel to Memorystore running on port 6379 (using this trick).

docker installation on ubuntu in virtualbox, cannot pull images

I have ubuntu 14.04.5 installed as guest os in virtualbox 5.0.26 running on windows 10. I am not aware of any issues with the ubuntu installation, it seems to run fine and has a bridged internet connection so gets its own ip.
I have installed docker following the directions on docker docs for linux. The installation goes fine without any errors and the docker daemon starts ok.
Here is the docker info:
root#ubuntu-z9:~# docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 1.12.0
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 0
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: overlay bridge host null
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor
Kernel Version: 4.2.0-27-generic
Operating System: Ubuntu 14.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 10
Total Memory: 31.42 GiB
Name: ubuntu-z9
ID: 7MPO:OHFW:3OBJ:KUVX:3YCS:XP4U:RE6W:SFC3:O4KK:GJJU:M6WJ:HYLY
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
127.0.0.0/8
The machine can see the internet fine and access hub.docker.com from a browser.
However, when I run the simple hello-world test the daemon hangs
root#ubuntu-z9:~# docker run hello-world
Unable to find image 'hello-world:latest' locally
with a timeout.
I can run docker-machine without any issues on the host windows 10 machine so I believe the issue lies in my setup of the ubuntu machine in virtualbox and docker.
Here is the logging output of the docker daemon on the ubuntu guest machine:
$ docker pull hello-world
DEBU[0093] Calling POST /v1.24/images/create?fromImage=hello-world&tag=latest
DEBU[0093] Trying to pull hello-world from https://registry-1.docker.io v2
DEBU[0094] Increasing token expiration to: 60 seconds
ERRO[0494] Error trying v2 registry: error parsing HTTP 408 response body: invalid character '<' looking for beginning of value: "<html><body><h1>408 Request Time-out</h1>\nYour browser didn't send a complete request in time.\n</body></html>\n\n"
ERRO[0494] Attempting next endpoint for pull after error: error parsing HTTP 408 response body: invalid character '<' looking for beginning of value: "<html><body><h1>408 Request Time-out</h1>\nYour browser didn't send a complete request in time.\n</body></html>\n\n"
DEBU[0494] Skipping v1 endpoint https://index.docker.io because v2 registry was detected
ERRO[0494] Handler for POST /v1.24/images/create returned error: error parsing HTTP 408 response body: invalid character '<' looking for beginning of value: "<html><body><h1>408 Request Time-out</h1>\nYour browser didn't send a complete request in time.\n</body></html>\n\n"
Any suggestions on a way forward to diagnose or fix the issue?
Many thanks.
It was a simple issue, undoubtedly documented somewhere but I missed it. I post an answer here in case someone else has the same.
The virtualbox os (ubuntu in my case) has to have a NAT network adapter and the NAT adapter has to have higher priority than a bridge adapter (if you have one). You don't need a bridged adapter to run docker (but if you want the virtualbox to have an ip on your local network then you do need to add a bridged adapter.)
VirtualBox configuration examples that work to run docker:
VBox Adapter 1: NAT (eth0), VBox Adapter 2: Host-only Adapter (eth1)
VBox Adapter 1: NAT (eth0), VBox Adapter 2: Bridged Adapter (eth1)
VirtualBox configuration examples that do not work to run docker:
VBox Adapter 1: Bridged Adapter (eth0)
VBox Adapter 1: Bridged Adapter (eth0), VBox Adapter 2: NAT (eth1)
Note in all four cases the virtualbox ubuntu os has access to the internet but docker can only pull images when NAT has priority over the bridged interface.

Connecting local cassandra cluster with docker container

I have set of microservices running as docker container. One microservice say A wants to connect to cassnadra running locally on my laptop. in order to do so i have below configurations
snippet from yaml file of service A
cassandra:
hosts: [127.0.0.1]
keyspace: "My keyspace"
protocol_version: 3
ports: 9042
In other side i ran cassandra by calling ./bin/cassandra . and then i connected to cqlsh locally whose output is as below
Connected to Test Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.0.6 | CQL spec 3.4.0 | Native protocol v4]
Use HELP for help.
now when my container comes up and try to connect to this running cassandra hosted on my machine it says as says connection refused . please see the trace below
File "cassandra/cluster.py", line 2076, in cassandra.cluster.ControlConnection._reconnect_internal (cassandra/cluster.c:36914)
cassandra.cluster.NoHostAvailable: ('Unable to connect to any servers', {'127.0.0.1': ConnectionRefusedError(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error: Connection refused")})
[start] application exit with code 1, killing container
more info
I am using apache-cassandra-3.0.6.
Please advise. Thanks
As Shibashis has mentioned, probably you cannot reach the host via docker container with 127.0.0.1
Please find the IP represented as HOST.
How to get the IP address of the docker host from inside a docker container
Start the cassandra instance by changing the conf\cassandra.yaml
listen_address
rpc_address
to the recongnized HOST IP.
Hope it helps!

Resources