CircleCI kubectl config error on deployment to gcloud - docker

I'm new to docker/kubernetes/dev ops in general and I was following a course that used Travis with Github, however I use BitBucket so I'm trying to implement a CI deployment to GKE with CircleCI.
Most of the tasks are working just fine but I'm reaching an error when it comes to kubectl (especifically on the deploy.sh script). Here's the error I'm getting:
unable to recognize "k8s/client-deployment.yml": Get http://localhost:8080/api?timeout=32s: dial tcp 127.0.0.1:8080: connect: connection refused
unable to recognize "k8s/database-persistent-volume-claim.yml": Get http://localhost:8080/api?timeout=32s: dial tcp 127.0.0.1:8080: connect: connection refused
unable to recognize "k8s/ingress-service.yml": Get http://localhost:8080/api?timeout=32s: dial tcp 127.0.0.1:8080: connect: connection refused
unable to recognize "k8s/postgres-cluster-ip-service.yml": Get http://localhost:8080/api?timeout=32s: dial tcp 127.0.0.1:8080: connect: connection refused
unable to recognize "k8s/postgres-deployment.yml": Get http://localhost:8080/api?timeout=32s: dial tcp 127.0.0.1:8080: connect: connection refused
unable to recognize "k8s/redis-cluster-ip-service.yml": Get http://localhost:8080/api?timeout=32s: dial tcp 127.0.0.1:8080: connect: connection refused
unable to recognize "k8s/redis-deployment.yml": Get http://localhost:8080/api?timeout=32s: dial tcp 127.0.0.1:8080: connect: connection refused
unable to recognize "k8s/server-cluster-ip-service.yml": Get http://localhost:8080/api?timeout=32s: dial tcp 127.0.0.1:8080: connect: connection refused
unable to recognize "k8s/server-deployment.yml": Get http://localhost:8080/api?timeout=32s: dial tcp 127.0.0.1:8080: connect: connection refused
unable to recognize "k8s/worker-deployment.yml": Get http://localhost:8080/api?timeout=32s: dial tcp 127.0.0.1:8080: connect: connection refused
I've managed to get this far tackling through problems, however I'm lost on this one so any help is appreciated.
Here's the config.yml for CircleCI (on MyUser I'm actually using my docker user, not an env or anything, it's simply to not disclose it):
version: 2
jobs:
build:
docker:
- image: node
working_directory: ~/app
steps:
- checkout
- setup_remote_docker
- run:
name: Install Docker Client
command: |
set -x
VER="18.09.2"
curl -L -o /tmp/docker-$VER.tgz https://download.docker.com/linux/static/stable/x86_64/docker-$VER.tgz
tar -xz -C /tmp -f /tmp/docker-$VER.tgz
mv /tmp/docker/* /usr/bin
- run:
name: Build Client Docker Image
command: docker build -t MY_USER/multi-docker-react -f ./client/Dockerfile.dev ./client
- run:
name: Run Tests
command: docker run -e CI=true MY_USER/multi-docker-react npm run test -- --coverage
deploy:
working_directory: ~/app
# Docker environment where we gonna run our build deployment scripts
docker:
- image: google/cloud-sdk
steps:
- checkout
- setup_remote_docker:
docker_layer_caching: true
# Set up Env
- run:
name: Setup Environment Variables
command: |
echo 'export GIT_SHA="$CIRCLE_SHA1"' >> $BASH_ENV
echo 'export CLOUDSDK_CORE_DISABLE_PROMPTS=1' >> $BASH_ENV
# Log in to docker CLI
- run:
name: Log in to Docker Hub
command: |
echo "$DOCKER_PASSWORD" | docker login -u "$DOCKER_ID" --password-stdin
# !!! This installs gcloud !!!
- run:
name: Installing GCL
working_directory: /
command: |
echo $GCLOUD_SERVICE_KEY | gcloud auth activate-service-account --key-file=-
gcloud --quiet config set project ${GOOGLE_PROJECT_ID}
gcloud --quiet config set compute/zone ${GOOGLE_COMPUTE_ZONE}
# !!! This runs a deployment
- run:
name: Deploying
command: bash ./deploy.sh
workflows:
version: 2
build:
jobs:
- deploy:
filters:
branches:
only:
- master
Here's the deploy.sh:
docker build -t MY_USER/multi-docker-client:latest -t MY_USER/multi-docker-client:$GIT_SHA -f ./client/Dockerfile ./client
docker build -t MY_USER/multi-docker-server:latest -t MY_USER/multi-docker-server:$GIT_SHA -f ./server/Dockerfile ./server
docker build -t MY_USER/multi-docker-worker:latest -t MY_USER/multi-docker-worker:$GIT_SHA -f ./worker/Dockerfile ./worker
docker push MY_USER/multi-docker-client:latest
docker push MY_USER/multi-docker-server:latest
docker push MY_USER/multi-docker-worker:latest
docker push MY_USER/multi-docker-client:$GIT_SHA
docker push MY_USER/multi-docker-server:$GIT_SHA
docker push MY_USER/multi-docker-worker:$GIT_SHA
kubectl apply -f k8s
kubectl set image deployments/client-deployment client=MY_USER/multi-docker-client:$GIT_SHA
kubectl set image deployments/server-deployment server=MY_USER/multi-docker-server:$GIT_SHA
kubectl set image deployments/worker-deployment worker=MY_USER/multi-docker-worker:$GIT_SHA
And here's my project structure:

So turns out I was only missing the next command:
gcloud --quiet container clusters get-credentials multi-cluster
As part of this task:
# !!! This installs gcloud !!!
- run:
name: Installing GCL
working_directory: /
command: |
echo $GCLOUD_SERVICE_KEY | gcloud auth activate-service-account --key-file=-
gcloud --quiet config set project ${GOOGLE_PROJECT_ID}
gcloud --quiet config set compute/zone ${GOOGLE_COMPUTE_ZONE}
gcloud --quiet container clusters get-credentials multi-cluster
Shout out to #DazWilkin for shedding a light

Related

Connection timeout using local kafka-connect cluster to connect on a remote database

I'm trying to run a local kafka-connect cluster using docker-compose.
I need to connect on a remote database and i'm also using a remote kafka and schema-registry.
I have enabled access to these remotes resources from my machine.
To start the cluster, on my project folder in my Ubuntu WSL2 terminal, i'm running
docker build -t my-connect:1.0.0
docker-compose up
The application runs successfully, but when I try to create a new connector, returns error 500 with timeout.
My Dockerfile
FROM confluentinc/cp-kafka-connect-base:5.5.0
RUN cat /etc/confluent/docker/log4j.properties.template
ENV CONNECT_PLUGIN_PATH="/usr/share/java,/usr/share/confluent-hub-components"
ARG JDBC_DRIVER_DIR=/usr/share/java/kafka/
RUN confluent-hub install --no-prompt confluentinc/kafka-connect-jdbc:5.5.0 \
&& confluent-hub install --no-prompt confluentinc/connect-transforms:1.3.2
ADD java/kafka-connect-jdbc /usr/share/confluent-hub-components/confluentinc-kafka-connect-jdbc/lib/
COPY java/kafka-connect-jdbc/ojdbc8.jar /usr/share/confluent-hub-components/confluentinc-kafka-connect-jdbc/lib/
ENTRYPOINT ["sh","-c","export CONNECT_REST_ADVERTISED_HOST_NAME=$(hostname -I);/etc/confluent/docker/run"]
My docker-compose.yaml
services:
connect:
image: my-connect:1.0.0
ports:
- 8083:8083
environment:
- CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL=http=//schema-registry:8081
- CONNECT_KEY_CONVERTER=io.confluent.connect.avro.AvroConverter
- CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL=http=//schema-registry:8081
- CONNECT_BOOTSTRAP_SERVERS=broker1.intranet:9092
- CONNECT_GROUP_ID=kafka-connect
- CONNECT_INTERNAL_KEY_CONVERTER=org.apache.kafka.connect.json.JsonConverter
- CONNECT_VALUE_CONVERTER=io.confluent.connect.avro.AvroConverter
- CONNECT_INTERNAL_VALUE_CONVERTER=org.apache.kafka.connect.json.JsonConverter
- CONNECT_OFFSET_STORAGE_TOPIC=kafka-connect.offset
- CONNECT_CONFIG_STORAGE_TOPIC=kafka-connect.config
- CONNECT_STATUS_STORAGE_TOPIC=kafka-connect.status
- CONNECT_CONNECTOR_CLIENT_CONFIG_OVERRIDE_POLICY=All
- CONNECT_LOG4J_ROOT_LOGLEVEL=INFO
- KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
- CONNECT_REST_ADVERTISED_HOST_NAME=localhost
My cluster it's up
~$ curl -X GET http://localhost:8083/
{"version":"5.5.0-ccs","commit":"606822a624024828","kafka_cluster_id":"OcXKHO7eT4m9NBHln6ACKg"}
Connector call
curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" localhost:8083/connectors/ -d
{
"name": "my-connector",
"config":
{
"connector.class" : "io.debezium.connector.oracle.OracleConnector",
"tasks.max": "1",
"database.user": "user",
"database.password": "pass",
"database.dbname":"SID",
"database.schema":"schema",
"database.server.name": "dbname",
"schema.include.list": "schema",
"database.connection.adapter":"logminer",
"database.hostname":"databasehost",
"database.port":"1521"
}
}
Error
{"error_code": 500,"message": "IO Error trying to forward REST request: java.net.SocketTimeoutException: Connect Timeout"}
## LOG
connect_1 | [2021-07-01 19:08:50,481] INFO Database Version: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
connect_1 | Version 19.4.0.0.0 (io.debezium.connector.oracle.OracleConnection)
connect_1 | [2021-07-01 19:08:50,628] INFO Connection gracefully closed (io.debezium.jdbc.JdbcConnection)
connect_1 | [2021-07-01 19:08:50,643] INFO AbstractConfig values:
connect_1 | (org.apache.kafka.common.config.AbstractConfig)
connect_1 | [2021-07-01 19:09:05,722] ERROR IO error forwarding REST request: (org.apache.kafka.connect.runtime.rest.RestClient)
connect_1 | java.util.concurrent.ExecutionException: java.net.SocketTimeoutException: Connect Timeout
Testing the connection to the database
$ telnet databasehostname 1521
Trying <ip>... Connected to databasehostname
Testing connection to kafka broker
$ telnet broker1.intranet 9092
Trying <ip>... Connected to broker1.intranet
Testing connection to remote schema-registry
$ telnet schema-registry.intranet 8081
Trying <ip>... Connected to schema-registry.intranet
What am I doing wrong? Do I need to configure something else to allow connection to this remote database?
You need to set correctly rest.advertised.host.name (or CONNECT_REST_ADVERTISED_HOST_NAME, if you’re using Docker).
This is how a Connect worker communicates with other workers in the cluster.
For more details see Common mistakes made when configuring multiple Kafka Connect workers by Robin Moffatt.
In your case try to remove CONNECT_REST_ADVERTISED_HOST_NAME=localhost from compose file.

Docker-in-Docker issues with connecting to internal container network (Anchore Engine)

I am having issues when trying to connect to a docker-compose network from inside of a container. These are the files I am working with. The whole thing runs when I ./run.sh.
Dockerfile:
FROM docker/compose:latest
WORKDIR .
# EXPOSE 8228
RUN apk update
RUN apk add py-pip
RUN apk add jq
RUN pip install anchorecli
COPY dockertest.sh ./dockertest.sh
COPY docker-compose.yaml docker-compose.yaml
CMD ["./dockertest.sh"]
docker-compose.yaml
services:
# The primary API endpoint service
engine-api:
image: anchore/anchore-engine:v0.6.0
depends_on:
- anchore-db
- engine-catalog
#volumes:
#- ./config-engine.yaml:/config/config.yaml:z
ports:
- "8228:8228"
..................
## A NUMBER OF OTHER CONTAINERS THAT ANCHORE-ENGINE USES ##
..................
networks:
default:
external:
name: anchore-net
dockertest.sh
echo "------------- INSTALL ANCHORE CLI ---------------------"
engineid=`docker ps | grep engine-api | cut -f 1 -d ' '`
engine_ip=`docker inspect $engineid | jq -r '.[0].NetworkSettings.Networks."cws-anchore-net".IPAddress'`
export ANCHORE_CLI_URL=http://$engine_ip:8228/v1
export ANCHORE_CLI_USER='user'
export ANCHORE_CLI_PASS='pass'
echo "System status"
anchore-cli --debug system status #This line throws error (see below)
run.sh:
#!/bin/bash
docker build . -t anchore-runner
docker network create anchore-net
docker-compose up -d
docker run --network="anchore-net" -v //var/run/docker.sock:/var/run/docker.sock anchore-runner
#docker network rm anchore-net
Error Message:
System status
INFO:anchorecli.clients.apiexternal:As Account = None
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 172.19.0.6:8228
Error: could not access anchore service (user=user url=http://172.19.0.6:8228/v1): HTTPConnectionPool(host='172.19.0.6', port=8228): Max retries exceeded with url: /v1
(Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',))
Steps:
run.sh builds container image and creates network anchore-net
the container has an entrypoint script, which does multiple things
firstly, it brings up the docker-compose network as detached FROM inside the container
secondly, nstalls anchore-cli so I can run commands against container network
lastly, attempts to get a system status of the anchore-engine (d.c network) but thats where I am running into HTTP request connection issues.
I am dynamically getting the IP of the api endpoint container of anchore-engine and setting the URL of the request to do that. I have also tried passing those variables from command line such as:
anchore-cli --u user --p pass --url http://$engine_ip/8228/v1 system status but that throws the same error.
For those of you who took the time to read through this, I highly appreciate any input you can give me as to where the issue may be lying. Thank you very much.

how to make ansible get access to an sshd container?

I use an ansible script to load & start the https://hub.docker.com/r/rastasheep/ubuntu-sshd/ container.
so it starts well of course :
bash-4.4$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8bedbd3b7d88 rastasheep/ubuntu-sshd "/usr/sbin/sshd -D" 37 minutes ago Up 36 minutes 0.0.0.0:49154->22/tcp test
bash-4.4$
so after ansible failure on ssh access to it I tested manually from shell
this is also ok.
bash-4.4$ ssh root#172.17.0.2
The authenticity of host '172.17.0.2 (172.17.0.2)' can't be established.
ECDSA key fingerprint is SHA256:YtTfuoRRR5qStSVA5UuznGamA/dvf+djbIT6Y48IYD0.
ECDSA key fingerprint is MD5:43:3f:41:e9:89:45:06:6f:f6:42:c4:6a:70:37:f8:1d.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.17.0.2' (ECDSA) to the list of known hosts.
root#172.17.0.2's password:
root#8bedbd3b7d88:~# logout
Connection to 172.17.0.2 closed.
bash-4.4$
so the step that failed is trying to get on it from ansible script & make access to ssh-copy-id
ansible error message is :
Fatal: [172.17.0.2]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Warning: Permanently added '172.17.0.2' (ECDSA) to the list of known hosts.\r\nPermission denied (publickey,password).\r\n", "unreachable": true}
---
- hosts: 127.0.0.1
tasks:
- name: start docker service
service:
name: docker
state: started
- name: load and start the container we wanna use
docker_container:
name: test
image: rastasheep/ubuntu-sshd
state: started
ports:
- "49154:22"
- name: Wait maximum of 300 seconds for ports to be available
wait_for:
host: 0.0.0.0
port: 49154
state: started
- hosts: 172.17.0.2
vars:
passwordadmin: $6$pbE6yznA$AeFIdI.....K0
passwordroot: $6$TMrxQUxT$I8.JIzR.....TV1
ansible_ssh_extra_args: "-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null"
tasks:
- name: Build test container root user rsa ssh-key
shell: docker exec test ssh-keygen -b 2048 -t rsa -f /root/.ssh/id_rsa -q -N ""
so I cannot even run the needed step to build ssh
how to do then ??
1st step (ansible task) : load docker container
2cd step (ansible task on only 172.17.0.2) : connect to it & setup it
there will be 3rd step to run application on it after that.
the problem occurs only when starting the 2cd step
Ok after many trys on a second container
conclusion is my procedure was bad
what I have done to solve that :
build a diroctory tree separating ./ ./inventory ./includes
build 1 yaml file by host (local, docker, labo)
build 1 main yaml file on ./
build 1 new host file in ./inventory
connect forced by sshpass to docker on default password
changed it
add the host key on authorized key to a login dedicated usage
installed pyhton (needed to answer ansible host else it makes
randomly module errors or refused connections depending on current
action)
setup a ssh login user in sudoers
then I can un the docker.yaml actions
then only at last I can run the labo.yaml actions.
Thanks for help
now I'm able to build the missing tools.

Docker Cloud Service Discovery Two Containers

In DockerCloud I am trying to get my container to speak with the other container. I believe the problem is the hostname not resolving (this is set in /conf.d/kafka.yaml shown below).
To get DockerCloud to have the two containers communicate, I have tried many variations including the full host-name kafka-development-1 and kafka-development-1.kafka, etc.
The error I keep getting is in the datadog-agent info:
Within the container I run ./etc/init.d/datadog-agent info and receive:
kafka
-----
- instance #kafka-kafka-development-9092 [ERROR]: 'Cannot connect to instance
kafka-development:9092 java.io.IOException: Failed to retrieve RMIServer stub:
javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is: \n\tjava.net.SocketException: Connection reset]' collected 0 metrics
- Collected 0 metrics, 0 events & 0 service checks
The steps I take for details
SSH Into Docker Node:
$ docker ps
CONTAINER | PORTS
datadog-agent-kafka-development-1.2fb73f62 | 8125/udp, 9001/tcp
kafka-development-1.3dc7c2d0 | 0.0.0.0:9092->9092/tcp
I log into the containers to see their values, this is the datadog-agent:
$ docker exec -it datadog-agent-kafka-development-1.2fb73f62 /bin/bash
$ > echo $DOCKERCLOUD_CONTAINER_HOSTNAME
datadog-agent-kafka-development-1
$ > tail /etc/hosts
172.17.0.7 datadog-agent-kafka-development-1
10.7.0.151 datadog-agent-kafka-development-1
This is the kafka container:
$ docker exec -it kafka-development-1.3dc7c2d0 /bin/bash
$ > echo $DOCKERCLOUD_CONTAINER_HOSTNAME
kafka-development-1
$ > tail /etc/hosts
172.17.0.6 kafka-development-1
10.7.0.8 kafka-development-1
$ > echo $KAFKA_ADVERTISED_HOST_NAME
kafka-development.c23d1d00.svc.dockerapp.io
$ > echo $KAFKA_ADVERTISED_PORT
9092
$ > echo $KAFKA_ZOOKEEPER_CONNECT
zookeeper-development:2181
Datadog conf.d/kafka.yaml:
instances:
- host: kafka-development
port: 9092 # This is the JMX port on which Kafka exposes its metrics (usually 9999)
tags:
kafka: broker
env: development
# ... Defaults Below
Can anyone see what I am doing wrong?

ERRO : Error trying to connect to local peer: grpc: timed out trying to connect

Unable to deploy chaincode example in my local hyperledger fabric.
system config: mac osx, Docker toolbox for mac
One validating peer is up and running using docker-compose.yaml
membersrvc:
image: hyperledger/fabric-membersrvc
command: membersrvc
vp0:
image: hyperledger/fabric-peer
environment:
- CORE_PEER_ADDRESSAUTODETECT=false
- CORE_VM_ENDPOINT=http://172.17.0.1:2375
- CORE_LOGGING_LEVEL=DEBUG
- CORE_PEER_ID=vp0
- CORE_SECURITY_ENROLLID=test_vp0
- CORE_SECURITY_ENROLLSECRET=MwYpmSRjupbT
links:
- membersrvc
command: sh -c "sleep 5; peer node start --peer-chaincodedev"
While deploying chaincode by running
CORE_CHAINCODE_ID_NAME=mycc CORE_PEER_ADDRESS=0.0.0.0:30303 ./test
Its showing the error as [shim] ERRO : Error trying to connect to local peer: grpc: timed out trying to connect
I tried replacing CORE_PEER_ADDRESS as suggested by grep timeout solution but no change in error.
First Validating peer output
Chaincode deployment error window
You need to use the correct port number on which peer process is listening to.
instead of using the following command,
CORE_CHAINCODE_ID_NAME=mycc CORE_PEER_ADDRESS=0.0.0.0:30303 ./test
try this instead,
CORE_CHAINCODE_ID_NAME=mycc CORE_PEER_ADDRESS=0.0.0.0:7052 ./test
if it doesn't work then run the following command to check your listening port and use that instead,
netstat -atp tcp | grep -i "listen"

Resources