Unable to connect to external network service from docker running inside github actions workflow - docker

I am building a docker compose based micro-service and am trying to execute integration tests from within github actions. Tests all pass locally and the app is currently deployed and working. However when I run the app in GHA it appears that the chat service is unable to reach the twitch irc servers. The strangest part is the chat service is capable of logging in but apparently not sending messages. I can only assume I am missing some critical network configuration in my compose file.
I am running the following compose file in GHA, any ideas folks?
# docker-compose.gha.yml
# Copyright (C) 2023
# Squidpie
version: "3.5"
services:
redis-test:
build:
context: ${STRAUSS_ROOT_DIR}
dockerfile: ${STRAUSS_ROOT_DIR}/tests/redis/Dockerfile
networks:
- strauss
depends_on:
- redis
chat-test:
build:
context: ${STRAUSS_ROOT_DIR}
dockerfile: ${STRAUSS_ROOT_DIR}/tests/chat/Dockerfile
networks:
- strauss
depends_on:
- chat
redis:
image: redis:7.0.6-alpine
container_name: redis
networks:
- strauss
chat:
image: strauss/chat:${STRAUSS_CHAT_PKG_VERSION}
container_name: chat
build:
context: ${STRAUSS_ROOT_DIR}
dockerfile: ${STRAUSS_ROOT_DIR}/services/chat/Dockerfile
args:
version: "prod"
STRAUSS_BUILD_DIR: ${STRAUSS_BUILD_DIR}
environment:
TWITCH_USER: ${TWITCH_USER}
TWITCH_TOKEN: ${TWITCH_TOKEN}
networks:
- strauss
volumes:
- ./strauss.yml:/strauss/strauss.yml
depends_on:
- redis
networks:
strauss:
name: strauss
driver: bridge
With the following Github Action Workflow:
# strauss-validate.yml
# Copyright (C) 2023
# Squidpie
name: strauss-validate
run-name: ${{ github.actor }} is validating strauss
on:
push:
branches:
[dev, stage/*]
pull_request:
types: [assigned, opened, synchronize, reopened]
branches:
[dev]
workflow_dispatch: {}
jobs:
validate:
name: Validate strauss
runs-on: ubuntu-latest
env:
STRAUSS_ROOT_DIR: ${{ github.workspace }}
STRAUSS_BUILD_DIR: target/debug
steps:
- name: Checkout
uses: actions/checkout#v3
with:
fetch-depth: 0
- name: Setup GitVersion
uses: gittools/actions/gitversion/setup#v0.9.15
with:
versionSpec: '5.x'
- name: Resolve Version
id: gitversion
uses: gittools/actions/gitversion/execute#v0.9.15
- name: Setup Ruby
uses: ruby/setup-ruby#v1
with:
ruby-version: '3.2'
bundler-cache: true
- name: Setup Strauss Env
run: |
./scripts/gen-gha-env.sh
cat .env > .env.runtime
echo ${{ secrets.STRAUSS_SECRETS }} >> .env.runtime
printf '%b\n' "services:\n chat:\n channel: <test channel>\n" > strauss.yml
- uses: actions/cache#v3
id: cache-rust
with:
path: |
~/.cargo/bin/
~/.cargo/registry/index/
~/.cargo/registry/cache/
~/.cargo/git/db/
target/
key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
- name: Build & Unit Test Services
run: docker compose --env-file .env -f strauss-build.debug.yml run build
- name: Run Integration Tests
run: |
docker compose --env-file .env.runtime \
-f docker-compose.gha.yml \
up -d
- name: Get Test Status
run: |
sleep 45
docker logs `docker container ls -a | grep chat-test | awk '{print $1}'` | grep "PASSED" && \
docker logs `docker container ls -a | grep redis-test | awk '{print $1}'` | grep "PASSED"
- name: Inspect Dockers
if: failure()
run: |
docker inspect `docker container ls -a | grep "strauss/chat" | awk '{print $1}'`
docker inspect `docker container ls -a | grep "redis:7" | awk '{print $1}'`
docker inspect `docker container ls -a | grep "chat-test" | awk '{print $1}'`
docker inspect `docker container ls -a | grep "redis-test" | awk '{print $1}'`
- name: Collect Docker Logs
if: failure()
uses: jwalton/gh-docker-logs#v2.2.1

Related

Cannot perform an interactive login from a non TTY device error from garygrossgarten/github-action-ssh#release

I'm trying to run docker commands in an ssh connection which was made from github actions. There I'm using even mention in the workflow. Here is my workflow
# This workflow will do a clean installation of node dependencies, cache/restore them, build the source code and run tests across different versions of node
# For more information see: https://help.github.com/actions/language-and-framework-guides/using-nodejs-with-github-actions
name: Node.js CI
on:
pull_request:
branches: ['main']
env:
REGISTRY: 'ghcr.io/testing/api-gateway'
IMAGE_NAME: ${{ format('{0}-{1}', github.event.repository.name, github.sha) }}
USERNAME: ${{ secrets.USERNAME }}
DOCKER_TOKEN: ${{ secrets.DOCKER_TOKEN }}
jobs:
build:
runs-on: ubuntu-latest
strategy:
matrix:
node-version: [16.x]
steps:
- uses: actions/checkout#v3
- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node#v3
with:
node-version: ${{ matrix.node-version }}
cache: 'npm'
- run: npm install
- run: npm run build
- name: Build & Push docker
run: docker build -t $REGISTRY:$IMAGE_NAME .
- name: Login to github package repository
run: echo $DOCKER_TOKEN | docker login ghcr.io -u $USERNAME --password-stdin
- name: Push docker image
run: docker push $REGISTRY:$IMAGE_NAME
deploy:
runs-on: ubuntu-latest
needs: build
steps:
- name: Deploy to Digital Ocean droplet via SSH action
uses: garygrossgarten/github-action-ssh#release
with:
host: ${{ secrets.DO_HOST }}
username: ${{ secrets.DO_USER }}
privateKey: ${{ secrets.DO_KEY }}
passphrase: ${{ secrets.DO_PASSPHRASE }}
command: |
echo $DOCKER_TOKEN | docker login ghcr.io -u $USERNAME --password-stdin
docker pull $REGISTRY:$IMAGE_NAME
docker run -p 3000:3000 $REGISTRY:$IMAGE_NAME
But I'm getting the following error when I run the workflow.
Run garygrossgarten/github-action-ssh#release
Establishing a SSH connection to ***.
using provided private key
(node:1514) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
🤝 Connected to ***.
Executing command: echo $DOCKER_TOKEN | docker login ghcr.io -u $USERNAME --password-stdin
docker pull $REGISTRY:$IMAGE_NAME
docker run -p 3000:3000 $REGISTRY:$IMAGE_NAME
Error: Cannot perform an interactive login from a non-TTY device
invalid reference format
docker: invalid reference format.
See 'docker run --help'.
⚠️ An error happened executing the command echo $DOCKER_TOKEN | docker login ghcr.io -u $USERNAME --password-stdin
docker pull $REGISTRY:$IMAGE_NAME
docker run -p 3000:3000 $REGISTRY:$IMAGE_NAME. Command exited with code 125
Error: Command exited with code 125
1: 0xa1a640 node::Abort() [/home/runner/runners/2.296.0/externals/node12/bin/node]
2: 0xa90649 [/home/runner/runners/2.296.0/externals/node12/bin/node]
3: 0xc06599 [/home/runner/runners/2.296.0/externals/node12/bin/node]
4: 0xc08387 v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) [/home/runner/runners/2.296.0/externals/node12/bin/node]
5: 0x140dd19 [/home/runner/runners/2.296.0/externals/node12/bin/node]
I was looking for a solution and I couldn't find a proper one. what would be the best solution for this?
According to this other thread about the Cannot perform an interactive login from a non-TTY device message:
docker login prints this error message when you use --password-stdin,
but don't actually send a password to the command's stdin.
Therefore, it seems the $DOCKER_TOKEN variable is empty.
You could try using ${{ env.DOCKER_TOKEN}}, or add env: DOCKER_TOKEN: ${{ env.DOCKER_TOKEN }} in the last step of your deploy job.
Something like this:
deploy:
runs-on: ubuntu-latest
needs: build
steps:
- name: Deploy to Digital Ocean droplet via SSH action
uses: garygrossgarten/github-action-ssh#release
env:
DOCKER_TOKEN: ${{ env.DOCKER_TOKEN }}
with:
host: ${{ secrets.DO_HOST }}
username: ${{ secrets.DO_USER }}
privateKey: ${{ secrets.DO_KEY }}
passphrase: ${{ secrets.DO_PASSPHRASE }}
command: |
echo $DOCKER_TOKEN | docker login ghcr.io -u $USERNAME --password-stdin
# or using directly ${{ env.DOCKER_TOKEN}} instead of $DOCKER_TOKEN
docker pull $REGISTRY:$IMAGE_NAME
docker run -p 3000:3000 $REGISTRY:$IMAGE_NAME

How can I use DooD in github actions

I want to automatically create an image using github actions and upload it after testing
However, when this build is executed inside the container, Unable to connect to newly launched container. (probably the execution was successful)
Of course it works when I try to do the same thing without using containers (on the machine).
Why the connection was failed when I tried in docker container?
jobs:
build_and_test:
runs-on: my-runner
container:
image: my-image:tag
credentials:
username: ${{ secrets.ID }}
password: ${{ secrets.PW }}
volumes:
- /var/run/docker.sock:/var/run/docker.sock
options: --user root
steps:
- uses: actions/checkout#v2
- name: build_image
run: |
...
# build image
...
- name: bvt_test
run: |
container=$(docker run --rm -itd -p ${port}:1234 ${new_image})
...
# test, but the connection failed
Thanks.

Error response from daemon: No such image

I am trying to run the following docker-compose yml in the github workflow, but I get the error of
Error response from daemon: No such image: ghcr.io/whats/app/backend/222243434353535353f
The error happens at the "Tag images" point below
The docker Ci file is
env:
WEB_IMAGE_BASE: ghcr.io/$(echo $GITHUB_REPOSITORY | tr '[:upper:]' '[:lower:]')/backend
WEB_IMAGE: ghcr.io/$(echo $GITHUB_REPOSITORY | tr '[:upper:]' '[:lower:]')/backend:$( echo $GITHUB_SHA )
jobs:
build:
name: Build Docker Images
runs-on: ubuntu-latest
steps:
- name: Prepare images
run: |
echo "WEB_IMAGE=$(echo ${{env.WEB_IMAGE}} )" >> $GITHUB_ENV
echo "WEB_IMAGE_BASE=$(echo ${{env.WEB_IMAGE_BASE}} )" >> $GITHUB_ENV
- name: Build images
run: |
docker-compose -f ci.yml build backend
- name: Tag images
run: |
docker tag ${{ env.WEB_IMAGE }} ${{ env.WEB_IMAGE_BASE }}:latest
The ci.yml is
​version​: ​"​3.9​"
​services​:
​  ​backend​: 
​    ​image​: ​backend_prod
​
What am I doing wrong or please indicate how to fix this?
You are trying to tag and image using as a source an image that doesn't exist.
The image ghcr.io/whats/app/backend/222243434353535353f wasn't created in any place.
You need to use as a source in tag command the built image result of docker-compose build.
Since you specify in your compose the image: backend_prod your image will be named by that.
Try to change the image in your docker compose to something like:
version: "3.9"
services:
backend:
build:
context: .
dockerfile: ./backend/dockerfile
image: backend_prod:local
And change your ci file:
- name: Tag images
run: |
docker tag backend_prod:local ${{ env.WEB_IMAGE_BASE }}:latest
You can read more about the image tag in docker-compose here

Github Actions db service container not reachable

I have the following Github Actions pipeline:
name: Elixir CI
on:
push:
branches:
- '*'
pull_request:
branches:
- '*'
jobs:
build:
name: Build and test
runs-on: ubuntu-latest
services:
postgres:
image: postgres
env:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: password
POSTGRES_PORT: 5432
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
ports:
- 5432:5432
steps:
- uses: actions/checkout#v2
- name: Docker Setup Buildx
uses: docker/setup-buildx-action#v1.6.0
with:
install: true
- name: building image
env:
DATABASE_HOST: postgres
DATABASE_PORT: 5432
run: |
docker build --build-arg DATABASE_HOST=$DATABASE_HOST -t backend:test -f Dockerfile.ci .
I have a single build step for an Elixir app: the dockerfile is a multistage one, the first stage runs the tests and builds the production app, and the second copies the application folder/tar.
DATABASE_HOST is the variable that my Elixir app looks for to connect to the test environment.
I have the need to run tests against Postgres, so I spawn a container service with it. I have executed the build both in a container and outside of it, but I always have the following error:
...
#19 195.9 14:10:58.624 [error] GenServer #PID<0.9316.0> terminating
#19 195.9 ** (DBConnection.ConnectionError) tcp connect (postgres:5432): non-existing domain - :nxdomain
#19 195.9 (db_connection 2.4.1) lib/db_connection/connection.ex:100: DBConnection.Connection.connect/2
#19 195.9 (connection 1.1.0) lib/connection.ex:622: Connection.enter_connect/5
#19 195.9 (stdlib 3.14.2.2) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
#19 195.9 Last message: nil
...
So apparently postgres:5432 is not reachable, am I missing something ?
The problem is in DATABASE_HOST: postgres I think.
The service container exports 5432 port to host, so for docker build, it should use host's ip address to visit that postgres service like next:
- name: building image
env:
DATABASE_PORT: 5432
run: |
DATABASE_HOST=$(ifconfig -a eth0|grep inet|grep -v 127.0.0.1|grep -v inet6|awk '{print $2}'|tr -d "addr:")
docker build --build-arg DATABASE_HOST=$DATABASE_HOST -t backend:test -f Dockerfile.ci .
Above will first use ifconfig to get virtual machine's ip(docker host's ip), then pass to docker build to let build container to visit the postgres.

How to target container in GitHub actions?

I am trying to use GitHub Actions to fire up a Postgres container for my tests. I have a script called build.sh that gets called when npm run build is called via GitHub actions. This script calls upon restore-schema.sh (shown below).
The issue here is when restore-schema.sh gets ran, I keep getting Error: no such container: postgres. GitHub actions is naming the container some arbitrary string. Is there not a way I can run docker exec on an image or somehow name the postgres container that GitHub actions is creating? I've looked through both documentations to no avail.
How should I go about this? I noticed that in the Docker run ps screenshot, it shows command docker-entrypoint.sh. Should I use this instead? Do I specify the Dockerfile inside .github/workflows/?
I tried to include as much relevant information as possible - comment if you need any other information please.
Screenshots from GitHub Actions
Initialize containers
Docker run ps <- docker ps showing name postgres
Run npm run build --if-present <- WHERE THE ISSUE IS OCCURING
build.sh
#!/bin/sh
# Import core db schema
./.deploy/postgres/restore-schema.sh
.deploy/postgres/restore-schema.sh
#!/bin/sh
docker exec -it postgres psql \
--username postgres \
--password dev \
coredb < .deploy/postgres/db-schema.sql
.github/workflows/test-api-gateway.yml
name: API Gateway CI
on:
push:
branches: [ master ]
pull_request:
branches: [ master, develop ]
jobs:
build:
runs-on: ubuntu-latest
services: # Serivce containers to run with `container-job`
# Label used to access the service container
postgres:
# Docker Hub image
image: postgres
# Provide the password for postgres
env:
POSTGRES_USER: postgres
POSTGRES_DB: coredb
POSTGRES_PASSWORD: dev
# Set health checks to wait until postgres has started
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
ports:
- 5432:5432
strategy:
matrix:
node-version: [14.x]
steps:
- uses: actions/checkout#v2
- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node#v1
with:
node-version: ${{ matrix.node-version }}
- run: docker ps
- run: chmod +x build.sh .deploy/postgres/restore-schema.sh
- run: npm ci
- run: npm run build --if-present
- run: npm test
Try the --name option
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
--name postgres
https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idservices
jobs.<job_id>.services.options: Additional Docker container resource options. For a list of options, see "docker create options."
Another solution I've seen is using last created container
docker exec -it $(docker ps --latest --quiet) bash

Resources