Here is my attempt at creating a Docker Host Context via GitHub Actions:
name: CICD
on:
push:
branches:
- main
- staging
workflow_dispatch:
jobs:
build_and_deploy_monitoring:
concurrency: monitoring
runs-on: [self-hosted, linux, X64]
steps:
- uses: actions/checkout#v2
- name: Save secrets to mon.env files
run: |
echo "DATA_SOURCE_NAME=${{ secrets.DB_DATASOURCE }}" >> mon.env
echo "GF_SECURITY_ADMIN_USER=${{ secrets.GF_ADMIN_USER }}" >> mon.env
echo "GF_SECURITY_ADMIN_PASSWORD=${{ secrets.GF_ADMIN_PASS }}" >> mon.env
echo "DISCORD_TOKEN=${{ secrets.DISCORD_TOKEN }}" >> mon.env
echo "PROMCORD_PREFIX=promcord_" >> mon.env
echo "DB_CONNECTION_STRING=${{ secrets.DBC_STRING }}" >> mon.env
# - name: Setup SSH stuff
# run: |
# sudo mkdir -p ~/.ssh/
# sudo echo "${{ secrets.SSH_KEY }}" >> ~/.ssh/tempest
# sudo chmod 0400 ~/.ssh/tempest
# sudo echo "${{ secrets.KNOWN_HOSTS }}" >> ~/.ssh/known_hosts
# sudo echo -e "Host ${{ secrets.SSH_HOST }}\n\tHostName ${{ secrets.SSH_HOST }}\n\tUser ${{ secrets.SSH_USER }}\n\tIdentityFile ~/.ssh/tempest" >> ~/.ssh/config
- name: Install docker-compose
run: sudo pip install docker-compose
- name: Create context for docker host
run: docker context create remote --docker
- name: Set default context for docker
run: docker context use remote
- name: Always build the monitoring stack
run: COMPOSE_PARAMIKO_SSH=1 COMPOSE_IGNORE_ORPHANS=1 docker-compose --context remote -f docker-compose-monitoring.yml up --build -d
The output is:
0s
Run docker context create remote --docker
docker context create remote --docker
shell: /usr/bin/bash -e {0}
/actions-runner/actions-runner/_work/_temp/05fc146a-237e-4a92-b27d-796451184c0c.sh: line 1: docker: command not found
Error: Process completed with exit code 127.
I am trying to create a workflow that is able to create a docker compose for some monitoring tools. I have set up GitHub runners to do this and it has been successful for everything until the docker host section. The error is given above. Can I get some help as I am completely stumped?
Related
My goal is to export data from a unit test inside a multistage docker container. I have a docker create, docker cp, and docker rm that work in my terminal but when I added it to my docker-image.yml it fails to run and displays this error "Error: Process completed with exit code .". Also, I added in the unit test code for a github action that can't be accessed since the build fails.
[enter image description here][1]
- name: Build the Docker image
run: |
echo "${{ env.app_version }}"
echo "${{ github.run_number }}"
BUILD_NUMBER=${{ github.run_number }}
VERSION_NUMBER=${{ env.app_version }}
FULL_VERSION=${VERSION_NUMBER}.${BUILD_NUMBER}
docker build . --file Dockerfile --tag placeholder/${SERVICE_NAME}:${FULL_VERSION} --build-arg BUILD_NUMBER=${BUILD_NUMBER}
docker tag placeholder/${SERVICE_NAME}:${FULL_VERSION} placeholder/${SERVICE_NAME}:latest
echo "full_version=$FULL_VERSION" >> $GITHUB_ENV
**docker create --name unit_test test-export
docker cp unit_test:/app/surefire-reports extracted
docker rm unit_test**
# Runs a set of commands using the runners shell
- name: Run a multi-line script
run: |
echo Add other actions to build,
echo test and deploy your project.
ls -lath target/surefire-reports/
- name: Publish Unit Test Results
# You may pin to the exact commit or the version.
# Uses: EnricoMi/publish-unit-test-result-action#4a00ba50806e7658e5005bb91acdb3274714595a
uses: EnricoMi/publish-unit-test-result-action#v1.31
with:
files: target/surefire-reports/*.xml
I am modifying my docker-publish file to build a docker image so it can work with Arm64. The previous version was working fine with x86 architecture, but now I need to make it work for Arm 64 so I just changed the way the docker builds the images.
The build process works fine but somehow the git push stopped working and I am getting the error
Error response from daemon: No such image: myimage-arm64:latest
This is my docker-publish.yml
name: Docker
on:
push:
# Publish `master` as Docker `latest` image.
branches:
- master
# Publish `v1.2.3` tags as releases.
tags:
- v*
# Run tests for any PRs.
pull_request:
env:
IMAGE_NAME: myimage-arm64
jobs:
# Push image to GitHub Packages.
# See also https://docs.docker.com/docker-hub/builds/
push:
runs-on: ubuntu-latest
if: github.event_name == 'push'
permissions:
contents: read
packages: write
steps:
- uses: actions/checkout#v2
- name: Set up QEMU
uses: docker/setup-qemu-action#v1
- name: Set up Docker Buildx
id: buildx
uses: docker/setup-buildx-action#v1
- name: Prepare multiarch docker
run: docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
- name: Builder create
run: docker buildx create --use
- name: Log into registry
run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login docker.pkg.github.com -u ${{ github.actor }} --password-stdin
- name: Build image
run: |
docker buildx build \
--tag $IMAGE_NAME \
--file Dockerfile \
--platform linux/arm64 .
- name: Push image
run: |
IMAGE_ID=docker.pkg.github.com/${{ github.repository }}/$IMAGE_NAME
# Change all uppercase to lowercase
IMAGE_ID=$(echo $IMAGE_ID | tr '[A-Z]' '[a-z]')
# Strip git ref prefix from version
# VERSION=$(echo "${{ github.ref }}" | sed -e 's,.*/\(.*\),\1,')
# I changed this so it takes the version from a file on my project
VERSION=$(cat version)
# Strip "v" prefix from tag name
[[ "${{ github.ref }}" == "refs/tags/"* ]] && VERSION=$(echo $VERSION | sed -e 's/^v//')
# Use Docker `latest` tag convention
[ "$VERSION" == "master" ] && VERSION=latest
echo IMAGE_ID=$IMAGE_ID
echo VERSION=$VERSION
###
The two previous echo print the correct stuff
I get the error in these last two lines
###
docker tag $IMAGE_NAME $IMAGE_ID:$VERSION
docker push $IMAGE_ID:$VERSION
Any help? The push phase was working fine previously and I haven't touched it to make it work with arm64
EDIT 1:
I modified the procedure following the answers but still it does not work (error: tag is needed when pushing to register)
- name: Log into registry
run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login docker.pkg.github.com -u ${{ github.actor }} --password-stdin
- name: Builder create
run: docker buildx create --use
- name: Build image
run: |
IMAGE_ID=docker.pkg.github.com/${{ github.repository }}/$IMAGE_NAME
VERSION=$(cat version)
echo TAG=$IMAGE_ID:$VERSION
docker buildx build --push \
--tag $IMAGE_ID:$VERSION \
--file Dockerfile \
--platform linux/arm64 .
Precisely, the logs are these ones:
Run IMAGE_ID=docker.pkg.github.com/GiamBoscaro/portfolio-website/$IMAGE_NAME
TAG=docker.pkg.github.com/UserName/RepoName/ImageName:1.2.0
#1 [internal] booting buildkit
#1 sha256:bfa0dddd89a9c970aa189079c1d31d17f7a75edd434bb19ad90432b27b266e3a
#1 pulling image moby/buildkit:buildx-stable-1
#1 pulling image moby/buildkit:buildx-stable-1 0.4s done
#1 creating container buildx_buildkit_intelligent_volhard0
#1 creating container buildx_buildkit_intelligent_volhard0 0.9s done
#1 DONE 1.3s
error: tag is needed when pushing to registry
Error: Process completed with exit code 1.
EDIT 2: Finally fixed the issue. Even if it's not the best way, here's the code that works. I switched over to the new container registry and moved the docker login in the same job of docker buildx:
jobs:
push:
runs-on: ubuntu-latest
if: github.event_name == 'push'
permissions:
contents: read
packages: write
steps:
- uses: actions/checkout#v2
- name: Set up QEMU
uses: docker/setup-qemu-action#v1
- name: Set up Docker Buildx
id: buildx
uses: docker/setup-buildx-action#v1
- name: Prepare multiarch docker
run: docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
- name: Builder create
run: docker buildx create --use
- name: Build image
run: |
IMAGE_ID=ghcr.io/${{ github.actor }}/$IMAGE_NAME
# Change all uppercase to lowercase
IMAGE_ID=$(echo $IMAGE_ID | tr '[A-Z]' '[a-z]')
VERSION=$(cat version)
echo TAG=$IMAGE_ID:$VERSION
echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
docker buildx build --push \
--tag $IMAGE_ID:$VERSION \
--file Dockerfile.arm \
--platform linux/arm64 .
Buildx runs builds within a separate container, not directly in your docker engine. And the output of buildx does not store the resulting image in the local docker engine. This doesn't work when you get into multi-platform images anyway, so you typically push directly to the registry. It's much more efficient to avoid moving layers around that didn't change in the registry, and allows you to manage multi-platform images (everything loaded into the docker engine is dereferenced to a single platform).
If you really want to save the output to the local docker engine, you can use --load in the buildx command. However, the preferred option is to use the build-push-action that builds your tag directly and pushes it in one step. This would mean reordering your steps to determine the versions and other variables first, and then run the build against that. You can see an example of this in my own project which was assembled from various other docker examples out there.
Here's a quick untested attempt to make that change:
- name: Prepare
id: prep
run: |
IMAGE_ID=docker.pkg.github.com/${{ github.repository }}/$IMAGE_NAME
# Change all uppercase to lowercase
IMAGE_ID=$(echo $IMAGE_ID | tr '[A-Z]' '[a-z]')
# Strip git ref prefix from version
# VERSION=$(echo "${{ github.ref }}" | sed -e 's,.*/\(.*\),\1,')
# I changed this so it takes the version from a file on my project
VERSION=$(cat version)
# Strip "v" prefix from tag name
[[ "${{ github.ref }}" == "refs/tags/"* ]] && VERSION=$(echo $VERSION | sed -e 's/^v//')
# Use Docker `latest` tag convention
[ "$VERSION" == "master" ] && VERSION=latest
echo IMAGE_ID=$IMAGE_ID
echo VERSION=$VERSION
echo ::set-output name=version::${VERSION}
echo ::set-output name=docker_tag::${IMAGE_ID}:${VERSION}
- name: Build and push
uses: docker/build-push-action#v2
with:
context: .
file: Dockerfile
platforms: linux/arm64
push: true
tags: ${{ steps.prep.outputs.docker_tag }}
From the updated question, this is the entire command being run:
docker buildx build --push
The next command to run would be:
--tag $IMAGE_ID:$VERSION ...
I'm sure you're saying "Wait, what? There's a trailing slash, that's a multi-line command!" But there's also whitespace after that slash, so instead of escaping a linefeed, you've escaped a space character. Docker treats that space as the one arg and will attempt to build with the context being a directory named . To fix, remove the trailing whitespace after the backslash.
I am trying to run Gitlab CI job of anchore engine to scan docker image. The command in script section fails with error of permission denied. I found out the command requires root user permissions. Sudo is not installed in the docker image I'm using as gitlab runner and only non sudo user anchore is there in the docker container.
Below is the CI job for container scanning.
container_scan:
stage: scan
image:
name: anchore/anchore-engine:latest
entrypoint: ['']
services:
- name: anchore/engine-db-preload:latest
alias: anchore-db
variables:
GIT_STRATEGY: none
ANCHORE_HOST_ID: "localhost"
ANCHORE_ENDPOINT_HOSTNAME: "localhost"
ANCHORE_CLI_USER: "admin"
ANCHORE_CLI_PASS: "foobar"
ANCHORE_CLI_SSL_VERIFY: "n"
ANCHORE_FAIL_ON_POLICY: "true"
ANCHORE_TIMEOUT: "500"
script:
- |
curl -o /tmp/anchore_ci_tools.py https://raw.githubusercontent.com/anchore/ci-tools/master/scripts/anchore_ci_tools.py
chmod +x /tmp/anchore_ci_tools.py
ln -s /tmp/anchore_ci_tools.py /usr/local/bin/anchore_ci_tools
- anchore_ci_tools --setup
- anchore-cli registry add "$CI_REGISTRY" gitlab-ci-token "$CI_JOB_TOKEN" --skip-validate
- anchore_ci_tools --analyze --report --image "$IMAGE_NAME" --timeout "$ANCHORE_TIMEOUT"
- |
if ; then
anchore-cli evaluate check "$IMAGE_NAME"
else
set +o pipefail
anchore-cli evaluate check "$IMAGE_NAME" | tee /dev/null
fi
artifacts:
name: ${CI_JOB_NAME}-${CI_COMMIT_REF_NAME}
paths:
- image-*-report.json
The CI job fails at ln -s /tmp/anchore_ci_tools.py /usr/local/bin/anchore_ci_tools in the script section.
I have tried to add user in the entrypoint section
name: anchore/anchore-engine:latest
entrypoint: ['bash', '-c', 'useradd myuser && exec su myuser -c bash']
but it did not allow to create a user. I have tried running the docker container in linux with docker run -it --user=root anchore/anchore-engine:latest /bin/bash and it run without any problem. How can I simulate the same in gitlab-ci job?
Im trying to run a cron job in a digital ocean kubernetes cluster. The image is hosted as a github package. I am getting an authenticatin error when the image is being pulled. I also get the same error when trying to pull directly from docker in the command line. Is that the same problem? Or do I need to auth in 2 differnet places?
docker publish github
source: https://github.com/actions/starter-workflows/blob/aa9d3bc6cc46ac11a53ca196e504d4f901a8de8d/ci/docker-publish.yml
name: Docker
on:
push:
# Publish `master` as Docker `latest` image.
branches:
- master
# Publish `v1.2.3` tags as releases.
tags:
- v*
# Run tests for any PRs.
pull_request:
env:
# TODO: Change variable to your image's name.
IMAGE_NAME: image
jobs:
# Run tests.
# See also https://docs.docker.com/docker-hub/builds/automated-testing/
test:
ru fi
# Push image to GitHub Packages.
# See also https://docs.docker.com/docker-hub/builds/
push:
# Ensure test job passes before pushing image.
needs: test
runs-on: ubuntu-latest
if: github.event_name == 'push'
steps:
- uses: actions/checkout#v2
- name: Build image
run: docker build . --file Dockerfile --tag $IMAGE_NAME
- name: Log into registry
run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login docker.pkg.github.com -u ${{ github.actor }} --password-stdin
- name: Push image
run: |
IMAGE_ID=docker.pkg.github.com/${{ github.repository }}/$IMAGE_NAME
# Change all uppercase to lowercase
IMAGE_ID=$(echo $IMAGE_ID | tr '[A-Z]' '[a-z]')
# Strip git ref prefix from version
VERSION=$(echo "${{ github.ref }}" | sed -e 's,.*/\(.*\),\1,')
# Strip "v" prefix from tag name
[[ "${{ github.ref }}" == "refs/tags/"* ]] && VERSION=$(echo $VERSION | sed -e 's/^v//')
# Use Docker `latest` tag convention
[ "$VERSION" == "master" ] && VERSION=latest
echo IMAGE_ID=$IMAGE_ID
echo VERSION=$VERSION
docker tag $IMAGE_NAME $IMAGE_ID:$VERSION
docker push $IMAGE_ID:$VERSION
docker build . --file Dockerfile
fi
# Push image to GitHub Packages.
# See also https://docs.docker.com/docker-hub/builds/
push:
# Ensure test job passes before pushing image.
needs: test
runs-on: ubuntu-latest
if: github.event_name == 'push'
steps:
- uses: actions/checkout#v2
- name: Build image
run: docker build . --file Dockerfile --tag $IMAGE_NAME
- name: Log into registry
run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login docker.pkg.github.com -u ${{ github.actor }} --password-stdin
- name: Push image
run: |
IMAGE_ID=docker.pkg.github.com/${{ github.repository }}/$IMAGE_NAME
# Change all uppercase to lowercase
IMAGE_ID=$(echo $IMAGE_ID | tr '[A-Z]' '[a-z]')
# Strip git ref prefix from version
VERSION=$(echo "${{ github.ref }}" | sed -e 's,.*/\(.*\),\1,')
# Strip "v" prefix from tag name
[[ "${{ github.ref }}" == "refs/tags/"* ]] && VERSION=$(echo $VERSION | sed -e 's/^v//')
# Use Docker `latest` tag convention
[ "$VERSION" == "master" ] && VERSION=latest
echo IMAGE_ID=$IMAGE_ID
echo VERSION=$VERSION
docker tag $IMAGE_NAME $IMAGE_ID:$VERSION
docker push $IMAGE_ID:$VERSION
cron service
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: test1
spec:
schedule: "*/15 * * * *"
jobTemplate:
spec:
template:
spec:
containers:
- name: data
image: docker.pkg.github.com/lambda-capital/k8s-test/image:latest
restartPolicy: OnFailure
You can't pull anonymously from docker.pkg.github.com, you'll have to create a dockercfg with some GitHub access token and use it as imagePullSecret.
If you want to publicly host your image on GitHub you can use the newer ghcr.io container registry, which provides anonymous pulling capabilities for public images.
I am a CircleCI user, and I am setting up an integration with Heroku.
I want to do the following, and setup security with integrations with dockerHub and also to Heroku from the CircleCI portal page, using this config.yml file.
The problem is that CircleCI doesn't seem to know what these variables should be set to, and instead just echos.
${HEROKU_API_KEY} ${HEROKU_APP}
config.yml
version: 2
jobs:
build:
working_directory: ~/springboot_swagger_example-master-cassandra
docker:
- image: circleci/openjdk:8-jdk-browsers
steps:
- checkout
- restore_cache:
key: springboot_swagger_example-master-cassandra-{{ checksum "pom.xml" }}
- run: mvn dependency:go-offline
- save_cache:
paths:
- ~/.m2
key: springboot_swagger_example-master-cassandra-{{ checksum "pom.xml" }}
- type: add-ssh-keys
- type: deploy
name: "Deploy to Heroku"
command: |
if [ "${CIRCLE_BRANCH}" == "master" ]; then
# Install Heroku fingerprint (this is heroku's own key, NOT any of your private or public keys)
echo 'heroku.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAu8erSx6jh+8ztsfHwkNeFr/SZaSOcvoa8AyMpaerGIPZDB2TKNgNkMSYTLYGDK2ivsqXopo2W7dpQRBIVF80q9mNXy5tbt1WE04gbOBB26Wn2hF4bk3Tu+BNMFbvMjPbkVlC2hcFuQJdH4T2i/dtauyTpJbD/6ExHR9XYVhdhdMs0JsjP/Q5FNoWh2ff9YbZVpDQSTPvusUp4liLjPfa/i0t+2LpNCeWy8Y+V9gUlDWiyYwrfMVI0UwNCZZKHs1Unpc11/4HLitQRtvuk0Ot5qwwBxbmtvCDKZvj1aFBid71/mYdGRPYZMIxq1zgP1acePC1zfTG/lvuQ7d0Pe0kaw==' >> ~/.ssh/known_hosts
# git push git#heroku.com:yourproject.git $CIRCLE_SHA1:refs/heads/master
# Optional post-deploy commands
# heroku run python manage.py migrate --app=my-heroku-project
fi
- run: mvn package
- run:
name: Install Docker client
command: |
set -x
VER="17.03.0-ce"
curl -L -o /tmp/docker-$VER.tgz https://get.docker.com/builds/Linux/x86_64/docker-$VER.tgz
tar -xz -C /tmp -f /tmp/docker-$VER.tgz
mv /tmp/docker/* /usr/bin
- run:
name: Build Docker image
command: docker build -t joethecoder2/spring-boot-web:$CIRCLE_SHA1 .
- run:
name: Push to DockerHub
command: |
docker login -u$DOCKERHUB_LOGIN -p$DOCKERHUB_PASSWORD
docker push joethecoder2/spring-boot-web:$CIRCLE_SHA1
- run:
name: Setup Heroku
command: |
curl https://cli-assets.heroku.com/install-ubuntu.sh | sh
chmod +x .circleci/setup-heroku.sh
.circleci/setup-heroku.sh
- run:
name: Deploy to Heroku
command: |
mkdir app
cd app/
heroku create
# git push https://heroku:$HEROKU_API_KEY#git.heroku.com/$HEROKU_APP.git master
echo ${HEROKU_API_KEY}
echo ${HEROKU_APP}
git push https://heroku:${HEROKU_API_KEY}#git.heroku.com/${HEROKU_APP}.git master
- store_test_results:
path: target/surefire-reports
- store_artifacts:
path: target/spring-boot-web-0.0.1-SNAPSHOT.jar
The problem is that CircleCI doesn't seem to know what these variables should be set to, and instead just echos.
${HEROKU_API_KEY}
${HEROKU_APP}
The question, and problem is why aren't these settings being detected automatically?
You need to set the value for the variables: https://circleci.com/docs/2.0/env-vars/
They are being echo'd because you're echoing them.
echo ${HEROKU_API_KEY}
echo ${HEROKU_APP}