Coming from this issue
I am using GitHub Actions for Gradle project with this given steps:
name: Java CI
on: [push]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v1
- name: Set up JDK 13
uses: actions/setup-java#v1
with:
java-version: 13
- run: ./gradlew bootJar
- name: Login to Github regestry
run: docker login docker.pkg.github.com -u xxxxx -p xxxxx
- name: Build the Docker image
run: docker build . -t docker.pkg.github.com/sulimanlab/realtime-chat/realtimechat-snapshot-0.$GITHUB_REF
- name: Push the image to github
run: docker push docker.pkg.github.com/sulimanlab/realtime-chat/realtimechat-snapshot-0.$GITHUB_REF
At the last step I get this error:
The push refers to repository
[docker.pkg.github.com/sulimanlab/realtime-chat/realtimechat-snapshot-0.refs/heads/master]
3aad04996f8f: Preparing
77cae8ab23bf: Preparing
error parsing HTTP 404 response body: invalid character 'p' after top-level value:
"404 page not found\n"
actually I was using the wrong environment variable to tag my images.
I used $GITHUB_REF what I should use $GITHUB_SHA
Related
I am trying to use GitHub workflow to build an ASP.NET 6 project using Dockerfile then push the image to a private Azure Registry using docker.
Here is my .yml file
name: Docker Image CI
on:
push:
branches: [ "main" ]
pull_request:
branches: [ "main" ]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v3
- name: Login To Azure Container Registr
uses: Azure/docker-login#v1
with:
login-server: ${{ secrets.ACR_HOST }}
username: ${{ secrets.ACR_USERNAME }}
password: ${{ secrets.ACR_PASSWWORD }}
- name: Build And Push Docker Images
uses: docker/build-push-action#v3.1.1
with:
push: true
file: ./Dockerfile
tags: companyname/projectname:${{ github.run_number }}
In the above, the Dockerfile is located in the root of my project's code.
However, the the build runs I get the following error
Error: buildx failed with: error: denied: requested access to the resource is denied
In the Secrets > Action section in my repository settings, I added ACR_HOST, ACR_USERNAME and ACR_PASSWORD secrets.
When viewing the logs, this issue seems to happen after this line in the logs
pushing companyname/projectname:2 with docker:
How can I solve this issue?
UPDATED
I changed the .yml script to the following
name: Docker Image CI
on:
push:
branches: [ "main" ]
pull_request:
branches: [ "main" ]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v3
- name: Login To Azure Container Registr
uses: Azure/docker-login#v1
with:
login-server: mycontainer.azurecr.io
username: "The admin username"
password: "The admin password"
- run: cat ${{ env.DOCKER_CONFIG }}/config.json
- name: Build And Push Docker Images
uses: docker/build-push-action#v3.1.1
with:
push: true
file: ./Dockerfile
tags: companyname/projectname:${{ github.run_number }}
The added step (i.e., cat ${{ env.DOCKER_CONFIG }}/config.json) displayed a json string that look like this
{"auths":{"mycontainer.azurecr.io":{"auth":"BASE64 string with the admin username:password as expected"}}}
The base64 string was formatted like this username:password
I am assuming that the step Azure/docker-login#v1 has no issue and stages the token for docker/build-push-action#v3.1.1 correctly.
If I set the push flag to false in the docker/build-push-action#v3.1.1 step, the workflow runs with no issue. So from what I can tell, the issue is when the step docker/build-push-action#v3.1.1 tries to push the created image to the Azure registry.
I use my local machine to login using the same credentials and all worked with no issue docker login mycontainer.azurecr.io
Additionally, the login request from my local machine is logged into Azure portal. However, I do not see the request when I run the workflow.
I think that main issue is that the step docker/build-push-action#v3.1.1 does not attempt to login before it pushes the image.
I followed the instructions here and it worked.
im running my cypress code coverage report with using "npx nyc report --reporter=lcov --reporter=text-summary" also i have script which is "yarn e2e:coverage" but i want to see result in github actions log;
**
- name: Save logs
continue-on-error: true
if: ${{ always() }}
env:
COMMIT_SHA: ${{ steps.vars.outputs.sha_short }}
run: |
docker ps
docker cp cypress_test:/cypress-coverage cypress-coverage
- name: Compress action step
uses: a7ul/tar-action#v1.1.0
id: compress
with:
command: c
cwd: .
files: |
cypress-coverage
outPath: test_cypress_coverage.tar
- name: Archive coverage
continue-on-error: true
if: ${{ always() }}
env:
COMMIT_SHA: ${{ steps.vars.outputs.sha_short }}
uses: actions/upload-artifact#v2
with:
name: "${{ steps.date.outputs.yyyymmdd }}_E2E_test_coverage_${{ env.COMMIT_SHA }}"
path: |
test_cypress_coverage.tar
retention-days: 7
**
Im using this part for it but there is a error "Error: No such container:path: cypress_test:/cypress-coverage" do you guys have a any idea how to find correct path ? How to see log of coverage result ? Btw i can get artifacts its working as expected but i got save logs error.
If the compress_test container has a bash, I would try and check:
if there is a log file produced at that stage
where it is.
That would be:
docker exec -it compress_test find / -name "cypress-coverage*"
That way, you can see for yourself where the file is.
I'm currently working on a small CICD project that will run a series of tests on Github Actions using dynamodb-local whenever I update my code and then package and deploy if the tests are successful.
I have the following workflow:
name: backend_actions
on:
workflow_dispatch:
push:
paths:
- 'backend/*'
branches:
- master
jobs:
test-locally:
runs-on: ubuntu-latest
outputs:
test-result: ${{ steps.run-tests.outputs.result }}
steps:
- uses: actions/checkout#v2
- uses: actions/setup-python#v2
with:
python-version: '3.9'
- uses: aws-actions/setup-sam#v1
- uses: aws-actions/configure-aws-credentials#v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: us-west-2
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
- name: Setup local DynamoDB
run: docker run -p 8000:8000 amazon/dynamodb-local
- name: Create table
run: aws dynamodb create-table --cli-input-json file://backend/src/test/make_table.json --endpoint-url http://localhost:8000
- name: start local API Gateway
run: sam local start-api --env-vars backend/env.json
- id: run-tests
name: Run tests
run: |
python backend/src/test_dynamoDB_lambda.py
echo "::set-output name=result::$?"
update_backend:
needs: test-locally
if: ${{ needs.test-locally.outputs.test-result == '0' }}
runs-on: ubuntu-latest
steps:
- name: Package and deploy
run: |
aws cloudformation package --s3-bucket cloud-resume-bucket \
--template-file backend/template.yaml --output-template-file backend/gen/template-gen.yaml
aws cloudformation deploy --template-file backend/gen/template-gen.yaml --stack-name cloud-formation-resume \
--capabilities CAPABILITY_IAM
When I try running the workflow in Github Actions, it will get to the 'Setup local DynamoDB' step, output the text below, and then hang.
Run docker run -p 8000:8000 amazon/dynamodb-local
Unable to find image 'amazon/dynamodb-local:latest' locally
latest: Pulling from amazon/dynamodb-local
2cbe74538cb5: Pulling fs layer
137077f50205: Pulling fs layer
58932e640a40: Pulling fs layer
58932e640a40: Verifying Checksum
58932e640a40: Download complete
2cbe74538cb5: Verifying Checksum
2cbe74538cb5: Download complete
137077f50205: Verifying Checksum
137077f50205: Download complete
2cbe74538cb5: Pull complete
137077f50205: Pull complete
58932e640a40: Pull complete
Digest: sha256:bdd26570dc0e0ae49e1ea9d49ff662a6a1afe9121dd25793dc40d02802e7e806
Status: Downloaded newer image for amazon/dynamodb-local:latest
Initializing DynamoDB Local with the following configuration:
Port: 8000
InMemory: true
DbPath: null
SharedDb: false
shouldDelayTransientStatuses: false
CorsParams: *
Seems like it can find the docker image and download it fine, but stops upon initializing? This is my first time working with Github Actions and Docker, so I'm not really sure why it's hanging on Github Actions and not when I run it on my own computer, so any help would be appreciated!
When you run the command docker run -p 8000:8000 amazon/dynamodb-local the process never exits, so the Github run block doesn't actually know when to move on to the next step—it just hangs there forever.
What I did in my project is simply background it, by using the & after the command:
- name: Setup local DynamoDB
run: docker run -p 8000:8000 amazon/dynamodb-local &
Github Workflows will start the Docker container and move to the next run step, and when all the steps are done it'll just kill the container as part of normal cleanup. Because of this, you don't need to worry about shutting it down at the end.
The difficulty with this approach is that it takes several seconds for DynamoDB-local to start up, but your next step relies on it and will likely throw ECONNREFUSED errors.
What I've done in my project is to have the next run step execute a script that attempts to list tables, retrying with a short delay until it gets back a response.
The bash command is simply (you would need to put this in a while+try/catch loop):
aws dynamodb list-tables --endpoint-url http://localhost:8000
As a guide, this is (roughly) what I do in JavaScript, using the aws-sdk and NodeJS#16:
// wait-for-dynamodb.js
import timers from 'timers/promises'
import AWS from 'aws-sdk'
const dynamodb = new AWS.DynamoDB()
const waitForDynamoDbToStart = async () => {
try {
await dynamodb.listTables().promise()
} catch (error) {
console.log('Waiting for Docker container to start...')
await timers.setTimeout(500)
return waitForDynamoDbToStart()
}
}
const start = Date.now()
waitForDynamoDbToStart()
.then(() => {
console.log(`DynamoDB-local started after ${Date.now() - start}ms.`)
process.exit(0)
})
.catch(error => {
console.log('Error starting DynamoDB-local!', error)
process.exit(1)
})
Then I simply have that in the run steps:
- name: Setup local DynamoDB
run: docker run -p 8000:8000 amazon/dynamodb-local &
- name: Wait for it to boot up
run: node ./wait-for-dynamodb.js
# now you're guaranteed to have DynamoDB-local running
I am trying to write a custom github-action that runs some commands in a docker container but allows the user to select which docker container they are run in (i.e. so I can run the same build instructions across different versions of the runtime environment)
My gut instinct was to have my .github/actions/main/action.yml file as
name: 'Docker container command execution'
inputs:
dockerfile:
default: Dockerfile_r_latest
runs:
using: 'docker'
image: '${{ inputs.dockerfile }}'
args:
- /scripts/commands.sh
However this errors with:
##[error](Line: 7, Col: 10): Unrecognized named-value: 'inputs'. Located at position 1 within expression: inputs.dockerfile
Any help would be appreciated !
File References
My .github/workflow/build_and_test.yml file is:
name: Test Package
on:
[push, pull_request]
jobs:
R_latest:
name: Test on latest
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#master
name: Checkout project
- uses: ./.github/actions/main
name: Build and test
with:
dockerfile: Dockerfile_r_latest
And my Dockerfile .github/actions/main/Dockerfile_r_latest is:
FROM rocker/verse:latest
ADD scripts /scripts
ENTRYPOINT [ "bash", "-c" ]
Interesting approach! I'm not sure if it's possible to use expressions in the image field of the action metadata. I would guess that the only fields that can take expressions instead of hardcoded strings are the args for the image so that the inputs can be passed.
For reference this is the args section of the action.yml metadata.
https://help.github.com/en/articles/metadata-syntax-for-github-actions#args
I think there are other ways to achieve what you want to do. Have you tried using the jobs.<job_id>.container syntax? That allows you to specify an image that the steps of a job will run in. It will require that you publish the image to a public repository, though. So take care not to include any secrets.
For example, if you published your image to Docker Hub at gowerc/r-latest your workflow might look something like this:
name: Test Package
on:
[push, pull_request]
jobs:
R_latest:
name: Test on latest
runs-on: ubuntu-latest
container: gowerc/r-latest
steps:
- uses: actions/checkout#master
name: Checkout project
- name: Build and test
run: ./scripts/commands.sh
ref: https://help.github.com/en/articles/workflow-syntax-for-github-actions#jobsjob_idcontainer
Alternatively, you can also specify your image at the step level with uses. You could then pass a command via args to execute your script.
name: my workflow
on: push
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#master
- name: Check container
uses: docker://alpine:3.8
with:
args: /bin/sh -c "cat /etc/alpine-release"
ref: https://help.github.com/en/github/automating-your-workflow-with-github-actions/workflow-syntax-for-github-actions#example-using-a-docker-hub-action
In addition to #peterevans answer, I would add there's a 3rd option where you can use a simple docker run command and pass any env that you have defined.
That helped to solve 3 things :
Reuse a custom docker image being build within the steps for testing actions. It seems not possible to do so with uses as it first tries to pull that image that doesn't exist yet in a Setup job step that occurs before any steps of the job.
This specific image can also be stored in a private docker registry
Be able to use a variable for the docker image
My workflow looks like this :
name: Build-Test-Push
on:
push:
branches:
- master
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
ECR_REGISTRY: ${{ secrets.AWS_ECR_REGISTRY }}
ECR_REPOSITORY: myproject/myimage
IMAGE_TAG: ${{ github.sha }}
jobs:
build-and-push:
runs-on: ubuntu-latest
steps:
- name: Checking out
uses: actions/checkout#v2
with:
ref: master
- name: Login to AWS ECR
id: login-ecr
uses: aws-actions/amazon-ecr-login#v1
- name: Build
run: |
docker pull $ECR_REGISTRY/$ECR_REPOSITORY || true
docker build . -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG -t $ECR_REGISTRY/$ECR_REPOSITORY:latest
- name: Test
run: |
docker run $ECR_REGISTRY/$ECR_REPOSITORY:latest /bin/bash -c "make test"
- name: Push
run: |
docker push $ECR_REGISTRY/$ECR_REPOSITORY
Here is another approach. The Docker image to use is passed to a cibuild shell script that takes care of pulling the right image.
GitHub workflow file:
name: 'GH Actions CI'
on:
push:
branches: ['*master', '*0.[0-9]?.x']
pull_request:
# The branches below must be a subset of the branches above
branches: ['*master', '*0.[0-9]?.x']
jobs:
build:
name: Build
runs-on: ubuntu-latest
strategy:
fail-fast: true
matrix:
include:
- FROM: 'ubuntu:focal'
- FROM: 'ubuntu:bionic'
- FROM: 'ubuntu:xenial'
- FROM: 'debian:buster'
- FROM: 'debian:stretch'
- FROM: 'opensuse/leap'
- FROM: 'fedora:33'
- FROM: 'fedora:32'
- FROM: 'centos:8'
steps:
- name: Checkout repository
uses: actions/checkout#v2
with:
# We must fetch at least the immediate parents so that if this is
# a pull request then we can checkout the head.
fetch-depth: 2
# If this run was triggered by a pull request event, then checkout
# the head of the pull request instead of the merge commit.
- run: git checkout HEAD^2
if: ${{ github.event_name == 'pull_request' }}
- name: Run CI
env:
FROM: ${{ matrix.FROM }}
run: script/cibuild
Bash script script/cibuild:
#!/bin/bash
set -e
docker run --name my-docker-container $FROM script/custom-script.sh
docker cp my-docker-container:/usr/src/my-workdir/my-outputs .
docker rm my-docker-container
echo "cibuild Done!"
Put your custom commands in script/custom-script.sh.
I'm trying to build and push image with drone.io's plugins/docker, but it seems cannot find my repo name.
Here is the last log about the build step.
---> Running in afca20280587
Removing intermediate container afca20280587
---> cb05c781a4c4
Successfully built cb05c781a4c4
Successfully tagged caa418f0605dc7a6b2bc84faebabac55a09a373b:latest
+ /usr/local/bin/docker tag caa418f0605dc7a6b2bc84faebabac55a09a373b :latest
Error parsing reference: ":latest" is not a valid repository/tag: invalid reference format
time="2019-01-02T02:05:18Z" level=fatal msg="exit status 1"
The sixth line should be
+ /usr/local/bin/docker tag caa418f0605dc7a6b2bc84faebabac55a09a373b registry.cn-beijing.aliyuncs.com/xxx/xxx_xxx:latest
But now it didn't find my repo name.
It's drone/drone:1.0.0-rc.3, and here is my .drone.yml
kind: pipeline
name: default
steps:
- name: build
image: python:3.6
commands:
- pip install -r requirements.txt
- python -m pytest app.py
when:
branch: master
event:
- push
- pull_request
- name: publish
image: plugins/docker
registry: registry.cn-beijing.aliyuncs.com
repo: xxx/xxx_xxx
tags: [ latest ]
username:
- from_secret: ali_username
password:
- from_secret: ali_password
Is there something wrong? Thanks for any tip!
When you define the repository you need to use the fully qualified image name:
- repo: xxx/xxx_xxx
+ repo: registry.cn-beijing.aliyuncs.com/xxx/xxx_xxx
In addition, all of the plugin settings need to be declared inside the settings block [1] like this:
- name: publish
image: plugins/docker
settings:
registry: registry.cn-beijing.aliyuncs.com
repo: registry.cn-beijing.aliyuncs.com/xxx/xxx_xxx
username:
- from_secret: ali_username
password:
- from_secret: ali_password
[1] http://plugins.drone.io/drone-plugins/drone-docker/