How can I make EKS kick working with CircleCI? - docker

I can push images to ECR but I am not even close sure what I should do next (what should be the flow) to make my images run on Kubernetes on EKS
jobs:
create-deployment:
executor: aws-eks/python3
parameters:
cluster-name:
description: |
Name of the EKS cluster
type: string
steps:
- checkout
- aws-eks/update-kubeconfig-with-authenticator:
cluster-name: << parameters.cluster-name >>
install-kubectl: true
- kubernetes/create-or-update-resource:
get-rollout-status: true
resource-file-path: tests/nginx-deployment/deployment.yaml
# resource-file-path: configs/k8s/prod-deployment.yaml
resource-name: deployment/prod-deployment
orbs:
aws-ecr: circleci/aws-ecr#6.15.0
aws-eks: circleci/aws-eks#1.1.0
kubernetes: circleci/kubernetes#0.4.0
version: 2.1
workflows:
deployment:
jobs:
- aws-ecr/build-and-push-image:
repo: bwtc-backend
tag: "${CIRCLE_BRANCH}-v0.1.${CIRCLE_BUILD_NUM}"
dockerfile: configs/Docker/Dockerfile.prod
path: .
filters:
branches:
ignore:
- master
- aws-eks/create-cluster:
cluster-name: eks-demo-deployment
requires:
- aws-ecr/build-and-push-image
- create-deployment:
cluster-name: eks-demo-deployment
requires:
- aws-eks/create-cluster
- aws-eks/update-container-image:
cluster-name: eks-demo-deployment
container-image-updates: 'nginx=nginx:1.9.1'
post-steps:
- kubernetes/delete-resource:
resource-names: nginx-deployment
resource-types: deployment
wait: true
record: true
requires:
- create-deployment
resource-name: deployment/nginx-deployment
- aws-eks/delete-cluster:
cluster-name: eks-demo-deployment
requires:
- aws-eks/update-container-image
That's what I've got in my config for now.
The problem I am facing at the moment is:
gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
Exited with code exit status 2
CircleCI received exit code 2
I am using a snippet from CircleCI Documentation, so I guess it should work.
I passed in all the params as I can see but I can't get what I've missed here.
I need your help guys!

The last update has a bug. It is failing due to an incorrect URL. The issue is still open, see here. The correct url should be
https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz
unlike the current
"https://github.com/weaveworks/eksctl/releases/download/latest_release/eksctl_$(uname -s)_amd64.tar.gz"
While you wait for it to be approved, use the following to install eksctl first.
- run:
name: Install the eksctl tool
command: |
if which eksctl > /dev/null; then
echo "eksctl is already installed"
exit 0
fi
mkdir -p eksctl_download
curl --silent --location --retry 5 "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" \
| tar xz -C eksctl_download
chmod +x eksctl_download/eksctl
SUDO=""
if [ $(id -u) -ne 0 ] && which sudo > /dev/null ; then
SUDO="sudo"
fi
$SUDO mv eksctl_download/eksctl /usr/local/bin/
rmdir eksctl_download
and then run the job
- aws-eks/create-cluster:
cluster-name: eks-demo-deployment
That should solve the issue.
An example
# Creation of Cluster
create-cluster:
executor: aws-eks/python3
parameters:
cluster-name:
description: |
Name of the EKS cluster
type: string
steps:
- run:
name: Install the eksctl tool
command: |
if which eksctl > /dev/null; then
echo "eksctl is already installed"
exit 0
fi
mkdir -p eksctl_download
curl --silent --location --retry 5 "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" \
| tar xz -C eksctl_download
chmod +x eksctl_download/eksctl
SUDO=""
if [ $(id -u) -ne 0 ] && which sudo > /dev/null ; then
SUDO="sudo"
fi
$SUDO mv eksctl_download/eksctl /usr/local/bin/
rmdir eksctl_download
- aws-eks/create-cluster:
cluster-name: eks-demo-deployment

Related

Why Cypress service is failing?

Below is my pipeline on which I'm trying to get Cypress job to run tests against
Nginx service (which points to the main app) which is built at the build stage
The below is based on official template from here https://gitlab.com/cypress-io/cypress-example-docker-gitlab/-/blob/master/.gitlab-ci.yml
:
image: docker:stable
services:
- docker:dind
stages:
- build
- test
cache:
key: ${CI_COMMIT_REF_SLUG}
paths:
- .npm
- cache/Cypress
- node_modules
job:
stage: build
script:
- export REACT_APP_USERS_SERVICE_URL=http://127.0.0.1
- apk add --update --no-cache gcc g++ make python2 python2-dev py-pip python3-dev docker-compose npm
- docker-compose up -d --build
e2e:
image: cypress/included:9.1.1
stage: test
script:
- export CYPRESS_VIDEO=false
- export CYPRESS_baseUrl=http://nginx:80
- npm i randomstring
- $(npm bin)/cypress run -t -v $PWD/e2e -w /e2e -e CYPRESS_VIDEO -e CYPRESS_baseUrl --network testdriven_default
- docker-compose down
Error output:
Cypress encountered an error while parsing the argument config
You passed: if [ -x /usr/local/bin/bash ]; then
exec /usr/local/bin/bash
elif [ -x /usr/bin/bash ]; then
exec /usr/bin/bash
elif [ -x /bin/bash ]; then
exec /bin/bash
elif [ -x /usr/local/bin/sh ]; then
exec /usr/local/bin/sh
elif [ -x /usr/bin/sh ]; then
exec /usr/bin/sh
elif [ -x /bin/sh ]; then
exec /bin/sh
elif [ -x /busybox/sh ]; then
exec /busybox/sh
else
echo shell not found
exit 1
fi
The error was: Cannot read properties of undefined (reading 'split')
What is wrong with this set up ?
From #jparkrr on GitHub : https://github.com/cypress-io/cypress-docker-images/issues/300#issuecomment-626324350
I had the same problem. You can specify entrypoint: [""] for the image in .gitlab-ci.yml.
Read more about it here: https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#overriding-the-entrypoint-of-an-image
In your case :
e2e:
image:
name: cypress/included:9.1.1
entrypoint: [""]

CircleCI branch build failing but tag build succeeds

I am building my project on CircleCI and I have a build job that looks like this:
build:
<<: *defaults
steps:
- checkout
- setup_remote_docker
- run:
name: Install pip
command: curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && sudo python get-pip.py
- run:
name: Install AWS CLI
command: curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip" && unzip awscli-bundle.zip && sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws
- run:
name: Login to Docker Registry
command: aws ecr get-login --no-include-email --region us-east-1 | sh
- run:
name: Install Dep
command: curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
- run:
name: Save Version Number
command: echo "export VERSION_NUM=${CIRCLE_TAG}.${CIRCLE_BUILD_NUM}" > deployment/dev/.env
- run:
name: Build App
command: source deployment/dev/.env && docker-compose -f deployment/dev/docker-compose.yml build
- run:
name: Test App
command: |
git config --global url."https://${GITHUB_PERSONAL_ACCESS_TOKEN} :x-oauth-basic#github.com/".insteadOf "https://github.com/"
dep ensure
go test -v ./...
- run:
name: Push Image
command: |
if [[ "${CIRCLE_TAG}" =~ ^v[0.9]+(\.[0-9]+)*-[a-z]*$ ]]; then
source deployment/dev/.env
docker-compose -f deployment/dev/docker-compose.yml push
else
echo 'No tag, not deploying'
fi
- persist_to_workspace:
root: .
paths:
- deployment/*
- tools/*
When I push a change to a branch, the build fails every time with Couldn't connect to Docker daemon at ... - is it running? when it reaches the Build App step of the build job.
Please help me figure out why branch builds are failing but tag builds are not.
I suspect you are hitting this docker-compose bug: https://github.com/docker/compose/issues/6050
The bug reports a misleading error (the one you're getting) when an image name in the docker-compose file is invalid.
If you use an environment variable for the image name or image tag, and that variable is set from a branch name, then it would fail on some branches, but not others.
The problem was occurring on the Save Version Number step. Sometimes that version would be .${CIRCLE_BUILD_NUM} since no tag was passed. Docker dislikes these tags starting with ., so I added a conditional check to see if CIRCLE_TAG was empty, and if it was, use some default version: v0.1.0-build.

CircleCI environmental variables for HEROKU not being set properly causing GIT to fail

I am a CircleCI user, and I am setting up an integration with Heroku.
I want to do the following, and setup security with integrations with dockerHub and also to Heroku from the CircleCI portal page, using this config.yml file.
The problem is that CircleCI doesn't seem to know what these variables should be set to, and instead just echos.
${HEROKU_API_KEY} ${HEROKU_APP}
config.yml
version: 2
jobs:
build:
working_directory: ~/springboot_swagger_example-master-cassandra
docker:
- image: circleci/openjdk:8-jdk-browsers
steps:
- checkout
- restore_cache:
key: springboot_swagger_example-master-cassandra-{{ checksum "pom.xml" }}
- run: mvn dependency:go-offline
- save_cache:
paths:
- ~/.m2
key: springboot_swagger_example-master-cassandra-{{ checksum "pom.xml" }}
- type: add-ssh-keys
- type: deploy
name: "Deploy to Heroku"
command: |
if [ "${CIRCLE_BRANCH}" == "master" ]; then
# Install Heroku fingerprint (this is heroku's own key, NOT any of your private or public keys)
echo 'heroku.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAu8erSx6jh+8ztsfHwkNeFr/SZaSOcvoa8AyMpaerGIPZDB2TKNgNkMSYTLYGDK2ivsqXopo2W7dpQRBIVF80q9mNXy5tbt1WE04gbOBB26Wn2hF4bk3Tu+BNMFbvMjPbkVlC2hcFuQJdH4T2i/dtauyTpJbD/6ExHR9XYVhdhdMs0JsjP/Q5FNoWh2ff9YbZVpDQSTPvusUp4liLjPfa/i0t+2LpNCeWy8Y+V9gUlDWiyYwrfMVI0UwNCZZKHs1Unpc11/4HLitQRtvuk0Ot5qwwBxbmtvCDKZvj1aFBid71/mYdGRPYZMIxq1zgP1acePC1zfTG/lvuQ7d0Pe0kaw==' >> ~/.ssh/known_hosts
# git push git#heroku.com:yourproject.git $CIRCLE_SHA1:refs/heads/master
# Optional post-deploy commands
# heroku run python manage.py migrate --app=my-heroku-project
fi
- run: mvn package
- run:
name: Install Docker client
command: |
set -x
VER="17.03.0-ce"
curl -L -o /tmp/docker-$VER.tgz https://get.docker.com/builds/Linux/x86_64/docker-$VER.tgz
tar -xz -C /tmp -f /tmp/docker-$VER.tgz
mv /tmp/docker/* /usr/bin
- run:
name: Build Docker image
command: docker build -t joethecoder2/spring-boot-web:$CIRCLE_SHA1 .
- run:
name: Push to DockerHub
command: |
docker login -u$DOCKERHUB_LOGIN -p$DOCKERHUB_PASSWORD
docker push joethecoder2/spring-boot-web:$CIRCLE_SHA1
- run:
name: Setup Heroku
command: |
curl https://cli-assets.heroku.com/install-ubuntu.sh | sh
chmod +x .circleci/setup-heroku.sh
.circleci/setup-heroku.sh
- run:
name: Deploy to Heroku
command: |
mkdir app
cd app/
heroku create
# git push https://heroku:$HEROKU_API_KEY#git.heroku.com/$HEROKU_APP.git master
echo ${HEROKU_API_KEY}
echo ${HEROKU_APP}
git push https://heroku:${HEROKU_API_KEY}#git.heroku.com/${HEROKU_APP}.git master
- store_test_results:
path: target/surefire-reports
- store_artifacts:
path: target/spring-boot-web-0.0.1-SNAPSHOT.jar
The problem is that CircleCI doesn't seem to know what these variables should be set to, and instead just echos.
${HEROKU_API_KEY}
${HEROKU_APP}
The question, and problem is why aren't these settings being detected automatically?
You need to set the value for the variables: https://circleci.com/docs/2.0/env-vars/
They are being echo'd because you're echoing them.
echo ${HEROKU_API_KEY}
echo ${HEROKU_APP}

Build FAILED but job status is SUCCESS in Gitlab

My Dockerfile:
FROM mm_php:7.1
ADD ./docker/test/source/entrypoint.sh /work/entrypoint.sh
ADD ./docker/wait-for-it.sh /work/wait-for-it.sh
RUN chmod 755 /work/entrypoint.sh \
&& chmod 755 /work/wait-for-it.sh
ENTRYPOINT ["/work/entrypoint.sh"]
entrypoint.sh:
#!/bin/bash -e
/work/wait-for-it.sh db:5432 -- echo "PostgreSQL started"
./vendor/bin/parallel-phpunit --pu-cmd="./vendor/bin/phpunit -c phpunit-docker.xml" tests
docker-compose.yml:
version: '2'
services:
test:
build:
context: .
args:
ssh_prv_key: ${ssh_prv_key}
application_env: ${application_env}
dockerfile: docker/test/source/Dockerfile
links:
- db
db:
build:
context: .
dockerfile: docker/test/postgres/Dockerfile
environment:
PGDATA: /tmp
.gitlab-ci.yml:
image: docker:latest
services:
- name: docker:dind
command: ["--insecure-registry=my.domain:5000 --registry-mirror=http://my.domain"]
before_script:
- apk add --no-cache py-pip
- pip install docker-compose
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- echo "$SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
- echo "$SSH_PRIVATE_KEY" > ~/.ssh/id_rsa
- chmod 600 ~/.ssh/id_rsa
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
test:
stage: test
script:
- export ssh_prv_key="$(cat ~/.ssh/id_rsa)" && export application_env="testing-docker" && docker-compose up --build test
All works good. But if tests are failed, status of job in Gitlab is SUCCESS instead of FAILED.
How to obtain status FAILED if tests are failed?
UPD
If I run docker-compose up locally, it return no error code:
$ export ssh_prv_key="$(cat ~/.ssh/id_rsa)" && export application_env="testing-docker" && docker-compose up --build test
Building db
Step 1/2 : FROM mm_postgres:9.6
...
test_1 | FAILURES!
test_1 | Tests: 1, Assertions: 1, Failures: 1.
test_1 | Success: 2 Fail: 2 Error: 0 Skip: 2 Incomplete: 0
mmadmin_test_1 exited with code 1
$ echo $?
0
It looks to me like it's reporting failed on the test without necessarily reporting failure on the return value of the docker-compose call. Have you tried capturing the return value of docker-compose when tests fail locally?
In order to get docker-compose to return the exit code from a specific service, try this:
docker-compose up --exit-code-from=service
When Gitlab CI runs something, if the process executed returns something different from zero, then, your build will fail.
In your case, you are running a docker-compose and this program returns zero when the container finish, what is correct.
You are trying to get phpunit's failure.
I think that is better you split your build in steps and not use docker-compose in this case:
gitlab.yml:
stages:
- build
- test
build:
image: docker:latest
stage: build
script:
- docker build -t ${NAME_OF_IMAGE} .
- docker push ${NAME_OF_IMAGE}
test:
image: ${NAME_OF_IMAGE}
stage: test
script:
- ./execute_your.sh

Are Instrumentation tests for Android Espresso available on CircleCi 2.0?

Are Instrumentation tests for Android Espresso available on CircleCI 2.0?
If yes, can anybody, please, help to configure config.yml file for me?
I’ve made thousand attempts and no luck. I can run unit tests, but not Instrumentation.
Thanks
The answer for this question is: yes. Instrumentation tests are possible for CircleCi. This is the configuration I have:
version: 2
jobs:
build:
working_directory: ~/code
docker:
- image: circleci/android:api-25-alpha
environment:
JVM_OPTS: -Xmx3200m
steps:
- checkout
- restore_cache:
key: jars-{{ checksum "build.gradle" }}-{{ checksum "app/build.gradle" }}
- run:
name: Chmod permissions #if permission for Gradlew Dependencies fail, use this.
command: sudo chmod +x ./gradlew
- run:
name: Download Dependencies
command: ./gradlew androidDependencies
- save_cache:
paths:
- ~/.gradle
key: jars-{{ checksum "build.gradle" }}-{{ checksum "app/build.gradle" }}
- run:
name: Setup emulator
command: sdkmanager "system-images;android-25;google_apis;armeabi-v7a" && echo "no" | avdmanager create avd -n test -k "system-images;android-25;google_apis;armeabi-v7a"
- run:
name: Launch emulator
command: export LD_LIBRARY_PATH=${ANDROID_HOME}/emulator/lib64:${ANDROID_HOME}/emulator/lib64/qt/lib && emulator64-arm -avd test -noaudio -no-boot-anim -no-window -accel on
background: true
- run:
name: Wait emulator
command: |
# wait for it to have booted
circle-android wait-for-boot
# unlock the emulator screen
sleep 30
adb shell input keyevent 82
- run:
name: Run Tests
command: ./gradlew connectedAndroidTest
- store_artifacts:
path: app/build/reports
destination: reports
- store_test_results:
path: app/build/test-results
The only problem with this configuration that it doesn't lead to successfull build because of not enough memory error. If somebody has better configuration, please, share.
I am running Android UI tests on CircleCI MacOS executor.
Here is my configuration:
version: 2
reference:
## Constants
gradle_cache_path: &gradle_cache_path
gradle_cache-{{ checksum "build.gradle" }}-{{ checksum "app/build.gradle" }}
workspace: &workspace
~/src
## Configurations
android_config: &android_config
working_directory: *workspace
macos:
xcode: "9.4.0"
shell: /bin/bash --login -eo pipefail
environment:
TERM: dumb
JVM_OPTS: -Xmx3200m
## Cache
restore_gradle_cache: &restore_gradle_cache
restore_cache:
key: *gradle_cache_path
save_gradle_cache: &save_gradle_cache
save_cache:
key: *gradle_cache_path
paths:
- ~/.gradle
## Dependency Downloads
download_android_dependencies: &download_android_dependencies
run:
name: Download Android Dependencies
command: ./gradlew androidDependencies
jobs:
ui_test:
<<: *android_config
steps:
- checkout
- run:
name: Setup environment variables
command: |
echo 'export PATH="$PATH:/usr/local/opt/node#8/bin:${HOME}/.yarn/bin:${HOME}/${CIRCLE_PROJECT_REPONAME}/node_modules/.bin:/usr/local/share/android-sdk/tools/bin"' >> $BASH_ENV
echo 'export ANDROID_HOME="/usr/local/share/android-sdk"' >> $BASH_ENV
echo 'export ANDROID_SDK_HOME="/usr/local/share/android-sdk"' >> $BASH_ENV
echo 'export ANDROID_SDK_ROOT="/usr/local/share/android-sdk"' >> $BASH_ENV
echo 'export QEMU_AUDIO_DRV=none' >> $BASH_ENV
echo 'export JAVA_HOME=/Library/Java/Home' >> $BASH_ENV
- run:
name: Install Android sdk
command: |
HOMEBREW_NO_AUTO_UPDATE=1 brew tap homebrew/cask
HOMEBREW_NO_AUTO_UPDATE=1 brew cask install android-sdk
- run:
name: Install emulator dependencies
command: (yes | sdkmanager "platform-tools" "platforms;android-26" "extras;intel;Hardware_Accelerated_Execution_Manager" "build-tools;26.0.0" "system-images;android-26;google_apis;x86" "emulator" --verbose) || true
- *restore_gradle_cache
- *download_android_dependencies
- *save_gradle_cache
- run: avdmanager create avd -n Pixel_2_API_26 -k "system-images;android-26;google_apis;x86" -g google_apis -d "Nexus 5"
- run:
name: Run emulator in background
command: /usr/local/share/android-sdk/tools/emulator #Pixel_2_API_26 -skin 1080x2066 -memory 2048 -noaudio
background: true
- run:
name: Run Tests
command: ./gradlew app:connectedAndroidTest
https://gist.github.com/DoguD/58b4b86a5d892130af84074078581b87
I hope it helps

Resources