I'm new to ansible (and docker). I would like to test my ansible playbook before using it on any staging/production servers.
Since I don't have access to an empty remote server, I thought the easiest way to test would be to use Docker container and then just run my playbook with the Docker container as the host.
I have a basic DockerFile that creates a standard ubuntu container. How would I configure the ansible hosts in order to run it against the docker container? Also, I suspect I would need to "run" the docker container to allow ansible to connect to it.
Running the playbook in a docker container may not actually be the best approach unless your stage and production servers are also Docker containers. The Docker ubuntu image is stripped down and will have some differences from a full installation. A better option might be to run the playbook in an Ubuntu VM that matches your staging and production installations.
That said, in order to run the ansible playbook within the container you should write a Dockerfile that runs your playbook. Here's a sample Dockerfile:
# Start with the ubuntu image
FROM ubuntu
# Update apt cache
RUN apt-get -y update
# Install ansible dependencies
RUN apt-get install -y python-yaml python-jinja2 git
# Clone ansible repo (could also add the ansible PPA and do an apt-get install instead)
RUN git clone http://github.com/ansible/ansible.git /tmp/ansible
# Set variables for ansible
WORKDIR /tmp/ansible
ENV PATH /tmp/ansible/bin:/sbin:/usr/sbin:/usr/bin
ENV ANSIBLE_LIBRARY /tmp/ansible/library
ENV PYTHONPATH /tmp/ansible/lib:$PYTHON_PATH
# add playbooks to the image. This might be a git repo instead
ADD playbooks/ /etc/ansible/
ADD inventory /etc/ansible/hosts
WORKDIR /etc/ansible
# Run ansible using the site.yml playbook
RUN ansible-playbook /etc/ansible/site.yml -c local
The ansible inventory file would look like
[local]
localhost
Then you can just docker build . (where . is the root of the directory where your playbooks and Dockerfile live), then docker run on the resulting image.
Michael DeHaan, the CTO of Ansible, has an informative blog post on this topic.
There's a working example regarding this: https://github.com/William-Yeh/docker-ansible
First, choose the base image you'd like to begin with from the following list:
williamyeh/ansible:debian8-onbuild
williamyeh/ansible:debian7-onbuild
williamyeh/ansible:ubuntu14.04-onbuild
williamyeh/ansible:ubuntu12.04-onbuild
williamyeh/ansible:centos7-onbuild
williamyeh/ansible:centos6-onbuild
Second, put the following Dockerfile along with your playbook directory:
FROM williamyeh/ansible:ubuntu14.04-onbuild
# ==> Specify playbook filename; default = "playbook.yml"
#ENV PLAYBOOK playbook.yml
# ==> Specify inventory filename; default = "/etc/ansible/hosts"
#ENV INVENTORY inventory.ini
# ==> Executing Ansible...
RUN ansible-playbook-wrapper
Third, docker build .
For more advanced usage, the role in Ansible Galaxy williamyeh/nginx also demonstrates how to do a simple integration test for a variety of Linux distributions on Travis CI’s Ubuntu 12.04 worker instances.
Disclosure: I am the author of the docker-ansible and wiliamyeh/nginx projects.
I've created a role for this vary scenario: https://github.com/chrismeyersfsu/provision_docker. Easily start Docker containers and use them in your role or playbook, as inventory, to test.
Includes:
Curated Dockerfile for Ubuntu 12.04 & 14.04 as well as CentOS 6 & 7 that put back in the distro-removed init systems
start ssh
Also note the examples all have a .travis.yml file to form a CI pipeline using Travis CI.
Examples:
Simple: https://github.com/chrismeyersfsu/provision_docker/tree/master/test
Simple: https://github.com/chrismeyersfsu/role-iptables/tree/master/test
Advanced: https://github.com/chrismeyersfsu/role-install_mongod/tree/master/test
Apart from provisioning localhost (the machine where you have Ansible installed), you can also tell Ansible to:
create a new docker container,
provision that container,
destroy that container.
For this to work you need such a hosts.yaml file:
all:
hosts:
mycontainer:
ansible_connection: docker
localhost:
ansible_connection: local
such a playbook.yaml file:
---
- name: Create a container to be provisioned later
hosts: localhost
tasks:
- name: create docker container
docker_container:
name: mycontainer
image: python:2.7.16-slim-stretch
command: ["sleep", "1d"]
- name: Provision the container created above
hosts: mycontainer
roles:
- simple
and another playbook file: destroy.yaml used to destroy the container:
---
- name: Destroy a container
hosts: localhost
tasks:
- name: destroy docker container
docker_container:
name: mycontainer
state: absent
Create also a simple role: roles/simple/taksks/main.yaml
---
- name: Create a file
copy:
content: "hi!!"
dest: /tmp/hello
force: yes
mode: 0555
And now to create a container and provision it, run:
ansible-playbook -i ./hosts.yaml ./playbook.yml
Verify that container was provisioned (the file was created):
docker exec mycontainer cat /tmp/hello
To destroy the container run:
ansible-playbook -i ./hosts.yaml ./destroy.yml
There are of course disadvantages:
the container must have python installed
some Ansible modules might not work, because additional python packages have to be installed. E.g. if you wanted to deploy docker containers (in the docker container), you have to install docker python SDK (pip3 install docker)
I was inspired by this blog post: https://medium.com/#andreilhicas/provision-docker-containers-with-ansible-30cc5ee6d950
Related
I successfully connected my Jenkins server to my Ansible server/control node and it built a stable and successful job after I made a change to my github too. But when I actually created an ansible playbook to create and tag a docker image and then push to dockerhub. I added 'ansible-playbook regapp.yml' to the Ansible configuration in my Jenkins server and it gives this error:
For context, this is the playbook I created:
- hosts: ansible
tasks:
- name: create docker image
command: docker build -t regapp:v2 .
args:
chdir: /opt/docker
- name: create tag to push image onto dockerhub
command: docker tag regapp:v2 codestein/regapp:v2
- name: push docker image
command: docker push codestein/regapp:v2
Things I've tried:
Resetting my docker container. I did rm -rf /var/lib/docker and then systemctl restart docker
Changed webapp/target/webapp.var on Jenkins config to '**/*war' and vice versa.
Gave ansadmin:ansadmin control/access of /opt/docker directory
Not sure how else to fix the issue. I'd appreciate any suggestions.
We want to use Paketo.io / CloudNativeBuildpacks (CNB) GitLab CI in the most simple way. Our GitLab setup uses an AWS EKS cluster with unprivileged GitLab CI Runners leveraging the Kubernetes executor. We also don't want to introduce security risks by using Docker in our builds. So we don't have our host’s /var/run/docker.sock exposed nor want to use docker:dind.
We found some guides on how to use Paketo with GitLab CI like this https://tanzu.vmware.com/developer/guides/gitlab-ci-cd-cnb/ . But as described beneath the headline Use Cloud Native Buildpacks with GitLab in GitLab Build Job WITHOUT Using the GitLab Build Template, the approach relies on Docker and pack CLI. We tried to resemble this in our .gitlab-ci.yml which looks like this:
image: docker:20.10.9
stages:
- build
before_script:
- |
echo "install pack CLI (see https://buildpacks.io/docs/tools/pack/)"
apk add --no-cache curl
(curl -sSL "https://github.com/buildpacks/pack/releases/download/v0.21.1/pack-v0.21.1-linux.tgz" | tar -C /usr/local/bin/ --no-same-owner -xzv pack)
build-image:
stage: build
script:
- pack --version
- >
pack build $REGISTRY_GROUP_PROJECT/$CI_PROJECT_NAME:latest
--builder paketobuildpacks/builder:base
--path .
But as outlined our setup does not support docker and we end up with the following error inside our logs:
...
$ echo "install pack CLI (see https://buildpacks.io/docs/tools/pack/)" # collapsed multi-line command
install pack CLI (see https://buildpacks.io/docs/tools/pack/)
fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/community/x86_64/APKINDEX.tar.gz
(1/4) Installing brotli-libs (1.0.9-r5)
(2/4) Installing nghttp2-libs (1.43.0-r0)
(3/4) Installing libcurl (7.79.1-r0)
(4/4) Installing curl (7.79.1-r0)
Executing busybox-1.33.1-r3.trigger
OK: 12 MiB in 26 packages
pack
$ pack --version
0.21.1+git-e09e397.build-2823
$ pack build $REGISTRY_GROUP_PROJECT/$CI_PROJECT_NAME:latest --builder paketobuildpacks/builder:base --path .
ERROR: failed to build: failed to fetch builder image 'index.docker.io/paketobuildpacks/builder:base': Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Cleaning up project directory and file based variables 00:01
ERROR: Job failed: command terminated with exit code 1
Any idea on how to use Paketo Buildpacks with GitLab CI without having Docker present inside our GitLab Kubernetes runners (which seems to be kind of a best practice)? We also don't want our setup to become to complex - e.g. by adding kpack.
TLDR;
Use the Buildpack's lifecycle directly inside your .gitlab-ci.yml here's a fully working example):
image: paketobuildpacks/builder
stages:
- build
# We somehow need to access GitLab Container Registry with the Paketo lifecycle
# So we simply create ~/.docker/config.json as stated in https://stackoverflow.com/a/41710291/4964553
before_script:
- mkdir ~/.docker
- echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_JOB_TOKEN\"}}}" >> ~/.docker/config.json
build-image:
stage: build
script:
- /cnb/lifecycle/creator -app=. $CI_REGISTRY_IMAGE:latest
The details: "using the lifecycle directly"
There are ongoing discussions about this topic. Especially have a look into https://github.com/buildpacks/pack/issues/564 and https://github.com/buildpacks/pack/issues/413#issuecomment-565165832. As stated there:
If you're looking to build images in CI (not locally), I'd encourage
you to use the lifecycle directly for that, so that you don't need
Docker. Here's an example:
The link to the example is broken, but it refers to the Tekton implementation on how to use buildpacks in a Kubernetes environment. Here we can get a first glue about what Stephen Levine referred to as "to use the lifecycle directly". Inside it the crucial point is the usage of command: ["/cnb/lifecycle/creator"]. So this is the lifecycle everyone is talking about! And there's good documentaion about this command that could be found in this CNB RFC.
Choosing a good image: paketobuildpacks/builder:base
So how to develop a working .gitlab-ci.yml? Let's start simple. Digging into the Tekton implementation you'll see that the lifecycle command is executed inside an environment defined in BUILDER_IMAGE, which itself is documented as The image on which builds will run (must include lifecycle and compatible buildpacks). That sound's familiar! Can't we simply pick the builder image paketobuildpacks/builder:base from our pack CLI command? Let's try this locally on our workstation before commiting to much noise into our GitLab. Choose a project you want to build (I created a example Spring Boot app if you'd like at gitlab.com/jonashackt/microservice-api-spring-boot you can clone) and run:
docker run --rm -it -v "$PWD":/usr/src/app -w /usr/src/app paketobuildpacks/builder bash
Now inside the paketobuildpacks/builder image powered container try to run the Paketo lifecycle directly with:
/cnb/lifecycle/creator -app=. microservice-api-spring-boot:latest
I only used the -app parameter of the many possible parameters for the creator command, since most of them have quite good defaults. But as the default app directory path is not the default /workspace - but the current directory, I configured it. Also we need to define an <image-name> at the end, which will simply be used as the resulting container image name.
The first .gitlab-ci.yml
Both commands did work at my local workstation, so let's finally create a .gitlab-ci.yml using this approach (here's a fully working example .gitlab-ci.yml):
image: paketobuildpacks/builder
stages:
- build
build-image:
stage: build
script:
- /cnb/lifecycle/creator -app=. $CI_REGISTRY_IMAGE:latest
docker login without docker
As we don't have docker available inside our Kubernetes Runners, we can't login into GitLab Container Registry as described in the docs. So the following error occured to me using this first approach:
===> ANALYZING
ERROR: failed to get previous image: connect to repo store "gitlab.yourcompanyhere.cloud:4567/yourgroup/microservice-api-spring-boot:latest": GET https://gitlab.yourcompanyhere.cloud/jwt/auth?scope=repository%3Ayourgroup%2Fmicroservice-api-spring-boot%3Apull&service=container_registry: DENIED: access forbidden
Cleaning up project directory and file based variables 00:01
ERROR: Job failed: command terminated with exit code 1
Using the approach described in this so answer fixed the problem. We need to create a ~/.docker/config.json containing the GitLab Container Registry login information - and then the Paketo build will pick them up, as stated in the docs:
If CNB_REGISTRY_AUTH is unset and a docker config.json file is
present, the lifecycle SHOULD use the contents of this file to
authenticate with any matching registry.
Inside our .gitlab-ci.yml this could look like:
# We somehow need to access GitLab Container Registry with the Paketo lifecycle
# So we simply create ~/.docker/config.json as stated in https://stackoverflow.com/a/41710291/4964553
before_script:
- mkdir ~/.docker
- echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_JOB_TOKEN\"}}}" >> ~/.docker/config.json
Our final .gitlab-ci.yml
As we're using the image: paketobuildpacks/builder at the top of our .gitlab-ci.yml, we can now leverage the lifecycle directly. Which is what we wanted to do in the first place. Only remember to use the correct GitLab CI variables to describe your <image-name> like this:
/cnb/lifecycle/creator -app=. $CI_REGISTRY_IMAGE:latest
Otherwise the Buildpack process analyser step will break and it finally won't get pushed to the GitLab Container Registry. So finally our .gitlab-ci.yml looks like this (here's the fully working example):
image: paketobuildpacks/builder
stages:
- build
# We somehow need to access GitLab Container Registry with the Paketo lifecycle
# So we simply create ~/.docker/config.json as stated in https://stackoverflow.com/a/41710291/4964553
before_script:
- mkdir ~/.docker
- echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_JOB_TOKEN\"}}}" >> ~/.docker/config.json
build-image:
stage: build
script:
- /cnb/lifecycle/creator -app=. $CI_REGISTRY_IMAGE:latest
Our builds should now run successfully using Paketo/Buildpacks without pack CLI and Docker:
See the full log of the example project here.
I am trying to build and run a docker container using ansible, but I am unable to pass the environment variable to the Dockerfile in the build state.
Below is my ansible file dev.yml
---
- name: setup docker
tasks:
- name: build dockerfile
community.docker.docker_container:
name: test
tag: v0
path: .
nocache: yes
env:
TEST_ENV: "SOME_TESTS_VARIABLE"
SSH_KEY: "{{LOCAL_SSH_KEY}}"
I am running ansible-playbook,
ansible-playbook -i hosts dev.yml -e "LOCAL_SSH_KEY='$(cat ~/.ssh/id_rsa)'"
I have figured out looking at the doc & the errors that community.docker.docker_container does not support nocache, path, tag and to build the container I should rather use docker_image which then does not supports env
Is there a way to build docker containers using the environment variables.
In other words how can I pass my ssh keys to the docker build step?
I have looked at other answers but those don't work for me, maybe because of ansible version 2.10 which I am using.
I would like to run ansible playbook on my local machine using ansible from a docker container.
Here is what my Ansible Dockerfile looks like:
FROM alpine:3.6
WORKDIR /ansible
RUN apk update \
&& apk add ansible
ENTRYPOINT ["ansible-playbook"]
playbook.yml:
---
- hosts: localhost
roles:
- osx
roles/osx/tasks/main.yml
---
- name: Welcome
shell: echo "Hello"
when: ansible_distribution == 'MacOSX'
Then I run it with:
docker build -t ansible_image:latest .
docker run --rm --network host \
-v $(pwd):/ansible \
ansible_image:latest ansible/playbook.yml
My host operating system is OS X. I expect that osx role will execute,
however it seems that playbook is run on alpine container.
I would like to ask how to indicate ansible in docker to deploy stuff on my local machine?
Your playbook is targeting localhost:
---
- hosts: localhost
roles:
- osx
This means that Ansible is going to target the local machine (which is
to say, your Ansible container) when running the playbook. Ansible is
designed to apply playbooks to remote machines as well, typically by
connecting to them using ssh. Assuming that it's possible to
connect from your Ansible container to your host using ssh, you
could just create an appropriate inventory file and then target your
playbook appropriately:
---
- hosts: my_osx_host
roles:
- osx
If you're just starting out with Ansible, you might want to start with
the Getting Started document and work your way from there. You'll find documentation on that site that should walk you through the process of creating an inventory file.
On my OS X host, I'm using Docker CE (18.06.1-ce-mac73 (26764)) with Kubernetes enabled and using Kubernetes orchestration. From this host, I can run a stack deploy to deploy a container to Kubernetes using this simple docker-compose file (kube-compose.yml):
version: '3.3'
services:
web:
image: dockerdemos/lab-web
volumes:
- "./web/static:/static"
ports:
- "9999:80"
and this command-line run from the directory containing the compose file:
docker stack deploy --compose-file ./kube-compose.yml simple_test
However, when I attempt to run the same command from my Jenkins container, Jenkins returns:
this node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again
I do not want the docker client in the Jenkins container to be initialized for a swarm since I'm not using Docker swarm on the host.
The Jenkins container is defined in a docker-compose to include a volume mount to the docker host socket endpoint:
version: '3.3'
services:
jenkins:
# contains embedded docker client & blueocean plugin
image: jenkinsci/blueocean:latest
user: root
ports:
- "8080:8080"
- "50000:50000"
volumes:
- ./jenkins_home:/var/jenkins_home
# run Docker from the host system when the container calls it.
- /var/run/docker.sock:/var/run/docker.sock
# root of simple project
- .:/home/project
container_name: jenkins
I have also followed this guide to proxy requests to the docker host with socat: https://github.com/docker/for-mac/issues/770 and here: Docker-compose: deploying service in multiple hosts.
Finally, I'm using the following Jenkins definition (Jenkinsfile) to call stack to deploy on my host. Jenkins has the Jenkins docker plug-in installed:
node {
checkout scm
stage ('Deploy To Kube') {
docker.withServer('tcp://docker.for.mac.localhost:1234') {
sh 'docker stack deploy app --compose-file /home/project/kube-compose.yml'
}
}
}
I've also tried changing the withServer signature to:
docker.withServer('unix:///var/run/docker.sock')
and I get the same error response. I am, however, able to telnet to the docker host from the Jenkins container so I know it's reachable. Also, as I mentioned earlier, I know the message is saying to run swarm init, but I am not deploying to swarm.
I checked the version of the docker client in the Jenkins container and it is the same version (Linux variant, however) as I'm using on my host:
Docker version 18.06.1-ce, build d72f525745
Here's the code I've described: https://github.com/ewilansky/localstackdeploy.git
Please let me know if it's possible to do what I'm hoping to do from the Jenkins container. The purpose for all of this is to provide a simple, portable demonstration of a pipeline and deploying to Kubernetes is the last step. I understand that this is not the approach that would be taken anywhere outside of a local development environment.
Here is an approach that's working well for me until the Jenkins Docker plug-in or the Kubernetes Docker Stack Deploy command can support the remote deployment scenario I described.
I'm now using the Kubernetes client kubectl from the Jenkins container. To minimize the size increase of the Jenkins container, I added just the Kubernetes client to the jenkinsci/blueocean image that was built on Alpine Linux. This DockerFile shows the addition:
FROM jenkinsci/blueocean
USER root
RUN curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.12.0/bin/linux/amd64/kubectl
RUN chmod +x ./kubectl
RUN mv ./kubectl /usr/local/bin/kubectl
RUN mkdir /root/.kube
COPY kube-config /root/.kube/config
I took this approach, which added ~100 mb to the image size rather than getting the Alpine Linux Kubernetes package, which almost doubled the size of the image in my testing. Granted, the Kubernetes package has all Kubernetes components, but all I needed was the Kubernetes client. This is similar to the requirement that the docker client be resident to the Jenkins container in order to run Docker commands on the host.
Notice in the DockerFile that there is reference to the Kuberenetes config file:
kube-config /root/.kube/config
I started with the Kubernetes configuration file on my host machine (the computer running Docker for Mac). I believe that if you enable Kubernetes in Docker for Mac, the Kubernetes client configuration will be present at ~/.kube/config. If not, install the Kubernetes client tools separately. In the Kubernetes configuration file that you will copy over to the Jenkins container via DockerFile, just change the server value so that the Jenkins container is pointing at the Docker for Mac host:
server: https://docker.for.mac.localhost:6443
If you're using a Windows machine, I think you can use docker.for.win.localhost. There's a discussion about this here: https://github.com/docker/for-mac/issues/2705 and other approaches described here: https://github.com/docker/for-linux/issues/264.
After recomposing the Jenkins container, I was then able to use kubectl to create a deployment and service for my app that's now running in the Kubernetes Docker for Mac host. In my case, here are the two commands I added to my Jenkins file:
stage ('Deploy To Kube') {
sh 'kubectl create -f /kube/deploy/app_set/sb-demo-deployment.yaml'
}
stage('Configure Kube Load Balancer') {
sh 'kubectl create -f /kube/deploy/app_set/sb-demo-service.yaml'
}
There are loads of options for Kubernetes container deployments. In my case, I simply needed to deploy my web app (with replicas) behind a load balancer. All of that is defined in the two yaml files called by kubectl. This is a bit more involved than docker stack deploy, but achieves the same end result.