I have a single git repository on github with:
a Dockerfile which builds multiple images meant to be used together. (a maven build produces a war file and sql files by downloading then from artifact repositories; a multi-stage build then creates a slim tomcat image with the war and a slim mysql image with the sql data preloaded).
a docker-compose.yml file that uses the "target" instruction to build and run containers on the images from the multi-stage build.
This works well during development. But it forces users to build images on their computer. I want the users to be able to only download images. The setup should also be using dockerhub's automated build to keep images up to date.
How can I set it up to achieve this ? What command(s) or file(s) do I give the users to allow them to download images and run containers ? If it is not possible, what can I do to make it possible (split the repo? copypaste the dockerfile? publish intermediate images to dockerhub and ensure a correct build order ? Don't use dockerhub's automated build ?)
To use dockerhub's automated builds you would need to build one image per Dockerfile and have one Dockerfile per repo. The image name comes from the source repository name (with the github org/user name as the docker image user name and the github repo name as the docker image name). Multistage builds work in automated builds but only one image is published per Dockerfile (the final image of the build).
You could build the images in your CI or even on your local machine and then push to dockerhub. You'd just need to have an account on dockerhub and be logged in to that account when you use the docker push command. When doing this push there doesn't have to be any mapping to GitHub repos but your image names should start with <dockerhub_user>/ as a kind of prefix (explained at https://docs.docker.com/docker-hub/repos/). It's ok if they are built under a different name as you could rename by retagging before pushing. This way you can also build the images however you like.
When you have images in dockerhub you can just refer to them in the docker-compose file using the form image: <dockerhub_user>/<dockerhub_image_name>:<tag>. The images will automatically be pulled when the user does docker-compose up.
Here are some tips and links that should help your situation:
Automated builds are a convenient way to deploy your images.
This part is pretty easy. You'll need accounts with Docker Hub and Github. You can register these accounts for free.
When you create a a repository on Docker Hub you can link it to your Github repository to automate the build.
Recommendations:
Split your services into separate Dockerfiles. Ideally you should use separate repositories: Docker Compose will pull them together at the end. A division of services will also help if anyone wants to implement e.g. a cloud database backend for their deployment.
Don't store database files inside a container. Containers should be ephemeral
For a robust design, test your builds.
Docker Hub automated builds are very flexible with the use of build hooks.
This part is a little tricky because I haven't found the best documentation. It also might not be necessary if you split your Dockerfile.
I've successfully created automated builds with multiple tags and targets using a hook at hooks/build but after reading the documentation it looks like you should also be able to use hooks/post_build.
Your hook could simply build the correct layer and push the tag to Docker Hub
For your repository that should look like:
#!/usr/bin/env bash
docker build --target lutece-mysql -t lutece/mysql .
docker push lutece/mysql
If you end up using hooks/build you might need to build the final target as the last step.
Recommendations:
If you need multiple tags for an image use a hook at hooks/post_push to add additional tags. That way each tag should link users to the same image. e.g.
#!/usr/bin/env bash
docker tag lutece/tomcat:master lutece/tomcat:latest
docker push lutece/tomcat:latest
Additionally you can use build hooks to label your image with things like build date and git commit.
Deployment with Docker Compose
Unfortunately I haven't done this part so I can't confirm how to get this to work.
With your repositories in Docker Hub and a working docker-compose.yml your clients may only need to run docker-compose up in the directory with your docker-compose.yml file. Docker Compose should pull in the images from Docker Hub.
Related
So we have a git repository that has a number of docker images (multiple Dockerfiles each of which used for a different type of application build in our Jenkins)
Now if one makes a change in one Dockerfile the Jenkins job will build and pushe all other Dockerfiles in the repository. I was wondering, if we were able to calcualte the digest id (SHA256) beforehand and compare that with our docker registry, if there is one already we can skip docker build and docker push.
I couldn't find any command in Docker user guide but here with this post I also wanted to know if we can raise a ticket for this new feature if this approach works and if there is no way of calculating this identifier
Any other suggestion is greatly appreciated
For the use case where a Dockerfile needs to be built for each platform it's on (a bit niche I know), is there a way possible for it to push itself to the registry, i.e. calling docker push from within the Dockerfile?
Currently, this is done:
docker build -t my-registry/<username>/<image>:<version> .
docker login my-registry
docker push <image>
Could the login and push steps be directly or cleverly built into the Dockerfile being built or with a combination of others?
Note: This would operate in a secure environment of trustworthy users (so all users being able to push to the registry is fine).
Note: This is an irregular use of Docker, not a good idea for building/packaging software in general, rather I am using Docker to share environments between developers.
I am wondering why can't you have a wrapper script file (say shell or bat) around the "Dockerfile" to do these steps
docker build -t my-registry/<username>/<image>:<version> .
docker login my-registry
docker push <image>
What is it so specific about "Dockerfile". I know, this is not addressing the question that you asked, I might have totally misunderstood your usecase, but I am just curious.
As others pointed out, this can be easily achieved using a CD systems like Drone.io/Travis/Jenkins etc.
At first this sounds to me like the decently-circulated "Nasa's Space pen Myth". But as I said earlier, you may have a proper valid use case which I am not aware of yet.
Docker build creates image using recipe provided in Dockerfile. Each line in Dockerfile creates new temporary image of file system with different checksum. Image after execution of last line of Dockerfile is your final image of build process and is tagged with provided name.
So it is not possible to put docker push command inside Dockerfile because image creation is not finished yet.
Having a Dockefile push it's own image will never work.
To explain a bit more:
What happens when you build and image: Docker will spawn a container and do everything the Dockerfile specifies. You can even see this when running docker ps during the build. If the exit status of the container is 0 (no errors), an image is created from the container.
We don't really have much control over this process other than the build parameters. It's definitely a chicken and egg problem.
Build systems should to this stuff
It's even fairly easy to do this in Jenkins. The Jenkins setup I have uses a docker plugin and executes build commands to a remote docker hosts.. so the Jenkins nodes only pull the repo, runs a build, then pushes the image to a private repo properly tagged (then deletes the local image). You can run unit tests in docker also by making a separate Dockerfile (gets a bit more complicated when you need external mock services)
Builds per branch/architecture is not too hard to set up. With remote hosts doing the build work we can boost up the job limit in Jenkins fairly high and it can run on cheap hardware.
You can also run Jenkins in docker and make it build images in the docker engine it runs in. I just do that through TLS or the old trick of mapping the socket file into the Jenkins container might still work.
I think I started with the CloudBees Docker Build and Publish plugin and it was fairly easy to use, but now I use a custom plugin so I have no idea of the alternatives.
I have multiple projects I need to build as part of the same CI flow - some are in java, some are nodejs, some are c++ etc.
We use Jenkins and slaves are supposed to run as docker containers.
My question is - should I create a jenkins slave container image per module type, i.e a dedicated slave image which would be able to build java, and a dedicated container to build nodejs with node installed etc. or a single container which can build anything - jave, node, etc.
If I look at it from vm perspective, I would most likely use the same vm to build anything - which means a centralized build slave. But I don't like this dependency, or if tomorrow I need to update the java version and keep the old one I might create huge images with little differences between them.
WDYT?
I personally would go down the route of a container-per-module-type because of the following:
I like to keep my containers as focussed as possible. They should do one thing and do it well e.g. build Java applications, build Node applications
Docker makes it incredibly easy to build Container images
It is incredibly easy to stop and start Containers
I'd probably create myself a separate project in Git that was structured something like this:
- /slaves
- /slaves/java
- /slaves/java/Dockerfile
- /slaves/node
- /slaves/node/Dockerfile
...
I have one Dockerfile that creates and builds the container image of the slave for the given "module type". I would make changes to this project via pull requests and each time a pull request is merged into master, push the resulting images up to DockerHub as the new version to be used as my Jenkins slaves.
I would have the above handled by another project running in my Jenkins instance that simply monitored my Git repository. When changes are made to the Git repository it just runs the build commands in order and then does a push of the new images to DockerHub:
docker build -f slaves/java/Dockerfile -t my-company/java-slave:$BUILD_NUMBER -t my-company/java-slave:latest
docker build -f slaves/node/Dockerfile -t my-company/node-slave:$BUILD_NUMBER -t my-company/node-slave:latest
docker push my-company/java-slave:$BUILD_NUMBER
docker push my-company/java-slave:latest
docker push my-company/node-slave:$BUILD_NUMBER
docker push my-company/node-slave:latest
You can then update your Jenkins configuration to the new image for the slaves when you're ready.
What is the right workflow for updating and storing images?
For example:
I download source code from GitHub (project with Docker files, docker-compose.yml)
I run "docker build"
And I push new image to Docker Hub (or AWS ECR)
I make some changes in source code
Push changes to GitHub
And what I should do now to update registry (Docker Hub)?
A) Should I run again "docker build" and then push new image (with new tag) to registry?
B) Should I somehow commit changes to existing image and update existing image on Docker Hub?
This will depend on what for you will use your docker image and what "releasing" policy you adopt.
My recommendation is that you sync the tags you keep on Docker Hub with the release/or tags you have in GitHub and automate as much as you can your production with a continuous integration tools like Jenkins and GitHub webooks.
Then your flow becomes :
You do your code modifications and integrate them in GitHub ideally using a pull request scheme. This means your codes will be merged into your master branch.
Your Jenkins is configured so that when master is changed it will build against your docker file and push it to Docker hub. This will erase your "latest" tag and make sure your latest tag in docker hub is always in sync with your master release on GitHub
If you need to keep additional tags, this will be typical because of different branches or releases of your software. You'll do the same as above with the tag hooked up through Jenkins and GitHub webhooks with a non-master branch. For this, take a look at how the official libraries are organized on GitHub (for example on Postgres or MySQL images).
I have created an automated build on dockerhub, but unfortunately I didn't find any proper documentation for those.
Basically, I am creating a system where docker automated build is triggered as soon as there is a commit in github repository. So how can I keep my docker tag a variable? Such that whenever there is a commit, the image being built is tagged with the latest commit's sha1.
I can put a regex in branch name or tag name, can't I put a regex in docker tag name? Here I wish to trigger the build by specifying the curl by docker tag name.
I don't think what you want is possible. The only variable you can use in the docker tag name is {sourceref}, which expands to the branch or tag name.
I presume this was deliberate -- you'd vastly increase the number of images that Docker Hub had to store if each commit was given a different docker tag.
You could try using a continuous integration/deployment service to build the images outside of Docker Hub. There are many to chose from, but Travis and Circle are popular ones that should be able to do what you want.