As described in the Cloud Native Buildpack features they add a rebasing capability with
Instant updates of base images without re-building.
In the buildpacks docs the rebase operation of Paketo's pack CLI is described to be executed like this:
pack rebase my-app:my-tag
As this is one of the key features of Cloud Native Buildpacks - and is a feature most microservice based architectures are in need of desparately - it would be great to know, how this could be automated inside a CI/CD pipeline such as GitHub Actions or a dependency management tool like renovate. Renovate already supports Docker, but because there's is no depencency management file for Paketo, it doesn't create Pull Requests right now.
So the question is how Paketo/Buildpacks rebase operation could be automated to create PRs without human interaction?
The pack rebase command is going to swap out the run image used by your container. The run image is part of the builder that you selected when you built your image.
For example, there is the Paketo paketobuildpacks/builder:base builder. You can run pack inspect-builder paketobuildpacks/builder:base and pack will give you a bunch of details about the builder. Included in that is a section telling you the run image for this builder and any images built using this builder.
Run Images:
index.docker.io/paketobuildpacks/run:base-cnb
gcr.io/paketo-buildpacks/run:base-cnb
Note there are two listed, but they are the same image just hosted in two different places.
Given this, you are going to want to set up your CI system to monitor for new versions of the run image for your builder. When there is a new run image, you'll want to pack rebase so you update your images to use the latest run image.
I haven't used Renovate but it sounds like the Docker support is probably what you want. Point it at your base image & use that to trigger the pack rebase command.
We have some Github actions which monitor Paketo images (not for this specific purpose, but the idea is the same). There's not a great way to do it (at least at the time I write this), but we use the schedule to periodically check for updates to the image. Then kick off workflows, in this case the workflow would basically be to run pack rebase.
Related
I started to use pack to build container images for my applications. I used different builders for different apps: gcr.io/buildpacks/builder for a Node.js app and paketobuildpacks/builder:tiny for a Clojure app.
Not having to write a Dockerfile is great, but I'm still not sure how Cloud Native Buildpacks work. Are buildpacks just a bunch of executables (ran by a builder) that achieve the same result of a developer who manually writes a Docker multi-stage build (i.e. multiple FROM instructions in a Dockerfile)?
There are a few concepts to keep in mind
A builder. A builder consists of a stack and one or more groups of buildpacks.
A stack is a base build and run image.
A buildpack is an OCI image with at least two binaries, detect and build.
The lifecycle. The lifecycle responsible for running all of the builders.
With that in mind, at a high level this is what happens
When you run pack build. The cli takes all your information and uses that to perform the build. To do a build, it will set up a container. The container uses the build image from your stack, which includes the groups of buildpacks from the builder, and usually some development libraries exclusive to that image to make building your app easier. It then passes off the settings you input via the cli and runs the lifecycle inside that container.
First, the lifecycle will run the detect script from each buildpack. The output of detect is a build plan, which the lifecycle uses to assemble the list of buildpacks that will participate in the build. This is how the pack cli and ultimately the lifecycle can build a Java app, a Node.js app or anything else supported by buildpacks. The build plan provides specific instructions on which buildpacks are required to do that.
Second, the lifecycle will take the buildpacks that are participating in the build, again which is determined by detect and the selected buildplan, and run them in order. Each buildpack then runs and performs the actions that are required to build your application. Exactly what happens, depends on your buildpack, but a buildpack can do virtually anything. The net result of a buildpack running is a set of layers that are populated with information & files.
The lifecycle then takes the run image from the builder and all the layers generated by the buildpacks that ran and marked as destined for the launch image and combines them into the output or launch image.
The lifecycle also handles storing build and cached layers such that subsequent builds will be able to take advantage and run much faster.
Dockerfiles or Buildpacks?
The answer is that it's not one or the other. Each tool is good for certain things and you should use the best tool for the job.
Buildpacks are targeting developers that have source code and wish to turn that into OCI images. Buildpacks eliminate the need to curate handcrafted, artisanal Dockerfiles for this task, freeing developers up to write more code and add value to their apps.
The buildpacks accomplish this by wrapping up the patterns and best practices for creating OCI images for each programming language into an encapsulated, well-tested, and easy-to-use to use tool.
Dockerfiles tend to be better suited for other tasks (i.e. not packaging up your application) such as making generic OS images or packing up servers like databases or message queues. In fact, Buildpacks use Dockerfiles to create the base build and run images that are part of the builder.
Can someone explain to me why the normal Docker process is to build an image from a Dockerfile and then upload it to a repository, instead of just moving the Dockerfile to and from the repository?
Let's say we have a development laptop and a test server with Docker.
If we build the image, that means uploading and downloading all of the packages inside the Dockerfile. Sometimes this can be very large (e.g. PyTorch > 500MB).
Instead of transporting the large imagefile to and from the server, doesn't it make sense to, perhaps compile the image locally to verify it works, but mostly transport the small Dockerfile and build the image on the server?
This started out as a comment, but it got too long. It is likely to not be a comprehensive answer, but may contain useful information regardless.
Often the Dockerfile will form part of a larger build process, with output files from previous stages being copied into the final image. If you want to host the Dockerfile instead of the final image, you’d also have to host either the (usually temporary) processed files or the entire source repo & build script.
The latter is often done for open source projects, but for convenience pre-built Docker images are also frequently available.
One tidy solution to this problem is to write the entire build process in the Dockerfile using multi-stage builds (introduced in Docker CE 17.05 & EE 17.06). But even with the complete build process described in a platform-independent manner in a single Dockerfile, the complete source repository must still be provided.
TL,DR: Think of a Docker image as a regular binary. It’s convenient to download and install without messing around with source files. You could download the source for a C application and build it using the provided Makefile, but why would you if a binary was made available for your system?
Instead of transporting the large imagefile to and from the server,
doesn't it make sense to, perhaps compile the image locally to verify
it works, but mostly transport the small Dockerfile and build the
image on the server?
Absolutely! You can, for example, set up an automated build on Docker Hub which will do just that every time you check in an updated version of your Dockerfile to your GitHub repo.
Or you can set up your own build server / CI pipeline accordingly.
IMHO, one of the reason for building the images concept and putting into repository is sharing with people too. For example we call Python's out of the box image for performing all python related stuff for a python program to run in Dockerfile. Similarly we could create a custom code(let's take example I did for apache installation with some custom steps(like ports changes and additionally doing some steps) I created its image and then finally put it to my company's repository.
I came to know after few days that may other teams are using it too and now when they are sharing it they need NOT to make any changes simply use my image and they should be done.
Although most of the time I am developing Java apps and am simply using Maven so my builds should be reproducible (at least that's what Maven says).
But say you are compiling a C++ program or something a little more involved, should you build inside of docker?
Or ideally use vagrant or another technology to produce reproduce able builds.
How do you manage reproducible build with docker?
You can, but not in your final image, as that would mean a much larger image than necessary: it would include all the compilation tool, instead of limiting to only what you need to execute the resulting binary.
You can see an alternative in "How do I build a Docker image for a Ruby project without build tools?"
I use an image to build,
I commit the resulting stopped container as a new image (with a volume including the resulting binary)
I use an execution image (one which only contain what you need to run), and copy the binary from the other image. I commit again the resulting container.
The final image includes the compiled binary and the execution environment.
I wanted to post an answer to this as well actually because to build on VonC's answer. Actually I just had Redhat Openshift training and they use a tool called Source to Image s2i, which uses docker to create docker images. And actually this strategy is great for managing a private (or public) cloud, where your build may be compiled on different machines, but you need to keep the build environment consistent.
I am new to Kubernetes and so I'm wondering what are the best practices when it comes to putting your app's source code into container run in Kubernetes or similar environment?
My app is a PHP so I have PHP(fpm) and Nginx containers(running from Google Container Engine)
At first, I had git volume, but there was no way of changing app versions like this so I switched to emptyDir and having my source code in a zip archive in one of the images that would unzip it into this volume upon start and now I have the source code separate in both images via git with separate git directory so I have /app and /app-git.
This is good because I do not need to share or configure volumes(less resources and configuration), the app's layer is reused in both images so no impact on space and since it is git the "base" is built in so I can simply adjust my dockerfile command at the end and switch to different branch or tag easily.
I wanted to download an archive with the source code directly from repository by providing credentials as arguments during build process but that did not work because my repo, bitbucket, creates archives with last commit id appended to the directory so there was no way o knowing what unpacking the archive would result in, so I got stuck with git itself.
What are your ways of handling the source code?
Ideally, you would use continuous delivery patterns, which means use Travis CI, Bitbucket pipelines or Jenkins to build the image on code change.
that is, every time your code changes, your automated build will get triggered and build a new Docker image, which will contain your source code. Then you can trigger a Deployment rolling update to update the Pods with the new image.
If you have dynamic content, you likely put this a persistent storage, which will be re-mounted on Pod update.
What we've done traditionally with PHP is an overlay on runtime. Basically the container will have a volume mounted to it with deploy keys to your git repo. This will allow you to perform git pull operations.
The more buttoned up approach is to have custom, tagged images of your code extended from fpm or whatever image you're using. That way you would run version 1.3 of YourImage where YourImage would contain code version 1.3 of your application.
Try to leverage continuous integration and continuous deployment. You can use Jenkins as CI/CD server, and create some jobs for building image, pushing image and deploying image.
I recommend putting your source code into docker image, instead of git repo. You can also extract configuration files from docker image. In kubernetes v1.2, it provides new feature 'ConfigMap', so we can put configuration files in ConfigMap. When running a pod, configuration files will be mounted automatically. It's very convenience.
(I understand this question is somewhat out of scope for stack overflow, because contains more problems and somewhat vague. Suggestions to ask it in the proper ways are welcome.)
I have some open source projects depending in each other.
The code resides in github, the builds happen in shippable, using docker images which in turn are built on docker hub.
I have set up an artifact repo and a debian repository where shippable builds put the packages, and docker builds use them.
The build chain looks like this in terms of deliverables:
pre-zenta docker image
zenta docker image (two steps of docker build because it would time out otherwise)
zenta debian package
zenta-tools docker image
zenta-tools debian package
xslt docker image
adadocs artifacts
Currently I am triggering the builds by pushing to github and sometimes rerunning failed builds on shippable after the docker build ran.
I am looking for solutions for the following problems:
Where to put Dockerfiles? Now they are in the repo of the package needing the resulting docker image for build. This way all information to build the package are in one place, but sometimes I have to trigger an extra build to have the package actually built.
How to trigger build automatically?
..., in a way supporting git-flow? For example if I change the code in zenta develop branch, I want to make sure that zenta-tools will build and test with the development version of it, before merging with master.
Are there a tool with which I can overview the health of the whole build chain?
Since your question is related to Shippable, I've created a support issue for you here - https://github.com/Shippable/support/issues/2662. If you are interested in discussing the best way to handle your scenario, you can also send me an email at support#shippable.com You can set up your entire flow, including building the docker images, using Shippable.