By default, dotnet core projects create a multi-stage build Dockerfile in which the application's source code is copied into a temporary container, built and published there, and then copied to a smaller runtime container as the final image.
I built an image this way, and then with a much simpler Dockerfile in which I pulled only the aspnetcore runtime image and then copied in the already-published application from my development machine. This produced an image I'm able to run on Docker Desktop. The size of the two images is exactly the same.
The direct-copy Dockerfile is much simpler than the multi-stage build file and eliminates the need for me to jump through some hoops so that the build inside the intermediate Docker container is able to see some custom resources that need to be pulled in.
Why is the multi-stage build used instead of simply copying the published application in the runtime folder? Are there scenarios in which the resulting image will not run correctly?
Related
Can someone explain to me why the normal Docker process is to build an image from a Dockerfile and then upload it to a repository, instead of just moving the Dockerfile to and from the repository?
Let's say we have a development laptop and a test server with Docker.
If we build the image, that means uploading and downloading all of the packages inside the Dockerfile. Sometimes this can be very large (e.g. PyTorch > 500MB).
Instead of transporting the large imagefile to and from the server, doesn't it make sense to, perhaps compile the image locally to verify it works, but mostly transport the small Dockerfile and build the image on the server?
This started out as a comment, but it got too long. It is likely to not be a comprehensive answer, but may contain useful information regardless.
Often the Dockerfile will form part of a larger build process, with output files from previous stages being copied into the final image. If you want to host the Dockerfile instead of the final image, you’d also have to host either the (usually temporary) processed files or the entire source repo & build script.
The latter is often done for open source projects, but for convenience pre-built Docker images are also frequently available.
One tidy solution to this problem is to write the entire build process in the Dockerfile using multi-stage builds (introduced in Docker CE 17.05 & EE 17.06). But even with the complete build process described in a platform-independent manner in a single Dockerfile, the complete source repository must still be provided.
TL,DR: Think of a Docker image as a regular binary. It’s convenient to download and install without messing around with source files. You could download the source for a C application and build it using the provided Makefile, but why would you if a binary was made available for your system?
Instead of transporting the large imagefile to and from the server,
doesn't it make sense to, perhaps compile the image locally to verify
it works, but mostly transport the small Dockerfile and build the
image on the server?
Absolutely! You can, for example, set up an automated build on Docker Hub which will do just that every time you check in an updated version of your Dockerfile to your GitHub repo.
Or you can set up your own build server / CI pipeline accordingly.
IMHO, one of the reason for building the images concept and putting into repository is sharing with people too. For example we call Python's out of the box image for performing all python related stuff for a python program to run in Dockerfile. Similarly we could create a custom code(let's take example I did for apache installation with some custom steps(like ports changes and additionally doing some steps) I created its image and then finally put it to my company's repository.
I came to know after few days that may other teams are using it too and now when they are sharing it they need NOT to make any changes simply use my image and they should be done.
I am using docker-compose with multi-stage Dockerfiles to build and run multiple services. This works, but the "build" portion of each multi-stage build is largely copy-and-pasted between each service's Dockerfile. I want to reduce the copy-and-paste / centralize the common build logic in one spot.
Reading https://engineering.busbud.com/2017/05/21/going-further-docker-multi-stage-builds/ I could create a local image with the shared build steps and have the service Docker files depend on it, but I want the development experience to be a simple docker-compose up. Creating a local build image means a developer would have to know to run docker build [common_build_image] first so that the build image exists locally and THEN run docker compose up to build and run all the services that depend on it.
There doesn't appear to be a way to include a Dockerfile into another Dockerfile. FROM does not appear to support local paths.
Is there a way to accomplish what I want? Of course I can use a shell script to tie everything together, but that is basically what multi-stage builds was trying to solve in the first place.
It turns out you can "compose" docker-compose: https://docs.docker.com/compose/extends/#adding-and-overriding-configuration which is what I was looking for.
Basically what the title says... Is there a best practice or an efficient way to develop a Golang app that will be Dockerized? I know you can mount volumes to point to your source code, and it works great for languages like PHP where you don't need to compile your code. But for Go, it seems like it would be a pain to develop alongside Docker since you pretty much only have two options I guess.
First would be to have a Dockerfile that is just onbuild so it starts the go app when a container is run, thus having to build a new image on every change (whether it be small or not). Or, you do mount your source code dir to the container dir, then attach to the container itself and do the manual go build/run yourself as if you would normally.
Those two ways are really the only way that I see it happening unless you just don't develop your Go app in a docker container. Just develop it as normal, then use the scratch image method where you pre build the Go into a binary then copy that into your container when you are ready to run it. I assume that is probably the way to go, but I wanted to ask more professional people on the subject and maybe get some feedback on the topic.
Not sure it's the best pratice but here is my way.
Makefile is MANDATORY
Use my local machine and my go tools for small iterations
Use a dedicated build container based on golang:{1.X,latest}, mount code directory to build a release, mainly to ensure that my code will build correctly on the CI. (Tips, here is my standard go build command for release build : CGO_ENABLED=0 GOGC=off go build -ldflags -s -w)
Test code
Then use a FROM scratch to build a release container (copy the bin + entrypoint)
Push you image to your registry
Steps 3 to 6 are tasks for the CI.
Important note : this is changing due to the new multistage builds feature : https://docs.docker.com/engine/userguide/eng-image/multistage-build/, no more build vs release containers.
Build container and release container will be merged in one multistage build so one Dockerfile with (not sure about the correct syntax but, you will get the idea) :
FROM golang:latest as build
WORKDIR /go/src/myrepos/myproject
RUN go build -o mybin
FROM scratch
COPY --from=build /go/src/myrepos/myproject/mybin /usr/local/bin/mybin
ENTRYPOINT [ "/usr/local/bin/mybin" ]
Lately, I've been using
https://github.com/thockin/go-build-template
As a base for all of my projects. The template comes with a Makefile that will build/test your application in a Docker.
As far as I understood from you question, you want to have a running container to develop a golang application. The same thing can be done in your host machine also. But good thing is that if you could build such application, then that will be consider as cloud Platform-as-a-Service(PaaS).
The basic requirement of the container will be: Ubuntu image and other packages such as editor, golang compiler and so on.
I would suggest to look on the docker development environment.
https://docs.docker.com/opensource/project/set-up-dev-env/
The docker development environment is running inside a container and the files are mounted from one of the host directory. The container image is build from Ubuntu scratch image and added required packages which are needed to compile docker source code.
I hope you almost got what you are looking for.
Although most of the time I am developing Java apps and am simply using Maven so my builds should be reproducible (at least that's what Maven says).
But say you are compiling a C++ program or something a little more involved, should you build inside of docker?
Or ideally use vagrant or another technology to produce reproduce able builds.
How do you manage reproducible build with docker?
You can, but not in your final image, as that would mean a much larger image than necessary: it would include all the compilation tool, instead of limiting to only what you need to execute the resulting binary.
You can see an alternative in "How do I build a Docker image for a Ruby project without build tools?"
I use an image to build,
I commit the resulting stopped container as a new image (with a volume including the resulting binary)
I use an execution image (one which only contain what you need to run), and copy the binary from the other image. I commit again the resulting container.
The final image includes the compiled binary and the execution environment.
I wanted to post an answer to this as well actually because to build on VonC's answer. Actually I just had Redhat Openshift training and they use a tool called Source to Image s2i, which uses docker to create docker images. And actually this strategy is great for managing a private (or public) cloud, where your build may be compiled on different machines, but you need to keep the build environment consistent.
I am gaining knowledge about Docker and I have the following questions
Where are Dockerfile's kept in a project?
Are they kept together with the source?
Are they kept outside of the source? Do you have an own Git repository just for the Dockerfile?
If the CI server should create a new image for each build and run that on the test server, do you keep the previous image? I mean, do you tag the previous image or do you remove the previous image before creating the new one?
I am a Java EE developer so I use Maven, Jenkins etc if that matter.
The only restriction on where a Dockerfile is kept is that any files you ADD to your image must be beneath the Dockerfile in the file system. I normally see them at the top level of projects, though I have a repo that combines a bunch of small images where I have something like
top/
project1/
Dockerfile
project1_files
project2/
Dockerfile
project2_files
The Jenkins docker plugin can point to an arbitrary directory with a Dockerfile, so that's easy. As for CI, the most common strategy I've seen is to tag each image built with CI as 'latest'. This is the default if you don't add a tag to a build. Then releases get their own tags. Thus, if you just run an image with no arguments you get the last image built by CI, but if you want a particular release it's easy to say so.
I'd recommend keeping the Dockerfile with the source as you would a makefile.
The build context issue means most Dockerfiles are kept at or near the top-level of the project. You can get around this by using scripts or build tooling to copy Dockerfiles or source folders about, but it gets a bit painful.
I'm unaware of best practice with regard to tags and CI. Tagging with the git hash or similar might be a good solution. You will want to keep at least one generation of old images in case you need to rollback.