I want to use go mod graph inside the docker container to generate dependency graph in go repository. I only use that command and not use the other go functionalities. I have try the other tools like godegraph, gomod, dept and nothing can do better than go mod graph . Currently I installed all the go functionalities and add about 400MB of my docker image size.
Question : Can I reduce the size of the docker container by installing spesific command in golang ? Or can I get the binary of go mod graph so its can reduce the size of the container?
If you don't need Go when executing your image (docker run), but only need it for building your image (docker build), then, as commented, use a multi-stage build.
Here is an example: "Create lean Docker images using the Builder Pattern" from Mike Kaperys:
Each FROM instruction defines a new base image for the following instructions, and begins a new build stage.
Before multi-stage builds it was possible to to achieve the same effect, but the process required 2 Dockerfiles.
Using multi-stage builds allows you to selectively copy files between build stages — this is the basis of the builder pattern.
See kaperys/blog/docker-builder-pattern
FROM golang:1.16
COPY . /go/src/github.com/kaperys/blog/docker-builder-pattern
WORKDIR /go/src/github.com/kaperys/blog/docker-builder-pattern
RUN go get && CGO_ENABLED=0 GOOS=linux go build -o server .
FROM scratch
LABEL maintainer="Mike Kaperys <mike#kaperys.io>"
COPY --from=0 /go/src/github.com/kaperys/blog/docker-builder-pattern/server /opt/kaperys/vision/server
COPY --from=0 /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
ADD html/ /opt/kaperys/vision/html
EXPOSE 8080
WORKDIR /opt/kaperys/vision
ENTRYPOINT [ "./server" ]
That way, you start from an image where Go is already installed (included go mod), then you build your actual image from any other one you want (including, in this example, "scratch").
So, I'm trying to get into embedded rust, for which I had to use the nightly version of rust, and modify my .cargo/config.toml to change the target device, and stuff. I decided to use docker, as I didn't want this interfering with my main installation. I don't know much about docker, but I'm assuming, it's quite similar to pipenv, where what I do with the docker image, doesn't affect anything outside it. (Unless I run the code)
So, this is how my Dockerfile looks
FROM jdrouet/rust-nightly:buster-slim AS builder
WORKDIR /usr/source/myapp
COPY . .
RUN cargo build --release
CMD cargo run
When I run sudo docker build . -t name It gives me the error I used to get before modifying my .cargo/config.toml file, which is a good thing, I'm guessing, cuz now I can revert to my original configuration, and make the changes to this image's config files. But I'm not able to find the configuration files for this docker image. I don't know what WORKDIR does, but there is no folder called /source in my /usr directory
So, I'm trying to get into embedded rust, for which I had to use the nightly version of rust, and modify my .cargo/config.toml to change the target device, and stuff
You can put a file in the folder wherever/your/project/is/.cargo/config.toml, and it will only impact the project(s) in that directory.
source: Cargo Book
I don't know much about docker, but I'm assuming, it's quite similar to pipenv
Docker is actually quite different to Pipenv. Cargo is similar to Pipenv in that it manages your dependencies for you (Cargo.toml vs Pipfile), distinguishes between regular dependencies vs dev dependencies vs build-time dependencies, etc. Docker is a level of isolation beyond this -- a Docker container is a completely different filesystem from your actual computer. The Dockerfile is a recipe that tells Docker how to build an image of your container, which Docker can run.
Basically, WORKDIR /usr/source/myapp creates a folder /usr/source/app in the Docker container's file system, and cd's into that for the rest of the Dockerfile. This means that the following line, COPY . ., will copy everything in the same folder as the Dockerfile into the folder in the container /usr/source/app.
I bet if you open a shell into the Docker container like so:
# Build the docker container
docker build . -t my-cool-project:latest
# Run it
docker run -it my-cool-project:latest bash
you should be able to cd /usr/source/app and see all your stuff.
Am trying to create a docker image where you can build/publish but also run a .net core website but having a real issue with
image mcr.microsoft.com/dotnet/sdk:5.0
If you build inside a container with that it publishes no problem but when you run you get issues.
By contrast if you run with
mcr.microsoft.com/dotnet/aspnet:5.0
then you get no issues, that runs the app fine, but of course you cant use that package to publish or build as its runtime only.
I have done an experiement to run a .net core web app using identical source code and dockerfiles except one is using the sdk and one is using aspnet.
Both dockerfiles build an image using pre-built dlls from a publish folder that was generated before. They look like this..
docker file 1
FROM mcr.microsoft.com/dotnet/aspnet:5.0
WORKDIR /app
#copy dlls from publish into working directory
COPY /publish .
ENTRYPOINT ["dotnet", "/app/Whipover.dll"]
docker file 2
FROM mcr.microsoft.com/dotnet/sdk:5.0
WORKDIR /app
#copy dlls from publish into working directory
COPY /publish .
ENTRYPOINT ["dotnet", "/app/Whipover.dll"]
Docker file 1 image runs the website no problem,
Docker file 2 image gives the error..
" Unable to bind to http://localhost:5000 on the IPv6 loopback interface: 'Cannot assign requested address'."
Do we know why that is? I thought the sdk image is supposed to be able to do everything the aspnet image can do.
As for the error I could modify docker file 2 to include
#ENV ASPNETCORE_URLS=http://+:80
#EXPOSE 80 5000
.. which runs the website but then it cant retrieve the contents of wwwroot.. so that would be another issue. Also I dont see why I need to manually expose ports when dockerfile 1 had no problem at all?
Thanks
The solution is to have a multipart dockerfile where you build with the sdk and then run with a runtime different image.
I was under the impression the sdk could do both.
I was looking at an out of date course on pluralsight. This is the best reference to use is
https://learn.microsoft.com/en-us/aspnet/core/host-and-deploy/docker/building-net-docker-images?view=aspnetcore-5.0
I need to create a docker container for an Asp.Net Core application (though the idea is the same for any application in any language) and there are 2 different approaches from what I can see:
From Microsoft docs:
FROM mcr.microsoft.com/dotnet/core/aspnet:3.1
COPY bin/Release/netcoreapp3.1/publish/ App/
WORKDIR /App
ENTRYPOINT ["dotnet", "NetCore.Docker.dll"]
From Docker docs:
FROM mcr.microsoft.com/dotnet/core/sdk:3.1 AS build-env
WORKDIR /app
# Copy csproj and restore as distinct layers
COPY *.csproj ./
RUN dotnet restore
# Copy everything else and build
COPY . ./
RUN dotnet publish -c Release -o out
# Build runtime image
FROM mcr.microsoft.com/dotnet/core/aspnet:3.1
WORKDIR /app
COPY --from=build-env /app/out .
ENTRYPOINT ["dotnet", "aspnetapp.dll"]
From what I can tell the difference is where the build of the application happens: either inside the docker itself or outside docker (in ci/cd tool e.g.) and only the output being copied in the docker file.
Is there any other technical difference/benefit between these approaches apart from the obvious ones of where the build commands are executed? (E.g. one would be faster, more reliable, etc.)
P.S. I plan to run CI/CD inside a containerized environment like Github Actions or Bitbucket pipelines, so the build should not be dependent on local develop settings in any case.
Personally I find if you build in the container there's fewer surprises as slight differences between developer machines become irrelevant, but there's a case to be made for both approaches. Sometimes the container build is slower, so you have to settle for a sub-optimal situation.
A build on your dev machine will have access to 100% of the CPU, generally speaking, but in a container it is limited by your Docker settings. Most people usually have that at 50% or less to avoid hogging the whole machine, so it's 50% as fast at best.
So the more reproducible approach is to let the container build everything.
The more performant approach is to build externally, then quickly package in the container.
Note that this equation changes considerably if you have a low-powered laptop but a very fast build server you can offload the build to. For example, there's no way a 2-core i3 laptop with 8GB can keep up with a 32-core Ryzen or Epyc server with 256GB.
I have a few Dockerfiles right now.
One is for Cassandra 3.5, and it is FROM cassandra:3.5
I also have a Dockerfile for Kafka, but t is quite a bit more complex. It is FROM java:openjdk-8-fre and it runs a long command to install Kafka and Zookeeper.
Finally, I have an application written in Scala that uses SBT.
For that Dockerfile, it is FROM broadinstitute/scala-baseimage, which gets me Java 8, Scala 2.11.7, and STB 0.13.9, which are what I need.
Perhaps, I don't understand how Docker works, but my Scala program has Cassandra and Kafka as dependencies and for development purposes, I want others to be able to simply clone my repo with the Dockerfile and then be able to build it with Cassandra, Kafka, Scala, Java and SBT all baked in so that they can just compile the source. I'm having a lot of issues with this though.
How do I combine these Dockerfiles? How do I simply make an environment with those things baked in?
You can, with the multi-stage builds feature introduced in Docker 1.17
Take a look at this:
FROM golang:1.7.3
WORKDIR /go/src/github.com/alexellis/href-counter/
RUN go get -d -v golang.org/x/net/html
COPY app.go .
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o app .
FROM alpine:latest
RUN apk --no-cache add ca-certificates
WORKDIR /root/
COPY --from=0 /go/src/github.com/alexellis/href-counter/app .
CMD ["./app"]
Then build the image normally:
docker build -t alexellis2/href-counter:latest
From : https://docs.docker.com/develop/develop-images/multistage-build/
The end result is the same tiny production image as before, with a significant reduction in complexity. You don’t need to create any intermediate images and you don’t need to extract any artifacts to your local system at all.
How does it work? The second FROM instruction starts a new build stage with the alpine:latest image as its base. The COPY --from=0 line copies just the built artifact from the previous stage into this new stage. The Go SDK and any intermediate artifacts are left behind, and not saved in the final image.
You can't combine dockerfiles as conflicts may occur. What you want to do is to create a new dockerfile or build a custom image.
TL;DR;
If your current development container contains all the tools you need and works, then save it as an image and upon it to a repo and create a dockerfile to pull from that image off that repo.
Details:
Building a custom image is by far easier than creating a dockerfile using a public image as you can store whatever hacks and mods into the image. To do so, start a blank container with a basic Linux image (or broadinstitute/scala-baseimage), install whatever tools you need and configure them until everything works correctly, then save it (the container) as an image. Create a new container off this image and test to see if you can build your code on top of it via docker-compose (or however you want to do/build it). If it works, than you have a working base image that you can upload to a repo so others can pull it.
To build a dockerfile with a public image, you will need to put all hacks, mods and setup on the dockerfile itself. That is, you will need to place every command line that you used into a text file and reduce whatever hacks, mods and setup into command lines. At the end, your dockerfile will create an image automatically and you don't need to store this image into a repo and all you need to do is to give others the dockerfile and they can spin the image up at their own docker.
Note that once you have a working dockerfile, you can tweak it easily as it will create a new image every time you use the dockerfile. With a custom image, you may run into issues where you need to rebuild the image due to conflicts. For example, all of your tools work with openjdk until you install one that doesn't work. The fix may involve uninstalling openjdk and use the oracle one, but all configuration you did for all the tools that you have installed broke.
The following answer applies to docker 1.7 and above:
I would prefer to use --from=NAME and from image as NAME
Why?
You can use --from=0 and above but this might get little hard to manage when you have many docker stages in dockerfile.
sample example:
FROM golang:1.7.3 as backend
WORKDIR /backend
RUN go get -d -v golang.org/x/net/html
COPY app.go .
RUN #install some stuff, compile assets....
FROM golang:1.7.3 as assets
WORKDIR /assets
RUN ./getassets.sh
FROM nodejs:latest as frontend
RUN npm install
WORKDIR /assets
COPY --from=assets /asets .
CMD ["./app"]
FROM alpine:latest as mergedassets
WORKDIR /root/
COPY --from=frontend . /
COPY --from=backend ./backend .
CMD ["./app"]
Note: Managing dockerfile properly will help to build a docker image much faster. Internally docker usings docker layer caching to help with this process, incase the image have to be rebuilt.
Yes, you can roll a whole lot of software into a single Docker image (GitLab does this, with one image that includes Postgres and everything else), but generalhenry is right - that's not the typical way to use Docker.
As you say, Cassandra and Kafka are dependencies for your Scala app, they're not part of the app, so they don't all belong in the same image.
Having to orchestrate many containers with Docker Compose adds an extra admin layer, but it gives you much more flexibility:
your containers can have different lifespans, so when you have a new version of your app to deploy, you only need to run a new app container, you can leave the dependencies running;
you can use the same app image in any environment, using different configurations for your dependencies - e.g. in dev you can run a basic Kafka container and in prod have it clustered on many nodes, your app container is the same;
your dependencies can be used by other apps too - so multiple consumers can run in different containers and all work with the same Kafka and Cassandra containers;
plus all the scalability, logging etc. already mentioned.
When might you want to "combine" Docker images?
As others are pointing out here, you typically don't want to put your database and you application into the same Docker image. Ideally you want a Docker image to wrap a "single process"/"runtime". This allows each process to be scaled up/down and restarted individually.
Let's say you want to use some shared C-libraries/executables that are not available in the package manager of the image you are using, but someone else has created an image where they are precompiled - and you might not want to recompile these binaries as part of your build (depending on how long this takes). Is there a way to quickly create a POC-Docker image containing all of these executables/libraries based on the existing images?
Docker and Composition
Relevant discussion: https://github.com/moby/moby/issues/3378
What Docker lacks is a good way of composing images. You can copy individual files or entire file systems from other images into your own using COPY --from=<image> <from-path> <to-path>. There is no builtin way of copying the environment variables from another image into your own.
That said, I have personally created a custom frontend/parser for Dockerfiles that adds an INCLUDE <image>-keyword. This copies the entire filesystem, along with the environment variables into your image:
DOCKER_BUILDKIT=1 docker build -t myimage .
#syntax=bergkvist/includeimage
FROM alpine:3.12.0
INCLUDE rust:1.44-alpine3.12
INCLUDE python:3.8.3-alpine3.12
nixpkgs.dockerTools
if you want truly composable Docker builds, I recommend checking out dockerTools in nixpkgs. This will also result in more reproducible (and typically very small) images. See https://nix.dev/tutorials/building-and-running-docker-images
docker load < $(nix-build docker-image.nix)
# docker-image.nix
let
pkgs = import <nixpkgs> {};
python = pkgs.python38;
rustc = pkgs.rustc;
in pkgs.dockerTools.buildImage {
name = "myimage";
tag = "latest";
contents = [ python rustc ];
}
Docker doesn't do merges of the images, but there isn't anything stopping you combining the dockerfiles if available, and rolling into them into a fat image which you'd need to build. There's times where this makes sense, however, as for running multiple processes in a container most Docker dogma will point to this as less desirable especially with microservice architecture (however rules are there to be broken right?)
You could not combine docker images into 1 container. See the detail discussions in Moby issue, How do I combine several images into one via Dockerfile.
For your case, it is better to not include the whole Cassandra and Kafka images. The application would only need the Cassandra Scala driver and Kafka Scala driver. The container should include the drivers only.
I needed docker:latest and python:latest images for Gitlab CI. Here is what I came up with:
FROM ubuntu:latest
RUN apt update
RUN apt install -y sudo
RUN sudo apt install -y docker.io
RUN sudo apt install -y python3-pip
RUN sudo apt install -y python3
RUN docker --version
RUN pip3 --version
RUN python3 --version
After I've build and pushed it to my Docker Hub repo:
docker build -t docker-hub-repo/image-name:latest path/to/Dockerfile
docker push docker-hub-repo/image-name:latest
Don't forget to docker login before push
Hope it helps