I have created a dockerfile that builds a scratch image from which it serves a static golang build. When I start a container from the image, I receive this error:
idea_service_prod | standard_init_linux.go:211: exec user process caused "no such file or directory"
idea_service_prod exited with code 1
The dockerfile:
FROM golang:1.14 as base
# install librdkafka
ENV LIBRDKAFKA_VERSION=1.4.2
RUN curl -Lk -o /root/librdkafka-${LIBRDKAFKA_VERSION}.tar.gz https://github.com/edenhill/librdkafka/archive/v${LIBRDKAFKA_VERSION}.tar.gz && \
tar -xzf /root/librdkafka-${LIBRDKAFKA_VERSION}.tar.gz -C /root && \
cd /root/librdkafka-${LIBRDKAFKA_VERSION} && \
./configure --prefix /usr && make && make install && make clean && ./configure --clean
# install inotify-tools
FROM base as dev
EXPOSE 80
RUN apt-get -y update \
&& apt-get install -y --no-install-recommends inotify-tools \
&& apt-get clean
FROM base as build
# install git
RUN apt-get -y update \
&& apt-get install -y --no-install-recommends git \
&& apt-get clean
# configure user
ENV USER=appuser
ENV UID=10001
RUN adduser \
--disabled-password \
--gecos "" \
--home "/nonexistent" \
--shell "/sbin/nologin" \
--no-create-home \
--uid "${UID}" \
"${USER}"
# download dependencies
WORKDIR $GOPATH/src/ideaservice
COPY . .
RUN go get -d -v
RUN go mod download
RUN go mod verify
# build binary
RUN GOOS=linux GOARCH=amd64 go build -tags static -ldflags="-w -s" -o /go/bin/main cmd/ideaservice/main.go
FROM scratch
EXPOSE 80
COPY --from=build /etc/passwd /etc/passwd
COPY --from=build /etc/group/ /etc/group
COPY --from=build /go/bin/main /go/bin/main
USER appuser:appuser
CMD ["/go/bin/main"]
What could be causing this? It seems that it should work - the binary is copied from the past stage in the build process, to a destination in the scratch image, where it is then executed from.
Dynamic linking issue. I would suggest you read article by Jerome. It is explains why does this append and how to avoid it, what is difference betwean base images for go dev, linux and libraries support provided by them.
Related
I have a docker file in which I do wget to copy something in the image. But the build is failing giving 'wget command not found'. WHen i googled I found suggestions to install wget like below
RUN apt update && apt upgrade
RUN apt install wget
Docker File:
FROM openjdk:17
LABEL maintainer="app"
ARG uname
ARG pwd
RUN useradd -ms /bin/bash -u 1000 user1
COPY . /app
WORKDIR /app
RUN ./gradlew build -PmavenUsername=$uname -PmavenPassword=$pwd
ARG YOURKIT_VERSION=2021.11
ARG POLARIS_YK_DIR=YourKit-JavaProfiler-2019.8
RUN wget https://www.yourkit.com/download/docker/YourKit-JavaProfiler-${YOURKIT_VERSION}-docker.zip --no-check-certificate -P /tmp/ && \
unzip /tmp/YourKit-JavaProfiler-${YOURKIT_VERSION}-docker.zip -d /usr/local && \
mv /usr/local/YourKit-JavaProfiler-${YOURKIT_VERSION} /usr/local/$POLARIS_YK_DIR && \
rm /tmp/YourKit-JavaProfiler-${YOURKIT_VERSION}-docker.zip
EXPOSE 10001
EXPOSE 8080
EXPOSE 5005
USER 1000
ENTRYPOINT ["sh", "/docker_entrypoint.sh"]
On doing this I am getting error app-get not found. Can some one suggest any solution.
The openjdk image you use is based on Oracle Linux which uses microdnf rather than apt as it's package manager.
To install wget (and unzip which you also need), you can add this to your Dockerfile:
RUN microdnf update \
&& microdnf install --nodocs wget unzip \
&& microdnf clean all \
&& rm -rf /var/cache/yum
The commands clean up the package cache after installing, to keep the image size as small as possible.
I have a project running with meteor and node.js in my local. The meteor version is 2.4, node.js version is 8.9.4, I have meteor/release file to make meteor version be 2.2 so that meteor and node can work together.
(base) xxx$ meteor --version
Meteor 2.4
(base) xxx$ node -v
v8.9.4
It seems fine so I deploy this project to docker container to server. The Dockerfile first line I wrote
# node version dependent on meteor version
FROM node:8.9.4
After successfully deployed, the docker logs shows error siad.
Waiting for mongodb server to start - sleeping
warn: --minUptime not set. Defaulting to: 1000ms
warn: --spinSleepTime not set. Your script will exit if it does not stay up for at least 1000ms
info: Forever processing file: /app/bundle/main.js
error: undefined
data: /app/bundle/main.js:34 - Meteor requires Node v12.0.0 or later.
data: /app/bundle/main.js:34 - error: Forever detected script exited with code: 1
I check inside docker, the node version is 8.9.4
(base) [xxx]$ docker exec -it -u root tblbuilder_meteor_1 /bin/bash -c 'node --version'
v8.9.4
So I assume it is meteor version. But first I dont know how to check meteor version inside the docker. And second why this happens? I am sure the release file is updated to push project folder.
With some great man help, I kinda understand it. In local I use meteor 2.2, in docker file I use node.js 8.9.4 work with meteor2.2. So the thing I left is to modify DOCKERFILE, change it from node 8.9.4 to node 12. Below is my Dockerfile file, I try to change it to node 12.22.2, but it keep give me error, I spent one day to solve them. Currently, I stack at install r-base part.
Is there some guide for change node 8 to node 12.
# node version dependent on meteor version
FROM node:8.9.4
# I am going to use 12.22.2
#FROM node:12.22.2
# (even if copied as root you still need to change)
# https://github.com/moby/moby/issues/6119
COPY ./compose/meteor/entrypoint.sh /entrypoint.sh
COPY ./compose/meteor/run_app.sh /run_app.sh
COPY ./compose/meteor/r-cran.pgp /r-cran.pgp
COPY ./settings/settings.json /app/settings.json
COPY ./requirements.txt /requirements.txt
COPY ./r_requirements.sh /r_requirements.sh
# set locale to utf8: https://github.com/docker-library/docs/pull/703/files
# added [check-valid-until=no] & Acquire::Check-Valid-Until "false"; https://unix.stackexchange.com/questions/508724/failed-to-fetch-jessie-backports-repository
# Needs work to bring it up-to-date
RUN \
echo "deb [check-valid-until=no] http://archive.debian.org/debian jessie-backports main" > /etc/apt/sources.list.d/jessie-backports.list && \
sed -i '/deb http:\/\/deb.debian.org\/debian jessie-updates main/d' /etc/apt/sources.list && \
apt-get -o Acquire::Check-Valid-Until=false update && \
\
sh -c 'echo "deb [check-valid-until=no] http://cran.rstudio.com/bin/linux/debian jessie-cran35/" >> /etc/apt/sources.list' && \
apt-key add /r-cran.pgp && \
\
apt-get -o Acquire::Check-Valid-Until=false update && \
apt-get -o Acquire::Check-Valid-Until=false install -y locales && \
\
localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8 && \
export LC_ALL=en_US.UTF-8 && \
export LANG=en_US.UTF-8 && \
export LANGUAGE=en_US.UTF-8
ENV LANG en_US.utf8
ENV LC_ALL en_US.UTF-8
# add rstudio debian install for R (requires version >3.3)
# https://cran.r-project.org/bin/linux/debian/
# install R from apt-get
# install python 3.6 from source :/
RUN apt install -y --force-yes r-base-core r-recommended r-base-html r-base-core
RUN apt-get install -y --force-yes wget bsdtar r-base r-base-dev && \
apt-get clean && \
\
wget https://www.python.org/ftp/python/3.6.5/Python-3.6.5.tgz && \
tar zxf Python-3.6.5.tgz && \
cd ./Python-3.6.5 && \
./configure && \
make && \
make altinstall && \
cd .. && \
rm Python-3.6.5.tgz && \
rm -rf ./Python-3.6.5
# create paths and users
# change executable permissions
RUN npm install forever -g && \
\
mkdir -p /app/production && \
mkdir -p /app/logs && \
mkdir -p /app/crons && \
\
groupadd -r app && \
useradd -m -d /home/app -g app app && \
\
chmod +x /entrypoint.sh && \
chmod +x /run_app.sh && \
chmod +x /r_requirements.sh && \
chmod +x /requirements.txt && \
\
chown -R app:app /app && \
chown app:app /entrypoint.sh && \
chown app:app /run_app.sh && \
chown app:app /r_requirements.sh &&\
chown app:app /requirements.txt
USER app
# 1) install R packages
# 2) install python packages
RUN export "R_LIBS=/home/app/R_libs" && \
mkdir /home/app/R_libs && \
bash /r_requirements.sh && \
\
/usr/local/bin/pip3.6 install --user -r /requirements.txt
USER root
COPY ./compose/meteor/src/src.tar.gz /app/src.tar.gz
COPY ./src/private /app/src/private
RUN chown -R app:app /app
USER app
RUN cd /app && \
bsdtar -xzvf src.tar.gz && \
npm install --prefix /app/bundle/programs/server --production
ENTRYPOINT ["/entrypoint.sh"]
There are many wrong understanding in your tests:
Your Meteor version is 2.2, because is the version inside your project;
To you see the Node version of this Meteor project, see this answers that many guys send to you in Meteor Docker Node.js version is not match
Usually, we build the Meteor, that mean transform it in a NodeJS package build, then, inside of the Docker you don't need Meteor.
We need see your Dockerfile and understand what process you do to build do Docker image.
docker image inspect <name>
gives me 16GB
and about 20 layers
When I am logged as root, this
du -hs /
show me just 2GB
FYI, there are already very multi-lines RUN commands in Dockerfile.
can I squash all layers into one layer without touching Dockerfile, rebuilding etc?
or possibly by adding extra action to Dockerfile which clear/improve caching
Dockerfile is
FROM heroku/heroku:18
ENV PYENV_ROOT="/pyenv"
ENV PATH="/pyenv/shims:/pyenv/bin:$PATH"
ENV PYTHON_VERSION 3.5.6
ENV GPG_KEY <value>
ENV PYTHONUNBUFFERED 1
ENV TERM xterm
ENV EDITOR vim
RUN apt-get update && apt-get install -y \
build-essential \
gdal-bin \
binutils \
iputils-ping \
libjpeg8 \
libproj-dev \
libjpeg8-dev \
libtiff-dev \
zlib1g-dev \
libfreetype6-dev \
liblcms2-dev \
libxml2-dev \
libxslt1-dev \
libssl-dev \
libncurses5-dev \
virtualenv \
python-pip \
python3-pip \
python-dev \
libmysqlclient-dev \
mysql-client-5.7 \
libpq-dev \
libcurl4-gnutls-dev \
libgnutls28-dev \
libbz2-dev \
tig \
git \
vim \
nano \
tmux \
tmuxinator \
fish \
sudo \
libnet-ifconfig-wrapper-perl \
ruby \
libssl-dev \
nodejs \
strace \
tcpdump \
# npm & grunt
&& curl -L https://npmjs.com/install.sh | sh \
&& npm install -g grunt-cli grunt \
# ruby & foreman
&& gem install foreman \
# installing pyenv
&& curl https://raw.githubusercontent.com/yyuu/pyenv-installer/master/bin/pyenv-installer | bash
COPY . /app
COPY ./requirements /requirements
COPY ./requirements.txt /requirements.txt
COPY ./docker/docker_compose/django/foreman.sh /foreman.sh
COPY ./docker/docker_compose/django/Procfile /Procfile
COPY ./docker/docker_compose/django/entrypoint.sh /entrypoint.sh
# ADD sudoer user django with password django
RUN groupadd -r django -g 1000 && \
useradd -ms /usr/bin/fish -p $(openssl passwd -1 django) --uid 1000 --gid 1000 -r -g django django && \
usermod -a -G sudo django && \
chown -R django:django /app
COPY --chown=django:django ./docker/docker_compose/django/fish /home/django/.config/fish
COPY --chown=django:django ./docker/docker_compose/django/tmuxinator /home/django/.tmuxinator
COPY ./docker/docker_compose/django/fish /root/.config/fish
WORKDIR /app
RUN sed -i 's/\r//' /entrypoint.sh \
&& sed -i 's/\r//' /foreman.sh \
&& chmod +x /entrypoint.sh \
&& chown django /entrypoint.sh \
&& chmod +x /foreman.sh \
&& chown django /foreman.sh \
&& chown -R django:django /home/django/ \
&& pyenv install ${PYTHON_VERSION%%} \
&& mkdir -p /app/log \
&& pyenv global ${PYTHON_VERSION%%} \
&& pyenv rehash \
&& ${PYENV_ROOT%%}/versions/${PYTHON_VERSION%%}/bin/pip install -U pip \
&& ${PYENV_ROOT%%}/versions/${PYTHON_VERSION%%}/bin/pip install -r /requirements.txt \
&& chown -R django:django /pyenv/ \
&& ${PYENV_ROOT%%}/versions/${PYTHON_VERSION%%}/bin/pip install -r /requirements/dev_requirements.txt
# this user receives ENVs from the top
USER django
ENTRYPOINT ["/entrypoint.sh"]
What I've tried so far:
The --squash option from experimental mode of docker build is rather not for me. That Dockerfile is one of more Dockerfiles inside docker-compose.
I've also checked this:
https://github.com/jwilder/docker-squash
but seems docker load cannot load a squashed image.
also, that squash gives me 8GB (still far away from expected ~2GB)
docker save <image_id> | docker-squash -t latest_tiny | docker load
update after answers:
when I've added this:
&& apt-get autoremove \ # ? to consider
&& apt-get clean \ # ? to consider
&& rm -rf /var/lib/apt/lists/*
to apt-get and --no-cache-dir to each pip, the result was 72GB (yes, even much more - docker images shows 36GB before pip command, and 72GB as final size).
my working directory is clear (regarding COPY). du -hs / (as a root) still has 2GB. And all images were removed before rebuilding.
Following the #Mihai approach, I was able to slim down the image from 16GB to 9GB.
There is a simple trick to get rid of the intermediate layers. It will bring down the size as well but with how much depends on how it was built.
Create a Dockerfile like this:
FROM your_image as initial
FROM your_image_base
COPY --from=initial / /
your_image_base should be something like 'alpine' - so the smallest image from which your image and its parents descend from.
Now build the image and check the history and size:
docker build -t your-image:2.0 .
docker image history your-image:2.0
docker image ls
This way you do create a new Dockerfile (if that is acceptable for your process) without touching the initial Dockerfile.
Let me know if this solves your issue.
UPDATE AFTER SEEING THE Dockerfile:
maybe I miss it but I don't see you cleaning up the apt-get cache after you perform the installations. Your big RUN command should end with "&& rm -rf /var/lib/apt/lists/*" on the same line so that it doesn't store the whole cache on the layer.
Definitely add && rm -rf /var/lib/apt/lists/* on the end of your main run command, like Mihai said. Another thing that may help (depending on how big your dependencies are) is installing with pip using the --no-cache-dir option . Also, make sure you understand build context and consider using either a .dockerignore or sending the context to another directory (totally depends on how you're directory is setup)
I've also had luck exploring an image using dive. Honestly this looks like a pretty big image so not sure how much you're going to be able to get it down
To squash a (Docker) container image, without re-building the image or manipulating the original Dockerfile,
You can extend from your image and squash it:
docker build --squash -t your_image_squashed - <<< "FROM your_image"
It's very easy, just use
docker commit YOUR_CONTAINER_ID NEW_IMAGE_ID
The docker will throw away the intermediate layers, you lost history but the size is small
I have some server and clients who are communicating using gRPC/golang. Now I want to containerize my application but the size of the docker image containing goland execution and grpc support is larger(more than 1GB). I would like to decrease the size of the docker image.
The required golang version is 1.9 and higher. Here is the Dockerfile script is given. If there is other way please suggest it.
FROM golang:1.11
RUN apt-get update && \
apt-get -y install git unzip build-essential autoconf libtool
RUN git clone https://github.com/google/protobuf.git && \
cd protobuf && \
./autogen.sh && \
./configure && \
make && \
make install && \
ldconfig && \
make clean && \
cd .. && \
rm -r protobuf
RUN go get google.golang.org/grpc
RUN go get github.com/golang/protobuf/protoc-gen-go
RUN ls -la
WORKDIR /helloworld
COPY . /helloworld
RUN protoc -I helloworld/ helloworld/helloworld.proto --go_out=plugins=grpc:helloworld
CMD ["go", "run", "helloworld/greeter_server/main.go"]
try to make a multistage docker image like this
# Compile stage
FROM golang:1.11 as build-env
RUN apt-get update && \
apt-get -y install git unzip build-essential autoconf libtool
RUN git clone https://github.com/google/protobuf.git && \
cd protobuf && \
./autogen.sh && \
./configure && \
make && \
make install && \
ldconfig && \
make clean && \
cd .. && \
rm -r protobuf
RUN go get google.golang.org/grpc
RUN go get github.com/golang/protobuf/protoc-gen-go
RUN ls -la
WORKDIR /helloworld
COPY . /helloworld
RUN protoc -I helloworld/ helloworld/helloworld.proto --go_out=plugins=grpc:helloworld
RUN go build -o server helloworld/greeter_server/main.go
# Making image
FROM alpine:3.8 AS host
RUN apk add --no-cache \
ca-certificates
COPY --from=build-env /helloworld/server /
# copy any other files you need
WORKDIR /
EXPOSE 8000
CMD ["server"]
You can try to use distroless base images and multi-stage builds. That might help you.
I have this file Dockerfile.nlu
FROM chatbot/spacy:latest
WORKDIR /app
COPY nlu ./agent_nlu
RUN python –m rasa_nlu.train --config agent_nlu/config.yml --data agent_nlu/data/ --path agent_nlu/agent --fixed_model_name default
and I get the error below:
]$ sudo docker build -t nlu:latest -f docker/Dockerfile.nlu .
Sending build context to Docker daemon 9.216kB
Step 1/4 : FROM chatbot/spacy:latest
---> 496dc6a38abb
Step 2/4 : WORKDIR /app
---> Using cache
---> 7f02012c8452
Step 3/4 : COPY nlu ./agent_nlu
COPY failed: stat /var/lib/docker/tmp/docker-builder363868051/nlu: no such file or directory
It doesn't look like Docker can find the nlu directory. Are you sure it exists? Are you sure that you are executing the command from the correct directory?
But you also aren't installing Rasa at all or any of it's requirements. Is there a reason you aren't using the pre-built Rasa images? available here with docs here.
Here is a fully functional Docker file pulled from their repo.
FROM python:3.6-slim
ENV RASA_NLU_DOCKER="YES" \
RASA_NLU_HOME=/app \
RASA_NLU_PYTHON_PACKAGES=/usr/local/lib/python3.6/dist-packages
# Run updates, install basics and cleanup
# - build-essential: Compile specific dependencies
# - git-core: Checkout git repos
RUN apt-get update -qq \
&& apt-get install -y --no-install-recommends build-essential git-core openssl libssl-dev libffi6 libffi-dev curl \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
WORKDIR ${RASA_NLU_HOME}
COPY . ${RASA_NLU_HOME}
# use bash always
RUN rm /bin/sh && ln -s /bin/bash /bin/sh
RUN pip install -r alt_requirements/requirements_spacy_sklearn.txt
RUN pip install -e .
RUN pip install https://github.com/explosion/spacy-models/releases/download/en_core_web_md-2.0.0/en_core_web_md-2.0.0.tar.gz --no-cache-dir > /dev/null \
&& python -m spacy link en_core_web_md en \
&& pip install https://github.com/explosion/spacy-models/releases/download/de_core_news_sm-2.0.0/de_core_news_sm-2.0.0.tar.gz --no-cache-dir > /dev/null \
&& python -m spacy link de_core_news_sm de
COPY sample_configs/config_spacy.yml ${RASA_NLU_HOME}/config.yml
VOLUME ["/app/projects", "/app/logs", "/app/data"]
EXPOSE 5000
ENTRYPOINT ["./entrypoint.sh"]
CMD ["start", "-c", "config.yml", "--path", "/app/projects"]