setpgid: Operation not permitted on Docker - docker

Any ideas why the following works (the Docker container runs without errors):
FROM alpine:3.14.0
RUN apk update && \
apk --no-cache add dcron
COPY entry.sh /entry.sh
RUN chmod +x /entry.sh
CMD /entry.sh
# entry.sh
#!/bin/sh
# start cron
/usr/sbin/crond -f
While the following
FROM alpine:3.14.0
RUN apk update && \
apk --no-cache add dcron
CMD /usr/sbin/crond -f
gives me the
cnt | setpgid: Operation not permitted
cnt exited with code 1
What am I missing here?

I think that is related to alpine:3.14.0.
Check it out:
https://gitlab.alpinelinux.org/alpine/aports/-/issues/12396
https://wiki.alpinelinux.org/wiki/Release_Notes_for_Alpine_3.14.0#faccessat2
Try to update you Docker version

Looks like it has something with how Linux process group ID and session ID works (more details in credentials(7)) and should be fixed in dcron.
At the moment dcron calls setpgid if started with -f option:
If dcron is started as PID1 in container - it works.
If dcron is started as non-PID1 in container - it also works.
But if some other process start as PID1 and then execve() to dcron - it doesn't work (setpgid returns EPERM for reasons explained in setpgid(2)).
UPDATE: See https://github.com/dubiousjim/dcron/issues/13

Related

su-exec: setgroups: Operation not permitted - error on Alpine container

H,
I'm re-working some container images that have been running the containers as root, which is non-ideal for security. I'm using the su-exec package (available through apk here) to run a bunch of commands that would require a superuser but are called with a non-root user that makes use of su-exec; note that this package is basically gosu that takes up less space.
To test out the command out I ran and exec'd into an alpine container with su-exec added via apk but I can't seem to run anything and continue to get the error
su-exec: setgroups: Operation not permitted
when simple commands like apk add <pkg> or sh are run with the non-root user:
$ su-exec root apk add --no-cache curl
su-exec: setgroups: Operation not permitted
$ su-exec root sh
su-exec: setgroups: Operation not permitted
$ su-exec --help
Usage: su-exec user-spec command [args]
Any insight is much appreciated!

How to set breakpoint in Dockerfile itself?

Searching up the above shows many results about how to set breakpoints for apps running in docker containers, yet I'm interested in setting a breakpoint in the Dockerfile itself, such that the docker build is paused at the breakpoint. For an example Dockerfile:
FROM ubuntu:20.04
RUN echo "hello"
RUN echo "bye"
I'm looking for a way to set a breakpoint on the RUN echo "bye" such that when I debug this Dockerfile, the image will build non-interactively up to the RUN echo "bye" point, exclusive. After then, I would be able to interactively run commands with the container. In the actual Dockerfile I have, there are RUNs before the breakpoint that change the file system of the image being built, and I want to analyze the filesystem of the image at the breakpoint by being able to interactively run commands like cd / ls / find at the time of the breakpoint.
You can't set a breakpoint per se, but you can get an interactive shell at an arbitrary point in your build sequence (between steps).
Let's build your image:
Sending build context to Docker daemon 2.048kB
Step 1/3 : FROM ubuntu:20.04
---> 1e4467b07108
Step 2/3 : RUN echo "hello"
---> Running in 917b34190e35
hello
Removing intermediate container 917b34190e35
---> 12ebbdc1e72d
Step 3/3 : RUN echo "bye"
---> Running in c2a4a71ae444
bye
Removing intermediate container c2a4a71ae444
---> 3c52993b0185
Successfully built 3c52993b0185
Each of the lines that says ---> 0123456789ab with a hex ID has a valid image ID. So from here you can
docker run --rm -it 12ebbdc1e72d sh
which will give you an interactive shell on the partial image resulting from the first RUN command.
There's no requirement that the build as a whole succeed. If a RUN step fails, you can use this technique to get an interactive shell on the image immediately before that step and re-run the command by hand. If you have a very long RUN command, you may need to break it into two to be able to get a debugging shell at a specific point within the command sequence.
I don't think this is possible directly - that feature has been discussed and rejected.
What I generally do to debug a Dockerfile is to comment all of the steps after the "breakpoint", then run docker build followed by docker run -it image bash or docker run -it image sh (depending on whether you have bash installed inside the container).
Then, I have an interactive shell, and I can run commands to debug why later stages are failing.
I agree that being able to set a breakpoint and poke around would be a handy feature, though.
You can run commands in intermediate containers using Remote shell debugging tricks.
Make sure your container images include basic utilities like netcat (nc) and fuser. These utilities enable "calling home" from any intermediate container image. At home you'll answer calls with netcat (or socat). This netcat will send your commands to containers, and print their outcomes. This debugging approach will work even on Dockerfiles that are built on unknown worker nodes somewhere in cloud.
Example:
FROM debian:testing-slim
# Set environment variables for calling home from breakpoints (BP)
ENV BP_HOME=<IP-ADDRESS-OF-YOUR-HOST>
ENV BP_PORT=33720
ENV BP_CALLHOME='BP_FIFO=/tmp/$BP.$BP_HOME.$BP_PORT; (rm -f $BP_FIFO; mkfifo $BP_FIFO) && (echo "\"c\" continues"; echo -n "($BP) "; tail -f $BP_FIFO) | nc $BP_HOME $BP_PORT | while read cmd; do if test "$cmd" = "c" ; then echo -n "" >$BP_FIFO; sleep 0.1; fuser -k $BP_FIFO >/dev/null 2>&1; break; else eval $cmd >$BP_FIFO 2>&1; echo -n "($BP) " >$BP_FIFO; fi; done'
# Install needed utils (netcat, fuser)
RUN apt update && apt install -y netcat psmisc
# Now you are ready to run "eval $BP_CALLHOME" wherever you want to call home.
RUN BP=before-hello eval $BP_CALLHOME
RUN echo "hello"
RUN BP=after-hello eval $BP_CALLHOME
RUN echo "bye"
Start waiting for and answering calls from a Dockerfile before launching a Docker build. On home host run nc -k -l -p 33720 (alternatively socat STDIN TCP-LISTEN:33720,reuseaddr,fork).
This is how above example looks like at home:
$ nc -k -l -p 33720
"c" continues
(before-hello) echo *
bin boot dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var
(before-hello) id
uid=0(root) gid=0(root) groups=0(root)
(before-hello) c
"c" continues
(after-hello)
...
The recent (May 2022) project ktock/buildg offers breakpoints.
See "Interactive debugger for Dockerfile" from Kohei Tokunaga
buildg is a tool to interactively debug Dockerfile based on BuildKit.
Source-level inspection
Breakpoints and step execution
Interactive shell on a step with your own debugigng tools
Based on BuildKit (needs unmerged patches)
Supports rootless
The command break, b LINE_NUMBER sets a breakpoint.
Example:
$ buildg.sh debug --image=ubuntu:22.04 /tmp/ctx
WARN[2022-05-09T01:40:21Z] using host network as the default
#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.1s
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 195B done
#2 DONE 0.1s
#3 [internal] load metadata for docker.io/library/busybox:latest
#3 DONE 3.0s
#4 [build1 1/2] FROM docker.io/library/busybox#sha256:d2b53584f580310186df7a2055ce3ff83cc0df6caacf1e3489bff8cf5d0af5d8
#4 resolve docker.io/library/busybox#sha256:d2b53584f580310186df7a2055ce3ff83cc0df6caacf1e3489bff8cf5d0af5d8 0.0s done
#4 sha256:50e8d59317eb665383b2ef4d9434aeaa394dcd6f54b96bb7810fdde583e9c2d1 772.81kB / 772.81kB 0.2s done
Filename: "Dockerfile"
2| RUN echo hello > /hello
3|
4| FROM busybox AS build2
=> 5| RUN echo hi > /hi
6|
7| FROM scratch
8| COPY --from=build1 /hello /
>>> break 2
>>> breakpoints
[0]: line 2
>>> continue
#4 extracting sha256:50e8d59317eb665383b2ef4d9434aeaa394dcd6f54b96bb7810fdde583e9c2d1 0.0s done
#4 DONE 0.3s
...
From PR 24:
Add --cache-reuse option which allows sharing the build cache among invocation of buildg debug to make the 2nd-time debugging faster.
This is useful to speed up running buildg multiple times for debugging an errored step.
Note that breakpoints on cached steps are ignored as of now.
Because of this limitation, this feature is optional as of now. We should fix this limitation and make it the default behaviour in the future.
Man, Docker makes things hard. Here's a workaround I cooked up:
Insert FROM scratch where you want the break point.
Run docker build . --stage=<n-1> where <n> is the number of FROM commands before your "breakpoint". Eg, if it's a single stage build, use --stage=0.
Alternatively, if you have already named the stage where you want the break point with FROM <image> AS <stage> then you can use --stage=<stage> instead.
Docker has cached all your successful layers anyway (even if you can't see them), and because the FROM "breakpoint" comes before the (potentially unsuccessful) point of interest, the build should all come from cache and be very fast.
So for example, if my Dockerfile looks like this:
FROM debian:bullseye AS build
RUN apt-get update && apt-get install -y \
build-essential cmake ninja-build \
libfontconfig1-dev libdbus-1-dev libfreetype6-dev libicu-dev libinput-dev libxkbcommon-dev libsqlite3-dev libssl-dev libpng-dev libjpeg-dev libglib2.0-dev
<SNIP lots of other setup commands>
ADD my_source.tar.xz /
WORKDIR /my_source
RUN ./configure -option1 -option2
RUN cmake --build . --parallel
RUN cmake --install .
FROM alpine
COPY --from=build /my_build /my_build
...
Then I can add a "breakpoint" like this:
FROM debian:bullseye AS build
RUN apt-get update && apt-get install -y \
build-essential cmake ninja-build \
libfontconfig1-dev libdbus-1-dev libfreetype6-dev libicu-dev libinput-dev libxkbcommon-dev libsqlite3-dev libssl-dev libpng-dev libjpeg-dev libglib2.0-dev
<SNIP lots of other setup commands>
ADD my_source.tar.xz /
WORKDIR /my_source
#### BREAKPOINT ###
FROM scratch
#### BREAKPOINT ###
RUN ./configure -option1 -option2
RUN cmake --build . --parallel
RUN cmake --install .
FROM alpine
COPY --from=build /my_build /my_build
...
and trigger it with docker build . --stage=build

Docker build error: failed to add the host

I'm facing an unexpected error when running docker build, and I say unexpected because I haven't changed my Dockerfile for a while, and it had worked fine for the last time two weeks ago, but now I'm getting the following error:
failed to create endpoint optimistic_spence on network bridge: failed to add the host (veth9fc3a03) <=> sandbox (veth15abfd6) pair interfaces: operation not supported
In case it is of any help:
Docker version is 18.06.0-ce, build 0ffa8257ec
I don't see any container with docker ps
Systemd returns an active status for the docker process (sudo systemctl status docker)
Build command is: docker build -t user/repo:tag .
Dockerfile looks like:
FROM alpine:3.4
LABEL version="current version"
LABEL description="A nice description."
LABEL maintainer="my#email.com"
RUN apk update && apk add \
gcc \
g++ \
make \
git \
&& git clone https://gitlab.com/user/repo.git \
&& cd repo \
&& make \
&& cp program /bin \
&& rm -r /repo \
&& apk del g++ make git
WORKDIR /tmp
ENTRYPOINT ["program"]
Does anybody understand what is going on? Thank you!
EDIT
When combined with the --network option, the error changes a little, but it won't fix the problem. For example, --network=host gives the following:
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.4/main: temporary error (try again later)
WARNING: Ignoring APKINDEX.167438ca.tar.gz: No such file or directory
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.4/community: temporary error (try again later)
WARNING: Ignoring APKINDEX.a2e6dac0.tar.gz: No such file or directory
2 errors; 11 distinct packages available
The command '/bin/sh -c apk update && apk add gcc g++ make git && git clone https://gitlab.com/user/repo.git && cd repo && make && cp program /bin && rm -r /repo && apk del g++ make git' returned a non-zero code: 2
Had the same error and systemctl restart docker nor pruning og images & system did not do it for me, I ended up rebooting my computer which seems o have resolved the issue.
It looks that something wrong happened with any docker network bridge and it doesn't let you to create the same because is "zombie".
Try with following steps:
docker network prune, and if it doesn't work, try with:
docker system prune <-- Careful, this also will purge your named volumes contents, i.e, volumes that are not assigned to a container. So, if you have volumes assigned to a container, you should have to re-build/create containers.
/etc/init.d/docker restart
Show me what happens and let's see, actually I need more info about your problem if it doesn't solve it to you.
Two times I have faced this issue and the way to fix it has always been the same. I'm posting in case it can be of any help for somebody:
First, make sure that the DNS server is properly set up (eg. setting DNS to 1.1.1.1).
Second, restarting the docker daemon.
For those using systemd in Linux, systemctl restart did not the job for me. I had to stop and start docker to make it work. After that, I could login and pull images again.

How can I run script automatically after Docker container startup

I'm using Search Guard plugin to secure an elasticsearch cluster composed of multiple nodes.
Here is my Dockerfile:
#!/bin/sh
FROM docker.elastic.co/elasticsearch/elasticsearch:5.6.3
USER root
# Install search guard
RUN bin/elasticsearch-plugin install --batch com.floragunn:search-guard-5:5.6.3-16 \
&& chmod +x \
plugins/search-guard-5/tools/hash.sh \
plugins/search-guard-5/tools/sgadmin.sh \
bin/init_sg.sh \
&& chown -R elasticsearch:elasticsearch /usr/share/elasticsearch
USER elasticsearch
To initialize SearchGuard (create internal users and assign roles). I need to run the script init_sg.sh after the container startup.
Here is the problem: Unless elasticsearch is running, the script will not initialize any security index.
The script's content is :
sleep 10
plugins/search-guard-5/tools/sgadmin.sh -cd config/ -ts config/truststore.jks -ks config/kirk-keystore.jks -nhnv -icl
Now, I just run the script manually after the container startup but since I'm running it on Kubernetes.. Pods may get killed or fail and get recreated automatically for some reason. In this case, the plugin have to be initialized automatically after the container startup!
So how to accomplish this? Any help or hint would be really appreciated.
The image itself has an entrypoint ENTRYPOINT ["/run/entrypoint.sh"] specified in the Dockerfile. You can replace it by your own script. So for example create a new script, mount it and first call /run/entrypoint.sh and then wait for start of elasticsearch before running your init_sg.sh.
Not sure this will solves your problem, but its worth check my repo'sDockerfile
I have created a simple run.sh file copied to docker image and in the Dockerfile I wrote CMD ["run.sh"]. In the same way define whatever you want in run.sh and write CMD ["run.sh"]. You can find another example like below
Dockerfile
FROM java:8
RUN apt-get update && apt-get install stress-ng -y
ADD target/restapp.jar /restapp.jar
COPY dockerrun.sh /usr/local/bin/dockerrun.sh
RUN chmod +x /usr/local/bin/dockerrun.sh
CMD ["dockerrun.sh"]
dockerrun.sh
#!/bin/sh
java -Dserver.port=8095 -jar /restapp.jar &
hostname="hostname: `hostname`"
nohup stress-ng --vm 4 &
while true; do
sleep 1000
done
This is addressed in the documentation here: https://docs.docker.com/config/containers/multi-service_container/
If one of your processes depends on the main process, then start your helper process FIRST with a script like wait-for-it, then start the main process SECOND and remove the fg %1 line.
#!/bin/bash
# turn on bash's job control
set -m
# Start the primary process and put it in the background
./my_main_process &
# Start the helper process
./my_helper_process
# the my_helper_process might need to know how to wait on the
# primary process to start before it does its work and returns
# now we bring the primary process back into the foreground
# and leave it there
fg %1
I was trying to solve the exact problem. Here's the approach that worked for me.
Create a separate shell script that checks for ES status, and only start initialization of SG when ES is ready:
Shell Script
#!/bin/sh
echo ">>>> Right before SG initialization <<<<"
# use while loop to check if elasticsearch is running
while true
do
netstat -uplnt | grep :9300 | grep LISTEN > /dev/null
verifier=$?
if [ 0 = $verifier ]
then
echo "Running search guard plugin initialization"
/elasticsearch/plugins/search-guard-6/tools/sgadmin.sh -h 0.0.0.0 -cd plugins/search-guard-6/sgconfig -icl -key config/client.key -cert config/client.pem -cacert config/root-ca.pem -nhnv
break
else
echo "ES is not running yet"
sleep 5
fi
done
Install script in Dockerfile
You will need to install the script in container so it's accessible after it starts.
COPY sginit.sh /
RUN chmod +x /sginit.sh
Update entrypoint script
You will need to edit the entrypoint script or run script of your ES image. So that it starts the sginit.sh in the background BEFORE starting ES process.
# Run sginit in background waiting for ES to start
/sginit.sh &
This way the sginit.sh will start in the background, and will only initialize SG after ES is started.
The reason to have this sginit.sh script starts before ES in the background is so that it's not blocking ES from starting. The same logic applies if you put it after starting of ES, it will never run unless you put the starting of ES in the background.
I would suggest to put the CMD in you docker file to execute the script when the container start
FROM debian
RUN apt-get update && apt-get install -y nano && apt-get clean
EXPOSE 8484
CMD ["/bin/bash", "/opt/your_app/init.sh"]
There is other way , but before using this look at your requirement,
ENTRYPOINT "put your code here" && /bin/bash
#exemple ENTRYPOINT service nginx start && service ssh start &&/bin/bash "use && to separate your code"
You can also use wait-for-it script. It will wait on the availability of a host and TCP port. It is useful for synchronizing the spin-up of interdependent services and works like a charm with containers. It does not have any external dependencies so you can just run it as an RUN command without doing anything else.
A Dockerfile example based on this thread:
FROM elasticsearch
# Make elasticsearch write data to a folder that is not declared as a volume in elasticsearchs' official dockerfile.
RUN mkdir /data && chown -R elasticsearch:elasticsearch /data && echo 'es.path.data: /data' >> config/elasticsearch.yml && echo 'path.data: /data' >> config/elasticsearch.yml
# Download wait-for-it
ADD https://raw.githubusercontent.com/vishnubob/wait-for-it/e1f115e4ca285c3c24e847c4dd4be955e0ed51c2/wait-for-it.sh /utils/wait-for-it.sh
# Copy the files you may need and your insert script
# Insert data into elasticsearch
RUN /docker-entrypoint.sh elasticsearch -p /tmp/epid & /bin/bash /utils/wait-for-it.sh -t 0 localhost:9200 -- path/to/insert/script.sh; kill $(cat /tmp/epid) && wait $(cat /tmp/epid); exit 0;

Docker doesn't seem to be mapping ports

I'm working with Hugo
Trying to run inside a Docker container to allow people to easily manage content.
My first task is to get Hugo running and people able to view the site locally.
Here's my Dockerfile:
FROM alpine:3.3
RUN apk update && apk upgrade && \
apk add --no-cache go bash git openssh && \
mkdir -p /aws && \
apk -Uuv add groff less python py-pip && \
pip install awscli && \
apk --purge -v del py-pip && \
rm /var/cache/apk/* && \
mkdir -p /go/src /go/bin && chmod -R 777 /go
ENV GOPATH /go
ENV PATH /go/bin:$PATH
RUN go get -v github.com/spf13/hugo
RUN git clone http://mygitrepo.com /app
WORKDIR /app
EXPOSE 1313
ENTRYPOINT ["hugo","server"]
I'm checking out the site repo then running Hugo - hugo server
I'm then running this container via:
docker run -d -p 1313:1313 --name app app
Which reports everything is starting OK however when I try to browse locally on localhost:1313 I see nothing.
Any ideas where I'm going wrong?
UPDATE
docker ps gives me:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9e1f12849044 app "hugo server" 16 minutes ago Up 16 minutes 0.0.0.0:1313->1313/tcp app
And docker logs 9e1 gives me:
Started building sites ...
Built site for language en:
0 draft content
0 future content
0 expired content
25 pages created
0 non-page files copied
0 paginator pages created
0 tags created
0 categories created
total in 64 ms
Watching for changes in /ltec/{data,content,layouts,static,themes}
Serving pages from memory
Web Server is available at http://localhost:1313/ (bind address 127.0.0.1)
Press Ctrl+C to stop
I had the same problem, but following this tutorial http://ahmedalani.com/post/so-recursive-it-hurts/, says about to use the param --bind from hugo server command.
Adding that param mentioned, and the ip 0.0.0.0 we have --bind=0.0.0.0
It works to me, I think this is a natural behavior from every container taking a localhost for self scope, but if you bind with 0.0.0.0 takes a visible scope to the main host.
This is because Docker is actually running in a VM. You need to navigate to the docker-machine ip instead of localhost.
curl $(docker-machine ip):1313
Delete EXPOSE 1313 in your Dockerfile. Dockerfile reference.

Resources