I'm learning docker and here is part of the docker-compose.yml file for a project that I'm learning from:
services:
...
...
redis:
build: ./redis
image: smartjoe:redis
container_name: smartjoe--redis
command: redis-server
ports:
- '9993:6379'
volumes:
- /redis
volumes_from:
- data
data:
build: ./data
image: smartjoe:data
container_name: smartjoe--data
volumes:
- /data
- /var/lib/mysql
- /var/lib/redis/data
command: tail -f /dev/null
...
...
In the line command: redis-server under redis:, I was wondering where the redis-servercame from. After some search I was able to find that it is in a entrypoint.sh file located within the smartjoe--redis container:
/ # ls -la
total 68
drwxr-xr-x 1 root root 4096 Mar 28 21:24 .
drwxr-xr-x 1 root root 4096 Mar 28 21:24 ..
-rwxr-xr-x 1 root root 0 Mar 28 21:24 .dockerenv
drwxr-xr-x 1 root root 4096 Jun 28 2017 bin
drwxr-xr-x 4 redis redis 4096 Aug 2 19:51 data
drwxr-xr-x 5 root root 340 Aug 5 14:54 dev
lrwxrwxrwx 1 root root 34 Jun 28 2017 entrypoint.sh -> usr/local/bin/docker-entrypoint.sh
drwxr-xr-x 1 root root 4096 Mar 28 21:24 etc
drwxr-xr-x 1 root root 4096 Jun 28 2017 home
drwxr-xr-x 1 root root 4096 Jun 28 2017 lib
drwxr-xr-x 5 root root 4096 Jun 25 2017 media
drwxr-xr-x 2 root root 4096 Jun 25 2017 mnt
dr-xr-xr-x 229 root root 0 Aug 5 14:54 proc
drwxr-xr-x 2 root root 4096 Sep 7 2018 redis
drwx------ 1 root root 4096 Aug 5 18:14 root
drwxr-xr-x 2 root root 4096 Jun 25 2017 run
drwxr-xr-x 1 root root 4096 Jun 28 2017 sbin
drwxr-xr-x 2 root root 4096 Jun 25 2017 srv
dr-xr-xr-x 13 root root 0 Aug 1 16:00 sys
drwxrwxrwt 1 root root 4096 Jun 28 2017 tmp
drwxr-xr-x 1 root root 4096 Jun 28 2017 usr
drwxr-xr-x 1 root root 4096 Jun 28 2017 var
And the content of entrypoint.sh looks like:
/ # cat entrypoint.sh
#!/bin/sh
set -e
# first arg is `-f` or `--some-option`
# or first arg is `something.conf`
if [ "${1#-}" != "$1" ] || [ "${1%.conf}" != "$1" ]; then
set -- redis-server "$#"
fi
# allow the container to be started with `--user`
if [ "$1" = 'redis-server' -a "$(id -u)" = '0' ]; then
chown -R redis .
exec su-exec redis "$0" "$#"
fi
exec "$#"
My question is: I don't see entrypoint.sh being referenced anywhere in the docker-compose.yml script, does docker just assume there is a such entrypoint.sh file and search the key word after command: in the file?
btw, the Dockerfile is quite simple:
Bob#MacBook-Pro:~/smartjoe/smartjoe-local/docker/redis$ cat Dockerfile
FROM redis:3.0-alpine
MAINTAINER smartjoe Engineering "engineering#smartjoe.com"
Bob#MacBook-Pro:~/smartjoe/smartjoe-local/docker/redis$
The entrypoint would have been defined in the image, smartjoe:redis. Setting from the Dockerfile are a default that you can override in the docker-compose.yml. You can see the image configuration by inspecting the image, e.g.
docker image inspect smartjoe:redis
Related
I have a cloudbuild.yaml file that looks like this:
steps:
- name: 'gcr.io/cloud-builders/gsutil'
args: [ "-m", "rsync", "-r", "gs://${_BUCKET}/maven-repository", "/cache/.m2" ]
volumes:
- path: '/cache/.m2'
name: 'm2_cache'
- name: docker/compose:debian-1.29.2
entrypoint: bash
args:
- -c
- |
./test.sh
volumes:
- path: '/cache/.m2'
name: 'm2_cache'
timeout: 2700s
substitutions:
_BUCKET: 'my-bucket'
In the first step we download our maven settings.xml file from GCS. This file is crucial for subsequent build steps since it contain the username/password to our Artifact Registry Maven repository (I've simplified this example as we don't actually store the credential in the settings.xml as plain text). Without these credentials, our Maven build won't run. Normally the script that we call in the second step starts several docker containers and then run our maven tests. But I've replaced it with test.sh to easier show what the problem is. The test.sh file is shown below:
#!/bin/bash
echo "### [Host] Contents in /cache/.m2"
ls -la /cache/.m2
mkdir ~/test
echo "Johan" > ~/test/ikk.txt
echo "### [Host] Contents in ~/test"
ls -la ~/test
docker run --rm -v /cache/.m2:/cache/.m2 -v ~/test:/root/test -w /usr/src/somewhere ubuntu bash -c 'echo "### [Docker] Contents in /cache/.m2" && ls -la /cache/.m2 && echo "### [Docker] Contents in /root/test" && ls -la /root/test'
I.e. we try to mount two volumes to the ubuntu container that we start in the test.sh file. I list the contents in two directors both outside (### [Host]) and inside (### [Docker]) the ubuntu container. Here's the relevant output of running this in cloud build:
### [Host] Contents in /cache/.m2
total 16
drwxr-xr-x 2 root root 4096 Sep 15 08:55 .
drwxr-xr-x 3 root root 4096 Sep 15 08:55 ..
-rw-r--r-- 1 root root 8063 Sep 13 11:03 settings.xml
### [Host] Contents in ~/test
total 12
drwxr-xr-x 2 root root 4096 Sep 15 08:55 .
drwxr-xr-x 6 root root 4096 Sep 15 08:55 ..
-rw-r--r-- 1 root root 6 Sep 15 08:55 ikk.txt
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
Digest: sha256:20fa2d7bb4de7723f542be5923b06c4d704370f0390e4ae9e1c833c8785644c1
Status: Downloaded newer image for ubuntu:latest
### [Docker] Contents in /cache/.m2
total 8
drwxr-xr-x 2 root root 4096 Sep 15 08:55 .
drwxr-xr-x 3 root root 4096 Sep 15 08:55 ..
### [Docker] Contents in /root/test
total 8
drwxr-xr-x 2 root root 4096 Sep 15 08:55 .
drwx------ 1 root root 4096 Sep 15 08:55 ..
As you can see, the volume mounts doesn't seem to work when I run the ubuntu container from the test.sh file in cloud build (since the contents of /root/test and /cache/.m2 are empty).
Running the test.sh locally on my machine yields the expected outcome:
### [Host] Contents in /cache/.m2
total 40
drwxr-xr-x 7 johan staff 224 Mar 15 2022 .
drwxr-x---+ 87 johan staff 2784 Sep 15 10:58 ..
-rw-r--r-- 1 johan staff 2344 Sep 14 11:37 copy_reference_file.log
drwxr-xr-x 221 johan staff 7072 Sep 14 10:52 repository
-rw-r--r-- 1 johan staff 327 Nov 24 2021 settings-docker.xml
-rw-r--r--# 1 johan staff 9842 Mar 15 2022 settings.xml
drwxr-xr-x 3 johan staff 96 Nov 19 2021 wrapper
### [Host] Contents in ~/test
total 8
drwxr-xr-x# 3 johan staff 96 Sep 15 10:53 .
drwxr-xr-x# 135 johan staff 4320 Sep 15 10:49 ..
-rw-r--r-- 1 johan staff 6 Sep 15 10:58 ikk.txt
### [Docker] Contents in /cache/.m2
total 24
drwxr-xr-x 7 root root 224 Mar 15 2022 .
drwxr-xr-x 3 root root 4096 Sep 15 08:58 ..
-rw-r--r-- 1 root root 2344 Sep 14 09:37 copy_reference_file.log
drwxr-xr-x 221 root root 7072 Sep 14 08:52 repository
-rw-r--r-- 1 root root 327 Nov 24 2021 settings-docker.xml
-rw-r--r-- 1 root root 9842 Mar 15 2022 settings.xml
drwxr-xr-x 3 root root 96 Nov 19 2021 wrapper
### [Docker] Contents in /root/test
total 8
drwxr-xr-x 3 root root 96 Sep 15 08:53 .
drwx------ 1 root root 4096 Sep 15 08:58 ..
-rw-r--r-- 1 root root 6 Sep 15 08:58 ikk.txt
Here you can see that the volumes are mounted correctly and I can access the files inside the ubuntu container.
How can I mount volumes inside a container in cloud build?
was trying to make a quick dbt container:
docker pull xemuliam/dbt
docker run -d --name dbt \
-p 8085:8085 \
-v /home/mypath/app:/usr/app \
xemuliam/dbt:latest
that didn't work, so i tried to inspect and saw the binding was rprivate, so i changed it to:
docker run -d --name dbt \
-p 8085:8085 \
-v /home/mypath/app:/usr/app,rshared:rshared \
xemuliam/dbt:latest
the issue is that /home/mypath/app ends up being empty, and if i go inside the container it has in the working directory of /usr/app:
drwxr-xr-x 15 root root 4096 Apr 29 12:24 .
drwxr-xr-x 1 root root 4096 Jun 21 20:14 ..
-rw-rw-r-- 1 root root 4895 Apr 28 16:54 ARCHITECTURE.md
-rwxrwxr-x 1 root root 16732 Apr 28 16:54 CHANGELOG.md
-rw-rw-r-- 1 root root 18081 Apr 28 16:54 CONTRIBUTING.md
-rw-rw-r-- 1 root root 2239 Apr 28 16:54 Dockerfile.test
-rw-rw-r-- 1 root root 11344 Apr 28 16:54 License.md
-rw-rw-r-- 1 root root 3271 Apr 28 16:54 Makefile
-rw-rw-r-- 1 root root 2365 Apr 28 16:54 README.md
drwxrwxr-x 5 root root 4096 Apr 28 16:54 core
drwxrwxr-x 4 root root 4096 Apr 29 12:23 dbt-core-1.1.0
-rw-rw-r-- 1 root root 205 Apr 28 16:54 dev-requirements.txt
drwxrwxr-x 2 root root 4096 Apr 28 16:54 docker
-rw-rw-r-- 1 root root 777 Apr 28 16:54 docker-compose.yml
drwxrwxr-x 3 root root 4096 Apr 28 16:54 docs
-rw-rw-r-- 1 root root 51 Apr 28 16:54 editable-requirements.txt
drwxrwxr-x 2 root root 4096 Apr 28 16:54 etc
drwxrwxr-x 3 root root 4096 Apr 28 16:54 events
-rw-rw-r-- 1 root root 65 Apr 28 16:54 mypy.ini
drwxrwxr-x 8 root root 4096 Apr 28 16:54 performance
drwxrwxr-x 6 root root 4096 Apr 28 16:54 plugins
-rw-r--r-- 1 root root 95 Apr 29 12:23 plugins.txt
-rw-rw-r-- 1 root root 242 Apr 28 16:54 pytest.ini
-rw-rw-r-- 1 root root 26 Apr 28 16:54 requirements.txt
drwxrwxr-x 2 root root 4096 Apr 28 16:54 scripts
drwxrwxr-x 5 root root 4096 Apr 28 16:54 test
drwxrwxr-x 7 root root 4096 Apr 28 16:54 tests
drwxrwxr-x 10 root root 4096 Apr 28 16:54 third-party-stubs
-rw-rw-r-- 1 root root 798 Apr 28 16:54 tox.ini
drwxr-xr-x 2 root root 4096 Apr 29 12:24 wheels
in my postgres db i am able to do a volume like this and it works:
-v /home/mypath/data:/var/lib/postgresql/data
i would like to access that stuff on the host so i can play around with editing it.
this is probably the worst way to get this done, so i would appreciate a better answer - but my solution:
step 1: pull image
$ podman pull xemuliam/dbt
step 2: create a container through cli instead of dockerfile
$ podman run -d --name db_dbt \
-p 8085:8085 \
-v /home/myusr/dbt_test/_official:/home/dbt_test/_official \
xemuliam/dbt:latest
step 3: copy files from /usr/app to /home/dbt_test/_official
$ cp -R /usr/app/* /home/dbt_test/_official
exit
$ podman stop db_dbt && podman rm db_dbt
step 4: copy files on the host
$ mkdir /home/myusr/dbt_test/app1
$ cp -R /home/myusr/dbt_test/_official/* /home/myusr/dbt_test/app1/
step 5: create a new volume
$ podman run -d --name db_dbt \
-p 8085:8085 \
-v /home/myusr/dbt_test/app1/:/usr/app/ \
xemuliam/dbt:latest
i really dont like this roundabout way, but now i am able to edit from the host and see what the container does.
I have a docker image https://github.com/carnellj/spmia-chapter1 which does not find its CMD ./run.sh executable although it is there in the file system.
I was able to run /bin/sh in the container, and I can ls -l:
D:\Dokumente\ws\spring-microservices\spmia-chapter1 (master)
λ docker run -i -t johncarnell/tmx-simple-service:chapter1 /bin/sh
/ # ls -l
total 56
drwxr-xr-x 2 root root 4096 Mar 3 11:20 bin
drwxr-xr-x 5 root root 360 Apr 22 07:10 dev
drwxr-xr-x 1 root root 4096 Apr 22 07:10 etc
drwxr-xr-x 2 root root 4096 Mar 3 11:20 home
drwxr-xr-x 1 root root 4096 Apr 22 06:01 lib
drwxr-xr-x 5 root root 4096 Mar 3 11:20 media
drwxr-xr-x 2 root root 4096 Mar 3 11:20 mnt
dr-xr-xr-x 123 root root 0 Apr 22 07:10 proc
drwx------ 1 root root 4096 Apr 22 07:10 root
drwxr-xr-x 2 root root 4096 Mar 3 11:20 run
-rwxr-xr-x 1 root root 245 Apr 22 06:50 run.sh
drwxr-xr-x 2 root root 4096 Mar 3 11:20 sbin
drwxr-xr-x 2 root root 4096 Mar 3 11:20 srv
dr-xr-xr-x 13 root root 0 Apr 22 07:10 sys
drwxrwxrwt 2 root root 4096 Mar 3 11:20 tmp
drwxr-xr-x 1 root root 4096 Mar 7 01:04 usr
drwxr-xr-x 1 root root 4096 Mar 7 01:04 var
/ # ./run.sh
/bin/sh: ./run.sh: not found
/ # ls run.sh
run.sh
/bin/sh does not find ./run.sh although it is there in the file system, as proven by ls run.sh. Also, cat shows the content of run.sh:
/ # cat run.sh
#!/bin/sh
echo "********************************************************"
echo "Starting simple-service "
echo "********************************************************"
java -jar /usr/local/simple-service/simple-service-0.0.1-SNAPSHOT.jar
When I run vi from sh and copy the content of run.sh into a new file myrun.sh and make myrun.sh executable, I can execute ./myrun.sh and the spring service starts.
What is going on here? Why would sh not see an executable which is there in the filesystem? Executables from PATH or executables which I add manually run fine.
I am running Docker on Windows 10.
OK the reason is, run.sh is created with Windows line endings in the docker image if you check out with automatic lf->crlf conversion. One possible solution is to tell git not to convert line endings.
I'm using a .dockerignore file that looks like this:
*
*/node_modules
*/bower_components
!www
!app
!inc
so here what I'm saying is: ignore all files in the current directory (this post is related to: How to include local libraries in build?) except for www, app and inc and ignore all directories named node_modules and bower_components in those subdirectories
when I run a build:
build --no-cache -t test -f app/Dockerfile .
(here's my Dockerfile):
FROM node:latest
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
ADD . /usr/src/app/
and then run a container:
run -i --entrypoint=/bin/bash test
if I look in the directory I see node_modules folders!
# ls -alF
total 20
drwxr-xr-x 5 root root 4096 Mar 19 03:39 ./
drwxr-xr-x 4 root root 4096 Mar 19 03:39 ../
drwxr-xr-x 4 root root 4096 Mar 18 03:26 inc/
drwxr-xr-x 3 root root 4096 Mar 19 00:57 app/
drwxr-xr-x 12 root root 4096 Mar 18 08:55 www/
# ls -alF app
total 36
drwxr-xr-x 3 root root 4096 Mar 19 00:57 ./
drwxr-xr-x 5 root root 4096 Mar 19 03:39 ../
-rw-r--r-- 1 root root 38 Mar 19 00:57 .dockerignore
-rw-r--r-- 1 root root 359 Mar 19 03:37 Dockerfile
-rw-r--r-- 1 root root 700 Mar 18 04:50 config.json
-rw-r--r-- 1 root root 3786 Mar 18 03:43 index.js
drwxr-xr-x 119 root root 4096 Mar 17 18:57 node_modules/
-rw-r--r-- 1 root root 2910 Mar 17 20:51 npm-debug.log
-rw-r--r-- 1 root root 348 Mar 19 03:24 package.json
so the first line of the .dockerignore worked fine, as did the 4th, 5th and 6th lines, but the subdirectories failed. for the sake of completeness I originally used **/node_modules but that failed too.
what is the correct way of expressing this?
ok, apparently the problem is that when I create exceptions to the ignore rules, I have to reapply the exceptions, so if I do this instead:
*
!www
!app
!inc
*/node_modules
*/bower_components
it works because */node_modules applies to "all of the above", which includes app. in my original declaration I was excluding the directories I do want to include from the rules
I have a docker image built from ubuntu base image with few softwares installed.
i have a startup script, as below
#!/bin/bash
/usr/local/sbin/process1 -d
/usr/local/sbin/process2 -d
/bin/bash
Now I use docker-py python library to start multiple of these containers from a python file.
c = docker.Client(base_url='unix://var/run/docker.sock',
version='1.12',
timeout=10)
container = c.create_container("p12", command="/startup.sh", hostname=None, user=None,
detach=False, stdin_open=False, tty=False, mem_limit=0,
ports=None, environment=None, dns=None, volumes=None,
volumes_from=None, network_disabled=False, name=None,
entrypoint=None, cpu_shares=None, working_dir=None,
memswap_limit=0)
c.start(container, binds=None, port_bindings=None, lxc_conf=None,
publish_all_ports=False, links=None, privileged=False,
dns=None, dns_search=None, volumes_from=None, network_mode=None,
restart_policy=None, cap_add=None, cap_drop=None)
This worked fine and I can start multiple (say 3) when I tested this on a Ubuntu Desktop, Ubuntu 14.04.1 LTS and with docker-py version of 1.10. It will start the dockers and I can do a docker attach later and work on the terminal.
Now i moved my testing environment to a Ubuntu Server edition with Ubuntu 14.04.1 LTS and with docker-py version of 1.12.
The issue i see is that, when I use the same script and try to start 3 dockers, after starting process1 and process 2 as background processes, all the dockers simply exit. It appears as if /bin/bash doesnt execute at all.
If i execute the same docker image as "docker run -t -i p14 /startup.sh --> then everything is fine again. The docker is started appropriately and i get the terminal access.
The only issue is when i execute this python library.
anybody has any similar issues...any idea on how to debug this problem...or any pointers for the fix ?
Thanks,
Kiran
The difference is you're in tty (-t) mode with an open stdin (-i) when you run your docker image with docker run -t -i p14 /startup.sh, whereas you set both stdin_open=False and tty=False in your docker-py configuration.
Because your docker container has no tty and can't take any input from stdin, your call to /bin/bash has nothing to do so exits with code 0.
Try it yourself:
An open stdin with a tty
$ docker run -t -i ubuntu:14.04 /bin/bash
root#1e7eda2bba03:/# ls -la
total 7184
drwxr-xr-x 21 root root 4096 Sep 19 21:30 .
drwxr-xr-x 21 root root 4096 Sep 19 21:30 ..
-rwxr-xr-x 1 root root 0 Sep 19 21:30 .dockerenv
-rwx------ 1 root root 7279686 Jul 21 10:50 .dockerinit
drwxr-xr-x 2 root root 4096 Sep 3 03:33 bin
drwxr-xr-x 2 root root 4096 Apr 10 22:12 boot
drwxr-xr-x 4 root root 360 Sep 19 21:30 dev
drwxr-xr-x 61 root root 4096 Sep 19 21:30 etc
drwxr-xr-x 2 root root 4096 Apr 10 22:12 home
drwxr-xr-x 12 root root 4096 Sep 3 03:33 lib
drwxr-xr-x 2 root root 4096 Sep 3 03:33 lib64
drwxr-xr-x 2 root root 4096 Sep 3 03:33 media
drwxr-xr-x 2 root root 4096 Apr 10 22:12 mnt
drwxr-xr-x 2 root root 4096 Sep 3 03:33 opt
dr-xr-xr-x 240 root root 0 Sep 19 21:30 proc
drwx------ 2 root root 4096 Sep 3 03:33 root
drwxr-xr-x 7 root root 4096 Sep 3 03:33 run
drwxr-xr-x 2 root root 4096 Sep 4 18:41 sbin
drwxr-xr-x 2 root root 4096 Sep 3 03:33 srv
dr-xr-xr-x 13 root root 0 Sep 19 18:44 sys
drwxrwxrwt 2 root root 4096 Sep 4 18:41 tmp
drwxr-xr-x 10 root root 4096 Sep 3 03:33 usr
drwxr-xr-x 11 root root 4096 Sep 3 03:33 var
root#1e7eda2bba03:/#
An open stdin with no tty (i.e., no prompt, but you can still send commands via stdin)
$ docker run -i ubuntu:14.04 /bin/bash
ls -la
total 7184
drwxr-xr-x 21 root root 4096 Sep 19 21:32 .
drwxr-xr-x 21 root root 4096 Sep 19 21:32 ..
-rwxr-xr-x 1 root root 0 Sep 19 21:32 .dockerenv
-rwx------ 1 root root 7279686 Jul 21 10:50 .dockerinit
drwxr-xr-x 2 root root 4096 Sep 3 03:33 bin
drwxr-xr-x 2 root root 4096 Apr 10 22:12 boot
drwxr-xr-x 4 root root 340 Sep 19 21:32 dev
drwxr-xr-x 61 root root 4096 Sep 19 21:32 etc
drwxr-xr-x 2 root root 4096 Apr 10 22:12 home
drwxr-xr-x 12 root root 4096 Sep 3 03:33 lib
drwxr-xr-x 2 root root 4096 Sep 3 03:33 lib64
drwxr-xr-x 2 root root 4096 Sep 3 03:33 media
drwxr-xr-x 2 root root 4096 Apr 10 22:12 mnt
drwxr-xr-x 2 root root 4096 Sep 3 03:33 opt
dr-xr-xr-x 243 root root 0 Sep 19 21:32 proc
drwx------ 2 root root 4096 Sep 3 03:33 root
drwxr-xr-x 7 root root 4096 Sep 3 03:33 run
drwxr-xr-x 2 root root 4096 Sep 4 18:41 sbin
drwxr-xr-x 2 root root 4096 Sep 3 03:33 srv
dr-xr-xr-x 13 root root 0 Sep 19 18:44 sys
drwxrwxrwt 2 root root 4096 Sep 4 18:41 tmp
drwxr-xr-x 10 root root 4096 Sep 3 03:33 usr
drwxr-xr-x 11 root root 4096 Sep 3 03:33 var
A closed stdin with a tty (you can see the prompt but you can't enter any commands)
$ docker run -t ubuntu:14.04 /bin/bash
root#95904c21e5a5:/# ls -la
hello
this does nothing :(
A closed stdin with no tty - /bin/bash has nothing to do
$ docker run ubuntu:14.04 /bin/bash
$