I am having a hard time trying to build a docker image for some python project, but at some point a RUN command resulting content is not available in the next RUN.
I mean I have a RUN command that generate a directory with some content and that directory is not available for the next RUN command.
Here is the Dockerfile snippet :
RUN poetry config virtualenvs.create true && \
poetry config virtualenvs.in-project true && \
poetry install -n --no-root --no-dev && \
ls -la
RUN ls -la
The first ls -la shows a .venv directory (see below) while it is not present anymore in the second one.
And here is the build output:
Step 9/14 : RUN poetry config virtualenvs.create true && poetry config virtualenvs.in-project true && poetry install -n --no-root --no-dev && ls -la
---> Running in 3a93a4994133
...
drwxrwxr-x 4 root odoo 4096 Sep 29 20:08 .
drwxr-xr-x 1 root root 4096 Sep 16 02:35 ..
drwxr-xr-x 4 odoo odoo 4096 Sep 29 20:08 .venv
-rw-rw-r-- 1 odoo odoo 15824 Sep 2 23:18 poetry.lock
-rw-rw-r-- 1 odoo odoo 2057 Sep 2 23:50 pyproject.toml
drwxr-xr-x 3 odoo odoo 4096 Sep 29 20:08 src
Removing intermediate container 3a93a4994133
---> 694c4e54863c
Step 10/14 : RUN ls -la
---> Running in 5d0ebcf8f1e9
total 32
drwxrwxr-x 3 root odoo 4096 Sep 29 20:08 .
drwxr-xr-x 1 root root 4096 Sep 16 02:35 ..
-rw-rw-r-- 1 odoo odoo 15824 Sep 2 23:18 poetry.lock
-rw-rw-r-- 1 odoo odoo 2057 Sep 2 23:50 pyproject.toml
drwxr-xr-x 3 odoo odoo 4096 Sep 29 20:08 src
Removing intermediate container 5d0ebcf8f1e9
---> 750255eac6bb
If someone can provide any lead, it would be really appreciated.
Thanks
Related
This is my Dockerfile:
FROM python:3.10.5-alpine
ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1
RUN adduser -D appuser
USER appuser
WORKDIR /home/appuser/
COPY requirements.txt .
RUN python -m pip install --user --no-cache-dir --disable-pip-version-check --requirement requirements.txt
COPY . .
ENTRYPOINT [ "./entrypoint.sh" ]
So I create a user called appuser and switch to it as soon as I can before copying anything (I've checked both user and its home folder is created).
But when I browse the filesystem of my image:
~ $ ls -l
total 156
-rwxr-xr-x 1 root root 335 Jul 28 10:57 Dockerfile
-rw-r--r-- 1 appuser appuser 131072 Jul 28 12:28 db.sqlite3
-rwxr-xr-x 1 root root 150 Jul 28 11:37 entrypoint.sh
-rwxr-xr-x 1 root root 685 Jul 28 10:04 manage.py
drwxr-xr-x 2 root root 4096 Jul 28 10:56 project
-rwxr-xr-x 1 root root 41 Jul 28 11:56 requirements.txt
drwxr-xr-x 2 root root 4096 Jul 28 11:50 static
drwxr-xr-x 5 root root 4096 Jul 28 10:05 venv
... almost everything belongs to root user and this gives me several permission denied errors.
What is my mistake because I assume Docker shouldn't operate under root when I've switched the user?
I know I can add RUN mkdir ~/static to the Dockerfile and get over it, but then what the documentation says about USER command doesn't make sense to me:
The USER instruction sets the user name (or UID) and optionally the user group (or GID) to use as the default user and group for the remainder of the current stage.
Use the optional flag --chown=<user>:<group> with either the ADD or COPY commands.
For example:
COPY --chown=appuser:appuser . .
docker docs
I am making a Docker image. I would like to have a ready-made environment in there as well as some ready-made directories. In this way, I only need to mount some of my directories and use them directly. I made the image using the Dockerfile below. In order to have the same permissions inside and outside the container (not root), I created a user user.
FROM matthewfeickert/docker-python3-ubuntu:latest
USER root
# Create an arbitrary non-root user; we don't care about its uid
# or other properties
RUN useradd --system user
RUN sudo pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
RUN set -x; \
sudo apt-get update \
&& DEBIAN_FRONTEND=noninteractive sudo apt-get install -y build-essential git-core m4 zlib1g zlib1g-dev libprotobuf-dev protobuf-compiler libprotoc-dev libgoogle-perftools-dev swig xz-utils gdb git \
&& sudo -H python3 -m pip install scons==3.0.1 \
&& sudo -H python3 -m pip install six
# RUN apt-get -y install gdb
RUN apt-get clean
RUN git config --global url."https://hub.fastgit.xyz/".insteadOf "https://github.com/"
WORKDIR /usr/local/src
RUN git clone https://github.com/gem5/gem5.git
RUN sudo chown user /usr/local/src/gem5 -R
USER user
# RUN mkdir -p /usr/local/src/gem5/build
# RUN sudo chown user /usr/local/src/gem5/build
WORKDIR /usr/local/src/gem5/
After making the image, I mount my directory into it.
docker run -it --rm \
-v my_dir/runScripts:/usr/local/src/gem5/runScripts \
-v my_dir/gem5/src:/usr/local/src/gem5/src \
-v my_dir/gem5/configs:/usr/local/src/gem5/configs \
-v my_dir/gem5/programs:/usr/local/src/gem5/programs
-v my_dir/gem5/build:/usr/local/src/gem5/build \
-v my_dir/gem5/results:/usr/local/src/gem5/results \
-v my_dir/gem5/update.sh:/usr/local/src/gem5/update.sh \
--security-opt seccomp=unconfined --user 1000:1000 gerrie/gem5:v1 "/bin/bash"
When I enter the docker container, I output the UID at this time.
$ echo $UID
1000
This is the same as outside the container.
What I think is that the inside and outside of the gem5 directory should be exactly the same user. But it's not.
$ ll
total 232
drwxr-xr-x 1 user root 4096 Jun 16 09:12 ./
drwxr-xr-x 1 root root 4096 Jun 16 03:16 ../
drwxr-xr-x 1 user root 4096 Jun 16 03:17 .git/
-rw-r--r-- 1 user root 984 Jun 16 03:17 .git-blame-ignore-revs
-rw-r--r-- 1 user root 645 Jun 16 03:17 .gitignore
-rw-r--r-- 1 user root 19339 Jun 16 03:17 .mailmap
-rw-r--r-- 1 user root 5595 Jun 16 03:17 CODE-OF-CONDUCT.md
-rw-r--r-- 1 user root 26112 Jun 16 03:17 CONTRIBUTING.md
-rw-r--r-- 1 user root 2332 Jun 16 03:17 COPYING
-rw-r--r-- 1 user root 1478 Jun 16 03:17 LICENSE
-rw-r--r-- 1 user root 7790 Jun 16 03:17 MAINTAINERS.yaml
-rw-r--r-- 1 user root 2133 Jun 16 03:17 README
-rw-r--r-- 1 user root 34435 Jun 16 03:17 RELEASE-NOTES.md
-rwxr-xr-x 1 user root 28876 Jun 16 03:17 SConstruct*
-rw-r--r-- 1 user root 8616 Jun 16 03:17 TESTING.md
drwxrwxr-x 2 docker docker 4096 Jun 16 08:52 build/
drwxr-xr-x 1 user root 4096 Jun 16 03:17 build_opts/
drwxr-xr-x 1 user root 4096 Jun 16 03:17 build_tools/
drwxrwxr-x 13 docker docker 4096 Jun 16 08:54 configs/
drwxr-xr-x 1 user root 4096 Jun 16 03:17 ext/
drwxr-xr-x 1 user root 4096 Jun 16 03:17 include/
drwxrwxr-x 2 docker docker 4096 Jun 16 09:03 programs/
-rw-rw-r-- 1 docker docker 0 Jun 16 08:58 results
drwxrwxr-x 2 docker docker 4096 Jun 16 02:33 runScripts/
drwxr-xr-x 1 user root 4096 Jun 16 03:17 site_scons/
drwxrwxr-x 17 docker docker 4096 Jun 16 02:33 src/
drwxr-xr-x 1 user root 4096 Jun 16 03:17 system/
drwxr-xr-x 1 user root 4096 Jun 16 03:17 tests/
-rw-rw-r-- 1 docker docker 0 Jun 16 08:58 update.sh
drwxr-xr-x 1 user root 4096 Jun 16 03:17 util/
All the directories I mount belong to the docker user, and all other directories are user.
I am able to create files inside my mounted directory. But for gem5's directory, I don't even have permission to create files.
But according to the Dockfile, I have clearly chown this directory to user. And when entering the container, I set the uid.
docker#7df3004beb2a:/usr/local/src/gem5$ touch test
touch: cannot touch 'test': Permission denied
docker#7df3004beb2a:/usr/local/src/gem5$ cd runScripts/
docker#7df3004beb2a:/usr/local/src/gem5/runScripts$ touch test
docker#7df3004beb2a:/usr/local/src/gem5/runScripts$ ll
total 8
drwxrwxr-x 2 docker docker 4096 Jun 16 09:13 ./
drwxr-xr-x 1 user root 4096 Jun 16 09:12 ../
-rw-r--r-- 1 docker docker 0 Jun 16 09:13 test
When I compile, this problem occurs. I think this is caused by a permissions issue. Where did I go wrong? How should I modify it? Thanks a lot!
FileNotFoundError: [Errno 2] No such file or directory: "/usr/local/src/gem5/fatal: unsafe repository ('/usr/local/src/gem5' is owned by someone else)\nTo add an exception for this directory, call:\n\n\tgit config --global --add safe.directory /usr/local/src/gem5/hooks":
As the question was tagged with podman (in addition to docker), here is a Podman solution to the problem of mapping users between the host and the container:
If you want to map your regular user on the host to a user with the same UID inside the container, you could add the Podman option --userns=keep-id. A more general solution (that also works when the UIDs are not the same) can be found in the troubleshooting.md tip and tip. The tips make use of the options --uidmap and --gidmap. (I wrote those tips).
The two options --uidmap and --gidmap may look to be a bit complicated to use, but as soon as you understand how rootless Podman maps UIDs and GIDs it will be pretty straight forward.
In a Dockerfile, I have existing folders from my derived image /home and /root. I cannot create a folder in the /home directory:
RUN mkdir -p /home/TEST_3
No error messages are display but I can see that the folder was not created.
The same command is working fine in the /root directory
Bother folder seems to be similar in term of permission:
drwxr-xr-x 2 root root 4096 Apr 24 2018 home
drwx------ 1 root root 4096 Jun 20 08:38 root
The hard link count is different. The problematic folder /homehas more permission
Another strange thing is that I can create a folder in /home using:
WORKDIR /home/TEST_2
but then any file that I create in this folder doesn't seems to exist.
Here a small Dockerfile to recreate this issue/feature
# Specify the base image and tag
FROM gcr.io/deeplearning-platform-release/pytorch-cpu:m27
RUN ls -la
RUN ls -la /home/
RUN ls -la /root/
## creation ofthe folder is working and the folder exist (and I can create a file in the new folder)
RUN mkdir -p /root/TEST_1
RUN ls -la /root/TEST_1
RUN touch /root/TEST_1/new_file.txt
RUN ls -la /root/TEST_1
## creation of the folder is working and the folder exist but the file created doesn't exist
WORKDIR /home/TEST_2
WORKDIR /
RUN ls -la /home/TEST_2
RUN touch /home/TEST_2/new_file.txt
RUN ls -la /home/TEST_2
## creation of the folder succeed (no error messages), but the folder doesn't exist
RUN mkdir -p /home/TEST_3
RUN ls -la /home/TEST_3
RUN touch /home/TEST_3/new_file.txt
RUN ls -la /home/TEST_3
and here the output:
docker build -f Dockerfile_test .
Sending build context to Docker daemon 37.89kB
Step 1/17 : FROM gcr.io/deeplearning-platform-release/pytorch-cpu:m27
---> 7eb6565a8cc6
Step 2/17 : RUN ls -la
---> Running in 1ced0ed2bf49
total 80
drwxr-xr-x 1 root root 4096 Oct 26 12:52 .
drwxr-xr-x 1 root root 4096 Oct 26 12:52 ..
-rwxr-xr-x 1 root root 0 Oct 26 12:52 .dockerenv
drwxr-xr-x 1 root root 4096 Jun 20 08:20 bin
drwxr-xr-x 2 root root 4096 Apr 24 2018 boot
drwxr-xr-x 5 root root 340 Oct 26 12:52 dev
-rwxr-x--x 1 root root 94 Jun 19 21:51 entrypoint.sh
-rwxr-xr-x 1 root root 0 Jun 20 08:38 env.sh
drwxr-xr-x 1 root root 4096 Oct 26 12:52 etc
drwxr-xr-x 2 root root 4096 Apr 24 2018 home
drwxr-xr-x 1 root root 4096 Jun 20 08:22 lib
drwxr-xr-x 2 root root 4096 Jun 12 16:55 lib64
drwxr-xr-x 2 root root 4096 Jun 12 16:54 media
drwxr-xr-x 2 root root 4096 Jun 12 16:54 mnt
drwxr-xr-x 1 root root 4096 Jun 20 08:23 opt
dr-xr-xr-x 226 root root 0 Oct 26 12:52 proc
drwx------ 1 root root 4096 Jun 20 08:38 root
drwxr-xr-x 1 root root 4096 Jun 20 08:33 run
-rwxr-x--x 1 root root 109 Jun 19 21:51 run_jupyter.sh
drwxr-xr-x 1 root root 4096 Jun 20 08:20 sbin
drwxr-xr-x 2 root root 4096 Jun 12 16:54 srv
dr-xr-xr-x 13 root root 0 Oct 26 12:52 sys
drwxrwxrwt 1 root root 4096 Jun 20 10:09 tmp
drwxr-xr-x 1 root root 4096 Jun 20 08:33 usr
drwxr-xr-x 1 root root 4096 Jun 12 16:55 var
Removing intermediate container 1ced0ed2bf49
---> 87dbdffa7315
Step 3/17 : RUN ls -la /home/
---> Running in 756f013caf7f
total 8
drwxr-xr-x 2 root root 4096 Apr 24 2018 .
drwxr-xr-x 1 root root 4096 Oct 26 12:52 ..
Removing intermediate container 756f013caf7f
---> 24419a25ac99
Step 4/17 : RUN ls -la /root/
---> Running in bde249ae0452
total 52
drwx------ 1 root root 4096 Jun 20 08:38 .
drwxr-xr-x 1 root root 4096 Oct 26 12:52 ..
-rw-r--r-- 1 root root 3106 Apr 9 2018 .bashrc
drwxr-xr-x 3 root root 4096 Jun 20 08:31 .cache
drwxrwsr-x 1 root root 4096 Jun 20 08:26 .conda
drwx------ 4 root root 4096 Jun 20 08:33 .config
-rw-r--r-- 1 root root 401 Jun 20 08:33 .gitconfig
drwxr-xr-x 3 root root 4096 Jun 20 08:38 .ipython
drwxr-xr-x 1 root root 4096 Jun 20 10:10 .jupyter
drwxr-xr-x 3 root root 4096 Jun 20 08:31 .npm
-rw-r--r-- 1 root root 148 Aug 17 2015 .profile
-rw-r--r-- 1 root root 405 Jun 20 10:10 .wget-hsts
drwxrwxrwx 1 root root 4096 Jun 20 10:09 miniconda3
Removing intermediate container bde249ae0452
---> f6e6b3780145
Step 5/17 : RUN mkdir -p /root/TEST_1
---> Running in 95ffe0bc574f
Removing intermediate container 95ffe0bc574f
---> a87cac09ec17
Step 6/17 : RUN ls -la /root/TEST_1
---> Running in b8fd10afa3ee
total 8
drwxr-xr-x 2 root root 4096 Oct 26 12:52 .
drwx------ 1 root root 4096 Oct 26 12:52 ..
Removing intermediate container b8fd10afa3ee
---> 87b173385d4b
Step 7/17 : RUN touch /root/TEST_1/new_file.txt
---> Running in cbdd9ac79955
Removing intermediate container cbdd9ac79955
---> 0f57f2c15855
Step 8/17 : RUN ls -la /root/TEST_1
---> Running in 0a7d46a44e9e
total 8
drwxr-xr-x 1 root root 4096 Oct 26 12:53 .
drwx------ 1 root root 4096 Oct 26 12:52 ..
-rw-r--r-- 1 root root 0 Oct 26 12:53 new_file.txt
Removing intermediate container 0a7d46a44e9e
---> 5413892c3b96
Step 9/17 : WORKDIR /home/TEST_2
---> Running in d0322094ae3b
Removing intermediate container d0322094ae3b
---> 9815f6870830
Step 10/17 : WORKDIR /
---> Running in f84feebf3db9
Removing intermediate container f84feebf3db9
---> 5c85f2fadd04
Step 11/17 : RUN ls -la /home/TEST_2
---> Running in 4644cbcd5c3c
total 8
drwxr-xr-x 2 root root 4096 Oct 26 12:53 .
drwxr-xr-x 3 root root 4096 Oct 26 12:53 ..
Removing intermediate container 4644cbcd5c3c
---> c82625e638d9
Step 12/17 : RUN touch /home/TEST_2/new_file.txt
---> Running in ac89ce336a51
Removing intermediate container ac89ce336a51
---> d9569d5ca42d
Step 13/17 : RUN ls -la /home/TEST_2
---> Running in 999f75b4ece1
total 8
drwxr-xr-x 2 root root 4096 Oct 26 12:53 .
drwxr-xr-x 3 root root 4096 Oct 26 12:53 ..
Removing intermediate container 999f75b4ece1
---> 57062c0ccf0a
Step 14/17 : RUN mkdir -p /home/TEST_3
---> Running in f9788ac58dc7
Removing intermediate container f9788ac58dc7
---> e88a503ae524
Step 15/17 : RUN ls -la /home/TEST_3
---> Running in 16ea32c637c6
ls: cannot access '/home/TEST_3': No such file or directory
The command '/bin/sh -c ls -la /home/TEST_3' returned a non-zero code: 2
I am using Docker version 19.03.4, build 9013bf5 on MacOS 10.14.6
My questions are the following:
where is comming the difference (be able to create a folder in/root but not in /home
how to fix this issue to be able to create a folder in /home and
be able to create files as well in the new folder.
TL;DR
what is the difference between /home and /root
/home has a volume associated with it
how to get around the issue
create the file on entry-point (after the container is built), OR create the file inside the volume (I use the first option in my long answer)
Long answer:
Running a docker inspect containerId after building the Image & container show theirs a volumes assocated with the home directory:
"Source": "/var/lib/docker/volumes/1447dbc568742221a757a650427f38fb485e42fc7c6e959f33966922fe862d13/_data",
"Destination": "/home",
The volume is RW (read write), so you can write to the file once inside the container. (I tested using docker exec -it containerId /bin/sh then touch /home/TEST_2/new_file.txt)
Try using an Entrypoint script to create the folders/files.
Edit:
I wanted to see for myself, but ran into a couple hurdles. The parent container (gcr.io/deeplearning-platform-release/pytorch-cpu:m27) already has an entrypoint.sh file.
Also, adding a CMD or ENTRYPOINT command will override the parents entrypoint container (which you probably want).
Working Solution:
From the host, COPY the entrypoint.sh file (Assuming you want to modify it on your host). docker cp containerId:/entrypoint.sh .
Modify the entrypoint, adding your TOUCH commands:
#!/bin/bash
touch /home/TEST_2/new_file.txt # <-- I added this
. /env.sh
. /root/miniconda3/etc/profile.d/conda.sh
conda activate base
exec "$#"
Change your dockerfile to copy the new entrypoint.sh, which will replace the existing one:
COPY entrypoint.sh /entrypoint.sh
Your listing commands on startup will be the same, however if you go into the container you will see your file:
# ls home/TEST_2
new_file.txt
#
I am trying to run a Symfony 3 "base" application (meaning non complexity at all and just a few bundles installed but not even enabled) in a "LAMP" stack using Docker and Docker Compose (I've removed MySQL from the post because it's not relevant). This is my docker-compose.yml file:
version: '2'
services:
php-fpm:
build: docker/php-fpm
ports:
- "80:80"
volumes:
- ./sources:/data/www
- ./data/logs/symfony:/data/www/var/logs
db:
image: mysql
environment:
MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
MYSQL_DATABASE: ${MYSQL_DATABASE}
MYSQL_USER: ${MYSQL_USER}
MYSQL_PASSWORD: ${MYSQL_PASSWORD}
volumes:
- sql-data:/var/lib/mysql
And this is the Dockerfile for the php-fpm container:
FROM reynierpm/docker-centos7-supervisord:latest
ENV TERM=xterm \
PATH="/root/.composer/vendor/bin:${PATH}" \
COMPOSER_ALLOW_SUPERUSER=1
RUN yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm \
https://rpms.remirepo.net/enterprise/remi-release-7.rpm
RUN yum install -y \
yum-utils \
git \
zip \
unzip \
nano \
httpd \
php71-php-fpm \
php71-php-cli \
php71-php-common \
php71-php-gd \
php71-php-intl \
php71-php-json \
php71-php-mbstring \
php71-php-mcrypt \
php71-php-mysqlnd \
php71-php-pdo \
php71-php-pear \
php71-php-xml \
php71-pecl-apcu \
php71-php-pecl-apfd \
php71-php-pecl-memcache \
php71-php-pecl-memcached \
php71-php-pecl-mongodb \
php71-php-pecl-redis \
php71-php-pecl-request \
php71-php-pecl-uploadprogress \
php71-php-pecl-xattr \
php71-php-pecl-zip && \
yum clean all && rm -rf /tmp/yum*
RUN rm -f /etc/httpd/conf/httpd.conf /etc/httpd/conf.d/* /etc/httpd/conf.modules.d/* && \
ln -sfF /opt/remi/php71/enable /etc/profile.d/php71-paths.sh && \
ln -sfF /opt/remi/php71/root/usr/bin/{pear,pecl,phar,php,php-cgi,phpize} /usr/local/bin/. && \
mv -f /etc/opt/remi/php71/php.ini /etc/php.ini && \
ln -s /etc/php.ini /etc/opt/remi/php71/php.ini && \
rm -rf /etc/php.d && \
mv /etc/opt/remi/php71/php.d /etc/. && \
ln -s /etc/php.d /etc/opt/remi/php71/php.d
RUN curl -LsS https://symfony.com/installer -o /usr/local/bin/symfony && \
chmod a+x /usr/local/bin/symfony
COPY container-files /
RUN curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer && \
composer global install --no-dev
RUN yum install -y php71-php-pecl-xdebug && \
yum clean all && rm -rf /tmp/yum* && \
php --version
RUN chmod +x /config/bootstrap.sh
RUN echo 'alias sf="php bin/console"' >> ~/.bashrc
WORKDIR /data/www
EXPOSE 80 9001
The thing is I am getting this error all the time if I try to access the dev environment using http://symfonyapp.local/app_dev.php:
php-fpm | [Sat Jan 14 15:09:27.655609 2017] [proxy_fcgi:error] [pid 13:tid 140600250660608] [client 172.18.0.1:43960] AH01071: Got error 'Unable to open primary script: /data/www/web/_wdt/210673 (No such file or directory)\n', referer: http://symfonyapp.local/app_dev.php
Having the error above I can think in:
Ownership|permissions issue at /data/www/web in the container which is odd since that folder is owned by root and well .... is root I don't need to explain
Something is failing with Symfony3 and I'm not aware of it also I couldn't find it so far
Apache|PHP-FPM can't write on such folder which leads to the first item on this list
Apache config is blocking the directory /web to be written. (mod_scurity is not running so I can't blame it)
This is what I've tried so far without success because I am still getting the same error all the time.
Change ownership/permissions in the container (this lead into an error in Linux since permissions in volumes are changed in host as well, the same doesn't happen in Windows). Below is an explanation in how do I achieve such thing.
Brief explanation in how do I change ownership/permissions:
The php-fpm Dockerfile is inherit from docker-centos7-supervisord which has this script as ENTRYPOINT. So I have created a file /container-files/config/init/20-permissions.sh with the following content:
#!/usr/bin/env bash
chown -R apache:root /data/www && \
find /data/www -type d -print0 | xargs -0 chmod 775 && \
find /data/www -type f -print0 | xargs -0 chmod 664
echo "Set up permissions finished"
exec "$#"
The file above gets executed after container starts and volumes are mounted. I did know the file is executed because I am seeing Set up permissions finished in the php-fpm container logs. It's weird though because checking ownership/permissions after show me the following:
> docker exec -it php-fpm ls -la /data/www/web
total 57
drwxr-xr-x 2 root root 4096 Jan 14 03:45 .
drwxr-xr-x 2 root root 4096 Jan 14 00:40 ..
-rwxr-xr-x 1 root root 3319 Jan 13 23:54 .htaccess
-rwxr-xr-x 1 root root 635 Jan 14 03:45 app.php
-rwxr-xr-x 1 root root 1184 Jan 14 03:45 app_dev.php
-rwxr-xr-x 1 root root 2092 Jan 13 23:54 apple-touch-icon.png
drwxr-xr-x 2 root root 0 Dec 13 13:36 bundles
-rwxr-xr-x 1 root root 21244 Jan 14 00:04 config.php
-rwxr-xr-x 1 root root 6518 Jan 13 23:54 favicon.ico
-rwxr-xr-x 1 root root 116 Jan 13 23:54 robots.txt
So in this case I am not sure if this is going well or if it's possible even. I have created a repository with two branches with all the necessary to gives this a try: master having httpd and php-fpm both in one container and httpd having them in separated containers. Although the result in both is the same.
To get everything up and running you should:
Run docker-compose up -d --build --force-recreate (the --force-recreate and --build are not necessary but is just in case)
Run docker exec -it php-fpm composer update so you download the libraries needed by the project.
Add symfonyapp.local to your hosts files
Currently I am using Docker in Windows, this is the info about it:
Version: 1.13.0-rc6-beta36 (9696)
Channel: Beta
Sha1: 64a715b54327a0ec8f28076d1a343f4c811856fb
Started on: 2017/01/13 18:34:34.519
Resources: C:\Program Files\Docker\Docker\Resources
OS: Windows 10 Pro
Edition: Professional
Id: 1607
Build: 14393
BuildLabName: 14393.693.amd64fre.rs1_release.161220-1747
But I have tested this in Linux as well and I have the same behavior meaning the error still there.
What is happening here? Can you give me some ideas or solution? At this point I am out of them and don't know what else to do.
TL;DR: The permission problem is being introduced during composer update. Possibly during one of the scripts (a list of which can be found in composer.json).
I started from scratch on a VM with your repository and followed your startup instructions.
git clone https://github.com/reypm/symfony3app
cd symfony3app
docker-compose up -d --build --force-recreate
At this point, the chown from 20-permissions.sh should have been run. To verify that, I looked inside the container. The error I have seen related to /data/www/var/cache/dev, so I looked at the permissions on every directory in that path.
[my-vm]# docker-compose exec php-fpm bash
[container]# ls -la /data/www{,/var{,/cache{,/dev}}}
ls: cannot access /data/www/var/cache/dev: No such file or directory
/data/www:
total 168
drwxrwsr-x 8 apache root 4096 Jan 15 19:26 .
drwxr-xr-x 8 root root 4096 Jan 15 19:27 ..
-rw-rw-r-- 1 apache root 248 Jan 15 19:26 .gitignore
-rw-rw-r-- 1 apache root 74 Jan 15 19:26 README.md
drwxrwsr-x 5 apache root 4096 Jan 15 19:27 app
drwxrwsr-x 2 apache root 4096 Jan 15 19:26 bin
-rw-rw-r-- 1 apache root 2387 Jan 15 19:26 composer.json
-rw-rw-r-- 1 apache root 119533 Jan 15 19:26 composer.lock
-rw-rw-r-- 1 apache root 978 Jan 15 19:26 phpunit.xml.dist
drwxrwsr-x 3 apache root 4096 Jan 15 19:26 src
drwxrwsr-x 3 apache root 4096 Jan 15 19:26 tests
drwxrwsr-x 4 apache root 4096 Jan 15 19:26 var
drwxrwsr-x 2 apache root 4096 Jan 15 19:26 web
/data/www/var:
total 52
drwxrwsr-x 4 apache root 4096 Jan 15 19:26 .
drwxrwsr-x 8 apache root 4096 Jan 15 19:26 ..
-rw-rw-r-- 1 apache root 34272 Jan 15 19:26 SymfonyRequirements.php
drwxrwsr-x 2 apache root 4096 Jan 15 19:26 cache
drwxrwsr-x 2 apache root 4096 Jan 15 19:26 sessions
/data/www/var/cache:
total 8
drwxrwsr-x 2 apache root 4096 Jan 15 19:26 .
drwxrwsr-x 4 apache root 4096 Jan 15 19:26 ..
-rw-rw-r-- 1 apache root 0 Jan 15 19:26 .gitkeep
So far, so good. The chown has set everything to apache:root and using the modes specified in the script.
Next, I exited the container and ran the composer update.
docker-compose exec php-fpm composer update
When prompted, I used the database parameters I found in the git repo, and everything installed fine. Next, I went back into the container to see if the permissions had changed.
[my-vm]# docker-compose exec php-fpm bash
[container]# ls -la /data/www{,/var{,/cache{,/dev}}}
/data/www:
total 164
drwxrwsr-x 9 apache root 4096 Jan 15 19:20 .
drwxr-xr-x 8 root root 4096 Jan 15 19:18 ..
-rw-rw-r-- 1 apache root 248 Jan 15 19:17 .gitignore
-rw-rw-r-- 1 apache root 74 Jan 15 19:17 README.md
drwxrwsr-x 5 apache root 4096 Jan 15 19:18 app
drwxrwsr-x 2 apache root 4096 Jan 15 19:21 bin
-rw-rw-r-- 1 apache root 2387 Jan 15 19:17 composer.json
-rw-rw-r-- 1 apache root 114331 Jan 15 19:20 composer.lock
-rw-rw-r-- 1 apache root 978 Jan 15 19:17 phpunit.xml.dist
drwxrwsr-x 3 apache root 4096 Jan 15 19:17 src
drwxrwsr-x 3 apache root 4096 Jan 15 19:17 tests
drwxrwsr-x 5 apache root 4096 Jan 15 19:21 var
drwxr-sr-x 25 root root 4096 Jan 15 19:21 vendor
drwxrwsr-x 3 apache root 4096 Jan 15 19:21 web
/data/www/var:
total 96
drwxrwsr-x 5 apache root 4096 Jan 15 19:21 .
drwxrwsr-x 9 apache root 4096 Jan 15 19:20 ..
-rw-rw-r-- 1 apache root 34272 Jan 15 19:21 SymfonyRequirements.php
-rw-r--r-- 1 root root 39637 Jan 15 19:21 bootstrap.php.cache
drwxrwsr-x 3 apache root 4096 Jan 15 19:21 cache
drwxr-sr-x 2 root root 4096 Jan 15 19:21 logs
drwxrwsr-x 2 apache root 4096 Jan 15 19:17 sessions
/data/www/var/cache:
total 12
drwxrwsr-x 3 apache root 4096 Jan 15 19:21 .
drwxrwsr-x 5 apache root 4096 Jan 15 19:21 ..
-rw-rw-r-- 1 apache root 0 Jan 15 19:17 .gitkeep
drwxr-sr-x 4 root root 4096 Jan 15 19:21 dev
/data/www/var/cache/dev:
total 636
drwxr-sr-x 4 root root 4096 Jan 15 19:21 .
drwxrwsr-x 3 apache root 4096 Jan 15 19:21 ..
-rw-r--r-- 1 root root 90 Jan 15 19:21 annotations.map
-rw-r--r-- 1 root root 277718 Jan 15 19:21 appDevDebugProjectContainer.php
-rw-r--r-- 1 root root 38062 Jan 15 19:21 appDevDebugProjectContainer.php.meta
-rw-r--r-- 1 root root 213247 Jan 15 19:21 appDevDebugProjectContainer.xml
-rw-r--r-- 1 root root 84170 Jan 15 19:21 appDevDebugProjectContainerCompiler.log
-rw-r--r-- 1 root root 4790 Jan 15 19:21 classes.map
drwxr-sr-x 3 root root 4096 Jan 15 19:21 doctrine
drwxr-sr-x 4 root root 4096 Jan 15 19:21 pools
As you can see, some things are now owned by root:root. As far as I can tell, this is simply because the container itself runs things as root. So when you exec a job inside, that job is run as root. Therefore, anything it creates is, by default, owned by root.
Meanwhile, Apache runs as the user "apache", because that is what supervisord is configured to do.
There are probably more elegant fixes for this problem, but this one was the simplest one I came up with:
docker-compose exec php-fpm chown -R apache:root /data/www/var/cache
docker-compose restart php-fpm
After that, the app returns
Welcome to
Symfony 3.2.2
Your application is now ready. You can start working on it at: /data/www/
I haven't tried to fix things any better than this. But my suggestion would be to try to have the startup run composer update for you and do the chown job after that. You probably don't need to chown all of /data/www, as Apache probably doesn't need write privs to everything in there. My guess was that the cache directory is one place it needs to write, so I chown'd that path.
After spent days and hours trying to get this working I finally got it thanks to Symfony #support channel in Slack and to the following channels on the IRC #symfony, #httpd, #php, #docker and last but not least to #DanLowe who takes the time to find a solution and help me out with the issue.
The facts|thoughts:
Is a Symfony 3.2.2 issue ... is not
Is a Docker running on Windows ... is not
Is a permission issue on the directory /web ... is not
Is a PHP-FPM problem ... is not
Is a Apache (httpd) problem ... is not
The problem: A miss configuration in PHP (.ini file) due to a copy & paste from Nginx setup to be used with Apache (previously I was using Nginx and then I moved to Apache leaving the PHP setup as it was).
// this work in Nginx but does not work in Apache
// cgi.fix_pathinfo is required to get PHP to adhere to the CGI spec
; Fix the cgi.fix_pathinfo directive
cgi.fix_pathinfo = 0
By default that line is commented in the php.ini file. But waits this could be an issue when PHP is using FastCGI and ProxyPass and so on .... well yes (I read it somewhere before) and no (if you research and set up your server as it should be). The solution to the problem above is comment out the line as it's by default:
; Fix the cgi.fix_pathinfo directive
; cgi.fix_pathinfo = 0
The solution to the "security" breach introduced above is and I've quoted from here:
If you have a recent version of PHP-FPM (~5.3.9+?), then you need to
do nothing, as the safe behaviour below is already the default.
Otherwise, find php-fpm's www.conf file (maybe
/etc/php-fpm.d/www.conf, depends on your system). Make sure you have
this:
security.limit_extensions = .php
Again, that's default in many places these days.
In my case I am using PHP 7.1 but even though I've secured my server by adding such line at my /etc/php-fpm.d/www.conf.
I'm building gogs from source and using
COPY gogs /home/git/go/src/github.com/gogits/gogs
to add the folder containing the built gogs bin and it's files to my container.
If I cd to /home/git/go/src/github.com/gogits/gogs I can see the file, I can chmod +x it , ls -l it to see permissions and ownership etc etc. However I can't seem to run it. Every way I try I get:
/home/git/go/src/github.com/gogits/gogs # cat /var/log/gogs/stderr.log
/bin/ash: /home/git/go/src/github.com/gogits/gogs/gogs: not found
/bin/ash: /home/git/go/src/github.com/gogits/gogs/gogs: not found
/bin/ash: /home/git/go/src/github.com/gogits/gogs/gogs: not found
Running ls -l . within the gogs directory:
/home/git/go/src/github.com/gogits/gogs # ls -l .
total 33148
-rw-r--r-- 1 git nogroup 613 Aug 27 09:22 Dockerfile
-rw-r--r-- 1 git nogroup 913 Aug 27 09:22 Dockerfile.rpi
-rw-r--r-- 1 git nogroup 1054 Aug 27 09:22 LICENSE
-rw-r--r-- 1 git nogroup 1792 Aug 27 09:22 Makefile
-rw-r--r-- 1 git nogroup 7852 Aug 27 09:22 README.md
-rw-r--r-- 1 git nogroup 5331 Aug 27 09:22 README_ZH.md
drwxr-xr-x 2 git nogroup 4096 Aug 28 08:33 cmd
drwxr-xr-x 10 git nogroup 4096 Aug 28 08:33 conf
drwxr-xr-x 5 git nogroup 4096 Aug 28 08:35 custom
drwxr-xr-x 2 git nogroup 4096 Aug 28 08:33 data
drwxr-xr-x 4 git nogroup 4096 Aug 28 08:33 docker
-rw-r--r-- 1 git nogroup 5359 Aug 27 09:22 glide.lock
-rw-r--r-- 1 git nogroup 1756 Aug 27 09:22 glide.yaml
-rwxr-xr-x 1 git nogroup 33831440 Aug 27 10:12 gogs
-rw-r--r-- 1 git nogroup 839 Aug 27 09:22 gogs.go
drwxr-xr-x 2 git nogroup 4096 Aug 28 08:33 log
drwxr-xr-x 4 git nogroup 4096 Aug 28 08:33 models
drwxr-xr-x 34 git nogroup 4096 Aug 28 08:33 modules
drwxr-xr-x 4 git nogroup 4096 Aug 28 08:33 packager
drwxr-xr-x 14 git nogroup 4096 Aug 28 08:33 public
drwxr-xr-x 14 git nogroup 4096 Aug 28 08:33 routers
drwxr-xr-x 12 git nogroup 4096 Aug 28 08:33 scripts
-rw-r--r-- 1 root root 2750 Aug 28 08:43 supervisord.log
-rw-r--r-- 1 root root 2 Aug 28 08:43 supervisord.pid
drwxr-xr-x 18 git nogroup 4096 Aug 28 08:33 templates
Other stuff I've tried:
/home/git/go/src/github.com/gogits/gogs # cp gogs /usr/bin
/home/git/go/src/github.com/gogits/gogs # /usr/bin/gogs
ash: /usr/bin/gogs: not found
/home/git/go/src/github.com/gogits/gogs # ls /usr/bin | grep gogs
gogs
/home/git/go/src/github.com/gogits/gogs # /usr/bin/gogs
ash: /usr/bin/gogs: not found
/home/git/go/src/github.com/gogits/gogs # ./usr/bin/gogs
ash: ./usr/bin/gogs: not found
/home/git/go/src/github.com/gogits/gogs # ls /usr/bin/gogs
/usr/bin/gogs
Exec'ng into the docker container with the command sh I start to get weird errors after sometime, it may be tend to happen after I've added to PATH the gogs bin, can't remember for sure, but this has happened before
/home/git/go/src/github.com/gogits/gogs # gogs
sh: gogs: not found
/home/git/go/src/github.com/gogits/gogs # ./gogs
sh: ./gogs: not found
/home/git/go/src/github.com/gogits/gogs # export PATH=PATH:/home/git/go/src/github.com/gogits/gogs
/home/git/go/src/github.com/gogits/gogs # gogs
sh: gogs: not found
/home/git/go/src/github.com/gogits/gogs # cp gogs /usr/bin
sh: cp: not found
/home/git/go/src/github.com/gogits/gogs # cp gogs /usr/bin
sh: cp: not found
/home/git/go/src/github.com/gogits/gogs # cp
sh: cp: not found
/home/git/go/src/github.com/gogits/gogs # ls
sh: ls: not found
Also when I did try to build gogs from source within the container I got a Segmentation Fault (core dump) sth like that. I tested it with valgrind and got sth like Bad permissions for mapped region at address 0x1340F80
Requested info:
FROM alpine:edge
MAINTAINER jonathan#saharacluster.com
ENV DEBIAN_FRONTEND noninteractive
COPY gogs /home/git/go/src/github.com/gogits/gogs
WORKDIR /home/git/go/src/github.com/gogits/gogs
RUN apk update && apk add --update \
openssh \
curl\
git\
supervisor ;\
rm -rf /var/lib/apt/lists/* ;\
adduser -S -g 'Gogs' git;\
mkdir -p custom/conf \
log\
data\
/var/log/gogs\
/home/git/.ssh\
/home/git/data\
/var/run/sshd\
/var/log/install\
/var/log/sshd;\
touch /home/git/.ssh/authorized_keys\
/home/git/setup.sh \
/home/git/data/gogs.db;\
chmod 700 /home/git/.ssh && chmod 600 /home/git/.ssh/authorized_keys;\
chown -R git: /var/log/gogs /var/log/sshd /home/git
COPY app.ini custom/conf/app.ini
COPY supervisord.conf /home/git/supervisord.conf
COPY docker-entrypoint.sh /home/git/docker-entrypoint.sh
COPY setup.sh /home/git/setup.sh
RUN chmod +x /home/git/docker-entrypoint.sh /home/git/setup.sh
EXPOSE 22 3000
ENV DEBIAN_FRONTEND dialog
ENTRYPOINT ["/home/git/docker-entrypoint.sh"]
env
GOGS_USER_PASSWORD=mypassword
HOSTNAME=e11c1f1b65cd
SHLVL=1
HOME=/root
DOMAIN=abc.com
GOGS_USER_EMAIL=gogsuser#abc.com
APP_NAME=sc
API_USER_PASSWORD=mypassword
TERM=xterm
GOGS_USER=gogsuser
API_USER_EMAIL=apiuser#abc.com
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DEBIAN_FRONTEND=dialog
PWD=/home/git/go/src/github.com/gogits/gogs
API_USER=apiuser
The main problem is that you're using an image which doesn't have the Go libraries needed. I suspect that you're also not compiling your Go application for your target OS.
Cross compile
If you're compiling your app in Mac or Windows, you have to cross
compile it. To target 64 bit linux (Debian/Ubuntu), it's like this.
GOOS=linux GOARCH=amd64 go build -o myapp_linux-amd64 .
The flags for other OS's are here https://golang.org/doc/install/source#environment.
It's a good idea to name your output binary with the name of the target platform (_linux-amd64).
Use the official golang:1.7 image
That's the simplest way to do it. The image will already include all the libraries needed.
Later on, you can look into optimising your image size. That's when static linking and Alpine becomes useful but they have their own sets of gotchas (no libc so DNS might not resolve correctly depending on your app). They can be worked around but it might not be worth your time now.