I have a profile page in my app which I am migrating to docker. I want to have a default profile picture for user who don't upload any picture for that I need to store that picture in my container.
I have my default profile pic stored at data/web/media/default.jpg and want it to copy to vol/web/media/default.jpg in my docker container.
I tried COPY but got this error
failed to solve: rpc error: code = Unknown desc = failed to compute cache key: "/data" not found: not found
My dockerfile :
FROM python:3.9-alpine3.13
LABEL maintainer="mRk"
ENV PYTHONUNBUFFERED 1
COPY ./requirements.txt /requirements.txt
COPY ./app /app
COPY ./scripts /scripts
COPY ./data /vol
WORKDIR /app
EXPOSE 8000
RUN python -m venv /py && \
/py/bin/pip install --upgrade pip && \
apk add --update --no-cache postgresql-client && \
apk add --update --no-cache --virtual .tmp-deps \
build-base postgresql-dev musl-dev linux-headers && \
apk add --virtual build-deps gcc python3-dev musl-dev && \
apk add jpeg-dev zlib-dev libjpeg && \
pip install Pillow && \
apk del build-deps && \
/py/bin/pip install -r /requirements.txt && \
apk del .tmp-deps && \
adduser --disabled-password --no-create-home app && \
mkdir -p /vol/web/static && \
mkdir -p /vol/web/media && \
cp ./data/web/media/default.jpg /vol/web/media/default.jpg && \
chown -R app:app /vol && \
chmod -R 755 /vol && \
chmod -R +x /scripts
ENV PATH="/scripts:/py/bin:$PATH"
USER app
CMD ["run.sh"]
My docker-compose file :
version: "3.9"
services:
app:
build:
context: .
restart: always
volumes:
- static-data:/vol/web
environment:
- DB_HOST=db
- DB_NAME=${DB_NAME}
- DB_USER=${DB_USER}
- DB_PASS=${DB_PASS}
- SECRET_KEY=${SECRET_KEY}
- ALLOWED_HOSTS=${ALLOWED_HOSTS}
- EMAIL_USER=${EMAIL_USER}
- EMAIL_PASS=${EMAIL_PASS}
depends_on:
- db
db:
image: postgres:13-alpine
restart: always
volumes:
- postgres-data:/var/lib/postgresql/data
environment:
- POSTGRES_DB=${DB_NAME}
- POSTGRES_USER=${DB_USER}
- POSTGRES_PASSWORD=${DB_PASS}
proxy:
build:
context: ./proxy
restart: always
depends_on:
- app
ports:
- 80:8000
volumes:
- static-data:/vol/static
volumes:
postgres-data:
static-data:
It looks like you try to copy something from the data dir:
cp ./data/web/media/default.jpg /vol/web/media/default.jpg && \
But this directory does not exist. However, you copied everything from the data dir on your host machine into your container dir vol on line:
COPY ./data /vol
That means, that all your files and dirs from data or copied into the vol dir in the container. So I geuss you do not need that line at all, as it is already in your container but in the vol directory.
This line isn't going to work since you set your WORKDIR to /app/ earlier.
cp ./data/web/media/default.jpg /vol/web/media/default.jpg && \
You have to change your working directory or use an absolute path
An other solution would be to put this line below the RUN command since you don't need to be in the /app/ directory to run it:
WORDIR = /app/
You can use docker volumes which help you to copy and read files from docker
https://docs.docker.com/storage/volumes/
Related
im not familiar with docker at all. im trying to use symfony with docker (windows). for docker i use (without changes, only +mysql)
https://github.com/dunglas/symfony-docker
it works well
problem that when i change files or create new files changes not sync, only after new build.
Please tell how to write proper dockerfile so files from host (and from docker) will be in sync
i need to use docker volumes, but idk how 😁
dockerfile
ARG PHP_VERSION=8.1
ARG CADDY_VERSION=2
# "php" stage
FROM php:${PHP_VERSION}-fpm-alpine AS symfony_php
# persistent / runtime deps
RUN apk add --no-cache \
acl \
fcgi \
file \
gettext \
git \
;
ARG APCU_VERSION=5.1.21
RUN set -eux; \
apk add --no-cache --virtual .build-deps \
$PHPIZE_DEPS \
icu-data-full \
icu-dev \
libzip-dev \
zlib-dev \
; \
\
docker-php-ext-configure zip; \
docker-php-ext-install -j$(nproc) \
intl \
zip \
; \
pecl install \
apcu-${APCU_VERSION} \
; \
pecl clear-cache; \
docker-php-ext-enable \
apcu \
opcache \
; \
\
runDeps="$( \
scanelf --needed --nobanner --format '%n#p' --recursive /usr/local/lib/php/extensions \
| tr ',' '\n' \
| sort -u \
| awk 'system("[ -e /usr/local/lib/" $1 " ]") == 0 { next } { print "so:" $1 }' \
)"; \
apk add --no-cache --virtual .phpexts-rundeps $runDeps; \
\
apk del .build-deps
RUN docker-php-ext-install mysqli pdo pdo_mysql && docker-php-ext-enable pdo_mysql
COPY docker/php/docker-healthcheck.sh /usr/local/bin/docker-healthcheck
RUN chmod +x /usr/local/bin/docker-healthcheck
HEALTHCHECK --interval=10s --timeout=3s --retries=3 CMD ["docker-healthcheck"]
RUN ln -s $PHP_INI_DIR/php.ini-production $PHP_INI_DIR/php.ini
COPY docker/php/conf.d/symfony.prod.ini $PHP_INI_DIR/conf.d/symfony.ini
COPY docker/php/php-fpm.d/zz-docker.conf /usr/local/etc/php-fpm.d/zz-docker.conf
COPY docker/php/docker-entrypoint.sh /usr/local/bin/docker-entrypoint
RUN chmod +x /usr/local/bin/docker-entrypoint
VOLUME /var/run/php
COPY --from=composer:latest /usr/bin/composer /usr/bin/composer
# https://getcomposer.org/doc/03-cli.md#composer-allow-superuser
ENV COMPOSER_ALLOW_SUPERUSER=1
ENV PATH="${PATH}:/root/.composer/vendor/bin"
WORKDIR /srv/app
# Allow to choose skeleton
ARG SKELETON="symfony/skeleton"
ENV SKELETON ${SKELETON}
# Allow to use development versions of Symfony
ARG STABILITY="stable"
ENV STABILITY ${STABILITY}
# Allow to select skeleton version
ARG SYMFONY_VERSION=""
ENV SYMFONY_VERSION ${SYMFONY_VERSION}
# Download the Symfony skeleton and leverage Docker cache layers
#RUN composer create-project "${SKELETON} ${SYMFONY_VERSION}" . --stability=$STABILITY --prefer-dist --no-dev --no-progress --no-interaction; \
# composer clear-cache
###> recipes ###
###> doctrine/doctrine-bundle ###
#RUN apk add --no-cache --virtual .pgsql-deps postgresql-dev; \
# docker-php-ext-install -j$(nproc) pdo_pgsql; \
# apk add --no-cache --virtual .pgsql-rundeps so:libpq.so.5; \
# apk del .pgsql-deps
###< doctrine/doctrine-bundle ###
###< recipes ###
COPY . .
RUN set -eux; \
mkdir -p var/cache var/log; \
composer install --prefer-dist --no-dev --no-progress --no-scripts --no-interaction; \
composer dump-autoload --classmap-authoritative --no-dev; \
composer symfony:dump-env prod; \
composer run-script --no-dev post-install-cmd; \
chmod +x bin/console; sync
VOLUME /srv/app/var
ENTRYPOINT ["docker-entrypoint"]
CMD ["php-fpm"]
FROM caddy:${CADDY_VERSION}-builder-alpine AS symfony_caddy_builder
RUN xcaddy build \
--with github.com/dunglas/mercure \
--with github.com/dunglas/mercure/caddy \
--with github.com/dunglas/vulcain \
--with github.com/dunglas/vulcain/caddy
FROM caddy:${CADDY_VERSION} AS symfony_caddy
WORKDIR /srv/app
COPY --from=dunglas/mercure:v0.11 /srv/public /srv/mercure-assets/
COPY --from=symfony_caddy_builder /usr/bin/caddy /usr/bin/caddy
COPY --from=symfony_php /srv/app/public public/
COPY docker/caddy/Caddyfile /etc/caddy/Caddyfile
docker-compose
version: "3.4"
services:
db:
image: mysql
# NOTE: use of "mysql_native_password" is not recommended: https://dev.mysql.com/doc/refman/8.0/en/upgrading-from-previous-series.html#upgrade-caching-sha2-password
# (this is just an example, not intended to be a production configuration)
command: --default-authentication-plugin=mysql_native_password
restart: always
environment:
MYSQL_ROOT_PASSWORD: root
adminer:
image: adminer
restart: always
ports:
- 8080:8080
php:
build:
context: .
target: symfony_php
args:
SYMFONY_VERSION: ${SYMFONY_VERSION:-}
SKELETON: ${SKELETON:-symfony/skeleton}
STABILITY: ${STABILITY:-stable}
restart: unless-stopped
volumes:
- php_socket:/var/run/php
healthcheck:
interval: 10s
timeout: 3s
retries: 3
start_period: 30s
environment:
MERCURE_URL: ${CADDY_MERCURE_URL:-http://caddy/.well-known/mercure}
MERCURE_PUBLIC_URL: https://${SERVER_NAME:-localhost}/.well-known/mercure
MERCURE_JWT_SECRET: ${CADDY_MERCURE_JWT_SECRET:-!ChangeMe!}
caddy:
build:
context: .
target: symfony_caddy
depends_on:
- php
environment:
SERVER_NAME: ${SERVER_NAME:-localhost, caddy:80}
MERCURE_PUBLISHER_JWT_KEY: ${CADDY_MERCURE_JWT_SECRET:-!ChangeMe!}
MERCURE_SUBSCRIBER_JWT_KEY: ${CADDY_MERCURE_JWT_SECRET:-!ChangeMe!}
restart: unless-stopped
volumes:
- php_socket:/var/run/php
- caddy_data:/data
- caddy_config:/config
ports:
# HTTP
- target: 80
published: ${HTTP_PORT:-80}
protocol: tcp
# HTTPS
- target: 443
published: ${HTTPS_PORT:-443}
protocol: tcp
# HTTP/3
- target: 443
published: ${HTTP3_PORT:-443}
protocol: udp
volumes:
php_socket:
caddy_data:
caddy_config:
This is not something you declare on the dockerfile level, the dockerfile is a layer you tell the container (or the host) to add the container in it's run time. (Installations, copies..)
this a flag on the docker API `docker run -v vol:vol .... ```
Or inside of the docker-compose file
I see your docker-compose file has a declaration of volumes for service caddy.
Try to init the volume as an empty one
# bottom of the docker-compose file
volumes:
php_socket: {}
Docker is built upon layers, every change you'll make, will create a new layer and will be saved on the volume.
My container is up an running successfully - i'm on a MacOS Catalina;
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
487b211c7300 laravel-demo_laravel-app "docker-php-entrypoi…" 6 seconds ago Up 5 seconds 9000/tcp, 9021/tcp, 0.0.0.0:8021->80/tcp laravel-app
My docker-compose.yml looks like;
version: '3'
services:
#Laravel App
laravel-app:
build:
context: .
dockerfile: ./Dockerfile
container_name: laravel-app
restart: unless-stopped
tty: true
environment:
SERVICE_NAME: laravel-app
SERVICE_TAGS: dev
working_dir: /var/www/html
ports:
- 8021:80
volumes:
- ./:/var/www/html
and my Dockerfile
FROM php:7.2-fpm-alpine
# Copy composer.lock and composer.json
COPY composer.lock composer.json /var/www/html/
# Set working directory
WORKDIR /var/www/html
# Install Additional dependencies
RUN apk update && apk add --no-cache \
build-base shadow vim curl \
php7 \
php7-fpm \
php7-common \
php7-pdo \
php7-pdo_mysql \
php7-mysqli \
php7-mcrypt \
php7-mbstring \
php7-xml \
php7-openssl \
php7-json \
php7-phar \
php7-zip \
php7-gd \
php7-dom \
php7-session \
php7-zlib
# Add and Enable PHP-PDO Extenstions
RUN docker-php-ext-install pdo pdo_mysql
RUN docker-php-ext-enable pdo_mysql
# Install PHP Composer
RUN curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --
filename=composer
# Remove Cache
RUN rm -rf /var/cache/apk/*
# Add UID '1000' to www-data
RUN usermod -u 1000 www-data
RUN usermod -u 501 www-data
# Copy existing application directory permissions
COPY --chown=www-data:www-data . /var/www/html
# Change current user to www
USER www-data
These two files are both on the root of my fresh Laravel install.
Internal/external ports look fine in my config and with docker ps reporting success - my app should launch in a browser no? In theory - I should see the Laravel splash-screen;-
Well, unfortunately it doesn't and it's kicking my ass.
Things I've tried;
EXPOSE 9000
CMD ["php-fpm", "--host", "http://localhost"]
and
#CMD ["php-fpm", "--host", "0.0.0.0"]
Plus, various implementations of docker-compose.yml - like removing the environment: parameter - but no joy.
By comparing my Dockerfile & docker-compose.yml - are you able to offer some clues?
Thanks
If you cant beat em - join em;-
https://hub.docker.com/r/bitnami/laravel
when I start my container with docker-compose up my app eventually gets stuck and repeats itself over and over with these lines:
I, [2019-12-09T15:19:35.396147 #342] INFO -- : Refreshing Gem list
E, [2019-12-09T15:19:47.019123 #342] ERROR -- : No such file or directory # rb_sysopen - /usr/src/app/config/environments/development.rb (Errno::ENOENT)
.
.
.
The thing is, when I open the container bash and use ls, I can see that the file is in the path that he can't seem to find.
Any advice on how to fix this problem? Thank you!
EDIT: Dockerfile
FROM ruby:2.6.3-alpine3.9
ENV APP_HOME /usr/src/app
ENV CONTAINERNAME_ENVIRONMENT docker
EXPOSE 80
WORKDIR $APP_HOME
COPY Gemfile* $APP_HOME/
# general dependencies
RUN apk update
RUN set -ex \
&& apk add --no-cache libpq imagemagick nodejs bash gcompat git
# poltergeist, see https://github.com/Overbryd/docker-phantomjs-alpine/releases
ENV PHANHOME /usr/share
RUN apk add --no-cache fontconfig curl && \
mkdir -p $PHANHOME \
&& curl -L https://github.com/Overbryd/docker-phantomjs-alpine/releases/download/2.11/phantomjs-alpine-x86_64.tar.bz2 | tar xj -C $PHANHOME \
&& ln -s $PHANHOME/phantomjs/phantomjs /usr/bin/phantomjs \
&& phantomjs --version
# build dependencies
RUN set -ex \
&& apk add --no-cache --virtual builddependencies \
linux-headers \
libpq \
tzdata \
build-base \
postgresql-dev \
imagemagick-dev \
&& bundle install \
&& apk del builddependencies
CMD ["bundle", "exec", "unicorn", "--port", "80"]
EDIT: docker-compose.yml
version: '3'
services:
postgresql:
image: postgres:9.6.2-alpine
container_name: postgresql-dev
environment:
- APPLICATION=myapp
- POSTGRES_PASSWORD=myapp
- POSTGRES_USER=my_app
- POSTGRES_DB=myapp
ports:
- 5432:5432
imimap:
build: .
image: myapp:dev
container_name: myapp-dev
environment:
- APPLICATION=myapp
- POSTGRES_PASSWORD=myapp
- POSTGRES_USER=my_app
- POSTGRES_DB=myapp
- RAILS_MASTER_KEY=${RAILS_MASTER_KEY}
- LDAP=${LDAP}
depends_on:
- postgresql
volumes:
- ./logc:/usr/src/app/log
- .:/usr/src/app
links:
- postgresql
ports:
- 8080:80
#entrypoint:
# - ci-cd/stall.sh
Changes in my requirements.txt are not being reflected when I run:
docker-compose -f docker-compose-dev.yml up -d
docker-compose-dev.yml
version: '3.6'
services:
web:
build:
context: ./services/web
dockerfile: Dockerfile-dev
volumes:
- './services/web:/usr/src/app'
ports:
- 5001:5000
environment:
- FLASK_ENV=development
- APP_SETTINGS=project.config.DevelopmentConfig
- DATABASE_URL=postgres://postgres:postgres#web-db:5432/web_dev
- DATABASE_TEST_URL=postgres://postgres:postgres#web-db:5432/web_test
depends_on:
- web-db
web-db:
build:
context: ./services/web/project/db
dockerfile: Dockerfile
ports:
- 5435:5432
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
nginx:
build:
context: ./services/nginx
dockerfile: Dockerfile-dev
restart: always
ports:
- 80:80
depends_on:
- web
- client
client:
build:
context: ./services/client
dockerfile: Dockerfile-dev
volumes:
- './services/client:/usr/src/app'
- '/usr/src/app/node_modules'
ports:
- 3007:3000
environment:
- NODE_ENV=development
- REACT_APP_WEB_SERVICE_URL=${REACT_APP_WEB_SERVICE_URL}
depends_on:
- web
Dockerfile-dev
# base image
FROM python:3.6-alpine
# install dependencies
RUN apk update && \
apk add --virtual build-deps gcc python-dev musl-dev && \
apk add libffi-dev && \
apk add postgresql-dev && \
apk add netcat-openbsd && \
apk add bind-tools && \
apk add --update --no-cache g++ libxslt-dev && \
apk add jpeg-dev zlib-dev
ENV PACKAGES="\
dumb-init \
musl \
libc6-compat \
linux-headers \
build-base \
bash \
git \
ca-certificates \
freetype \
libgfortran \
libgcc \
libstdc++ \
openblas \
tcl \
tk \
libssl1.0 \
"
ENV PYTHON_PACKAGES="\
numpy \
matplotlib \
scipy \
scikit-learn \
nltk \
"
RUN apk add --no-cache --virtual build-dependencies python3 \
&& apk add --virtual build-runtime \
build-base python3-dev openblas-dev freetype-dev pkgconfig gfortran \
&& ln -s /usr/include/locale.h /usr/include/xlocale.h \
&& python3 -m ensurepip \
&& rm -r /usr/lib/python*/ensurepip \
&& pip3 install --upgrade pip setuptools \
&& ln -sf /usr/bin/python3 /usr/bin/python \
&& ln -sf pip3 /usr/bin/pip \
&& rm -r /root/.cache \
&& pip install --no-cache-dir $PYTHON_PACKAGES \
&& pip3 install 'pandas<0.21.0' \
&& apk del build-runtime \
&& apk add --no-cache --virtual build-dependencies $PACKAGES \
&& rm -rf /var/cache/apk/*
# set working directory
WORKDIR /usr/src/app
# add and install requirements
COPY ./requirements.txt /usr/src/app/requirements.txt # <--- refer to EDIT
RUN pip install -r requirements.txt
# add entrypoint.sh
COPY ./entrypoint.sh /usr/src/app/entrypoint.sh
RUN chmod +x /usr/src/app/entrypoint.sh
# add app
COPY . /usr/src/app
# run server
CMD ["/usr/src/app/entrypoint.sh"]
what am I missing?
EDIT
Like the accepted answer in [Docker how to run pip requirements.txt only if there was a change?, I'm already copying the requirements.txt file in a separate build step before adding the entire application into the image, but it does not seem to work.
I think the problem likely is that $ docker-compose up alone will not rebuild your images if you make changes. In order to get docker-compose to include your changes to your requirements.txt you will need to pass the --build flag to docker-compose.
I.e instead run:
docker-compose -f docker-compose-dev.yml up --build -d
Which will force a docker-compose rebuild the image. However this will rebuild all images in the docker-compose file which may or may not be desired.
If you only want to rebuild the image of a single service you can first run docker-compose -f docker-compose-dev.yml build web, then afterwards just run your original docker-compose command.
More info on the build command here.
Try to install requirements from the copied file
https://docs.docker.com/develop/develop-images/dockerfile_best-practices/
It is an example of their Dockerfile
COPY requirements.txt /tmp/
RUN pip install --requirement /tmp/requirements.txt
This is what you have
RUN pip install -r requirements.txt
Then after you have changed your docker file, you have to stop your container, remove your image, build a new one, and run container from it.
Stop container and remove the image.
docker-compose down
docker-compose --rmi all
--rmi all - removes all images. You might want to use --rmi IMAGE_NAME
And to start it (if you use not default parameters, change these commands with your arguments).
docker-compose up
Update
In case you have running docker and you do not want to stop it and rebuild an image (if you just want to install a package or run some commands or even start a new application), you can connect the container from your local machine and run command line commands.
docker exec -it [CONTAINER_ID] bash
To get [CONTAINER_ID], run
docker ps
Note docker-compose ps will give you containers names, but you need container id to ssh the container.
I would like to have a shared directory between my containers: ftp and s3fs. Todo so, I have created a volume in my docker-compose file called s3.
If I stop s3fs from running in my s3fs container, then I can create files in the ftp container and they will show up in side s3fs under /home/files.
However, when running s3fs the directory /home/files remains empty whilst I create files in the ftp.
This is what my /proc/mounts file looks like:
/dev/sda2 /home/files ext4 rw,relatime,data=ordered 0 0
s3fs /home/files fuse.s3fs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
I belive fuse maybe overriding my docker volume, has anyone encountered this problem before?
docker-compose.yml
version: "3"
services:
ftp:
image: app/proftpd:latest
volumes:
- s3:/home/files
ports:
- 2222:2222
s3fs:
image: app/s3fs:latest
command: start
env_file:
- s3fs/aws.env
volumes:
- s3:/home/files
cap_add:
- SYS_ADMIN
devices:
- "/dev/fuse"
environment:
ENVIRONMENT: "dev"
volumes:
s3:
s3fs - Dockerfile
FROM ubuntu:16.04
RUN apt-get update -qq
RUN apt-get install -y \
software-properties-common
RUN apt-get update -qq
RUN apt-get install -y \
automake \
autotools-dev \
fuse \
g++ \
git \
libcurl4-openssl-dev \
libfuse-dev \
libssl-dev \
libxml2-dev \
make \
pkg-config \
curl
RUN curl -L https://github.com/s3fs-fuse/s3fs-fuse/archive/v1.84.tar.gz | tar zxv -C /usr/src
RUN cd /usr/src/s3fs-fuse-1.84 && ./autogen.sh && ./configure --prefix=/usr --with-openssl && make && make install
COPY entrypoint.sh /opt/s3fs/bin/entrypoint.sh
RUN mkdir -p /home/files
WORKDIR /opt/s3fs/bin
ENTRYPOINT ["/bin/sh", "./entrypoint.sh"]
s3fs - entrypoint.sh
#!/usr/bin/env bash
case $1 in
start)
echo "Starting S3Fs: "
s3fs mybucket /home/files -o allow_other,nonempty -d -d
;;
esac
ftp - Dockerfile
FROM ubuntu:16.04
RUN apt-get update && apt-get install -y \
openssh-server \
proftpd-basic \
proftpd-mod-mysql
COPY proftpd.conf /etc/proftpd/proftpd.conf
COPY sftp.conf /etc/proftpd/conf.d/sftp.conf
COPY setup.sh /etc/proftpd/setup.sh
RUN chmod 500 /etc/proftpd/setup.sh && /etc/proftpd/setup.sh
EXPOSE 2222
ENTRYPOINT ["/bin/sh", "/etc/proftpd/entrypoint.sh"]
You can mount s3 in your docker container in next way
1.Add to Dockerfile
RUN apt-get install -y fuse s3fs
RUN mkdir /root/.aws
RUN touch /root/.aws/.passwd-s3fs && chmod 600 /root/.aws/.passwd-s3fs
COPY entrypoint.sh ./
RUN chmod 700 entrypoint.sh
ENTRYPOINT entrypoint.sh
2.Create entrypoint.sh with next script
#!/bin/sh
echo "$AWS_CREDS" > /root/.aws/.passwd-s3fs
echo "$BUCKET_NAME /srv/files fuse.s3fs _netdev,allow_other,passwd_file=/root/.aws/.passwd-s3fs 0 0" > /etc/fstab
mount -a
<your old CMD or ENTRYPOINT>
3.In docker-compose.yml add next
<your-container-name>:
image: ...
build: ...
environment:
- AWS_ID="AKI..."
- AWS_KEY="omIE..."
- AWS_CREDS=AKI...:2uMZ...
- BUCKET_NAME=<YOUR backed name>
devices:
- "/dev/fuse"
cap_add:
- SYS_ADMIN
security_opt:
- seccomp:unconfined