how to get docker-compose populate old volumes? - docker

Im running a docker container where i copy the content of current folder in /app in the container.
Then I put in a volume /app/media folder of the container.
However, when the volume is already created from a previous docker-compose build, i dont find all the new files put in my ./media folder, supposed to be copied to /app/media in the container...
Therefore i'm wondering how docker is populating the volume ? Is it not supposed to check in the container folder new files and put them in the volume?
I had the issue first and it was /media folder in the .dockerignore file, but now it's doing this again with other files in /media folder
Following What is the right way to add data to an existing named volume in Docker? I ve tried to do :
docker run -v mediafiles:/data --name helper busybox true
cd ./media && docker cp . helper:/data
docker rm helper
And it is now working
Thank you
Here is my docker-compose.yml
version: '3.7'
services:
nginx:
build:
context: .
dockerfile: ./compose/production/nginx/Dockerfile
restart: always
ports:
- 80:80
depends_on:
- backend
- frontend
volumes:
- staticfiles:/app/static
- mediafiles:/app/media
networks:
spa_network:
frontend:
build:
context: .
dockerfile: ./compose/production/frontend/Dockerfile
restart: always
stdin_open: true
command: yarn start
ports:
- 3000:3000
depends_on:
- backend
networks:
spa_network:
ipv4_address: 172.20.128.3
backend:
build:
context: .
dockerfile: ./compose/production/django/Dockerfile
restart: always
command: /start
volumes:
- staticfiles:/app/static
- mediafiles:/app/media
- sqlite_db:/app/db
ports:
- 8000:8000
env_file:
- ./env/prod-sample
networks:
spa_network:
ipv4_address: 172.20.128.2
networks:
spa_network:
ipam:
config:
- subnet: 172.20.0.0/16
volumes:
sqlite_db:
staticfiles:
mediafiles:
Here is my dockerfile for backend (where i dont find the /app/media files)
FROM python:3.8-slim-buster
ENV PYTHONUNBUFFERED 1
RUN apt-get update \
# dependencies for building Python packages
&& apt-get install -y build-essential netcat \
# psycopg2 dependencies
&& apt-get install -y libpq-dev \
# Translations dependencies
&& apt-get install -y gettext \
# cleaning up unused files
&& apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false \
&& rm -rf /var/lib/apt/lists/*
RUN addgroup --system django \
&& adduser --system --ingroup django django
# Requirements are installed here to ensure they will be cached.
COPY ./requirements.txt /requirements.txt
RUN pip install -r /requirements.txt
#COPY ./compose/production/django/entrypoint /entrypoint
#RUN sed -i 's/\r$//g' /entrypoint
#RUN chmod +x /entrypoint
#RUN chown django /entrypoint
COPY ./compose/production/django/start /start
RUN sed -i 's/\r$//g' /start
RUN chmod +x /start
RUN chown django /start
WORKDIR /app
# avoid 'permission denied' error
# copy project code
COPY . .
RUN chown -R django:django /app
#USER django
#ENTRYPOINT ["/entrypoint"]

If the volume is new, then docker will copy any files in the image to the new volume.
If the volume isn't new, then nothing is copied and you get the existing, old, contents of the volume.
More info here: https://docs.docker.com/storage/volumes/#populate-a-volume-using-a-container

Related

I solve the file owner problem that is pecuilar to linux user.What is the wrong in my dockerfile?

As mentioned above, I have a file owner problem with Laravel setup.
https://www.fullstaq.com/knowledge-hub/blogs/docker-and-the-host-filesystem-owner-matching-problem
This site has the solution.but the example of dockerfile in this site is for debian.
I want to create apache dockerfile.So Not suitable for what I want to do.
below is my dockerfile and docker-compose.yml.
# Dockerfile
FROM php:7.3-apache
COPY ./000-default.conf /etc/apache2/sites-available/000-default.conf
COPY --from=composer:2.0 /usr/bin/composer /usr/bin/composer
RUN apt-get update \
&& apt-get -y install \
git \
zip \
unzip \
vim \
&& docker-php-ext-install pdo_mysql bcmath mbstring \
&& a2enmod rewrite
WORKDIR /var/www/html
EXPOSE 80
# I added below the code,but An error occurred with the code.
ARG uid
RUN useradd -G www-data,root -u $uid -d /home/devuser devuser
RUN mkdir -p /home/devuser/.composer && \
chown -R devuser:devuser /home/devuser
# docker-compose.yml
version: '3'
services:
backend:
build:
context: ./docker/php
dockerfile: Dockerfile
ports:
- "80:80"
volumes:
- ./src/backend:/var/www/html
# - /etc/group:/etc/group:ro
# - /etc/user:/etc/user:ro
depends_on:
- db
db:
build:
context: ./docker/mysql
dockerfile: Dockerfile
command: --default-authentication-plugin=mysql_native_password
ports:
- "3306:3306"
volumes:
- ./docker/mysql/data:/var/lib/mysql
environment:
- MYSQL_ROOT_PASSWORD=pass
What is the wrong in my dockerfile?
Thank you for your help.
The error your getting here is
useradd: invalid user ID '-d'
This is because with
ARG uid
RUN useradd -G www-data,root -u $uid -d /home/devuser devuser
you specified a build ARG which is used in the following RUN directive. But when building the image with docker-compose there is no actual value specified for that ARG.
So when building it is substituted with an empty string and the executed command becomes
useradd -G www-data,root -u -d /home/devuser devuser
You need to specify a value for uid (i.e. the uid of your user on the host system) with the args attribute in the docker-compose.yml:
services:
backend:
build:
context: ./docker/php
dockerfile: Dockerfile
args:
uid: 1000
# or using env variables:
# uid: $HOST_UID

After I run docker compose up, my mac returns error stating that it can't find my mix phx.server. How do I show docker where my mix.exs file is?

When I'm running Docker Compose up, I receive an error
** (Mix) The task "phx.server" could not be found
Note no mix.exs was found in the current directory
I believe it's the very last step I need to run the project. This is a phoenix/Elixir Docker project. Mix.exs is a top level file in my project, same level as my dockerfile/docker-compose file.
Dockerfile
FROM elixir:1.13.1
# Build Args
ARG PHOENIX_VERSION=1.6.6
ARG NODEJS_VERSION=16.x
# Apt
RUN apt-get update && apt-get upgrade -y
RUN apt-get install -y apt-utils
RUN apt-get install -y build-essential
RUN apt-get install -y inotify-tools
# Nodejs
RUN curl -sL https://deb.nodesource.com/setup_${NODEJS_VERSION} | bash
RUN apt-get install -y nodejs
# Phoenix
RUN mix local.hex --force
RUN mix archive.install --force hex phx_new #{PHOENIX_VERSION}
RUN mix local.rebar --force
# App Directory
ENV APP_HOME /app
RUN mkdir -p $APP_HOME
WORKDIR $APP_HOME
COPY . .
# App Port
EXPOSE 4000
# Default Command
CMD ["mix", "phx.server"]
Docker-compose.yml
version: "3"
services:
book-search:
build: .
volumes:
- ./src:/app
ports:
- "4000:4000"
depends_on:
- db
db:
image: postgres:9.6
environment:
POSTGRES_DB: "db"
POSTGRES_HOST_AUTH_METHOD: "trust"
POSTGRES_USER: tmclean
POSTGRES_PASSWORD: tmclean
PGDATA: /var/lib/postgresql/data/pgdata
restart: always
volumes:
- ./pgdata:/var/lib/postgresql/data
Let me know what other questions I can answer
The problem is your docker-compose.yml file.
volumes:
- ./src:/app
You are overwriting the app with a probably non-existant src directory. Change it to:
volumes:
- .:/app
and it should work. However, if you do that, there is no point in copying the files in your Dockerfile, so you can also remove the
COPY . .
Alternatively, leave the COPY if you want the source files to be in the image, and remove the volumes section from the book-search service in docker-compose.yml.

Docker Compose with custom django app, ngnix, postgres, certbot, and letsencrypt

I am building my first customs Docker with Docker compose and I feel I am very close to finishing it but I have having an issue with what seem to be the entrypoint
FYI i am tryng to deploy a django app with postgres, nginx, certbot, letsencrypt
this is what I am seeing:
certbot_1 | /data/entrypoint.sh: exec: line 14: certbot: not found
nginx_1 | /data/entrypoint.sh: exec: line 14: run: not found
EDIT: I was able to make them run with the editted code but Ngnix exits with a code 0 and I don't know why and wont run
I have tried changing path with no luck
I am not sure what I am doing wrong
any advice you can provide would be great!
docker compose file:
version: '3.8'
services:
web:
build: .
command: gunicorn FleetOptimal.wsgi:application --bind 0.0.0.0:8000
environment:
- TZ=America/Toronto
volumes:
- /home/littlejiver/src/FleetOptimal/FleetOptimal/FleetOptimal/:/manage/
- /home/littlejiver/src/FleetOptimal/FleetOptimal/:/data/
- /home/littlejiver/src/FleetOptimal/FleetOptimal/FleetOptimal/staticfiles/:/static_volume/
- /home/littlejiver/src/FleetOptimal/FleetOptimal/FleetOptimal/images/:/media_volume/
expose:
- 8000
env_file:
- ./.env.dev
depends_on:
- db
db:
image: postgres:13.0-alpine
volumes:
- /home/littlejiver/docker/postgres/postgres_data:/postgres_data/
environment:
- PUID=1000
- PGID=1000
- TZ=Canada/Toronto
- POSTGRES_USER=someusername
- POSTGRES_PASSWORD=#somepassword
- POSTGRES_DB=somedb
nginx-proxy:
tty: true
image: nginx:latest
container_name: nginx-proxy
build: .
command: nginx -g "daemon off"
restart: always
environment:
- NGINX_DOCKER_GEN_CONTAINER=nginx-proxy-letsencrypt
ports:
- 443:443
- 80:80
volumes:
- /home/littlejiver/src/FleetOptimal/FleetOptimal/:/data/
- /home/littlejiver/src/FleetOptimal/FleetOptimal/FleetOptimal/staticfiles/:/static_volume/
- /home/littlejiver/src/FleetOptimal/FleetOptimal/FleetOptimal/images/:/media_volume/
- certs:/etc/nginx/certs
- html:/usr/share/nginx/html
- vhost:/etc/nginx/vhost.d
- /var/run/docker.sock:/tmp/docker.sock:ro
depends_on:
- web
nginx-proxy-letsencrypt:
image: jrcs/letsencrypt-nginx-proxy-companion
env_file:
- ./.env.dev
environment:
- NGINX_DOCKER_GEN_CONTAINER=nginx-proxy-letsencrypt
volumes:
- /home/littlejiver/src/FleetOptimal/FleetOptimal/:/data/
- /var/run/docker.sock:/var/run/docker.sock:ro
- certs:/etc/nginx/certs
- html:/usr/share/nginx/html
- vhost:/etc/nginx/vhost.d
- acme:/etc/acme.sh
depends_on:
- nginx-proxy
volumes:
postgres_data:
static_volume:
media_volume:
certs:
html:
vhost:
acme:
DockerFile for Ngnix:
FROM nginx:latest
COPY vhost.d/default /etc/nginx/vhost.d/default
COPY custom.conf /etc/nginx/conf.d/custom.conf
DockerFile for Web:
###########
# BUILDER #
###########
# pull official base image
FROM python:3.9.6-alpine as builder
# set work directory
WORKDIR /manage
# set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
# install psycopg2 dependencies
RUN apk update \
&& apk add postgresql-dev gcc python3-dev musl-dev
# lint
RUN apk add zlib-dev jpeg-dev gcc musl-dev
RUN pip install --upgrade pip
# RUN pip install flake8==3.9.2
COPY . .
# RUN flake8 --ignore=E501,F401 /manage
# install dependencies
COPY ./requirements.txt .
RUN pip wheel --no-cache-dir --no-deps --wheel-dir /usr/src/app/wheels -r requirements.txt
#########
# FINAL #
#########
# pull official base image
FROM python:3.9.6-alpine
# create directory for the app user
# create the app user
RUN addgroup -S littlejiver && adduser -S littlejiver -G littlejiver
# create the appropriate directories
ENV HOME=/manage
ENV APP_HOME=/manage
WORKDIR $APP_HOME
# install dependencies
RUN apk add zlib-dev jpeg-dev gcc musl-dev
COPY --from=builder /usr/src/app/wheels /wheels
COPY --from=builder /manage/requirements.txt .
RUN pip install --no-cache /wheels/*
# copy entrypoint.prod.sh
COPY ./entrypoint.sh .
RUN sed -i 's/\r$//g' $APP_HOME/entrypoint.sh
RUN chmod +x $APP_HOME/entrypoint.sh
# copy project
COPY . $APP_HOME
# chown all the files to the app user
RUN chown -R littlejiver:littlejiver $APP_HOME
# change to the app user
USER littlejiver
WORKDIR /manage
# run entrypoint.prod.sh
ENTRYPOINT ["sh", "/data/entrypoint.sh"]
entrypoint.sh
#!/bin/sh
if [ "$DATABASE" = "postgres" ]
then
echo "Waiting for postgres..."
while ! nc -z $SQL_HOST $SQL_PORT; do
sleep 0.1
done
echo "PostgreSQL started"
fi
exec "$#"
Thanks
-littlejiver
roughly based of this guide:

build path either does not exist, is not accessible, or is not a valid URL

I tryna figure out docker to run my django rest framework + vue.js project in clouds. I built Dockerfile and docker-compose.yml file to start an ubuntu machine and run the postgresql, vue.js and drf containers. But when I try run docker-compose build I get the following message:
build path either does not exist, is not accessible, or is not a valid URL
Here is my Dockerfile:
RUN apt-get update && upt-get install -y \
gcc \
musl-dev \
node.js \
postgresql-server-dev-10 \
apt-utils \
python3.7 \
python3.7-dev \
python3-pip \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
RUN npm install webpack#2.9
WORKDIR /app
COPY requirements.txt /app
RUN pip3 install --no-cache-dir -r requirements.txt
COPY . /app
docker-compose.yml:
version: '3.5'
services:
postgres:
image: postgres:10
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: 8599
POSTGRES_DB: adserver
volumes:
- adserver-data/postgresql/data:/var/lib/postgresql/data
restart: always
rest_framework:
build:
context: ./app/adserver
dockerfile: Dockerfile
depends_on:
- postgres
command: ['python manage.py runserver']
restart: always
vue:
build:
context: ./app/adserver-vue
depends_on:
- rest_framework
command: ['npm run watch']
Please tell me what am I doing wrong?
Verify the name of folders because the folder app/adserver-vue needs to exist with the name equal in docker-compose.yml

New code changes exist in live container but are not reflected in the browser

I am using Docker with the open source BI tool Apache Superset. I have added a new file, specifically a .geojson file in the CountryMap directory. Now, when I try to build using docker-compose up --build or make changes in the frontend, Docker is not fully updated, and I get a file not found error when trying to run a query. When I look inside the container via docker exec -it container_id bash, the new file is there.
Dockerfile:
FROM python:3.6-jessie
RUN useradd --user-group --create-home --no-log-init --shell /bin/bash superset
# Configure environment
ENV LANG=C.UTF-8 \
LC_ALL=C.UTF-8
RUN apt-get update -y
# Install dependencies to fix `curl https support error` and `elaying package configuration warning`
RUN apt-get install -y apt-transport-https apt-utils
# Install superset dependencies
# https://superset.incubator.apache.org/installation.html#os-dependencies
RUN apt-get install -y build-essential libssl-dev \
libffi-dev python3-dev libsasl2-dev libldap2-dev libxi-dev
# Install extra useful tool for development
RUN apt-get install -y vim less postgresql-client redis-tools
# Install nodejs for custom build
# https://superset.incubator.apache.org/installation.html#making-your-own-build
# https://nodejs.org/en/download/package-manager/
RUN curl -sL https://deb.nodesource.com/setup_10.x | bash - \
&& apt-get install -y nodejs
WORKDIR /home/superset
COPY requirements.txt .
COPY requirements-dev.txt .
COPY contrib/docker/requirements-extra.txt .
RUN pip install --upgrade setuptools pip \
&& pip install -r requirements.txt -r requirements-dev.txt -r requirements-extra.txt \
&& rm -rf /root/.cache/pip
RUN pip install gevent
COPY --chown=superset:superset superset superset
ENV PATH=/home/superset/superset/bin:$PATH \
PYTHONPATH=/home/superset/superset/:$PYTHONPATH
USER superset
RUN cd superset/assets \
&& npm ci \
&& npm run build \
&& rm -rf node_modules
COPY contrib/docker/docker-init.sh .
COPY contrib/docker/docker-entrypoint.sh /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
HEALTHCHECK CMD ["curl", "-f", "http://localhost:8088/health"]
EXPOSE 8088
docker-compose.yml:
version: '2'
services:
redis:
image: redis:3.2
restart: unless-stopped
ports:
- "127.0.0.1:6379:6379"
volumes:
- redis:/data
postgres:
image: postgres:10
restart: unless-stopped
environment:
POSTGRES_DB: superset
POSTGRES_PASSWORD: superset
POSTGRES_USER: superset
ports:
- "127.0.0.1:5432:5432"
volumes:
- postgres:/var/lib/postgresql/data
superset:
build:
context: ../../
dockerfile: contrib/docker/Dockerfile
restart: unless-stopped
environment:
POSTGRES_DB: superset
POSTGRES_USER: superset
POSTGRES_PASSWORD: superset
POSTGRES_HOST: postgres
POSTGRES_PORT: 5432
REDIS_HOST: redis
REDIS_PORT: 6379
# If using production, comment development volume below
#SUPERSET_ENV: production
SUPERSET_ENV: development
# PYTHONUNBUFFERED: 1
user: root:root
ports:
- 8088:8088
depends_on:
- postgres
- redis
volumes:
# this is needed to communicate with the postgres and redis services
- ./superset_config.py:/home/superset/superset/superset_config.py
# this is needed for development, remove with SUPERSET_ENV=production
- ../../superset:/home/superset/superset
volumes:
postgres:
external: false
redis:
external: false
Why is there a not found error?
try to use absolute path in volumes:
volumes:
- /home/me/my_project/superset_config.py:/home/superset/superset/superset_config.py
- /home/me/my_project/superset:/home/superset/superset
It is because docker-compose is utilizing cache. If the dockerfile and the docker-compose.yml in not changed it does not recreate the container image. To avoid this you should use the following flag:
--force-recreate
--force-recreate
Recreate containers even if their configuration and image haven't
changed.
For development purposes I like to use the following switch as well:
-V, --renew-anon-volumes
Recreate anonymous volumes instead of retrieving data from the previous containers.

Resources