I'm trying to mount a volume using docker-compose so I can hot reload some C code when developing. I've used Docker a couple times before and specifically hit this use case while working on a nodejs website but I'm completely at a loss here.
My docker-compose.yml and Dockerfile have been stripped down entirely to just the bare minimum. I would like to mount my current directory (all source code) into the container. My Dockerfile just installs some dependencies, sets the working directory and then attempts to run the makefile while my docker-compose.yml adds the volume. The result is a container that cannot access the mounted volume and resulting code (it's nothing wrong with the Makefile as it works on the host directory and when copied in, instead of using a volume). Does anyone see anything wrong with either of these files? It appears the /cfs folder isn't even being created in the container. I tried mounting it to the home directory to no avail.
docker-compose.yml
version: '3'
services:
cfs:
volumes:
- ./:/cfs
build:
context: ./
dockerfile: ./Dockerfile
networks:
- default
networks:
default:
internal: true
Dockerfile
FROM ubuntu:20.04
# install dependencies
RUN apt-get -qy update \
&& apt-get -y install \
cmake=3.16.3-1ubuntu1 \
make=4.2.1-1.2 \
gcc=4:9.3.0-1ubuntu2 \
g++=4:9.3.0-1ubuntu2
WORKDIR /cfs
RUN make prep
RUN make
RUN make install
Most Compose settings aren't visible during an image build. volumes: aren't mounted, environment: variables aren't set, networks: aren't accessible. Only the settings within the immediate build: block have an effect.
That means you should look at the Dockerfile in isolation, ignoring the docker-compose.yml file. At that point, the /cfs directory is empty (you don't COPY any source code into it), so the RUN make ... commands will fail. It doesn't matter that the directory will eventually have something else mounted over it.
If you're just planning to recompile the application when the source code changes, delete the volumes:, COPY the source into the image, and run docker-compose build at the point you'd typically run make. If you do have a setup that can rebuild the application when its source changes, you need to set the image's CMD to launch that, but if you don't COPY the source in, you can't build it at image build time. (...and if you're going to overwrite the entire interesting content of the image with a volume, it will get lost anyways.)
Related
I intend to build and run a dockerized container using an image that is built locally and not using docker hub. My use case is the following :
Cloned an open source repo source code(https://github.com/jitsi/jitsi-meet).
They have their own dockerized version too which build from the source and is deployed
on Docker Hub:(https://github.com/jitsi/docker-jitsi-meet)
Renamed the filenames and the contents inside the filenames of jitsi-meet for my own ease of use.
Packaged as a 7z packaged with the final changes.
I now Require to build the image locally using the code from the 7z package/ OR source code folder itself without uploading the image to the public docker hub.
RUN specific set of commands inside the Dockerfile.
my Dockerfile:
ARG MYPROJECT_REPO=myproject
ARG BASE_TAG=stable
FROM ${MYPROJECT_REPO}/base:${BASE_TAG}
LABEL org.opencontainers.image.title="Myproject"
LABEL org.opencontainers.image.url="https://myproject.org/myproject-meet/"
LABEL org.opencontainers.image.source="https://github.com/myproject/docker-myproject-meet"
LABEL org.opencontainers.image.documentation="https://myproject.github.io/handbook/"
ADD https://raw.githubusercontent.com/acmesh-official/acme.sh/2.8.8/acme.sh /opt
COPY rootfs/ /
RUN apt-dpkg-wrap apt-get update && \
apt-dpkg-wrap apt-get install -y cron nginx-extras myproject-meet-web socat curl jq && \
mv /usr/share/myproject-meet/interface_config.js /defaults && \
rm -f /etc/nginx/conf.d/default.conf && \
apt-cleanup
EXPOSE 80 443
VOLUME ["/config", "/usr/share/myproject-meet/transcripts"]
My docker-compose.yml (Only uploading relevant parts):
services:
# Frontend
myproject_webserver:
container_name: myproject-webserver
build:
dockerfile: ./Dockerfile
context: ./
#image: jitsi/web:${JITSI_IMAGE_VERSION:-unstable}
restart: ${RESTART_POLICY:-unless-stopped}
ports:
- '${HTTP_PORT}:80'
- '${HTTPS_PORT}:443'
volumes:
- ${CONFIG}/web:/config:Z
- ${CONFIG}/web/crontabs:/var/spool/cron/crontabs:Z
- ${CONFIG}/transcripts:/usr/share/myproject-meet/transcripts:Z
environment:
- AMPLITUDE_ID
- ANALYTICS_SCRIPT_URLS
As you can see i have commented out the public docker image of jitsi from docker hub and used build context instead. I need to build a local image and deploy to the DockerFile.
My core problem stems from the issue of renaming files/folders and the contents of the same.
Kindly correct my understanding of the following :
If i had used the core code i could have made minute changes to the code itself which are necessary without renaming and used a COPY command in DockerFile which would be used instead of the core file keeping everything else intact and also keeping the image line in docker-compose.yml as is.
So if the original repo has folder A/filenamea.js running inside a container :
Can docker COPY command be used if I have folder A1/filenamea1.js' as
renamed files to replace and run instead of the ones inside the container folder
A/filenamea.js?
I'm using the following Dockerfile for development of an Angular project:
FROM node:18-alpine
WORKDIR /code
COPY package*.json /code/
RUN npm ci --quiet
It gets started with docker compose. My code folder is mounted as a volume so the development server inside the container detects changes when editing and keeps live updates going:
version: "3"
services:
ui:
build: ./PathOnHostWithProjectRepo
command: sh -c "npm start"
ports:
- 4200:4200
volumes:
- ./PathOnHostWithProjectRepo:/code
- node_modules:/code/node_modules
volumes:
node_modules:
node_modules gets created when the image is created and, to my understanding, would only update if my package.json is changed. However, today I updated package.json with a new dependency and it is not being installed inside of the volume. I have tried everything I can think of. docker compose down, docker system prune -a -f, and rebuilding. Every time the container starts there is an error that it cannot find the new dependency added. If I step into the container and inspect the node_modules folder the library isn't there. It is present on my host machine if I run npm install locally without Docker, so I know the package and imports must be correct.
With this setup your node_modules will never be updated. Docker will completely ignore any changes in your package.json file. You've told it that directory contains user data that must not be modified.
For the setup you show you don't need Docker at all. It's straightforward to install Node and OS package managers like Debian/Ubuntu APT or MacOS Homebrew generally have a prepackaged version. If you use Node directly then you won't have problems like this; everything will work normally.
If you must use Docker here, the most straightforward thing to do is to make sure all of your application code is in a subdirectory; then you can mount only the subdirectory containing the code and leave the image's node_volumes directory intact.
$ ls -F
Dockerfile
docker-compose.yml
node_modules/
package.json
package-lock.json
src/
# Dockerfile
FROM node:lts
WORKDIR /code
COPY package*.json ./
RUN npm ci
COPY src/ ./src/
# RUN npm build
CMD ["npm", "start"]
# docker-compose.yml
version: '3.8'
services:
ui:
build: .
ports:
- '4200:4200'
volumes:
- ./src:/code/src
Mounting only the src subdirectory avoids the trouble of storing node_modules in a named volume (or an anonymous one). If you change your package.json file you will need to re-run docker-compose build, but since you're directly using the library tree in your image then this will in fact get updated.
If you're going to deploy this image somewhere, remember to delete the volumes: block during your local integration testing so that you're actually running the image you're going to deploy, and not a hybrid of an image and your potentially-modified local code.
I am trying to make the binary file /bin/wkhtmltopdf from the container wkhtmltopdf available in the web container. I try to achieve this with a named volume.
I have the following docker container setup in my docker-compose.yml:
services:
web:
image: php:7.4-apache
command: sh -c "mkdir -p /usr/local/bin && touch /usr/local/bin/wkhtmltopdf"
entrypoint: sh -c "exec 'apache2-foreground'"
volumes:
- wkhtmltopdfvol:/usr/local/bin/wkhtmltopdf
wkhtmltopdf:
image: madnight/docker-alpine-wkhtmltopdf
command: sh -c "touch /bin/wkhtmltopdf"
entrypoint: sh -c "tail -f /dev/null" # workaround to keep container running
volumes:
- wkhtmltopdfvol:/bin/wkhtmltopdf
volumes:
wkhtmltopdfvol:
However, I get the following error when running docker-compose up:
ERROR: for wkhtmltopdf Cannot create container for service wkhtmltopdf:
source /var/lib/docker/overlay2/42e7082b8024ae4ebb13a4f0003a9e17bc18b33ef0677431dd002da3c21dde88/merged/bin/wkhtmltopdf is not directory
.../bin/wkhtmltopdf is not directory
Does that mean that I can't share one file between containers but only directories through a named volume? How do I achieve this?
Edit: I also noticed that /usr/local/bin/wkhtmltopdf inside the web container is a directory and not a file as I expected.
It can be tricky to share binaries between containers like this. Volumes probably aren't the mechanism you're looking for.
If you look at the Docker Hub page for the php image you can see that php:7.4-apache is an alias for (currently) php:7.4.15-apache-buster, where "Buster" is the name of a Debian release. You can then search on https://packages.debian.org/ to discover that Debian has a prepackaged wkhtmltopdf package. You can install this using a custom Dockerfile:
FROM php:7.4-apache
RUN apt-get update \
&& DEBIAN_FRONTEND=noninteractive \
apt-get install --assume-yes --no-install-recommends \
wkhtmltopdf
# COPY ...
# Base image provides EXPOSE, CMD
Then your docker-compose.yml file needs to build this image:
version: '3.8'
services:
web:
build: .
# no image:, volumes:, or command: override
Just in terms of the mechanics of sharing binaries like this, you can run into trouble where a binary needs a shared library that's not present in the target container. The apt-get install mechanism handles this for you. There are also potential troubles if a container has a different shared-library ecosystem (especially Alpine-based containers), or using host binaries from a different operating system.
The Compose file you show mixes several concepts in a way that doesn't really work. A named volume is always a directory, so trying to mount that over the /bin/wkhtmltopdf file in the second container causes the error you see. There's a dependency issue for which container starts up first and gets to create the volume. A container only runs a single command, and if you have both entrypoint: and command: then the command gets passed as extra arguments to the entrypoint (and if the entrypoint is an sh -c ... invocation, effectively ignored).
If you really wanted to try this approach, you should make web: {depends_on: [wkhtmltopdf]} to force the dependency order. The second container should mount the volume somewhere else, it probably shouldn't have an entrypoint:, and it should do something like command: cp -a /bin/wkhtmltopdf /export. (It will exit immediately once this cp finishes, but that shouldn't matter.) The first container can then mount the volume on, say, /usr/local/bin, and not specially set command: or entrypoint:. There will still be a minor race condition (you're not guaranteed the cp command will complete before Apache starts) but it probably wouldn't be a practical problem.
I am working on a docker app. The purpose of this repo is to output some json into a volume. I am using a Dockerfile, docker-compose and a Makefile. I'll show the contents of each file below. Goal/desired outcome is that when I run using make up that the container runs and outputs the json.
Directory looks like this:
docker-compose.yaml
Dockerfile
Makefile
main/ # a directory
Here are the contents of directory Main:
example.R
Not sure the best order to show these files. Throughout my setup I refer to a variable $PROJECTS_DIR which is a global on the host / local:
echo $PROJECTS_DIR
/home/doug/Projects
Here are my files:
docker-compose.yaml:
version: "3.5"
services:
nextzen_ga_extract_marketing:
build:
context: .
environment:
start_date: "2020-11-18"
start_date: "2020-11-19"
volumes:
- ${PROJECTS_DIR}/Zen/nextzen_google_analytics_extract_pipeline:/home/rstudio/Projects/nextzen_google_analytics_extract_pipeline
Dockerfile:
FROM rocker/tidyverse:latest
ADD main main
WORKDIR "/main"
RUN apt-get update && apt-get install -y \
less \
vim
ENTRYPOINT ["Rscript", "example.R"]
Makefile:
.PHONY: build
build:
docker-compose build
.PHONY: up
up:
docker-compose pull
docker-compose up -d
.PHONY: restart
restart:
docker-compose restart
.PHONY: down
down:
docker-compose down
Here is the contents of the 'main' file of the Docker app, example.R:
library(jsonlite)
unlink("../output_data", recursive = TRUE) # delete any existing data from previous runs
dir.create('../output_data')
write(toJSON(mtcars), '../output_data/ga_tables.json')
If I navigate into ${PROJECTS_DIR}/Zen/nextzen_google_analytics_extract_pipeline/main and then run sudo Rscript example.R then the file runs and outputs the json in '../output_data/ga_tables.json as expected.
I am struggling to get this to happen when running the container. If I navigate into ${PROJECTS_DIR}/Zen/nextzen_google_analytics_extract_pipeline/ and then in the terminal run make up for:
docker-compose pull
docker-compose up -d
I then see:
make up
docker-compose pull
docker-compose up -d
Creating network "nextzengoogleanalyticsextractpipeline_default" with the default driver
Creating nextzengoogleanalyticsextractpipeline_nextzen_ga_extract_marketing_1 ...
Creating nextzengoogleanalyticsextractpipeline_nextzen_ga_extract_marketing_1 .
It 'looks' like everything ran as expected with no errors. Except no output appears in directory output_data as expected?
I guess I'm misunderstanding or misusing ENTRYPOINT in the Dockerfile with ENTRYPOINT ["Rscript", "example.R"]. My goal is that this file would run when the container is run.
How can I 'run' (if that's the correct terminology) my app so that it outputs json into /output_data/ga_tables.json?
Not sure what other info to provide? Any help much appreciated, I'm still getting to grips with docker.
If you run your application from /main and its output is supposed to go into ../output_data (so effectively /output_data), you need to bind mount this directory to have this output available on host. Therefore I would update your docker-compose.yaml to read something like this:
volumes:
- /path/to/output_data/on/host:/output_data
Bear in mind however that your script will not be able to remove /output_data when bind-mounted this way, so you might want to change your step to removing directory contents and not directory itself.
In my case, I got this working when I used full paths as opposed to relative paths.
I'm going crazy here.
I've been working on a Dockerfile and docker-compose.yml file for my project. I recently updated my project's dependencies. When I build the project outside of a container using composer install, it builds with the correct dependencies. However, when I build the project inside a docker container, it downloads and installs the latest dependencies, but then somehow runs the application using obsolete dependencies!
First of all, this is what my Dockerfile looks like:
FROM composer
# Set the working directory within the docker container
WORKDIR /app
# Copy in the app, then install dependencies.
COPY . /app
RUN composer install
I have excluded the composer.lock file and the vendor directory in my .dockerignore:
vendor
composer.lock
Here's my docker-compose.yml:
version: "3"
services:
app:
build: .
volumes:
- app:/app
webserver:
image: richarvey/nginx-php-fpm
volumes:
- app:/var/www/html
volumes:
app:
Note that the build process occurs within the app volume. I don't think this should be part of the problem, as I run docker system prune each time, to purge all existing volumes.
This is what I do to run the container. While troubleshooting, I have been running these commands to eliminate any cached files before starting the container:
$ docker system prune
$ docker-compose build --no-cache
$ docker-compose up --force-recreate
As I watch the dependencies install and download, I can see that it is downloading and installing the right versions! So it must have the correct composer.json file at some point in the process.
Yet somehow, once the build is complete and the application starts, I get the same old warnings about obsolete dependencies, and sure enough, and the composer.json inside the container is obsolete!
So my questions are:
How TF is the composer.json file in the container obsolete?
WHERE is it getting the obsolete file from, since it no longer exists in any image or cache??
How TF is it managing to install the latest dependencies with this obsolete composer.json file, but then not using them, and in fact reverting the composer.json file and the dependencies??
I think the problem is, that you copy your local files into the app-container and run composer install on this copy. Since this will not affect your host system, your webserver, which will actually serve your project will still use the outdated local version, instead of the copy from your other image.
You could try using multi-stage builds or something like this:
COPY FROM app:latest /app /var/www/html
This will copy the artifact from your "build-container", i.e. your project with the installed dependency in app, into the actual container that is running the code, i.e. webserver. Unfortunately, I don't think this will work (well) with your setup, where you mount the volume into that location.
Well, I finally fixed this issue, although parts of my original problem still confuse me.
Here's what I learned:
The docker-compose up process goes in this order:
If an image already exists, use it, even if the Dockerfile (or files used by it) has changed. (This can be avoided with docker-compose up --build).
If there is no existing image, build the image from the Dockerfile.
Mount the volumes specified in the docker-compose file.
A huge part of my problem was that I thought that the volumes were mounted before the build process, and that my application would be installed into this volume as a result of these commands:
COPY . /app
RUN composer install
However, these files were later overwritten when the volume was mounted at the same location within the container (/app).
Now, since I was not mounting a host directory, just an ephemeral, named volume, the /app directory should have been empty. I still don't understand why it wasn't, considering I was clearing my existing Docker volumes with docker system prune before each build. Whatever.
In the end, I used #dbrumann's solution. This was simpler, did not require the use of any Docker volumes, and avoids having a live composer container after the build process has completed (this would be bad for production). My Dockerfile now looks like this:
Dockerfile:
# Install dependencies using the composer image
FROM composer AS composer
# Set the working directory within the docker container
WORKDIR /app
# Copy in the app, then install dependencies.
COPY . .
RUN composer install
# Start the nginx server
FROM richarvey/nginx-php-fpm
# Copy over files from the composer image, which is then discarded automatically
WORKDIR /var/www/html
COPY --from=composer /app .
And the new docker-compose.yml:
version: "3.7"
services:
webserver:
build: .
tty: true
ports:
- "80:80"
- "443:443"