I've nignx container and one asset container which have all my assets build from grunt or some other tools.
Now in docker compose file, i want to mount asset container's 's folder path into nginx container so nginx can serve that files.
How can we do that? i don't remember but i think there is a option where we can share path of one container with another.
Suppose if i scale up nginx to 2 container then will that mount works for all instance of nginx?
if i scale up asset container then what will happen?
i also want to mount that with my host so development can be done be easily.
What you want to do is use a volume, and then mount that volume into whatever containers you want it to appear in.
Completely within Docker
You can do this completely inside of Docker.
Here is an example (stripped-down - your real file would have much more than this in it, of course).
version: '3'
services:
nginx:
volumes:
- asset-volume:/var/lib/assets
asset:
volumes:
- asset-volume:/var/lib/assets
volumes:
asset-volume:
At the bottom is a single volume defined, named "asset-volume".
Then in each of your services, you tell Docker to mount that volume at a certain path. I show example paths inside the container, just adjust these to be whatever path you wish them to be in the container.
The volume is an independent entity not owned by any particular container. It is just mounted into each of them, and is shared. If one container modifies the contents, then they all see the changes.
Note that if you prefer only one can make changes, you can always mount the volume as read-only in some services, by adding :ro to the end of the volume string.
services:
servicename:
volumes:
- asset-volume:/var/lib/assets:ro
Using a host directory
Alternately you can use a directory on the host and mount that into the containers. This has the advantage of you being able to work directly on the files using your tools outside of Docker (such as your GUI text editor and other tools).
It's the same, except you don't define a volume in Docker, instead mounting the external directory.
version: '3'
services:
nginx:
volumes:
- ./assets:/var/lib/assets
asset:
volumes:
- ./assets:/var/lib/assets
In this example, the local directory "assets" is mounted into both containers using the relative path ./assets.
Using both depending on environment
You can also set it up for a different dev and production environment. Put everything in docker-compose.yml except the volume mounts. Then make two more files.
docker-compose.dev.yml
docker-compose.prod.yml
In these files put only the minimum config to define the volume mount. We'll mix this with the docker-compose.yml to get a final config.
Then use this. It will use the config from docker-compose.yml, and use anything in the second file as an override or supplemental config.
docker-compose -f docker-compose.yml \
-f docker-compose.dev.yml \
up -d
And for production, just use the prod file instead of the dev file.
The idea here is to keep most of the config in docker-compose.yml, and only the minimum set of differences in the alternative files.
Example:
docker-compose.prod.yml
version: '3'
services:
nginx:
volumes:
- asset-volume:/var/lib/assets
docker-compose.dev.yml
version: '3'
services:
nginx:
volumes:
- ./assets:/var/lib/assets
Related
I am writing a docker-compose.yaml file for my project. I have checked the volumes documentation here .
I also understand the concept of volume in docker that I can mount a volume e.g. -v my-data/:/var/lib/db where my-data/ is a directory on my host machine while /var/lib/db is the path inside database container.
My confuse is with the link I put above. There it has the following sample:
version: "3.9"
services:
db:
image: db
volumes:
- data-volume:/var/lib/db
backup:
image: backup-service
volumes:
- data-volume:/var/lib/backup/data
volumes:
data-volume:
I wonder does it mean that I have to create a directory named data-volume on my host machine? What if I have a directory on my machine with path temp/my-data/ and I want to mount that path to the database container /var/lib/db ? Should I do something like below?
version: "3.9"
services:
db:
image: db
volumes:
- temp/my-data/:/var/lib/db
volumes:
temp/my-data/:
My main confusion is the volumes: section at the bottom, I am not sure whether the volume name should be the path of my directory or should be just literally a name I give & if it is the latter case then how could the given name be mapped with temp/my-data/ on my machine? The sample doesn't indicate that & is ambiguous to clarify that.
Could someone please clarify it for me?
P.S. I tried with above docker-compose I guessed, ended up with the error:
ERROR: The Compose file './docker-compose.yaml' is invalid because:
volumes value 'temp/my-data/' does not match any of the regexes: '^[a-zA-Z0-9._-]+$'
Mapped volumes can either be files/directories on the host machine (sometimes called bind mounts in the documentation) or they can be docker volumes that can be managed using docker volume commands.
The volumes: section in a docker-compose file specify docker volumes, i.e. not files/directories. The first docker-compose in your post uses such a volume.
If you want to map a file or directory (like in your last docker-compose file), you don't need to specify anything in the volumes: section.
Docker volumes (the ones specified in the volumes: section or created using docker volume create) are of course also stored somewhere on your host computer, but docker manages that and you shouldn't normally need to know where or what the format is.
This part of the documentation is pretty good about explaining it, I think https://docs.docker.com/storage/volumes/
As #HansKilian mentions, you don't need both volumes and services.volumes. To use services.volumes, map the host directory to the container directory like this:
services:
db:
image: db
volumes:
- /host/path/lib/db:/container/path/lib/db
With that, the directory /host/path/lib/db on the host machine will be used by the container and available at /container/path/lib/db.
Now, if you're like me, I get really confused with fake examples, so let's say the real directory on your host machine is /var/lib/db and you just want to see it at /db when you run a shell in Docker (i.e., docker exec -it /bin/bash container-id).
docker-compose.yaml would look like this:
services:
db:
image: db
volumes:
- /var/lib/db:/db
Now when you run the shell, cd /logs and ls, you'll see the same results as if you'd cd /var/lib/db on the host.
If you want to use the volumes section to indicate a global volume to use, you first have to create that volume using docker volume create. The documentation Hans linked includes steps to do this. The syntax of /host/path:/container/path is replaced by volume-name:/container/path. Then, once defined, you'd alter your docker-compose.yaml to be more like this:
services:
db:
image: db
volumes:
- your-global-volume-name:/db
volumes:
your-global-volume-name:
external: true
Note that I have not tested or used the this configuration. I'm assuming it's correct based on the other method working and the few changes I can identify in the docs.
I have a problem that I just can't understand. I am using docker to run certain containers, but I have problems with at least one Volume, where I't like to ask if anybody can give me a hint what I am doing wrong. I am using Nifi-Ingestion as example, but it affects even more container volumes.
First, let's talk about the versions I use:
Docker version 19.03.8, build afacb8b7f0
docker-compose version 1.27.4, build 40524192
Ubuntu 20.04.1 LTS
Now, let's show the volume in my working docker-compose-file:
In my container, it is configured as followed:
volumes:
- nifi-ingestion-conf:/opt/nifi/nifi-current/conf
Below my docker-compose file it is defined as a normal named volume:
volumes:
nifi-ingestion-conf:
This is a snippet from the docker-compose that I'd like to get working
In my container, it is configured in this case as followed (having my STORAGE_VOLUME_PATH defined as /mnt/storage/docker_data):
volumes:
- ${STORAGE_VOLUME_PATH}/nifi-ingestion-conf:/opt/nifi/nifi-current/conf
On the bottom I guess there is something to do but I don't know what I could need to do here. In this case it is the same as in the working docker-compose:
volumes:
nifi-ingestion-conf:
So, now whats my problem?
I have two docker-compose files. One uses the normal named volumes, and one uses the volumes in my extra mount path. When I run the containers, the volumes seem to work different since files are written in the first style, but not in the second. My mount paths are generated in the second version so there is nothing wrong with my environment variables in the .env-file.
Hint: the /mnt/storage/docker_data is an NFS-mount but my machine has the full privileges on that share.
Here is my fstab-entry to mount that volume (maybe I have to set other options):
10.1.0.2:/docker/data /mnt/storage/docker_data nfs auto,rw
Bigger snippets
Here is a bigger snipped if the docker-compose (i need to cut and remove confident data, my problem is not that it does not work, it is only that the volume acts different. Everything for this one volume is in the code.):
version: "3"
services:
nifi-ingestion:
image: my image on my personal repo
container_name: nifi-ingestion
ports:
- 0000
labels:
- app-specivic
volumes:
- ${STORAGE_VOLUME_PATH}/nifi-ingestion-conf:/opt/nifi/nifi-current/conf
#working: - nifi-ingestion-conf:/opt/nifi/nifi-current/conf
environment:
- app-specivic
networks:
- cnetwork
volumes:
nifi-ingestion-conf:
networks:
cnetwork:
external: false
ipam:
driver: default
config:
- subnet: 192.168.1.0/24
And here of the env (only the value we are using)
STORAGE_VOLUME_PATH=/mnt/storage/docker_data
if i understand your question correctly, you wonder why the following docker-compose snippet works for you
version: "3"
services:
nifi-ingestion:
volumes:
- nifi-ingestion-conf:/opt/nifi/nifi-current/conf
volumes:
nifi-ingestion-conf:
and the following docker-compose snippet does not work for you
version: "3"
services:
nifi-ingestion:
volumes:
- ${STORAGE_VOLUME_PATH}/nifi-ingestion-conf:/opt/nifi/nifi-current/conf
what makes them different is how you use volumes. you need to differentiate between mount host paths and mount named volumes
You can mount a host path as part of a definition for a single service, and there is no need to define it in the top level volumes key.
But, if you want to reuse a volume across multiple services, then define a named volume in the top-level volumes key.
named volumes are managed by docker
If you start a container with a volume that does not yet exist, Docker creates the volume for you.
also, would advise you to read this answer
update:
you might also want to read about docker nfs volumes
I have a Docker Compose setup to run a Nginx serving pictures (jpg files). The port, protocol, etc. is not relevant. My Nginx works correctly and serve those files:
web:
image: myweb
restart: always
volumes:
- ./logs/nginx:/var/log/nginx
ports:
...
The image myweb inherits from nginx and add all the configuration needed by this server, basically, the web root is:
index index.html index.php;
root /var/www/myweb/public;
I'm also using php:7.0-fpm and fastcgi locations but it is not the issue here.
What I would want is to mount a host volume with the pictures (instead of copying them into the docker image) in order to have the possibility of update it externally (Dropbox sync or whatever).
web:
image: myweb
restart: always
volumes:
- ./webcode:/var/www
- ./logs/nginx:/var/log/nginx
- ./images-data/catalog-images:/var/www/myweb/public/catalog/images
ports:
...
The host path webcode (relative to docker-compose.yml location) is the static web, also mounted on /var/www.
The host path images-data/catalog-images (relative to docker-compose.yml location) is the pictures directory, mounted on /var/www/myweb/public/catalog/images.
I'm not sure it this is a good practice (mount a host directory on the same path where other host directory was already mounted). Anyway, I also tried keeping webcode as part of the myweb image having only one mounted volume (the pictures one).
Mounting webcode and including pictures on it, works. And without any volume mount, just including all the stuff (website and pictures), also works. But I would need to have an isolated volume for the pictures.
I also tried symlinking within the container /var/www/myweb/public/catalog/images towards the pictures volume. But Nginx does not render it, perhaps because it it a different volume than /var/www.
Should it belong to the root of nginx server configuration ?
The thing is that, those images are not loaded in the browser. Going into the container, I confirm that the ownership for the volume is correct (www-data:www-data).
From the point of view of the container, ownership and content is the same if I mount the volume or if I copy in the container filesystem. But Nginx only can see the files within the Docker container file system.
Is this the normal behaviour ? Am I missing any Nginx configuration to work normally with host volumes inside the container ?
I think I finally solved.
When I said
I'm also using php:7.0-fpm and fastcgi locations but it is not the
issue here.
I was totally wrong: it was the key: the php execution. The problem was that the pictures was loaded by an index.php. Then it was sent to the fpm service where the pictures were not available !!.
This is my docker-compose.yml definition for the php service:
fpm:
image: php7fpm:latest
restart: always
volumes:
- ./webcode:/var/www
The thing is that the pictures volume was not visible there. I just added such volume to fpm volume list, and problem is solved:
fpm:
image: php7fpm:latest
restart: always
volumes:
- ./webcode:/var/www
- ./images-data/catalog-images:/var/www/myweb/public/catalog/images
Sorry for the inconvenience.
I have for example this service and volume defined in my docker-compose file
postgres:
image: postgres:9.4
volumes:
- db_data:/var/lib/postgresql/data
volumes:
blue_prod_db:
driver: rancher-nfs
Then. if you define a volume inside a Dockerfile like this:
RUN mkdir /stuff
COPY ./stuff/* /stuff/
VOLUME /stuff
How can you later access it through the docker-compose configuration and add it to a container?
When configured in the Dockerfile, a volume will result in any container started from that image, including temporary containers later in the build process from the RUN command, to have a volume defined at the specified location, e.g. /stuff. If you do not define a source for that volume at run time, you will get an anonymous volume created by docker for you at that location. However, you can always define a volume with a source at run time (even without the volume being defined) by specifying the location in your compose file:
version: "3"
services:
app:
image: your_image
volumes:
- data:/stuff
volumes:
data:
Note that there are two volumes sections, one for a specific service that specifies where the volume is mounted inside the container, and another at the top level where you can specify the source of the volume. Without specifying a source, you'll get a local volume driver with a directory under /var/lib/docker bind mounted into the container.
I do not recommend specifying volumes inside the Dockerfile in general, it breaks the ability to extend the image in later steps for child images, and clutters the filesystem with anonymous volumes that are not easy to track back to their origin. It's best to define them at runtime with something like a compose file.
I am having problems with writing files out from inside a docker container to my host computer. I believe this is a privilege issue and prefer not to set privileged: True. A work around for writing out files is by pre-pending ../ to a volume in my docker-compose.yml file. For example,
version: '3'
services:
example:
volumes:
- ../:/example
What exactly is ../ doing here? Is it taking from the container's privileges and "going up" a directory to the host machine? Without ../, I am unable to write out files to my host machine.
Specifying a path as the source, as opposed to a volume name, bind mounts a host path to a path inside the container. In your example, ../ will be visible inside the container at /example on a recent version of docker.
Older versions of docker can only access the directory it is in and lower, not higher, unless you specify the higher directory as the context.
To run the docker build from the parent directory:
docker build -f /home/me myapp/Dockerfile
As opposed to
docker build -f /home/me/myapp Dockerfile
Doing the same in composer:
#docker-compose.yml
version: '3.3'
services:
yourservice:
build:
context: /home/me
dockerfile: myapp/Dockerfile
Or with your example:
version: '3'
services:
build:
context: /home/me/app
dockerfile: docker/Dockerfile
example:
volumes:
- /home/me/app:/example
Additionally you have to supply full paths, not relative paths. Ie.
- /home/me/myapp/files/example:/example
If you have a script that is generating the Dockerfile from an unknown path, you can use:
CWD=`pwd`; echo $CWD
To refer to the current working directory. From there you can append /..
Alternately you can build the image from a directory one up, or use a volume which you can share with an image that is run from a higher directory, or you need to output your file to stdout and redirect the output of the command to the file you need from the script that runs it.
See also: Docker: adding a file from a parent directory
The statement volumes: ['../:/example'] makes the parent directory of the directory containing docker-compose.yml on the host (../) visible inside the container at /example. Host directory bind-mounts like this, plus some equivalent constructs using a named volume attached to a specific host directory, are the only way a container can write out to the host filesystem.