I am trying to sync a directory on my local machine to a docker container. I want to be able to read and modify a directory on my local machine from a docker container and those changes be seen on the local machine. Adding a relative path in the volumes section of the docker-compose.yml file works, but a direct path does not. Here is what I have
volumes:
- ../../logs:/opt/airflow/logs
but what I want to do is
volumes:
- /Users/bob/airflow/logs:/opt/airflow/logs
Go to your logs directory on your local machine and pwd to see if you have mistyped the absolute path.
Related
Let's say, for example, I have a python script that I want to run as task A. It is located in C:\Users\Name\Desktop\Folder1\Script.py
I also want to move some csv files to C:\Users\Name\Desktop\Folder\CSVs through python script.
How do I map those 2 folders through volume? My compose.yaml currently looks like this:
volumes:
- ./dags:/opt/airflow/dags
- ./logs:/opt/airflow/logs
- ./plugins:/opt/airflow/plugins
When I try to indicate the local folder path as is, it's giving me "no such directory" error.
Figured I have to indicate an absolute path to any local directory on Windows to map it on container.
For example if I need to use this folder C:\Users\Name\Desktop\Folder\CSVs, map it with the airflow container by modifying it in docker-compose.yaml, then compose up to apply changes
volumes:
/c/Users/Name/Desktop/Folder/CSVs:/opt/airflow/CSVs
I'm running Jenkins in a Docker container. Following this article, I'm bind mounting the Docker socket in order to interact with it from the dockerized Jenkins. I'm also bind mounting the container directory jenkins_home. Here is a quick recap on my volumes:
# Jenkins
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- /usr/local/bin/docker-compose:/usr/local/bin/docker-compose
- ./bar:/var/jenkins_home
I run this from the directory /home/foo/ of the host, therefore the following directory is created in the host file system (and mounted):
/home/foo/bar
Now, I have a Jenkins pipeline (mypipe) that runs a docker-compose file spinning up a MySQL container with the following volume:
# MySQL created from Jenkins
volumes:
- ./data:/var/lib/mysql
Weirdly enough, it ends up mounting:
/var/jenkins_home/workspace/mypipe/data < /var/lib/mysql
instead of:
/home/foo/bar/workspace/mypipe/data < /var/lib/mysql
Here is a graphical recap:
Searching stackoverflow, it turned out that it happens since:
The volume source path (left of :) does not refer to the middle container, but to the host filesystem!
And that's ok, but my question is:
Why there?
I mean why does .data is translated exactly into the path: /var/jenkins_home/workspace/…/data, since the MySQL container is not aware of the path /var/jenkins_home?
When Docker creates a bind mount, it is always from an absolute path in the host filesystem to an absolute path in the container filesystem.
When your docker-compose.yml names a relative path, Compose first expands that path before handing it off to the Docker daemon. In your example, you're trying to bind-mount ./bar from a file /var/jenkins_home/workspace/mypipe/docker-compose.yml, so Compose fills in the absolute path you see when it invokes the Docker API. Compose has no idea that the current directory is actually a bind-mount from a different path in the Docker daemon's context.
If you look in the Jenkins logs at what scripted pipeline invocations like docker.inside { ... } do, mounts the workspace directory to an identical path inside the container it launches. Probably the easiest way to work around the mapping problem you're having is to use an identical /var/jenkins_home path on the host system, so the filesystem path is the same in every context.
I need a way to configure docker-compose to create a volume if it's missing, or in case it exists, use it.
I need it to be persistent between versions, but I cannot assure it'll be configured upon initial configuration.
volumes:
my_volume:
external: true
I need to mount docker volume and not host directory.
something like:
-v my_volume:/my_files
what's the best solution for such use-case?
You can use volume for each application or services you set in docker-compose file. For instance, I set a volume for my nginx server as like.
volumes:
- ./web/public:/srv/www/static
- ./default.conf:/etc/nginx/conf.d/default.conf
The left side before colon are the path of the files or folder I want to store inside my docker image as volume whereas on right side I wrote the path where the files will be stored.
When you build the file for first time it will create volume if in case if doesn't exist or use existing volume if it exist
hope this helps.
It's probably a very simple thing. I just can't seem to get an answer by googling. I keep getting error when using Docker-Compose to mount a folder 2 level down of my current folder as a volume into a container.
I'm running Docker on Windows 10. I got two containers. The first one does some web scraping and generates a table in html and dump it into the mounted volume. The 2nd container runs flask and present the html through web portal.
By default the flask container loads html files from its templates folder. So for the scraper container, I tries to mount a volume points to that templates folder. But as soon as I put more than 2 levels of path in the mount path, the container will just fail to run with this error:
invalid mount config for type "volume": invalid mount path: 'static' mount path must be absolute
I then tried put in the absolute path as indicated, but getting the same error.
volumes:
- 'C:/shared/project/testProject/webfront/templates:/usr/src/app/data'
Here is docker-compose file snippet
scraper:
build: './scraper'
env_file:
- '.env'
volumes:
- './webfront/templates:/usr/src/app/data'
Here's the folder structure
./
├───scraper
└───webfront
├───templates
I have tested mounting the volume with docker run command, and was able to mount the volume successfully after provided the absolute path. So it appears this is an issue/bug to do with docker-compose only. I will try raise a issue in their Github repo.
I want to convert my docker-compose.yml to a tutum stackfile.
In the docker-compose.yml I'm using a relative path for the volume:
web
volumes:
- './web:/web'
Which accomplishes, that the local folder ./web, where the docker-compose.yml resides, is added as a volume to the web docker service.
stackfiles from tutum on the other hand only allow absolute paths.
Changing it to
web
volumes:
- '/web:/web'
has the result, that my Dockerfile can't find the folder /web anymore.
How to accomplish this with a stackfile?
Thanks!
You can't use volumes anymore because the host running the container isn't going to have access to your source code. You need to build the image with the source code inside it using COPY or ADD in the Dockerfile.