I'm running https://github.com/jupyterhub/jupyterhub-deploy-docker on a VM. I correctly set up everything, but now I need the spawned notebook to contain a shared folder with some data that I have on the VM.
I edited the docker-compose file in order to have the "shared" folder initially on the VM also present on the JupyterHub container, adding the last line in this snippet:
volumes:
# Bind Docker socket on the host so we can connect to the daemon from
# within the container
- "/var/run/docker.sock:/var/run/docker.sock:rw"
# Bind Docker volume on host for JupyterHub database and cookie secrets
- "data:${DATA_VOLUME_CONTAINER}"
#Add a "shared" folder on the VM to the JupyterHub container
- "./shared:/shared"
But then I don't know how to link that folder to the notebook. I tried with
c.DockerSpawner.volumes = { 'jupyterhub-user-{username}': notebook_dir, '/shared': {"bind": '/home/jovyan/work/shared', "mode": "ro"}}
inside the jupyterhub_config.py, but nothing happens.
Any suggestions?
Related
I've docker container build from debian:latest image.
I need to execute a bash script that will start several services.
My host machine is Windows 10 and I'm using Docker Desktop, I've found configuration files in
docker-desktop-data wsl2 drive in data\docker\containers\<container_name>
I've 2 config files there:
config.v2.json and hostcongih.json
I've edited the first of them and replaced:
"Entrypoint":null with "Entrypoint":["/bin/bash", "/opt/startup.sh"]
I have done it while the container was down, when I restarted it the script was not executed. When I opened config.v2.json file again the Entrypoint was set to null again.
I need to run this script at every container start.
Additional strange thing is that this container doesn't have any volume appearing in docker desktop. I can checkout this container and start another one, but I need to preserve current state of this container (installed packages, files, DB content). How can I change the entrypoint or run the script in other way?
Is there anyway to export the container to image alongside with it's configuration? I need to expose several ports and run the startup script. Is there anyway to make every new container made from the image exported from current container expose the same ports and run same startup script?
Docker's typical workflow involves containers that only run a single process, and are intrinsically temporary. You'd almost never create a container, manually set it up, and try to persist it; instead, you'd write a script called a Dockerfile that describes how to create a reusable image, and then launch some number of containers from that.
It's almost always preferable to launch multiple single-process containers than to try to run multiple processes in a single container. You can use a tool like Docker Compose to describe the multiple containers and record the various options you'd need to start them:
# docker-compose.yml
# Describe the file version. Required with the stable Python implementation
# of Compose. Most recent stable version of the file format.
version: '3.8'
# Persistent storage managed by Docker; will not be accessible on the host.
volumes:
dbdata:
# Actual containers.
services:
# The database.
db:
# Use a stock Docker Hub image.
image: postgres:15
# Persist its data.
volumes:
- dbdata:/var/lib/postgresql/data
# Describe how to set up the initial database.
environment:
POSTGRES_PASSWORD: passw0rd
# Make the container accessible from outside Docker (optional).
ports:
- '5432:5432' # first port any available host port
# second port MUST be standard PostgreSQL port 5432
# Reverse proxy / static asset server
nginx:
image: nginx:1.23
# Get static assets from the host system.
volumes:
- ./static:/usr/share/nginx/html
# Make the container externally accessible.
ports:
- '8000:80'
You can check this file into source control with your application. Also consider adding a third container that build: an image containing the actual application code; that probably will not have volumes:.
docker-compose up -d will start this stack of containers (without -d, in the foreground). If you make a change to the docker-compose.yml file, re-running the same command will delete and recreate containers as required. Note that you are never running an unmodified debian image, nor are you manually running commands inside a container; the docker-compose.yml file completely describes the containers, their startup sequences (if not already built into the images), and any required runtime options.
Also see Networking in Compose for some details about how to make connections between containers: localhost from within a container will call out to that same container and not one of the other containers or the host system.
I have mounted a volume shared to my service main.
Now I am trying to mount that same volume to another container client, that is started with docker-compose up client from within the main container (Docker-in-Docker):
version: "3.8"
# set COMPOSE_PROJECT_NAME=default before running `docker-compose up main`
services:
main:
image: rbird/docker-compose:python-3.9-slim-buster
privileged: true
entrypoint: docker-compose up client # start client
volumes:
- //var/run/docker.sock:/var/run/docker.sock
- ./docker-compose.yml:/docker-compose.yml
- ./shared:/shared
client:
image: alpine
entrypoint: sh -c "ls shared*"
profiles:
- do-not-run-directly
volumes:
- /shared:/shared1
- ./shared:/shared2
The output I get is:
[+] Running 2/2
- Network test_default Created 0.0s
- Container test_main_1 Started 0.9s
Attaching to main_1
Recreating default_client_1 ... done
Attaching to default_client_1
main_1 | client_1 | shared1:
main_1 | client_1 |
main_1 | client_1 | shared2:
main_1 | default_client_1 exited with code 0
main_1 exited with code 0
So the folders /shared2 and /shared2 are empty, although they contain files in the host directory as well as in the main container.
How do I re-share volumes between containers?
Or is there a way to share a host directory between all containers, even the ones started by one of the containers?
The cleanest answer here is to delete the main: container and the profiles: block for the client: container, and run docker-compose on the host directly.
The setup you have here uses the host's Docker socket. (It is not "Docker-in-Docker"; that setup generally is the even more confusing case of running a second Docker daemon in a container.) This means that the Docker Compose instance inside the container sends instructions to the host's Docker daemon telling it what containers to start. You're mounting the docker-compose.yml file in the container's root directory, so the ./shared path is interpreted relative to / as well.
This means the host's Docker daemon is receiving a request to create a container with /shared mounted on /shared1 inside the new container, and also with /shared (./shared, relative to the path /) mounted on /shared2. The host's Docker daemon creates this container using host paths. If you look on your host system, you will probably see an empty /shared directory in the host filesystem root, and if you create files there they will appear in the new container's /shared1 and /shared2 directories.
In general, there is no way to mount one container's filesystem to another. If you're trying to run docker (or docker-compose) from a container, you have to have external knowledge of which of your own filesystems are volumes mounts and what exactly has been mounted.
If you can, avoid both the approaches of containers launching other containers and of sharing volumes between containers. If it's possible to launch another container, and that other container can mount arbitrary parts of the host filesystem, then you can pretty trivially root the entire host. In addition to the security concerns, the path complexities you note here are difficult to get around. Sharing volumes doesn't work well in non-Docker environments (in Kubernetes, for example, it's hard to get a ReadWriteMany volume and containers generally won't be on the same host as each other) and there are complexities around permissions and having multiple readers and writers on the same files.
Instead, launch docker and docker-compose commands on the host only (as a privileged user on a non-developer system). If one container needs one-way publishing of read-only content to another, like static assets, create a custom image COPY --from= one image to the other. Otherwise consider using purpose-built network-accessible storage (like a database) that doesn't specifically depend on a filesystem and knows how to handle concurrent access.
I am trying to setup a gitea container and while checking the official docs, for the volumes sections, the following is defined:
volumes:
- ./gitea:/data
- /etc/timezone:/etc/timezone:ro
- /etc/localtime:/etc/localtime:ro
I know that the volumes section is used to configure DB in docker-compose but I could not find why this specific configuration is done here. Can someone explain to me what do we achieve with the lines added here in the volumes section?
To be more specific, what do we achieve with ./gitea:/data, /etc/timezone:/etc/timezone:ro and /etc/localtime:/etc/localtime:ro and why is this needed?
Thanks.
The volume section is a way to share files and directories between the host system and the container. With :ro the shared files can be made read-only to the container.
One must understand, that a container is just a snapshot of a current build from e.g. docker hub. Whenever you delete this container, all data are deleted too.
So volumes are also used to create a place for data which shall be persistent and not affected by the removal of the container.
So what happens here:
With /etc/timezone:/etc/timezone:ro the file /etc/timezone on the host system (where the docker daemon is running on) is made available under /etc/timezone (:ro means readonly) inside the container. And the same for /etc/localtime.
Those files define the timezone used on the host. By sharing it with the container it can be used inside to recognize the systems timezone.
Now about the line ./gitea:/data.
The same way you can share files you also can share directories. In your case it's expected, that in whatever directory you are currently in, there is a folder gitea (./ means >here<). And if your execute the docker command the folder ./getea on the host is mapped to /data inside the container.
So when you start the container, the apps inside the container will write the data to /data - and you would be also able to access those data on the host under ./gitea.
I'm running Jenkins in a Docker container. Following this article, I'm bind mounting the Docker socket in order to interact with it from the dockerized Jenkins. I'm also bind mounting the container directory jenkins_home. Here is a quick recap on my volumes:
# Jenkins
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- /usr/local/bin/docker-compose:/usr/local/bin/docker-compose
- ./bar:/var/jenkins_home
I run this from the directory /home/foo/ of the host, therefore the following directory is created in the host file system (and mounted):
/home/foo/bar
Now, I have a Jenkins pipeline (mypipe) that runs a docker-compose file spinning up a MySQL container with the following volume:
# MySQL created from Jenkins
volumes:
- ./data:/var/lib/mysql
Weirdly enough, it ends up mounting:
/var/jenkins_home/workspace/mypipe/data < /var/lib/mysql
instead of:
/home/foo/bar/workspace/mypipe/data < /var/lib/mysql
Here is a graphical recap:
Searching stackoverflow, it turned out that it happens since:
The volume source path (left of :) does not refer to the middle container, but to the host filesystem!
And that's ok, but my question is:
Why there?
I mean why does .data is translated exactly into the path: /var/jenkins_home/workspace/…/data, since the MySQL container is not aware of the path /var/jenkins_home?
When Docker creates a bind mount, it is always from an absolute path in the host filesystem to an absolute path in the container filesystem.
When your docker-compose.yml names a relative path, Compose first expands that path before handing it off to the Docker daemon. In your example, you're trying to bind-mount ./bar from a file /var/jenkins_home/workspace/mypipe/docker-compose.yml, so Compose fills in the absolute path you see when it invokes the Docker API. Compose has no idea that the current directory is actually a bind-mount from a different path in the Docker daemon's context.
If you look in the Jenkins logs at what scripted pipeline invocations like docker.inside { ... } do, mounts the workspace directory to an identical path inside the container it launches. Probably the easiest way to work around the mapping problem you're having is to use an identical /var/jenkins_home path on the host system, so the filesystem path is the same in every context.
I am trying to create a jenkins and nexus integration using docker compose file. Where in my jenkins updated with few plugins using Dockerfile and volume created under /var/lib/jenkins/.
VOLUME ["/var/lib/jenkins/"]
in compose file am trying to map my volume to local store /opt/jenkins/
jenkins:
build: ./jenkins
ports:
- 9090:8080
volumes:
- /opt/jenkins/:/var/lib/jenkins/
But Nothing is copying to my persistence directory(/opt/jenkins/).
I can see in all my jenkins jobs created under _data/jobs/ directory under some volume. not in my volume defined /var/lib/jenkins/
Can any one help me on this why this is happening?
From the documentation:
Volumes are initialized when a container is created. If the container’s base image contains data at the specified mount point, that existing data is copied into the new volume upon volume initialization. (Note that this does not apply when mounting a host directory.)
And in the mount a host directory as data volume:
This command mounts the host directory, /src/webapp, into the container at /webapp. If the path /webapp already exists inside the container’s image, the /src/webapp mount overlays but does not remove the pre-existing content. Once the mount is removed, the content is accessible again. This is consistent with the expected behavior of the mount command.
So basically you are overlaying (hiding) anything that was in var/lib/jenkins. Can your image function if those things are hidden?