I'm having trouble demonstrating that data I generate on a shared volume is persistent, and I can't figure out why. I have a very simple docker-compose file:
version: "3.9"
# Define network
networks:
sorcernet:
name: sorcer_net
# Define services
services:
preclean:
container_name: cleaner
build:
context: .
dockerfile: DEESfile
image: dees
networks:
- sorcernet
volumes:
- pgdata:/usr/share/appdata
#command: python run dees.py
process:
container_name: processor
build:
context: .
dockerfile: OASISfile
image: oasis
networks:
- sorcernet
volumes:
- pgdata:/usr/share/appdata
volumes:
pgdata:
name: pgdata
Running the docker-compose file to keep the containers running in the background:
vscode ➜ /com.docker.devenvironments.code $ docker compose up -d
[+] Running 4/4
⠿ Network sorcer_net Created
⠿ Volume "pgdata" Created
⠿ Container processor Started
⠿ Container cleaner Started
Both images are running:
vscode ➜ /com.docker.devenvironments.code $ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
oasis latest e2399b9954c8 9 seconds ago 1.09GB
dees latest af09040befd5 31 seconds ago 1.08GB
and the volume shows up as expected:
vscode ➜ /com.docker.devenvironments.code $ docker volume ls
DRIVER VOLUME NAME
local pgdata```
Running the docker container, I navigate to the volume folder. There's nothing in the folder -- this is expected.
vscode ➜ /com.docker.devenvironments.code $ docker run -it oasis
[root#049dac037802 opt]# cd /usr/share/appdata/
[root#049dac037802 appdata]# ls
[root#049dac037802 appdata]#
Since there's nothing in the folder, I create a file in called "dog.txt" and recheck the folder contents. The file is there. I exit the container.
[root#049dac037802 appdata]# touch dog.txt
[root#049dac037802 appdata]# ls
dog.txt
[root#049dac037802 appdata]# exit
exit
To check the persistence of the data, I re-run the container, but nothing is written to the volume.
vscode ➜ /com.docker.devenvironments.code $ docker run -it oasis
[root#1787d76a54b9 opt]# cd /usr/share/appdata/
[root#1787d76a54b9 appdata]# ls
[root#1787d76a54b9 appdata]#
What gives? I've tried defining the volume as persistent, and I know each of the images have a folder location at /usr/share/appdata.
If you want to check the persistence of the data in the containers defined in your docker compose, the --volumes-from flag is the way to go
When you run
docker run -it oasis
This newly created container shares the same image, but it doesn't know anything about the volumes defined.
In order to link the volume to the new container run this
docker run -it --volumes-from $CONTAINER_NAME_CREATED_FROM_COMPOSE oasis
Now this container shares the volume pgdata.
You can go ahead and create files at /usr/share/appdata and validate their persistence
Related
After reading the docker docs on volumes and how to use them in docker-compose, I believed this minimal example would work to mount a local directory as a volume:
docky/Dockerfile
FROM alpine
ENTRYPOINT ["ash"]
docky/docker-compose.yml
version: '3'
services:
server:
build:
context: .
dockerfile: Dockerfile
volumes:
- ./bindme:/bound
But the volume isn't mounted:
$ docker-compose build
Building server
Step 1/2 : FROM alpine
---> b14afc6dfb98
Step 2/2 : ENTRYPOINT ["ash"]
---> Using cache
---> 68fcd94074c9
Successfully built 68fcd94074c9
Successfully tagged docky_server:latest
$ docker run -it 68fcd94074c9
/ # ls /bound
ls: /bound: No such file or directory
$ docker inspect 68fcd94074c9 | grep -e Volumes -e Binds -e Mounts
"Volumes": null,
"Volumes": null,
The directory ./bindme exists and contains a file. What am I missing/doing wrong?
Docker version 20.10.0, build 7287ab3
docker-compose version 1.25.0
You used Docker Compose to build your image, but you run it with Docker command, so your container configuration isn't used at all for runtime.
Use docker-compose up -d to launch your containers using your Docker Compose configuration. Then you shall have the volumes properly mounted :) !
Beginner mistake: I was not running compose with the options to keep stdin open as a TTY, so the container was exiting immediately, and I misread what was happening. The solution was just to add:
server:
stdin_open: true
tty: true
I am trying to run bamboo on server using docker containers. When i running on local machine work normally and volume save datas successfully. But when i run same docker compose file on server, volume data not save my datas.
docker-compose.yml
version: '3.2'
services:
bamboo:
container_name: bamboo-server_test
image: atlassian/bamboo-server
volumes:
- ./volumes/bamboo_test_vol:/var/atlassian/application-data/bamboo
ports:
- 8085:8085
volumes:
bamboo_test_vol:
Run this compose file on local machine
$ docker-compose up -d
Creating network "test_default" with the default driver
Creating volume "test_bamboo_test_vol" with default driver
Creating bamboo-server_test ... done
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
916c98ca1a9d atlassian/bamboo-server "/entrypoint.sh" 24 minutes ago Up 24 minutes 0.0.0.0:8085->8085/tcp, 54663/tcp bamboo-server_test
$ ls
docker-compose.yml volumes
$ cd volumes/bamboo_test_vol/
$ ls
bamboo.cfg.xml logs
localhost:8085
Run this compose file on server
$ ssh <name>#<ip_address>
password for <name>:
$ docker-compose up -d
Creating network "test_default" with the default driver
Creating volume "test_bamboo_test_vol" with default driver
Creating bamboo-server_test ... done
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
38b77e1b736f atlassian/bamboo-server "/entrypoint.sh" 12 seconds ago Up 11 seconds 0.0.0.0:8085->8085/tcp, 54663/tcp bamboo-server_test
$ ls
docker-compose.yml volumes
$ cd volumes/
$ cd bamboo_test_vol/
$ ls
$ # VOLUME PATH IS EMPTY
server_ip:8085
I didn't have this problem when I tried the same process for jira-software. Why can't it work through the bamboo server even though I use the exact same compose file?
I had the same problem when I wanted to upgrade my Bamboo server instance with my mounted host volume for the bamboo-home directory.
The following was in my docker-compose file:
version: '2.2'
bamboo-server:
image: atlassian/bamboo-server:${BAMBOO_VERSION}
container_name: bamboo-server
environment:
TZ: 'Europe/Berlin'
restart: always
init: true
volumes:
- ./bamboo/bamboo-server/data:/var/atlassian/application-data/bamboo
ports:
- "8085:8085"
- "54663:54663"
When i started with docker-compose up -d bamboo-server, the container never took the files from the host system. So I tried it first without docker-compose, following the instructions of Atlassian Bamboo with the following command:
docker run -v ./bamboo/bamboo-server/data:/var/atlassian/application-data/bamboo --name="bamboo-server" --init -d -p 54663:54663 -p 8085:8085 atlassian/bamboo-server:${BAMBOO_VERSION}
The following error message was displayed:
docker: Error response from daemon: create ./bamboo/bamboo-server/data: "./bamboo/bamboo-server/data" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If you intended to pass a host directory, use absolute path.
So I converted the error message and took the absolute path:
docker run -v /var/project/bamboo/bamboo-server/data:/var/atlassian/application-data/bamboo --name="bamboo-server" --init -d -p 54663:54663 -p 8085:8085 atlassian/bamboo-server:${BAMBOO_VERSION}
After the successful start, I switched to the docker container via SSH and all files were as usual in the docker directory.
I transferred the whole thing to the docker-compose file and took the absolute path in the volumes section. Subsequently it also worked with the docker-compose file.
My docker-compose file then looked like this:
[...]
init: true
volumes:
- /var/project/bamboo/bamboo-server/data:/var/atlassian/application-data/bamboo
ports:
[...]
Setting up a containerized Bamboo Server is not supported for these reasons;
Repository-stored Specs (RSS) are no longer processed in Docker by default. Running RSS in Docker was not possible because;
there is no Docker capability added on the Bamboo server by default,
the setup would require running Docker in Docker.
Short version:
I want to add files in a docker container in docker-compose or Dockerfile and I want to make it accessible from other containers that I made in docker-compose file. How can I do that?
Long version:
I have a Python app in a container that uses a .csv file to generate a POJO machine learning model.
I also have a Java app in a container that uses the POJO machine learning model and appends the .csv file. The java app has a fileWatcher() method implemented.
The containers are made from the docker-compose file that calls Dockerfiles for each one of them. So I want to add them this way and not with CMD docker commands.
You can add the same named volume to different containers:
docker volume create --name volume_data
docker run -t -i -v volume_data:/public debian:jessie /bin/bash
docker run -t -i -v volume_data:/public2 debian:jessie /bin/bash
or as docker-compose.yml
services:
assets:
image: any_asset_image
volumes:
- assets:"/public/assets"
proxy:
image: nginx
volumes:
- assets
volumes:
- assets
I'm executing two docker containers using docker compose.
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
eef95ca1b59b gogent_qaf "/bin/sh -c ./slave.s" 14 seconds ago Up 12 seconds 4242/tcp, 7000-7005/tcp, 9999/tcp, 0.0.0.0:30022->22/tcp coreqafidm_qaf_1
a01373e893eb gogent_master "/bin/sh -c ./master." 15 seconds ago Up 13 seconds 4242/tcp, 0.0.0.0:27000->7000/tcp, 0.0.0.0:27001->7001/tcp, 0.0.0.0:27002->7002/tcp, 0.0.0.0:27003->7003/tcp, 0.0.0.0:29999->9999/tcp coreqafidm_master_1
When I try to use:
docker exec -it coreqafidm_qaf_1 /bin/bash
I get the error:
docker exec -it coreqafidm_qaf_1 /bin/bash
no such file or directory
Here is the docker-compose file:
version: '2'
services:
master:
image: gogent_master
volumes:
- .:/d1/apps/qaf
- ./../core-idm-gogent/:/d1/apps/gogent
ports:
- "27000-27003:7000-7003"
- "29999:9999"
build:
context: .
dockerfile: Dockerfile.master
qaf:
image: gogent_qaf
ports:
- "30022:22"
volumes:
- .:/d1/apps/qaf
- ./../core-idm-gogent/:/d1/apps/gogent
depends_on: [master]
build:
context: .
dockerfile: Dockerfile.qaf
Both Docker files involved have as their last WORKDIR command:
WORKDIR /d1/apps/qaf
If there is a REAL directory /d1/apps/qaf on the machine's natural file system docker exec works, to some degree. It will open up a shell. However, the mapped in volumes are not available to this shell and the files I see are the ones in the real, natural directory, not what should be the mapped in volume.
$ mkdir /d1/apps/qaf
$ docker exec -it coreqafidm_qaf_1 /bin/bash
root#eef95ca1b59b:/d1/apps/qaf#
root#eef95ca1b59b:/d1/apps/qaf# ls /d1/apps/gogent
ls: cannot access /d1/apps/gogent: No such file or directory
The volumes work correctly from within the docker-compose context. I have scripts executing in their and they work. It's just docker exec that fails to see the volumes.
The error stems from a the container not finding /bin/bash, hence the no such file or directory error. The docker exec works fine though.
Try with /bin/sh.
Well, I installed docker-compose etc. on a different machine and this problem was not there. Go figure. This is just one of those things I don't have time to track down.
I realize other people have had similar questions but this uses v2 compose file format and I didn't find anything for that.
I want to make a very simple test app to play around with MemSQL but I can't get volumes to not get deleted after docker-compose down. If I've understood Docker Docs right, volumes shouldn't be deleted without explicitly telling it to. Everything seems to work with docker-compose up but after going down and then up again all data gets deleted from the database.
As recommended as a good practice, I'm using separate memsqldata service as a separate data layer.
Here's my docker-compose.yml:
version: '2'
services:
app:
build: .
links:
- memsql
memsql:
image: memsql/quickstart
volumes_from:
- memsqldata
ports:
- "3306:3306"
- "9000:9000"
memsqldata:
image: memsql/quickstart
command: /bin/true
volumes:
- memsqldatavolume:/data
volumes:
memsqldatavolume:
driver: local
I realize this is an old and solved thread where the OP was pointing to a directory in the container rather than the volume they had mounted, but wanted to clear up some of the misinformation I'm seeing.
docker-compose down does not remove volumes, you need to run docker-compose down -v if you also want to delete volumes. Here's the help text straight from docker-compose (note the "by default" list):
$ docker-compose down --help
Stops containers and removes containers, networks, volumes, and images
created by `up`.
By default, the only things removed are:
- Containers for services defined in the Compose file
- Networks defined in the `networks` section of the Compose file
- The default network, if one is used
Networks and volumes defined as `external` are never removed.
Usage: down [options]
Options:
...
-v, --volumes Remove named volumes declared in the `volumes` section
of the Compose file and anonymous volumes
attached to containers.
...
$ docker-compose --version
docker-compose version 1.12.0, build b31ff33
Here's a sample yml with a named volume to test and a dummy command:
$ cat docker-compose.vol-named.yml
version: '2'
volumes:
data:
services:
test:
image: busybox
command: tail -f /dev/null
volumes:
- data:/data
$ docker-compose -f docker-compose.vol-named.yml up -d
Creating volume "test_data" with default driver
Creating test_test_1
After starting the container, the volume is initialized empty since the image is empty at that location. I created a quick hello world in that location:
$ docker exec -it test_test_1 /bin/sh
/ # ls -al /data
total 8
drwxr-xr-x 2 root root 4096 May 23 01:24 .
drwxr-xr-x 1 root root 4096 May 23 01:24 ..
/ # echo "hello volume" >/data/hello.txt
/ # ls -al /data
total 12
drwxr-xr-x 2 root root 4096 May 23 01:24 .
drwxr-xr-x 1 root root 4096 May 23 01:24 ..
-rw-r--r-- 1 root root 13 May 23 01:24 hello.txt
/ # cat /data/hello.txt
hello volume
/ # exit
The volume is visible outside of docker and is still there after a docker-compose down:
$ docker volume ls | grep test_
local test_data
$ docker-compose -f docker-compose.vol-named.yml down
Stopping test_test_1 ... done
Removing test_test_1 ... done
Removing network test_default
$ docker volume ls | grep test_
local test_data
Recreating the container uses the old volume with the file still visible inside:
$ docker-compose -f docker-compose.vol-named.yml up -d
Creating network "test_default" with the default driver
Creating test_test_1
$ docker exec -it test_test_1 /bin/sh
/ # cat /data/hello.txt
hello volume
/ # exit
And running a docker-compose down -v finally removes both the container and the volume:
$ docker-compose -f docker-compose.vol-named.yml down -v
Stopping test_test_1 ... done
Removing test_test_1 ... done
Removing network test_default
Removing volume test_data
$ docker volume ls | grep test_
$
If you find your data is only being persisted if you use a stop/start rather than a down/up, then your data is being stored in the container (or possibly an anonymous volume) rather than your named volume, and the container is not persistent. Make sure the location for your data inside the container is correct to avoid this.
To debug where data is being stored in your container, I'd recommend using docker diff on a container. That will show all of the files created, modified, or deleted inside that container which will be lost when the container is deleted. E.g.:
$ docker run --name test-diff busybox \
/bin/sh -c "echo hello docker >/etc/hello.conf"
$ docker diff test-diff
C /etc
A /etc/hello.conf
You are using docker-compose down and if you look at the docs here
Stop containers and remove containers, networks, volumes, and images
created by up. Only containers and networks are removed by default.
You are right, it should not remove volumes (by default). It may be a bug or you may have changed the default configuration. But I think the right command for you is docker-compose stop. I will try to make some tests with simplier cases for down command.
This was traced back to a bad documentation from MemSQL. MemSQL data path in memsql/quickstart container is /memsql and not /var/lib/memsql like in a stand-alone installation (and in MemSQL docs), and definitely not /data like somebody told me.
Not sure if this helps or not. When you use docker-compose up -d the container is downloaded and images created. To stop the docker images, use docker-compose down, and the images will remain and can be restarted with docker-compose start
I was using the up/down commands and kept losing my data until I tried the stop/start and now data persists.
The simplest solution is to use docker-compose stop instead of docker-compose down. And then docker-compose start to restart.
According to the docs, down "stops containers and removes containers, networks, volumes, and images created by up."