I'm having a bit of bother with the following docker compose setup:
version: '2.4'
services:
graph_dev:
image: neo4j:3.5.14-enterprise
container_name: amlgraph_dev
user: 100:101
ports:
- 7474:7474
- 7473:7473
- 7687:7687
volumes:
- neo4jbackup:/backup
- neo4jdata:/data
volumes:
neo4jbackup:
neo4jdata:
I would like to run the neo4j-admin command, which must be run as the user 100 (_apt). However, the volume I need to backup to neo4jbackup, is mounted as root and _apt can't write there.
How do I create a volume that _apt can write to? The user _apt:neo4j obviously does not exist on the host. There are no users for which I have root on the docker image.
I can think of two options,
run neo4j docker container as a valid LINUX user and group and give that user access to a backup folder. Here is what my script looks like (I don't use compose currently) to run neo4j in docker under the current user
docker run
--user "$(id -u):$(id -g)"
Here is an article that covers doing the same thing with compose
https://medium.com/faun/set-current-host-user-for-docker-container-4e521cef9ffc
(hacky?) but you could run neo4j-admin outside docker, or in another container in a process that does have access to the backup volume? (I hear you want to run it as root?)
but of course I'm wondering why the backup process or db backup would be owned by root (as opposed to owned by a db owner or backup account...) Personally I feel it is best practice to avoid using root account, whenever possible.
I ended up solving this problem by running the command as _apt as required (docker-compose run graph_dev) and the using docker exec -it -u neo4j:neo4j graph_dev /bin/bash to copy the file over to the backup directory. Not elegant but works.
Related
I have a docker-compose which runs 3 containers:
selenium/hub
selenium/node-chrome
My own image of a java program that uses the 2 above containers to log into a website, navigate to a download page, click on a check-box, then click on a submit button, that will cause a file to be downloaded.
Everything runs fine on my pc, but on an EC2 instance the chrome node gets the error:
mkdir: cannot create directory '/home/selsuser'
and then other errors trying to create sub-directories.
How can I give a container mkdir permissions?
I would like to run this as an ECS-Fargate task, so I would also need to give a container mkdir permissions within that task.
Thanks
Well,
Thank you for the details. It seems indeed you need rights you do not have. What you can try is to create a user group and share it accross your container.
To do so,
Create a groupe user with a GID that does not already exists (enter id on your terminal to see all the existing GID). We will assume 500 is not already used:
chown :500 Downloads
Then, give the appropriate rights to your new group and make all the subfolders having the right of your created group:
chmod 665 Downloads && chmod g+s Downloads
(If you want to be at ease you can always give full permission, up to you)
Then share the rights with a group created in the container thanks to a Dockerfile (replace <username> and <group_name> by whatever you want:
FROM selenium/node-chrome:3.141.59
RUN addgroup --gid 500 <group_name> &&\
adduser --disabled-password --gecos "" --force-badname --ingroup 500 <username>
USER <username>
Then of course don't forget to edit your docker-compose file:
selenium:
build:
context: <path_to_your_dockerfile>
Hoping it will work :)
(From the author of question)
I do have volume mapping, but I do not think there is any connection there to the problem I have. The problem is the selenium/node-chrome container wants to create the directory. On my pc, there are no problems, on EC2 it causes an error that it cannot create the directory. I assume on EC2 you need root privs to do anything on /home.
Here is the complete docker-compose file:
version: "3"
services:
hub:
image: selenium/hub:3.141.59
ports:
- "4444:4444"
chrome:
image: selenium/node-chrome:3.141.59
shm_size: '1gb'
depends_on:
- hub
environment:
- HUB_HOST=hub
volumes:
- ./Downloads:/home/seluser/Downloads
migros-module:
image: freiburgbill/migros-docker
depends_on:
- chrome
environment:
- HUB_HOST=hub
- BROWSER=chrome
volumes:
- ./migros-output:/usr/share/udemy/test-output
- ./Downloads:/usr/share/udemy/Downloads
Thanks again to Paul Barrie for your input and help to get me looking closer at permissions.
For running the docker-compose file that worked on my pc, but did not work on an EC2 instance, I created a /tmp/download directory and gave it full rights (sudo chmod -R 2775 /tmp/Downloads), then it ran without any problems!
For trying to do the same thing as an ECS-Fargate Task. I created an EFS, attached the EFS to an EC2 instance so I could go into it and set the permissions on the whole EFS (sudo chmod -R 777 /mnt/efs/fs1, where that is the default path connecting the EFS to the EC2). I then created ECS-Fargate Task attaching the EFS as a volume. Then everything worked!
So in summery, the host where the docker-compose is running has to have permissions for writing the file. With Fargate we cannot access the host, so an EFS has to be given permissions for writing the file.
I know there must be a better way of locking down the security to just what is needed, but the open permissions does work.
It would of been good if I could of changed the permissions of the Fargate temporary storage and used the bind mount, but I could not find a way to do that.
Is there better way to avoid folder permission issues when a relative folder is being set in a docker compose file when using manjaro?
For instance, take the bitnami/elasticsearch:7.7.0 image as an example:
This image as an example will always throw the ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/bitnami/elasticsearch/data/nodes]; error.
I can get around in by:
create the data directory with sudo, followed by chmod 777
attaching a docker volume
But I am looking for a bit easier to manage solution, similar to the docker experience in Ubuntu and OSX which I do not have to first create a directory with root in order for folder mapping to work.
I have made sure that my user is in the docker group by following the post install instructions on docker docs. I have no permission issues when accessing docker info, or sock.
docker-compose.yml
version: '3.7'
services:
elasticsearch:
image: bitnami/elasticsearch:7.7.0
container_name: elasticsearch
ports:
- 9200:9200
networks:
- proxy
environment:
- ELASTICSEARCH_HEAP_SIZE=512m
volumes:
- ./data/:/bitnami/elasticsearch/data
- ./config/elasticsearch.yml:/opt/bitnami/elasticsearch/config/elasticsearch.yml
networks:
proxy:
external: true
I am hoping for a more seamless experience when using my compose files from git which works fine in other systems, but running into this permission issue on the data folder on manjaro.
I did check other posts on SO, some some are temporary, like disabling selinux, while other require running docker with the --privileged flag, but I am trying to do with from compose.
This has nothing to do with the Linux distribution but is a general problem with Docker and bind mounts. A bind mount is when you mount a directory of your host into a container. The problem is that the Docker daemon creates the directory under the user it runs with (root) and the UID/GIDs are mapped literally into the container.
Not that it is advisable to run as root, but depending on your requirements, the official Elasticsearch image (elasticsearch:7.7.0) runs as root and does not have this problem.
Another solution that would work for the bitnami image is to make the ./data directory owned by group root and group writable, since it appears the group of the Elasticsearch process is still root.
A third solution is to change the GID of the bitnami image to whatever group you had the data created with and make it group writable.
I am trying to wrap my mind around Docker volumes but I must have some things missing to understand it.
Let's say I have a Python app that require some initialisation depending on env variables. What I'm trying to achieve is having a "Code only image" from which I can start containers that would be mounted at executions. The entrypoint script of the Main container will then read and generate some files from/on the Code only container.
I tried to create an image to have a copy of the code
FROM ubuntu
COPY ./app /usr/local/code/app
Then docker create --name code_volume
And with docker-compose:
app:
image: python/app
hostname: app
ports:
- "2443:443"
environment:
- ENV=stuff
volumes_from:
- code_volume
I get an error from the app container saying it can't find a file in /usr/local/code/app/src but when I run code_volume with bash then ls into the folder, the file is sitting there...
I tried to change access rights, add /bin/true (seeing it in some examples) but I just can't get what I want to be working. I checked the docker volume create feature but it seems to be for storing/sharing data afterward
What am I missing ? Is the entrypoint script executed before volumes are mounted ? Is there any best practices for cases like this that don't involve mounting folders and keeping one copy for every container ? Should I be thinking my containers over again ?
You do not declare the volume on code_volume container upon creation.
docker create -v /usr/local/code/app --name code_volume
I have installed OrientDB inside Docker.I want to run scripts inside the bin folder .But I am not able to find any way to access the directory of OrientDB like a normal explorer. Is there any way I can use the Docker installation like a local installation so that I can see and interact with all the folders of OrientDB installation.Thanks
If you want to access them inside docker container, you can do this:
start the container, then docker exec -i -t CONTAINER_NAME bash or docker exec -i -t CONTAINER_NAME /bin/sh. If bash/sh is installed in this particular image, you will get to the shell and can what you want there.
Another way, and I think it's what you want is to use docker volumes. You map your host path to a container path, so it sees whatever changes you do outside.
map some folder on your host system to the location orientdb expects and it will create files there.
mapping excerpt from docker-compose.yml for mysql:
alldbhost:
ports:
- "3306:3306"
image: percona:5.5
volumes:
- ./etc/timezone:/etc/timezone
- /dev/shm/mysql/:/var/lib/mysql
- ./etc/mysql/:/etc/mysql
- /home/user/temp/mysql_replication:/local/mysql/binlog
environment:
TERM: xterm
actually, orientdb manual provides these instructions:
docker run --name orientdb -d -v <config_path>:/opt/orientdb/config -v <databases_path>:/opt/orientdb/databases -v <backup_path>:/opt/orientdb/backup -p 2424 -p 2480 nesrait/orientdb-2.0
-v <databases_path>:/opt/orientdb/databases is a path on your host system where database files will be located
If you installed orientdb inside some container (ubuntu, for example) you should find orientdb config files, find where it stores databases and, again, map your host directory to container's
I created a rails app in a docker environment and it links to a postgres instance. I edited the
postgres container to add initial data (by running rake db:setup from the rails app). Now I commited the postgres database, but it doesn't seem to remember my data when I create a new container (of the commited postgres image).
Isn't it possible to save data in a commit and then reuse it afterwards?
I used the postgres image: https://registry.hub.docker.com/_/postgres/
The problem is that the postgres Dockerfile declares "/var/lib/postgresql/data" as a volume. This is a just a normal directory that lives outside of the Union File System used by images. Volumes live until no containers link to them and they are explicitly deleted.
You have a few choices:
Use the --volumes-from command to share data with new containers. This will only work if there is only one running postgres image at a time, but it is the best solution.
Write your own Dockerfile which creates the data before declaring the volume. This data will then be copied into the volume when the container is created.
Write an entrypoint or cmd script which populates the database at run time.
All of these suggestions require you to use Volumes to manage the data once the container is running. Alternatively, you could write your own Dockerfile and simply not declare a volume. You could then use docker commit to create a new image after adding data. This will probably work in the short term, but is definitely not how you should work with containers - it isn't repeatable and you will eventually run out of layers in the Union File System.
Have a look at the official Docker docs on managing data in containers for more info.
Create a new Dockerfile and change PGDATA:
FROM postgres:9.2.10
RUN mkdir -p /var/lib/postgresql-static/data
ENV PGDATA /var/lib/postgresql-static/data
You should be all set with the following command. The most important part is the PGDATA location, which should be anything but the default.
docker run -e PGDATA=/var/lib/postgresql/pgdata -e POSTGRES_PASSWORD=YourPa$$W0rd -d postgres
It is not possible to save data during a commit since the data resides on a mount which is specific for that container and will get removed once you run docker rm <container ID> but you can use data volumes to share and reuse data between container and the changes made are directly on the volume.
You can use docker run -v /host/path:/Container/path to mount the volume to the new container.
Please refer to: https://docs.docker.com/userguide/dockervolumes/
For keeping permanent data such as databases, you should define these data volumes as external, therefore it will not be removed or created automatically every time you run docker-compose up or down commands, or redeploy your stack to the swarm.
...
volumes:
db-data:
external: true
...
then you should create this volume:
docker volume create db-data
and use it as data volume for your databse:
...
db:
image: postgres:latest
volumes:
- db-data:/var/lib/postgresql/data
ports:
- 5432:5432
...
In production, there are many factors to consider when using docker for keeping permanent data safely, specially in swarm mode, or in kubernetes cluster.