Docker for Mac and mkdir permisions - docker

Using the docker for mac app. Just installed everything yesterday. Finally got the app going.
But I can't run migrations till I install postgis. So I dropped the official postgres dockerhub image for postgis:11-alpine image. But I keep on getting a permission denied issue when docker tries to mkdir for the pg_data volume.
Dockerfile:
version: '3'
# Containers we are going to run
services:
# Our Phoenix container
phoenix:
# The build parameters for this container.
build:
# Here we define that it should build from the current directory
context: .
environment:
# Variables to connect to our Postgres server
PGUSER: postgres
PGPASSWORD: postgres
PGDATABASE: gametime_dev
PGPORT: 5432
# Hostname of our Postgres container
PGHOST: db
ports:
# Mapping the port to make the Phoenix app accessible outside of the container
- "4000:4000"
depends_on:
# The db container needs to be started before we start this container
- db
- redis
redis:
image: "redis:alpine"
ports:
- "6379:6379"
sysctls:
net.core.somaxconn: 1024
db:
# We use the predefined Postgres image
image: mdillon/postgis:11-alpine
environment:
# Set user/password for Postgres
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
# Set a path where Postgres should store the data
PGDATA: /var/lib/postgresql/data/pgdata
restart: always
volumes:
- pgdata:/usr/local/var/postgres_data
# Define the volumes
volumes:
pgdata:
Error I'm getting:
db_1 | mkdir: can't create directory '/var/lib/postgresql/data/pgdata': Permission denied
This does not happen though when using the postgres(official) image. I have googled high and low. I did read something about Docker for Mac running commands on the containers it creates in a VM as the current user's localhost user and not root. But that doesn't make sense to me - how do I get around this, if that's the case?
[Extra note:] - I did try the :z and :Z - still got the exact same error as above.
Appreciate the the time - in advance.

Your environment variables for the db service state that PGDATA is in /var/lib/postgresql/data/pgdata but you are mounting a pgdata volume in the container at /usr/local/var/postgres_data.
My guess is that when postgres starts, it is looking at the env vars and expecting a dir in /var/lib/postgresql/data/pgdata. Since it probably does not exists, it is trying to create it as postgres user which does not have the right to do it.
Use the same path for both vars and I'm quite sure it will fix the error.

Related

Containerizing Cordapp with Docker Image and Docker Compose

When running Corda in docker with external Postgres DB configurations, I get insufficient privileges to access error.
Note:
Corda: 4.6 Postgresql: 9.6
Docker engine 20.10.6
Docker-compose: docker-compose version 1.29.1, build c34c88b2
docker-compose.yml file:
version: '3.3'
services:
partyadb:
hostname: partyadb
container_name: partyadb
image: "postgres:9.6"
environment:
POSTGRES_PASSWORD: postgres
POSTGRES_USER: postgres
POSTGRES_DB: partyadb
ports:
- 5432
partya:
hostname: partya
# image: corda/corda-zulu-java1.8-4.7:RELEASE
image: corda/corda-zulu-java1.8-4.6:latest
container_name: partya
ports:
- 10006
- 2223
command: /bin/bash -c "java -jar /opt/corda/bin/corda.jar run-migration-scripts -f /etc/corda/node.conf --core-schemas --app-schemas && /opt/corda/bin/run-corda"
volumes:
- ./partya/node.conf:/etc/corda/node.conf:ro
- ./partya/certificates:/opt/corda/certificates:ro
- ./partya/persistence.mv.db:/opt/corda/persistence/persistence.mv.db:rw
- ./partya/persistence.trace.db:/opt/corda/persistence/persistence.trace.db:rw
# - ./partya/logs:/opt/corda/logs:rw
- ./shared/additional-node-infos:/opt/corda/additional-node-infos:rw
- ./shared/cordapps:/opt/corda/cordapps:rw
- ./shared/drivers:/opt/corda/drivers:ro
- ./shared/network-parameters:/opt/corda/network-parameters:rw
environment:
- ACCEPT_LICENSE=${ACCEPT_LICENSE}
depends_on:
- partyadb
Error:
[ERROR] 12:41:24+0000 [main] internal.NodeStartupLogging. - Exception during node startup. Corda started with insufficient privileges to access /opt/corda/additional-node-infos/nodeInfo-5B........................................47D
The corda/corda-zulu-java1.8-4.6:latest image runs under the user corda, not root. This user has user id 1000, and also is in a group called corda, also with gid 1000:
corda#5bb6f196a682:~$ id -u corda
1000
corda#5bb6f196a682:~$ groups corda
corda : corda
corda#5bb6f196a682:~$ id -G corda
1000
The problem here seems to be that the file you are mounting into the docker container (./shared/additional-node-infos/nodeInfo-5B) does not have permissions setup in such a way as to allow this user to access it. I'm assuming the user needs read and write access. A very simple fix would be to give other read and write access to this file:
$ chmod o+rw ./shared/additional-node-infos/nodeInfo-5B
There are plenty of other ways to manage this kind of permissions issue in docker, but remember that the permissions are based on uid/gid which usually do not map nicely from your host machine into the docker container.
So the error itself describes that it's a permission problem.
I don't know if you crafted this dockerfile yourself, you may want to take a look at generating them with the dockerform task (https://docs.corda.net/docs/corda-os/4.8/generating-a-node.html#use-cordform-and-dockerform-to-create-a-set-of-local-nodes-automatically)
This permission problem could be that you're setting only read / write within the container:
- ./shared/additional-node-infos:/opt/corda/additional-node-infos:rw
or it could be that you need to change the permissions on the shared folder. Try changing the permissions of shared to 777 and see if that works, then restrict your way back down to permissions you're comfortable with.
I just configure the image to be run as root. This works but may not be safe. Simply add
services:
cordaNode:
user: root
to the service configuration.
Ref: How to configure docker-compose.yml to up a container as root

Docker volume associated to postgres image empty and not persistent

I have a docker-compose file to build a web server with django and a postgres database. It basically looks like that :
version: '3'
services:
server:
build:
context: .
dockerfile: ./docker/server/Dockerfile
image: backend
volumes:
- ./api:/app
ports:
- 8000:8000
depends_on:
- postgres
- redis
environment:
- PYTHONUNBUFFERED=1
postgres:
image: kartoza/postgis:11.0-2.5
volumes:
- pg_data:/var/lib/postgresql/data:rw
environment:
POSTGRES_DB: "gis,backend"
POSTGRES_PORT: "5432"
POSTGRES_USER: "user"
POSTGRES_PASS: "pass"
POSTGRES_MULTIPLE_EXTENSIONS: "postgis,postgis_topology"
ports:
- 5432:5432
redis:
image: "redis:alpine"
volumes:
pg_data:
I'm using a volume to make my data persistent
I managed to run my containers and add data to the database. A volume has successfully been created : docker volume ls
DRIVER VOLUME NAME
local server_pg_data
But this volume is empty as the output of docker system df -v shows:
Local Volumes space usage:
VOLUME NAME LINKS SIZE
server_pg_data 1 0B
Also, if I want or need to build the containers once again using docker-compose down and docker-compose up, data has been purged from my database. Yet, I thought that volumes were used to make data persistent on disk…
I must be missing something in the way I'm using docker and volumes but I don't get what:
why does my volume appears empty while there is some data in my postgres container ?
why does my volume does not persist after doing docker-compose down ?
This thread (How to persist data in a dockerized postgres database using volumes) looked similar but the solution does not seem to apply.
The kartoza/postgis image isn't configured the same way as the standard postgres image. Its documentation notes (under "Cluster Initializations"):
By default, DATADIR will point to /var/lib/postgresql/{major-version}. You can instead mount the parent location like this: -v data-volume:/var/lib/postgresql
If you look at the Dockerfile in GitHub, you will also see that parent directory named as a VOLUME, which has some interesting semantics here.
With the setting you show, the actual data will be stored in /var/lib/postgresql/11.0; you're mounting the named volume on a different directory, /var/lib/postgresql/data, which is why it stays empty. Changing the volume mount to just /var/lib/postgresql should address this:
volumes:
- pg_data:/var/lib/postgresql:rw # not .../data

Where do I specify to Postgres within Docker that it should run and accept connections?

I'm trying to dockerize an existing Rails app that uses Postgresql 9.5 as its database. In my docker-compose.yml. After a successful "docker-compose build" I can run the "docker-compose up" command and see the connection but when I navigate to localhost I get the following error.
PG::ConnectionBad
could not connect to server: No such file or directory Is the server running >locally and accepting connections on Unix domain socket >"/var/run/postgresql/.s.PGSQL.5432"?
Here is what is in my docker-compose.yml
version: '2'
services:
db:
image: postgres:9.5
restart: always
volumes:
- ./tmp/db:/var/lib/postgresql/9.5/main
environment:
POSTGRES_USER: username
POSTGRES_PASSWORD: password
POSTGRES_DB: hardware_development
web:
build: .
command: bash -c "rm -f tmp/pids/server.pid && bundle exec rails s -p 3000 -b '0.0.0.0'"
volumes:
- .:/myapp
ports:
- "3000:3000"
depends_on:
- db
From what I've seen I need to do some specification somewhere in my dockerfile or the docker-compose.yml but I either don't see a change or end up back at the same error.
I've been able to use Docker's own docs to use Docker Compose to create a new rails app with postgres where I see the "yay you're on rails!" page but now with my own code I can't see anything. Running the app outside of docker shows me the test page as well so its not the code within my rails app or the Postgres evnironment outside of Docker.
Your db docker-compose entry isn't exposing any ports. It needs to expose 5432. Add a ports line for that just like you have for web.
Edit: also I don't know why you added restart: always to your database container, but I wouldn't recommend that for rails or pretty much anything.

Unable to connect mysql from docker container?

I have created a docker-compose file it has two services with Go and Mysql. It creates container for go and mysql. Now i am running code which try to connect to mysql database which is running as a docker container. but i get error.
docker-compose.yml
version: "2"
services:
app:
container_name: golang
restart: always
build: .
ports:
- "49160:8800"
links:
- "mysql"
depends_on:
- "mysql"
mysql:
image: mysql
container_name: mysql
volumes:
- dbdata:/var/lib/mysql
restart: always
environment:
- MYSQL_ROOT_PASSWORD=root
- MYSQL_DATABASE=testDB
- MYSQL_USER=root
- MYSQL_PASSWORD=root
ports:
- "3307:3306"
volumes:
dbdata:
Error while connecting to mysql database
golang | 2019/02/28 11:33:05 dial tcp 127.0.0.1:3306: connect: connection refused
golang | 2019/02/28 11:33:05 http: panic serving 172.24.0.1:49066: dial tcp 127.0.0.1:3306: connect: connection refused
golang | goroutine 19 [running]:
Connection with MySql Database
func DB() *gorm.DB {
db, err := gorm.Open("mysql", "root:root#tcp(mysql:3306)/testDB?charset=utf8&parseTime=True&loc=Local")
if err != nil {
log.Panic(err)
}
log.Println("Connection Established")
return db
}
EDIT:Updated docker file
FROM golang:latest
RUN go get -u github.com/gorilla/mux
RUN go get -u github.com/jinzhu/gorm
RUN go get -u github.com/go-sql-driver/mysql
COPY ./wait-for-it.sh .
RUN chmod +x /wait-for-it.sh
WORKDIR /go/src/app
ADD . src
EXPOSE 8800
CMD ["go", "run", "src/main.go"]
I am using gorm package which lets me connet to the database
depends_on is not a verification that MySQL is actually ready to receive connections. It will start the second container once the database container is running regardless it was ready for connections or not which could lead to such an issue with your application as it expects the database to be ready which might not be true.
Quoted from the documentation:
depends_on does not wait for db and redis to be “ready” before starting web - only until they have been started.
There are many tools/scripts that can be used to solve this issue like wait-for which sh compatible in case your image based on Alpine for example (You can use wait-for-it if you have bash in your image)
All you have to do is to add the script to your image through Dockerfile then use this command in docker-compose.yml for the service that you want to make it wait for the database.
What comes after -- is the command that you would normally use to start your application
version: "2"
services:
app:
container_name: golang
...
command: ["./wait-for", "mysql:3306", "--", "go", "run", "myapplication"]
links:
- "mysql"
depends_on:
- "mysql"
mysql:
image: mysql
...
I have removed some parts from the docker-compose for easier readability.
Modify this part go run myapplication with the CMD of your golang image.
See Controlling startup order for more on this problem and strategies for solving it.
Another issue that will rise after you solve the connection issue will be as the following:
Setting MYSQL_USER with root value will cause a failure in MySQL with this error message:
ERROR 1396 (HY000) at line 1: Operation CREATE USER failed for 'root'#'%'
This is because this user already exist in the database and it tries to create another. if you need to use the root user itself you can use only this variable MYSQL_ROOT_PASSWORD or change the value of MYSQL_USER so you can securely use it in your application instead of the root user.
Update: In case you are getting not found and the path was correct, you might need to write the command as below:
command: sh -c "./wait-for mysql:3306 -- go run myapplication"
First, if you are using latest version of docker compose you don't need the link argument in you app service. I quote the docker compose documentation Warning: The --link flag is a legacy feature of Docker. It may eventually be removed. Unless you absolutely need to continue using it, https://docs.docker.com/compose/compose-file/#links
I think the solution is to use the networks argument. This create a docker network and add each service to it.
Try this
version: "2"
services:
app:
container_name: golang
restart: always
build: .
ports:
- "49160:8800"
networks:
- my_network
depends_on:
- "mysql"
mysql:
image: mysql
container_name: mysql
volumes:
- dbdata:/var/lib/mysql
restart: always
networks:
- my_network
environment:
- MYSQL_ROOT_PASSWORD=root
- MYSQL_DATABASE=testDB
- MYSQL_USER=root
- MYSQL_PASSWORD=root
ports:
- "3307:3306"
volumes:
dbdata:
networks:
my_network:
driver: bridge
By the way, if you only connect to Mysql from your app service you don't need to expose the mysql port. If the containers runs in the same network they can reach all ports inside this network.
If my example doesn't works try this
run the docker compose and next go into the app container using
docker container exec -it CONTAINER_NAME bash
Install ping in order to test connection and then run ping mysql.

Docker compose error while creating mount source path

I have created a docker-compose.yml using cloudestuary. After downloading it and putting it in my Laravel project folder and running docker-compose up -d the download takes place and then I get this message:
ERROR: for worker-1 Cannot start service worker-1: error while creating mount source path '/var/www/html/lensin/html': mkdir /var/www: read-only file system
ERROR: for nginx Cannot start service nginx: error while creating mount source path '/var/www/html/lensin/html': mkdir /var/www: read-only file system
ERROR: for app Cannot start service app: error while creating mount source path '/var/www/html/lensin/html': mkdir /var/www: read-only file system
ERROR: for workspace Cannot start service workspace: error while creating mount source path '/var/www/html/lensin/html': mkdir /var/www: read-only file system
ERROR: Encountered errors while bringing up the project.
I`m on Ubuntu 17, and have tried even to set 777 to all folders, and running it with sudo, but the result is the same. I have also tried to move the file and to edit the volumes in yml.
Here is my docker compose file:
version: '2'
services:
nginx:
image: 'cloudestuary/nginx:mainline-fpm'
restart: always
environment:
CLIENT_MAX_BODY_SIZE: 100m
DOCUMENT_ROOT: /var/www/html/public
INDEX_FILE: index.php
PHP_FPM: app
networks:
- app
volumes:
- './html:/var/www/html'
ports:
- '80:80'
app:
image: 'cloudestuary/php-fpm:7.1'
restart: always
environment:
MAX_UPLOAD_FILE_SIZE: 100m
APP_URL: 'http://lensin.localhost'
APP_KEY: 'base64:2X9U1HiBdmfbwvZ4UkwUP/25svg7439HXKWL1F8Xn1c='
DB_CONNECTION: mysql
DB_HOST: mysql
DB_PORT: '3306'
DB_DATABASE: cloudestuary
DB_USER: cloudestuary
DB_PASSWORD: secret
networks:
- app
volumes:
- './html:/var/www/html'
workspace:
image: 'cloudestuary/php-workspace:7.1'
restart: always
ports:
- '2222:22'
environment:
MAX_UPLOAD_FILE_SIZE: 100m
APP_URL: 'http://lensin.localhost'
APP_KEY: 'base64:2X9U1HiBdmfbwvZ4UkwUP/25svg7439HXKWL1F8Xn1c='
DB_CONNECTION: mysql
DB_HOST: mysql
DB_PORT: '3306'
DB_DATABASE: cloudestuary
DB_USER: cloudestuary
DB_PASSWORD: secret
SSH_PASSWORD: xsKEVWXPrdAeg
networks:
- app
volumes:
- './html:/var/www/html'
worker-1:
image: 'cloudestuary/php-cli:7.1'
restart: always
networks:
- app
environment: { }
volumes:
- './html:/var/www/html'
command: 'php artisan queue:work'
mysql:
image: 'mysql:5.7'
restart: always
networks:
- app
environment:
MYSQL_ROOT_PASSWORD: toor
MYSQL_PASSWORD: secret
MYSQL_USER: cloudestuary
MYSQL_DATABASE: cloudestuary
volumes:
- 'mysql-data:/var/lib/mysql'
volumes:
mysql-data: { }
networks:
app: { }
It's likely a pathing issue with Docker when installed with snap, you're better off installing it with the official documentation from Docker.
Remove docker from snap
snap remove docker
Remove the docker directory, and old version (It's okay if these don't exist already)
rm -R /var/lib/docker
sudo apt-get remove docker docker-engine docker.io
Install the official docker package: https://docs.docker.com/install/linux/docker-ce/ubuntu/
Update: Since posting this answer, I've learnt that tools installed using snap are installed in a sandbox with limited permissions outside of that sandbox. This is likely the cause as docker won't have access to the external filesystem from its isolated sandbox environment.
Restart your docker service. Then the problem will solve.
sudo systemctl restart docker
What led me here was the Kubernetes V1VolumeMount. When deploying my application I was getting the same error format:
ERROR: for <pod_name> Cannot start <service_name>: error while creating mount source path '<source_path>': mkdir <dir_path>: read-only file system
At the start I was thinking permissions error as well, hence the message is a bit misleading. I turned out that I was trying to mount something that didn't exist in the source image. Hence, my uneducated suggestion would be, verify that what you are trying to mount does exist, if it doesn't you probably don't need that mount path.
P.S. I saw that there wasn't an accepted answer, so I am hoping that my contribution is not causing unnecessary cluttering.
Got this error but the issue was that the source path was a symlink. For some reason docker does not seem to like it, even after restarting the service.
Had to use a real path and then it worked just fine with Docker version 20.10.8 installed with snap.
For Docker on Windows 10, sometimes you have just to wait a while (1-5 min) before executing docker-compose up again.
Hope this will help someone else.

Resources