I am unable to mount the gcs bucket after upgrading gcp composer1 to composer2. Getting the following error while mounting.
daemonize.Run: readFromProcess: sub-process: mountWithArgs: mountWithConn: Mount: mount: running /bin/fusermount: exit status 1
composer1 version: composer-1.17.5-airflow-1.10.15
gcsfuse : gcsfuse/0.37.0 (Go version go1.17.2)
composer2 version: composer-2.0.29-airflow-2.2.5
gcsfuse : gcsfuse/0.41.7 (Go version go1.18.4)
i tried to mount with this command
gcsfuse --implicit-dirs $INPUT_FILE_BUCKET gcp_inbound
Related
I am facing an issue with mounting a host directory into docker container with both -v and --mount options.
Using mount:
docker run --mount type=bind,source=/home/myuser/docker_test/out_dir,target=/home/out_dir --user 12345:1000 -it docker-name:0.1 bash
docker: Error response from daemon: invalid mount config for type "bind": stat /home/myuser/docker_test/out_dir: permission denied.
But I am able to do stat on this directory.
stat /home/myuser/docker_test/out_dir
File: '/home/myuser/docker_test/out_dir'
Size: 4096 Blocks: 8 IO Block: 32768 directory
Device: 33h/51d Inode: 9275022755226025350 Links: 2
Access: (0770/drwxrwx---) Uid: (12345/ myuser) Gid: ( 1000/ hercules)
Access: 2022-12-01 02:12:54.430582000 -0500
Modify: 2022-12-01 02:12:38.239629000 -0500
Change: 2022-12-01 02:12:38.239629000 -0500
Birth: -
Using -v:
docker run -v /home/myuser/docker_test/out_dir:/home/out_dir --user 12345:1000 -it docker-name:0.1:0.1 bash
docker: Error response from daemon: error while creating mount source path '/home/myuser/docker_test/out_dir': mkdir /home/myuser/docker_test: permission denied.
ERRO[0000] error waiting for container: context canceled
I don't know why it's trying to do mkdir but /home/myuser/docker_test already exists and is writable for the current user.
Am I missing something here?
BTW - /home is a NFS mounted directory.
EDIT: mounting /tmp worked. So this means it is related to the NFS mounted directory /home.
EDIT 2
I am working on a network machine where I don’t have root (sudo) access.
The docker service is installed by root user.
/home/myuser/docker_test/out_dir has 700 (rwx------) permissions. If I change the permission to 755, it will work. But I can’t change the directory permissions.
My question is why stat is failing when the user starting the docker has the permissions to access the source directory?
Is the stat being called by the docker executable as some ‘other’ user?
Use:
sudo docker run -v /home/myuser/docker_test/out_dir:/home/out_dir --user 12345:1000 -it docker-name:0.1:0.1 bash
I am trying Aerospike Connect to Kafka.
OS: Windows
I referred https://enterprise.aerospike.com/docs/connect/streaming-from-asdb/installing/from-asdb-to-kafka-installing.html.
I referred: '# With Aerospike 5.0 and later' for deploying on docker
I have these running in docker:
My Windows path for the file: C:/aerospike-kafka-outbound/aerospike-kafka-outbound.yml
C:\Users\Nupur>docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1114b8a97406 aerospike/aerospike-kafka-outbound:4.0.0 "/bin/sh -c '/opt/${…" 37 seconds ago Up 29 seconds compassionate_hofstadter
93343be1d481 aerospike/aerospike-tools "wrapper aql -h 172.…" 4 hours ago Up 4 hours dreamy_mendeleev
5f00ee532c4e aerospike/amc "/opt/amc/amc ' -con…" 8 days ago Up 4 hours 0.0.0.0:8081->8081/tcp aerospike-consoles
a81ff04dbd74 aerospike/aerospike-server-enterprise "/usr/bin/tini -r SI…" 8 days ago Up 4 hours 0.0.0.0:3000-3002->3000-3002/tcp aerospike
Error:
C:\Users\Nupur>docker run -p 8080:8080 -v /C:/aerospike-kafka-outbound/aerospike-kafka-outbound.yml:/etc/aerospike-kafka-outbound/aerospike-kafka-outbound.yml aerospike/aerospike-kafka-outbound:4.0.0
docker: Error response from daemon: invalid mode: /etc/aerospike-kafka-outbound/aerospike-kafka-outbound.yml.
See 'docker run --help'.
As for the front slash used in Windows path, on using backslash:
C:\Users\Nupur>docker run -p 8080:8080 -v C:\\aerospike-kafka-outbound\\aerospike-kafka-outbound.yml:/etc/aerospike-kafka-outbound/aerospike-kafka-outbound.yml aerospike/aerospike-kafka-outbound:4.0.0
docker: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "/run/desktop/mnt/host/c/aerospike-kafka-outbound/aerospike-kafka-outbound.yml" to rootfs at "/etc/aerospike-kafka-outbound/aerospike-kafka-outbound.yml" caused: mount through procfd: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type.
This is my docker directory:
C:\Users\Nupur>docker exec -it 1114b8a97406 bash
root#1114b8a97406:/# ls
bin boot dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var
root#1114b8a97406:/# cd /etc
root#1114b8a97406:/etc# ls
adduser.conf ca-certificates.conf environment host.conf ld.so.cache machine-id pam.conf rc1.d resolv.conf shells timezone
aerospike-kafka-outbound cron.daily fstab hostname ld.so.conf mke2fs.conf pam.d rc2.d rmt skel update-motd.d
alternatives debconf.conf gai.conf hosts ld.so.conf.d motd passwd rc3.d securetty ssl xattr.conf
apt debian_version group init.d libaudit.conf mtab passwd- rc4.d security subgid
bash.bashrc default group- issue localtime nsswitch.conf profile rc5.d selinux subuid
bindresvport.blacklist deluser.conf gshadow issue.net login.defs opt profile.d rc6.d shadow systemd
ca-certificates dpkg gshadow- kernel logrotate.d os-release rc0.d rcS.d shadow- terminfo
root#1114b8a97406:/etc# cd aerospike-kafka-outbound
root#1114b8a97406:/etc/aerospike-kafka-outbound# ls
aerospike-kafka-outbound.yml
root#1114b8a97406:/etc/aerospike-kafka-outbound#
I've been binding the host docker socket and cli so that I can run docker and compose commands from within running containers for over a year without issue but since updating to docker version 20.10.7 and compose version 1.29.2 I can't get my containerised environment to launch without the following error:
invalid mount config for type "bind": bind source path does not exist: /usr/local/bin/docker
Nothing has changed other than I updated Docker Desktop.
The location of the docker binary (symlink) on the host is still present:
0 lrwxr-xr-x 1 aadams-mbp staff 54 3 Aug 2018 /usr/local/bin/docker -> /Applications/Docker.app/Contents/Resources/bin/docker
The target of the symlink permissions look like this:
133608 -rwxr-xr-x 1 root admin 68405888 7 Jul 17:59 /Applications/Docker.app/Contents/Resources/bin/docker
This snippet is from my docker-compose.yaml file:
volumes:
# Bind docker CLI so can run docker commands
# from inside the container. Double check the
# location of the source binary on hosts that
# are not Mac OS. Docker might be in /usr/bin/docker,
# but on Mac OS it is at /usr/local/bin/docker.
- type: bind
source: /var/run/docker.sock
target: /var/run/docker.sock
- type: bind
source: ${DOCKER_BIN_SRC}
target: /usr/bin/docker
The ${DOCKER_BIN_SRC} is pulled in from a .env file (snippet):
##
# Docker bind
#
DOCKER_BIN_SRC=/usr/local/bin/docker
I am running on Mac OS Mojave version 10.14.6
On mac, /a/b is with below permissions:
$ ls -l /a/b
total 0
drwxrwxrwx 2 root wheel 64 13 Jan 08:50 b
$ whoami
user1
$
Below is the docker-compose file to mount /a/b from docker container:
version: '2'
services:
someapp:
build:
context: .
args:
DOCKER_GID: ${DOCKER_GID}
DOCKER_VERSION: ${DOCKER_VERSION}
DOCKER_COMPOSE: ${DOCKER_COMPOSE}
volumes:
- /a/b:/var/some_mount
- /var/run/docker.sock:/var/run/docker.sock
ports:
- "8080:8080"
On running docker-compose up -d someapp, I see below error:
ERROR: for docker-folder_someapp_1 Cannot start service someapp: b'Mounts denied: \r\nThe path /a/b\r\n is not shared from OS X and is not known to Docker.\r\nYou can configure shared paths from Docker -> Preferences... -> File Sharing.\r\nSee https://docs.docker.com/docker-for-mac/osxfs/#namespaces for more info.\r\n.'
ERROR: for someapp Cannot start service someapp: b'Mounts denied: \r\nThe path /a/b\r\nis not shared from OS X and is not known to Docker.\r\nYou can configure shared paths from Docker -> Preferences... -> File Sharing.\r\nSee https://docs.docker.com/docker-for-mac/osxfs/#namespaces for more info.\r\n.'
ERROR: Encountered errors while bringing up the project.
Following the instructions when I add /a/b using File sharing option to existing list:
I get another error popup: The export path /Users/user1/Documents/:a/:a:b overlaps with the export /Users
Another observation is, installing docker on MacOS, using VMWare to run docker, unlike ubuntu :
$ ps -eaf | grep docker
0 11100 1 0 9:02am ?? 0:00.07 /Library/PrivilegedHelperTools/com.docker.vmnetd
1873530912 11108 11038 0 9:02am ?? 0:01.45 /Applications/Docker.app/Contents/MacOS/com.docker.supervisor -watchdog fd:0
I do not see such mount deny issues, running docker daemon in Ubuntu.
1)
How to mount path(/a/b) of docker host to docker container's(/var/some_mount) ? in macos
2)
Is the explicit file sharing needed from docker host, because, docker installation on MacOS makes docker host run on VMWare and docker client run on MacOS?
When I run the command ./byfn.sh -m generate everything is fine.
I am trying to bring up the network using the command
./byfn.sh -m up
But I get an error:
2018-03-28 08:47:19.896 UTC [main] main -> INFO 001 Exiting.....
LOCAL_VERSION=1.1.0
DOCKER_IMAGE_VERSION=1.1.0
orderer.example.com is up-to-date
Creating peer1.org2.example.com ...
Creating peer0.org2.example.com ...
Creating peer1.org1.example.com ... error
Creating peer0.org1.example.com ...
Creating peer1.org2.example.com ... error
ERROR: for peer1.org1.example.com Cannot create container for service peer1.org 1.example.com: b'Mount denied:\nThe source path "\\\\var\\\\run:/host/var/run/"\ nis not a valid Windows path'
ERROR: for peer1.org2.example.com Cannot create container for service peer1.org 2.example.com: b'Mount denied:\nThe source path "\\\\var\\\\run:/host/var/run/"\ Creating peer0.org1.example.com ... error
Creating peer0.org2.example.com ... error
1.example.com: b'Mount denied:\nThe source path "\\\\var\\\\run:/host/var/run/"\ nis not a valid Windows path'
ERROR: for peer0.org2.example.com Cannot create container for service peer0.org 2.example.com: b'Mount denied:\nThe source path "\\\\var\\\\run:/host/var/run/"\ nis not a valid Windows path'
ERROR: for peer1.org1.example.com Cannot create container for service peer1.org 1.example.com: b'Mount denied:\nThe source path "\\\\var\\\\run:/host/var/run/"\ nis not a valid Windows path'
ERROR: for peer1.org2.example.com Cannot create container for service peer1.org 2.example.com: b'Mount denied:\nThe source path "\\\\var\\\\run:/host/var/run/"\ nis not a valid Windows path'
ERROR: for peer0.org1.example.com Cannot create container for service peer0.org 1.example.com: b'Mount denied:\nThe source path "\\\\var\\\\run:/host/var/run/"\ nis not a valid Windows path'
ERROR: for peer0.org2.example.com Cannot create container for service peer0.org 2.example.com: b'Mount denied:\nThe source path "\\\\var\\\\run:/host/var/run/"\ nis not a valid Windows path'
Encountered errors while bringing up the project.
ERROR !!!! Unable to start network
I tried to resolve it and I think there is something wrong with docker.Especially after I updated it.
$ docker --version
Docker version 18.03.0-ce, build 0520e24
$ docker-compose --version
docker-compose version 1.20.1, build 5d8c71b2
I re-installed docker but now I get the error below:
I've had the same problem.
If you've have this issue at Windows, try this.
open .env file in fabric-samples/first-network/
and add this
COMPOSE_CONVERT_WINDOWS_PATHS=1
It works for me with steps below:
$ cd [to your working folder of Fabric]
$ curl -sSL https://raw.githubusercontent.com/hyperledger/fabric/master/scripts/bootstrap.sh | bash -s 1.1.0
$ cd fabric-examples/
$ export FABRIC_CFG_PATH=$PWD
$ cd first-network
$ vi .env
--and add COMPOSE_CONVERT_WINDOWS_PATHS=1
$ ./byfn.sh -m generate
$ ./byfn.sh -m up
Special thanks to 권하경