Docker userns remap give permission issues within defined range - docker

Question

/etc/subuid
ubuntu:1000:1
ubuntu:165533:65536
/etc/subgid
ubuntu:999:1
ubuntu:165536:65536
So I am expecting files created by root in the container to map to my username on the host, which avoids permission issues with bind mounts directories on the host.
This works fine, except when I docker-compose up anchore-engine
This creates a named volume with these permissions:
The anchore services immediately terminates and exit unless I manually correct the permissions with chown to ubuntu:docker on the _data directory.
I was expecting that the 166531 is within the range defined in the subuid file. What's wrong?
docker-compose.yaml
version: '2.1'
volumes:
anchore-db-volume:
# Set this to 'true' to use an external volume. In which case, it must be created manually with "docker volume create anchore-db-volume"
external: false
services:
# The primary API endpoint service
api:
image: anchore/anchore-engine:v0.8.2
depends_on:
- db
- catalog
ports:
- "8228:8228"
logging:
driver: "json-file"
options:
max-size: 100m
environment:
- ANCHORE_ENDPOINT_HOSTNAME=api
- ANCHORE_DB_HOST=db
- ANCHORE_DB_PASSWORD=mysecretpassword
command: ["anchore-manager", "service", "start", "apiext"]
# Catalog is the primary persistence and state manager of the system
catalog:
image: anchore/anchore-engine:v0.8.2
depends_on:
- db
logging:
driver: "json-file"
options:
max-size: 100m
expose:
- 8228
environment:
- ANCHORE_ENDPOINT_HOSTNAME=catalog
- ANCHORE_DB_HOST=db
- ANCHORE_DB_PASSWORD=mysecretpassword
command: ["anchore-manager", "service", "start", "catalog"]
queue:
image: anchore/anchore-engine:v0.8.2
depends_on:
- db
- catalog
expose:
- 8228
logging:
driver: "json-file"
options:
max-size: 100m
environment:
- ANCHORE_ENDPOINT_HOSTNAME=queue
- ANCHORE_DB_HOST=db
- ANCHORE_DB_PASSWORD=mysecretpassword
command: ["anchore-manager", "service", "start", "simplequeue"]
policy-engine:
image: anchore/anchore-engine:v0.8.2
depends_on:
- db
- catalog
expose:
- 8228
logging:
driver: "json-file"
options:
max-size: 100m
environment:
- ANCHORE_ENDPOINT_HOSTNAME=policy-engine
- ANCHORE_DB_HOST=db
- ANCHORE_DB_PASSWORD=mysecretpassword
command: ["anchore-manager", "service", "start", "policy_engine"]
analyzer:
image: anchore/anchore-engine:v0.8.2
depends_on:
- db
- catalog
expose:
- 8228
logging:
driver: "json-file"
options:
max-size: 100m
environment:
- ANCHORE_ENDPOINT_HOSTNAME=analyzer
- ANCHORE_DB_HOST=db
- ANCHORE_DB_PASSWORD=mysecretpassword
volumes:
- /analysis_scratch
command: ["anchore-manager", "service", "start", "analyzer"]
db:
image: "postgres:9"
volumes:
- anchore-db-volume:/var/lib/postgresql/data
environment:
- POSTGRES_PASSWORD=mysecretpassword
expose:
- 5432
logging:
driver: "json-file"
options:
max-size: 100m
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres"]
Logs from one of the stopped containers:
/usr/local/lib/python3.6/site-packages/yosai/core/conf/yosaisettings.py:100: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
config = yaml.load(stream)
Traceback (most recent call last):
File "/usr/local/bin/twistd", line 11, in <module>
sys.exit(run())
File "/usr/local/lib64/python3.6/site-packages/twisted/scripts/twistd.py", line 31, in run
app.run(runApp, ServerOptions)
File "/usr/local/lib64/python3.6/site-packages/twisted/application/app.py", line 674, in run
runApp(config)
File "/usr/local/lib64/python3.6/site-packages/twisted/scripts/twistd.py", line 25, in runApp
runner.run()
File "/usr/local/lib64/python3.6/site-packages/twisted/application/app.py", line 383, in run
self.logger.start(self.application)
File "/usr/local/lib64/python3.6/site-packages/twisted/application/app.py", line 184, in start
observer = self._observerFactory()
File "/usr/local/lib/python3.6/site-packages/anchore_engine/subsys/twistd_logger.py", line 14, in logger
f = logfile.LogFile(thefile, '/var/log/', rotateLength=10000000, maxRotatedFiles=10)
File "/usr/local/lib64/python3.6/site-packages/twisted/python/logfile.py", line 170, in __init__
BaseLogFile.__init__(self, name, directory, defaultMode)
File "/usr/local/lib64/python3.6/site-packages/twisted/python/logfile.py", line 45, in __init__
self._openFile()
File "/usr/local/lib64/python3.6/site-packages/twisted/python/logfile.py", line 175, in _openFile
BaseLogFile._openFile(self)
File "/usr/local/lib64/python3.6/site-packages/twisted/python/logfile.py", line 85, in _openFile
self._file = open(self.path, "wb+", 0)
PermissionError: [Errno 13] Permission denied: '/var/log/anchore/anchore-api.log'
[MainThread] [anchore_manager.cli.service/start()] [INFO] Loading DB routines from module (anchore_engine)
[MainThread] [anchore_manager.util.db/connect_database()] [INFO] DB params: {"db_connect_args": {"connect_timeout": 86400}, "db_pool_size": 30, "db_pool_max_overflow": 100, "db_echo": false, "db_engine_args": null}
[MainThread] [anchore_manager.util.db/connect_database()] [INFO] DB connection configured: True
[MainThread] [anchore_manager.util.db/connect_database()] [INFO] DB attempting to connect...
[MainThread] [anchore_manager.util.db/connect_database()] [INFO] DB connected: True
[MainThread] [anchore_manager.util.db/init_database()] [INFO] DB compatibility check: running...
[MainThread] [anchore_manager.util.db/init_database()] [INFO] DB compatibility check success
[MainThread] [anchore_manager.util.db/init_database()] [INFO] DB post actions: running...
[MainThread] [anchore_manager.cli.service/start()] [INFO] DB version and code version in sync.
[MainThread] [anchore_manager.cli.service/start()] [INFO] Starting services: ['anchore-api']
[MainThread] [anchore_manager.cli.service/terminate_service()] [INFO] Looking for pre-existing service (anchore-api) pid from pidfile (/var/run/anchore/anchore-api.pid)
[MainThread] [anchore_manager.cli.service/start()] [INFO] waiting for service pidfile /var/run/anchore/anchore-api.pid to exist 0/30
[anchore-api] [anchore_manager.cli.service/startup_service()] [INFO] cleaning up service: anchore-api
[anchore-api] [anchore_manager.cli.service/terminate_service()] [INFO] Looking for pre-existing service (anchore-api) pid from pidfile (/var/run/anchore/anchore-api.pid)
[anchore-api] [anchore_manager.cli.service/startup_service()] [INFO] starting service: anchore-api
[anchore-api] [anchore_manager.cli.service/startup_service()] [INFO] /usr/local/bin/twistd --logger=anchore_engine.subsys.twistd_logger.logger --pidfile /var/run/anchore/anchore-api.pid -n anchore-api --config /config
[MainThread] [anchore_manager.cli.service/start()] [INFO] waiting for service pidfile /var/run/anchore/anchore-api.pid to exist 1/30
[MainThread] [anchore_manager.cli.service/start()] [INFO] waiting for service pidfile /var/run/anchore/anchore-api.pid to exist 2/30
[MainThread] [anchore_manager.cli.service/start()] [INFO] waiting for service pidfile /var/run/anchore/anchore-api.pid to exist 3/30
[MainThread] [anchore_manager.cli.service/start()] [INFO] waiting for service pidfile /var/run/anchore/anchore-api.pid to exist 4/30
[MainThread] [anchore_manager.cli.service/start()] [INFO] waiting for service pidfile /var/run/anchore/anchore-api.pid to exist 5/30
[MainThread] [anchore_manager.cli.service/start()] [INFO] waiting for service pidfile /var/run/anchore/anchore-api.pid to exist 6/30
[MainThread] [anchore_manager.cli.service/start()] [INFO] waiting for service pidfile /var/run/anchore/anchore-api.pid to exist 7/30
[MainThread] [anchore_manager.cli.service/start()] [INFO] waiting for service pidfile /var/run/anchore/anchore-api.pid to exist 8/30
[MainThread] [anchore_manager.cli.service/start()] [INFO] waiting for service pidfile /var/run/anchore/anchore-api.pid to exist 9/30
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/anchore_manager/cli/service.py", line 165, in startup_service
raise Exception("process exited: " + str(rc))
Exception: process exited: 1
[anchore-api] [anchore_manager.cli.service/startup_service()] [ERROR] service process exited at (Tue Dec 1 16:30:54 2020): process exited: 1
[anchore-api] [anchore_manager.cli.service/startup_service()] [FATAL] Could not start service due to: process exited: 1
[anchore-api] [anchore_manager.cli.service/startup_service()] [INFO] exiting service thread
[MainThread] [anchore_manager.cli.service/start()] [INFO] waiting for service pidfile /var/run/anchore/anchore-api.pid to exist 10/30
[MainThread] [anchore_manager.cli.service/start()] [INFO] service thread has stopped anchore-api
[MainThread] [anchore_manager.cli.service/start()] [INFO] auto_restart_services setting: False
[MainThread] [anchore_manager.cli.service/start()] [INFO] checking for startup failure pidfile=False, is_alive=False
[MainThread] [anchore_manager.cli.service/start()] [WARN] service start failed - exception: service thread for (anchore-api) failed to start
[MainThread] [anchore_manager.cli.service/start()] [FATAL] one or more services failed to start. cleanly terminating the others
[MainThread] [anchore_manager.cli.service/terminate_service()] [INFO] Looking for pre-existing service (anchore-api) pid from pidfile (/var/run/anchore/anchore-api.pid)
/usr/local/lib/python3.6/site-packages/yosai/core/conf/yosaisettings.py:100: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
config = yaml.load(stream)
Traceback (most recent call last):
File "/usr/local/bin/twistd", line 11, in <module>
sys.exit(run())
File "/usr/local/lib64/python3.6/site-packages/twisted/scripts/twistd.py", line 31, in run
app.run(runApp, ServerOptions)
File "/usr/local/lib64/python3.6/site-packages/twisted/application/app.py", line 674, in run
runApp(config)
File "/usr/local/lib64/python3.6/site-packages/twisted/scripts/twistd.py", line 25, in runApp
runner.run()
File "/usr/local/lib64/python3.6/site-packages/twisted/application/app.py", line 383, in run
self.logger.start(self.application)
File "/usr/local/lib64/python3.6/site-packages/twisted/application/app.py", line 184, in start
observer = self._observerFactory()
File "/usr/local/lib/python3.6/site-packages/anchore_engine/subsys/twistd_logger.py", line 14, in logger
f = logfile.LogFile(thefile, '/var/log/', rotateLength=10000000, maxRotatedFiles=10)
File "/usr/local/lib64/python3.6/site-packages/twisted/python/logfile.py", line 170, in __init__
BaseLogFile.__init__(self, name, directory, defaultMode)
File "/usr/local/lib64/python3.6/site-packages/twisted/python/logfile.py", line 45, in __init__
self._openFile()
File "/usr/local/lib64/python3.6/site-packages/twisted/python/logfile.py", line 175, in _openFile
BaseLogFile._openFile(self)
File "/usr/local/lib64/python3.6/site-packages/twisted/python/logfile.py", line 85, in _openFile
self._file = open(self.path, "wb+", 0)
PermissionError: [Errno 13] Permission denied: '/var/log/anchore/anchore-api.log'
[MainThread] [anchore_manager.cli.service/start()] [INFO] Loading DB routines from module (anchore_engine)
[MainThread] [anchore_manager.util.db/connect_database()] [INFO] DB params: {"db_connect_args": {"connect_timeout": 86400}, "db_pool_size": 30, "db_pool_max_overflow": 100, "db_echo": false, "db_engine_args": null}
[MainThread] [anchore_manager.util.db/connect_database()] [INFO] DB connection configured: True
[MainThread] [anchore_manager.util.db/connect_database()] [INFO] DB attempting to connect...
[MainThread] [anchore_manager.util.db/connect_database()] [INFO] DB connected: True
[MainThread] [anchore_manager.util.db/init_database()] [INFO] DB compatibility check: running...
[MainThread] [anchore_manager.util.db/init_database()] [INFO] DB compatibility check success
[MainThread] [anchore_manager.util.db/init_database()] [INFO] DB post actions: running...
[MainThread] [anchore_manager.cli.service/start()] [INFO] DB version and code version in sync.
[MainThread] [anchore_manager.cli.service/start()] [INFO] Starting services: ['anchore-api']
[MainThread] [anchore_manager.cli.service/terminate_service()] [INFO] Looking for pre-existing service (anchore-api) pid from pidfile (/var/run/anchore/anchore-api.pid)
[anchore-api] [anchore_manager.cli.service/startup_service()] [INFO] cleaning up service: anchore-api
[anchore-api] [anchore_manager.cli.service/terminate_service()] [INFO] Looking for pre-existing service (anchore-api) pid from pidfile (/var/run/anchore/anchore-api.pid)
[anchore-api] [anchore_manager.cli.service/startup_service()] [INFO] starting service: anchore-api
[anchore-api] [anchore_manager.cli.service/startup_service()] [INFO] /usr/local/bin/twistd --logger=anchore_engine.subsys.twistd_logger.logger --pidfile /var/run/anchore/anchore-api.pid -n anchore-api --config /config
[MainThread] [anchore_manager.cli.service/start()] [INFO] waiting for service pidfile /var/run/anchore/anchore-api.pid to exist 0/30
[MainThread] [anchore_manager.cli.service/start()] [INFO] waiting for service pidfile /var/run/anchore/anchore-api.pid to exist 1/30
[MainThread] [anchore_manager.cli.service/start()] [INFO] waiting for service pidfile /var/run/anchore/anchore-api.pid to exist 2/30
[MainThread] [anchore_manager.cli.service/start()] [INFO] waiting for service pidfile /var/run/anchore/anchore-api.pid to exist 3/30
[MainThread] [anchore_manager.cli.service/start()] [INFO] waiting for service pidfile /var/run/anchore/anchore-api.pid to exist 4/30
[MainThread] [anchore_manager.cli.service/start()] [INFO] waiting for service pidfile /var/run/anchore/anchore-api.pid to exist 5/30
[MainThread] [anchore_manager.cli.service/start()] [INFO] waiting for service pidfile /var/run/anchore/anchore-api.pid to exist 6/30
[MainThread] [anchore_manager.cli.service/start()] [INFO] waiting for service pidfile /var/run/anchore/anchore-api.pid to exist 7/30
[MainThread] [anchore_manager.cli.service/start()] [INFO] waiting for service pidfile /var/run/anchore/anchore-api.pid to exist 8/30
[MainThread] [anchore_manager.cli.service/start()] [INFO] waiting for service pidfile /var/run/anchore/anchore-api.pid to exist 9/30
[MainThread] [anchore_manager.cli.service/start()] [INFO] waiting for service pidfile /var/run/anchore/anchore-api.pid to exist 10/30
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/anchore_manager/cli/service.py", line 165, in startup_service
raise Exception("process exited: " + str(rc))
Exception: process exited: 1
[anchore-api] [anchore_manager.cli.service/startup_service()] [ERROR] service process exited at (Tue Dec 1 16:32:07 2020): process exited: 1
[anchore-api] [anchore_manager.cli.service/startup_service()] [FATAL] Could not start service due to: process exited: 1
[anchore-api] [anchore_manager.cli.service/startup_service()] [INFO] exiting service thread
[MainThread] [anchore_manager.cli.service/start()] [INFO] waiting for service pidfile /var/run/anchore/anchore-api.pid to exist 11/30
[MainThread] [anchore_manager.cli.service/start()] [INFO] service thread has stopped anchore-api
[MainThread] [anchore_manager.cli.service/start()] [INFO] auto_restart_services setting: False
[MainThread] [anchore_manager.cli.service/start()] [INFO] checking for startup failure pidfile=False, is_alive=False
[MainThread] [anchore_manager.cli.service/start()] [WARN] service start failed - exception: service thread for (anchore-api) failed to start
[MainThread] [anchore_manager.cli.service/start()] [FATAL] one or more services failed to start. cleanly terminating the others
[MainThread] [anchore_manager.cli.service/terminate_service()] [INFO] Looking for pre-existing service (anchore-api) pid from pidfile (/var/run/anchore/anchore-api.pid)

Answer 1

I was expecting that the 166531 is within the range defined in the subuid file. What's wrong?
The 166531 is within the subuid range, it maps to uid 999 (165533-1+999) which matches the uid of the postgres user within the postgres image:
$ docker run -it --rm --entrypoint /bin/bash postgres:9
root#d99b2bbb3d48:/# ls -al /var/lib/postgresql/data
total 8
drwxrwxrwx 2 postgres postgres 4096 Nov 18 08:39 .
drwxr-xr-x 1 postgres postgres 4096 Nov 18 08:39 ..
root#d99b2bbb3d48:/# id postgres
uid=999(postgres) gid=999(postgres) groups=999(postgres),101(ssl-cert)
By default, docker will initialize a new/empty named volume to match the contents of the image, including file ownership and permissions. This is expected behavior for postgres. We'd need to see logs of the failing containers to give more details of why you're seeing containers exit.