How to run nodejs and nginx with dyanmic Docker Option? - docker

I'm fairly new to node and nginx. I've a task of building a simple webserver which host dynamic contents. A very crucial part of the webserver is to take inputs from user about ports to be used , any custom domain to be used (in place of localhost) , SSL certificates etc. from installer [Its supposed to be built for docker ] but I have no idea how to execute a script such that is passes the variable entered by user ( like $SERVER_URI) to nginx.conf and node file and overwrite current data

I will suggest to create a config file and read the value from them so everything will be dynamic.
Here is how you can achieve SSL certificate and other ENV and port dynamically also docker name and image name will be get and set.
Create file docker.config which contain ports, ENV, path mapping, hosts values and links if you wish to link container. leave them blank
if you do not need them. remove host_port:container_port this entry
just for comment purpose.
docker.config
START_PORT_MAPPINGS
host_port:container_port
8080:80
END_PORT_MAPPINGS
START_PATH_MAPPINGS
/path_to_code/:/var/www/htlm/test
/path_to_nginx_config1:/etc/nginx/nginx.conf
/path_to_ssl_certs:/container_path_to_Certs
END_PATH_MAPPINGS
START_LINKING
db:db-server
END_LINKING
START_HOST_MAPPINGS
test.com:192.168.1.23
test2.com:192.168.1.23
END_HOST_MAPPINGS
START_ENV_VARS
MYSQL_ROOT_PASSWORD=1234
OTHER_ENV_VAR=value
END_ENV_VARS
create start.sh this will read the values from docker.config and will run command your docker container.
Need two arguments 1st: docker name and 2nd: image name.
function read_connfig() {
docker_name="${1}"
input="docker.config"
option_key=$(echo "${2}" | cut -d':' -f1)
config_name=$(echo "${2}" | cut -d':' -f2)
post_fix=$(echo "${2}" | cut -d':' -f3)
while IFS=$' \t\n\r' read -r line; do
if [[ $line == END_"${config_name}" ]] ; then
read_prop="no"
fi
if [[ $read_prop == "yes" ]] ; then
echo -n "${option_key}${line}${post_fix} "
fi
if [[ $line == START_"${config_name}" ]] ; then
read_prop="yes"
fi
done < "$input"
}
function get_run_configs() {
docker_name=${1}
declare -a configs=("-p :PORT_MAPPINGS:" "-v :PATH_MAPPINGS:" "--add-host=:HOST_MAPPINGS:" "-e :ENV_VARS:" "--link :LINKING:")
local run_command=""
for config in "${configs[#]}"
do
config_vals=($(read_connfig "${docker_name}" "${config}"))
if [ ! -z "${config_vals}" ];
then
for config_val in "${config_vals[#]}"
do
run_command="${run_command} ${config_val}"
done
else
echo >&2 "No config found for ${config}"
fi
done
echo "${run_command}"
}
container_name=$1;
image_name=$2
docker_command=$(get_run_configs $docker_name)
echo $docker_command
docker run --name $container_name $docker_command -dit $image_name
Resulting command will be. ./start.sh test test
docker run --name test -p host_port:container_port -p 8080:80 -v /path_to_code/:/var/www/htlm/test -v /path_to_nginx_config1:/etc/nginx/nginx.conf -v /path_to_ssl_certs:/container_path_to_Certs --add-host=test.com:192.168.1.23 --add-host=test2.com:192.168.1.23 -e MYSQL_ROOT_PASSWORD=1234 -e OTHER_ENV_VAR=value --link db:db-server -dit test

Related

Docker volume: rename or copy operation

As per documentation Docker volumes are advertised this way:
Volumes are the preferred mechanism for persisting data generated by and used by Docker containers. While bind mounts are dependent on the directory structure and OS of the host machine, volumes are completely managed by Docker.
But if they are so good, why there are no operations to manage them like: copy, rename?
the command:
docker volume --help
gives only these options:
Usage: docker volume COMMAND
Manage volumes
Commands:
create Create a volume
inspect Display detailed information on one or more volumes
ls List volumes
prune Remove all unused local volumes
rm Remove one or more volumes
Documentation also states no other commands, nor any workarounds for having the copy or rename functionality.
I would like to rename currently existing volume and create another (blank) in place of the originally named volume and populate it with the new data for test.
After doing my test I may want (or not) to remove the newly created volume and rename the other one to its previous (original) name to restore the volume setup as it was before.
I would like to not create a backup of the original volume that I want to rename. Renaming is good enough for me and much faster than creating the backup and restoring form it.
Editing the docker-compose file and changing the name of the volume there is something I would like to avoid as well.
Is there any workaround that can work for renaming of a volume?
Can low level manual management from the shell targeting the Docker Root Dir: /var/lib/docker and volumes sub-dir be a solution or that approach may lead to some docker demon data inconsistency?
Not really the answer but I'll post this copy example because I couldn't find any before and searching for it took me to this question.
Docker suggest --volumes-from for backup purposes here.
For offline migration (stopped container) I don't see the point in using --volumes-from. So I just used a middle container with both volumes mounted and a copy command.
To finish off the migration a new container can use the new volume
Here's a quick test
Prepare a volume prova
docker run --name myname -d -v prova:/usr/share/nginx/html nginx:latest
docker exec myname touch /usr/share/nginx/html/added_file
docker stop myname
Verify the volume has nginx data + our file added_file
sudo ls /var/lib/docker/volumes/prova/_data
Output:
50x.html added_file index.html
Migrate the data to volume prova2
docker run --rm \
-v prova:/original \
-v prova2:/migration \
ubuntu:latest \
bash -c "cp -R /original/* /migration/"
Verify the new volume has the same data
sudo ls /var/lib/docker/volumes/prova2/_data
Output:
50x.html added_file index.html
Run a new container with the migrated volume:
docker run --name copyname -d -v prova2:/user/share/nginx/html nginx:latest
Verify the new container sees the migrated data at the original volume moint point:
docker exec copyname ls -al /user/share/nginx/html
For next searchers, I made a script that can do a copy of volume by #Lennonry example. Here it is https://github.com/KOYU-Tech/docker-volume-copy
Script itself for history:
#!/bin/bash
if (( $# < 2 )); then
echo ""
echo "No arguments provided"
echo "Use command example:"
echo "./dcv.sh OLD_VOLUME_NAME NEW_VOLUME_NAME"
echo ""
exit 1
fi
OLD_VOLUME_NAME="$1"
NEW_VOLUME_NAME="$2"
echo "== From '$OLD_VOLUME_NAME' to '$NEW_VOLUME_NAME' =="
function isVolumeExists {
local isOldExists=$(docker volume inspect "$1" 2>/dev/null | grep '"Name":')
local isOldExists=${isOldExists#*'"Name": "'}
local isOldExists=${isOldExists%'",'}
local isOldExists=${isOldExists##*( )}
if [[ "$isOldExists" == "$1" ]]; then
return 1
else
return 0
fi
}
# check if old volume exists
isVolumeExists ${OLD_VOLUME_NAME}
if [[ "$?" -eq 0 ]]; then
echo "Volume $OLD_VOLUME_NAME doesn't exist"
exit 2
fi
# check if new volume exists
isVolumeExists ${NEW_VOLUME_NAME}
if [[ "$?" -eq 0 ]]; then
echo "creating '$NEW_VOLUME_NAME' ..."
docker volume create ${NEW_VOLUME_NAME} 2>/dev/null 1>/dev/null
isVolumeExists ${NEW_VOLUME_NAME}
if [[ "$?" -eq 0 ]]; then
echo "Cannot create new volume"
exit 3
else
echo "OK"
fi
fi
# most important part, data migration
docker run --rm --volume ${OLD_VOLUME_NAME}:/source --volume ${NEW_VOLUME_NAME}:/destination ubuntu:latest bash -c "echo 'copying volume ...'; cp -R /source/* /destination/"
if [[ "$?" -eq 0 ]]; then
echo "Done successfuly 🎉"
else
echo "Some error occured 😭"
fi

How to Import Streamsets pipeline in Dockerfile without container exiting

I am trying to import a pipeline into streamsets, during container start up, by using the Docker CMD command in Dockerfile. The image builds, but while creating the container there is no error but it exits with code 0. So it never comes up. Here is what I did:
Dockerfile:
FROM streamsets/datacollector:3.18.1
COPY myPipeline.json /pipelinejsonlocation/
EXPOSE 18630
ENTRYPOINT ["/bin/sh"]
CMD ["/opt/streamsets-datacollector-3.18.1/bin/streamsets","cli","-U", "http://localhost:18630", \
"-u", \
"admin", \
"-p", \
"admin", \
"store", \
"import", \
"-n", \
"myPipeline", \
"--stack", \
"-f", \
"/pipelinejsonlocation/myPipeline.json"]
Build image:
docker build -t cmp/sdc .
Run image:
docker run -p 18630:18630 -d --name sdc cmp/sdc
This outputs the container id. But the container is in the Exited status as shown below.
docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
537adb1b05ab cmp/sdc "/bin/sh /opt/stream…" 5 seconds ago Exited (0) 3 seconds ago sdc
When I do not specify the CMD command in the Dockerfile, the streamsets container spins up and then when I run the streamsets import command in the running container in shell, it works. But how do I get it done during provisioning itself? Is there something I am missing in the Dockerfile?
In your Dockerfile you overwrite the default CMD and ENTRYPOINT from the StreamSets Data Collector Dockerfile. So the container only executes your command during startup and exits without errors afterwards. This is the reason why your container is in Exited (0) status.
In general this is good and expected behavior. If you want to keep your container alive you need to execute another command in the foreground, which never ends. But unfortunately, you cannot run multiple CMDs in your docker file.
I dug a little deeper. The default entry point of the image is ENTRYPOINT ["/docker-entrypoint.sh"]. This script sets up a few things and starts the Data Collector.
It is required that the Data Collector is running before the pipeline is imported. So a solution could be to copy the default docker-entrypoint.sh and modify it to start the Data Collector and import the pipeline afterwards. You could to it like this:
Dockerfile:
FROM streamsets/datacollector:3.18.1
COPY myPipeline.json /pipelinejsonlocation/
# Replace docker-entrypoint.sh
COPY docker-entrypoint.sh /docker-entrypoint.sh
EXPOSE 18630
docker-entrypoint.sh (https://github.com/streamsets/datacollector-docker/blob/master/docker-entrypoint.sh):
#!/bin/bash
#
# Copyright 2017 StreamSets Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
set -e
# We translate environment variables to sdc.properties and rewrite them.
set_conf() {
if [ $# -ne 2 ]; then
echo "set_conf requires two arguments: <key> <value>"
exit 1
fi
if [ -z "$SDC_CONF" ]; then
echo "SDC_CONF is not set."
exit 1
fi
grep -q "^$1" ${SDC_CONF}/sdc.properties && sed 's|^#\?\('"$1"'=\).*|\1'"$2"'|' -i ${SDC_CONF}/sdc.properties || echo -e "\n$1=$2" >> ${SDC_CONF}/sdc.properties
}
# support arbitrary user IDs
# ref: https://docs.openshift.com/container-platform/3.3/creating_images/guidelines.html#openshift-container-platform-specific-guidelines
if ! whoami &> /dev/null; then
if [ -w /etc/passwd ]; then
echo "${SDC_USER:-sdc}:x:$(id -u):0:${SDC_USER:-sdc} user:${HOME}:/sbin/nologin" >> /etc/passwd
fi
fi
# In some environments such as Marathon $HOST and $PORT0 can be used to
# determine the correct external URL to reach SDC.
if [ ! -z "$HOST" ] && [ ! -z "$PORT0" ] && [ -z "$SDC_CONF_SDC_BASE_HTTP_URL" ]; then
export SDC_CONF_SDC_BASE_HTTP_URL="http://${HOST}:${PORT0}"
fi
for e in $(env); do
key=${e%=*}
value=${e#*=}
if [[ $key == SDC_CONF_* ]]; then
lowercase=$(echo $key | tr '[:upper:]' '[:lower:]')
key=$(echo ${lowercase#*sdc_conf_} | sed 's|_|.|g')
set_conf $key $value
fi
done
# MODIFICATIONS:
#exec "${SDC_DIST}/bin/streamsets" "$#"
check_data_collector_status () {
watch -n 1 ${SDC_DIST}/bin/streamsets cli -U http://localhost:18630 ping | grep -q 'version' && echo "Data Collector has started!" && import_pipeline
}
function import_pipeline () {
sleep 1
echo "Start to import pipeline"
${SDC_DIST}/bin/streamsets cli -U http://localhost:18630 -u admin -p admin store import -n myPipeline --stack -f /pipelinejsonlocation/myPipeline.json
echo "Finished importing pipeline"
}
# Start checking if Data Collector is up (in background) and start Data Collector
check_data_collector_status & ${SDC_DIST}/bin/streamsets $#
I commented out the last line exec "${SDC_DIST}/bin/streamsets" "$#" of the default docker-entrypoint.sh and added two functions. check_data_collector_status () pings the Data Collector service until it is available. import_pipeline () imports your pipeline.
check_data_collector_status () runs in background and ${SDC_DIST}/bin/streamsets $# is started in foreground as before. So the pipeline is imported after the Data Collector service is started.
Run this image with sleep command:
docker run -p 18630:18630 -d --name sdc cmp/sdc sleep 300
300 is the time to sleep in seconds.
Then exec your script manually within the docker container and find out what's wrong.

Couchbase in docker for integration tests: Make the ports 8092, 8093, 8094 and 8095 configurable to be able to use docker’s random ports

I am using Couchbase java client SDK 2.7.9 and am running into a problem while trying to run automated integration tests. In such test we usually use random ports to be able to run the same thing on the same Jenkins slave (using docker for example).
But, with the client, we can specify many custom ports but not the 8092, 8093, 8094 and 8095.
The popular TestContainers modules mention as well that those port have to remain static in their Couchbase module: https://www.testcontainers.org/modules/databases/couchbase/ 1
Apparently it is also possible to change those ports at the server level.
Example:
Docker-compose.yml
version: '3.0'
services:
rapid_test_cb:
build:
context: ""
dockerfile: cb.docker
ports:
- "8091"
- "8092"
- "8093"
- "11210"
The docker image is ‘couchbase:community-5.1.1’
Internally the ports are the ones written above but externally they are random. At the client level you can set up bootstrapHttpDirectPort and bootstrapCarrierDirectPort but apparently the 8092 and 8093 ports are taken from the server-side (who does not know which port was assigned to him).
I would like to ask you whether it is possible to change those ports at the client level and, if not, to seriously consider adding that feature.
So, as discussed with the Couchbase team here,
it is not really possible. So we found a way to make it work using Gradle's docker compose plugin but I imagine it would work in different situations (TestContainer could use a similar system).
docker-compose.yml:
version: '3.0'
services:
rapid_test_cb:
build:
context: ""
dockerfile: cb.docker
ports:
- "${COUCHBASE_RANDOM_PORT_8091}:${COUCHBASE_RANDOM_PORT_8091}"
- "${COUCHBASE_RANDOM_PORT_8092}:${COUCHBASE_RANDOM_PORT_8092}"
- "${COUCHBASE_RANDOM_PORT_8093}:${COUCHBASE_RANDOM_PORT_8093}"
- "${COUCHBASE_RANDOM_PORT_11210}:${COUCHBASE_RANDOM_PORT_11210}"
environment:
COUCHBASE_RANDOM_PORT_8091: ${COUCHBASE_RANDOM_PORT_8091}
COUCHBASE_RANDOM_PORT_8092: ${COUCHBASE_RANDOM_PORT_8092}
COUCHBASE_RANDOM_PORT_8093: ${COUCHBASE_RANDOM_PORT_8093}
COUCHBASE_RANDOM_PORT_11210: ${COUCHBASE_RANDOM_PORT_11210}
cb.docker:
FROM couchbase:community-5.1.1
COPY configure-node.sh /opt/couchbase
#HEALTHCHECK --interval=5s --timeout=3s CMD curl --fail http://localhost:8091/pools || exit 1
RUN chmod u+x /opt/couchbase/configure-node.sh
RUN echo "{rest_port, 8091}.\n{query_port, 8093}.\n{memcached_port, 11210}." >> /opt/couchbase/etc/couchbase/static_config
CMD ["/opt/couchbase/configure-node.sh"]
configure-node.sh:
#!/bin/bash
poll() {
# The argument supplied to the function is invoked using "$#", we check the return value with $?
"$#"
while [ $? -ne 0 ]
do
echo 'waiting for couchbase to start'
sleep 1
"$#"
done
}
set -x
set -m
if [[ -n "${COUCHBASE_RANDOM_PORT_8092}" ]]; then
sed -i "s|8092|${COUCHBASE_RANDOM_PORT_8092}|g" /opt/couchbase/etc/couchdb/default.d/capi.ini
fi
if [[ -n "${COUCHBASE_RANDOM_PORT_8091}" ]]; then
sed -i "s|8091|${COUCHBASE_RANDOM_PORT_8091}|g" /opt/couchbase/etc/couchbase/static_config
fi
if [[ -n "${COUCHBASE_RANDOM_PORT_8093}" ]]; then
sed -i "s|8093|${COUCHBASE_RANDOM_PORT_8093}|g" /opt/couchbase/etc/couchbase/static_config
fi
if [[ -n "${COUCHBASE_RANDOM_PORT_11210}" ]]; then
sed -i "s|11210|${COUCHBASE_RANDOM_PORT_11210}|g" /opt/couchbase/etc/couchbase/static_config
fi
/entrypoint.sh couchbase-server &
poll curl -s localhost:${COUCHBASE_RANDOM_PORT_8091:-8091}
# Setup index and memory quota
curl -v -X POST http://127.0.0.1:${COUCHBASE_RANDOM_PORT_8091:-8091}/pools/default --noproxy '127.0.0.1' -d memoryQuota=300 -d indexMemoryQuota=300
# Setup services
curl -v http://127.0.0.1:${COUCHBASE_RANDOM_PORT_8091:-8091}/node/controller/setupServices --noproxy '127.0.0.1' -d services=kv%2Cn1ql%2Cindex
# Setup credentials
curl -v http://127.0.0.1:${COUCHBASE_RANDOM_PORT_8091:-8091}/settings/web --noproxy '127.0.0.1' -d port=${couchbase_random_port_8091:-8091} -d username=Administrator -d password=password
# Load the rapid_test bucket
curl -X POST -u Administrator:password -d name=rapid_test -d ramQuotaMB=128 --noproxy '127.0.0.1' -d authType=sasl -d saslPassword=password -d replicaNumber=0 -d flushEnabled=1 -v http://127.0.0.1:${COUCHBASE_RANDOM_PORT_8091:-8091}/pools/default/buckets
fg 1
Gradle's docker compose configuration:
def findRandomOpenPortOnAllLocalInterfaces = {
new ServerSocket(0).withCloseable { socket ->
return socket.getLocalPort().intValue()
}
}
dockerCompose {
environment.put 'COUCHBASE_RANDOM_PORT_8091', findRandomOpenPortOnAllLocalInterfaces()
environment.put 'COUCHBASE_RANDOM_PORT_8092', findRandomOpenPortOnAllLocalInterfaces()
environment.put 'COUCHBASE_RANDOM_PORT_8093', findRandomOpenPortOnAllLocalInterfaces()
environment.put 'COUCHBASE_RANDOM_PORT_11210', findRandomOpenPortOnAllLocalInterfaces()
}
integTest.doFirst {
systemProperty 'com.couchbase.bootstrapHttpDirectPort', couchbase_random_port_8091
systemProperty 'com.couchbase.bootstrapCarrierDirectPort', couchbase_random_port_11210
}

issues in accessing docker environment variables in systemd service files

1) I am running a docker container with following cmd (passing few env variables with -e option)
$ docker run --name=xyz -d -e CONTAINER_NAME=xyz -e SSH_PORT=22 -e NWMODE=HOST -e XDG_RUNTIME_DIR=/run/user/0 --net=host -v /mnt:/mnt -v /dev:/dev -v /etc/sysconfig/network-scripts:/etc/sysconfig/network-scripts -v /:/hostroot/ -v /etc/hostname:/etc/host_hostname -v /etc/localtime:/etc/localtime -v /var/run/docker.sock:/var/run/docker.sock --privileged=true cf3681e04bfb
2) After running the container as above, i check the env variable NWMODE inside the container, and it shows correctly as shown below :
$ docker exec -it xyz bash
$ env | grep NWMODE
NWMODE=HOST
3) Now, i created a sample service 'b' shown below which executes a script b.sh (where i try to access NWMODE) :
root#ubuntu16:/etc/systemd/system# cat b.service
[Unit]
Description=testing service b
[Service]
ExecStart=/bin/bash /etc/systemd/system/b.sh
root#ubuntu16:/etc/systemd/system# cat b.sh
#!/bin/bash`
systemctl import-environment
echo "NWMODE:" $NWMODE`
4) Now if i start service 'b' and see its logs, it shows that it is not able to access NWMODE env variable
$ systemctl start b
$ journalctl -fu b
...
systemd[1]: Started testing service b.
bash[641]: NWMODE: //blank for $NWMODE here`
5) Now rather than having 'systemctl import-environment' in b.sh, if i do following then the b.service logs show the correct value of NWMODE env variable:
$ systemctl import-environment
$ systemctl start b
Though the step 5 above works i can't go for it, as all the services in my system will be started automatically by systemd. In that case, can anyone please let me know how can i access the environment variables (passed using 'docker run...' cmd above) in a service file (say for e.g. in b.sh above). Can this be achieved somehow with systemctl import-environment or there is some other way ?
systemd unsets all environment variables to provide a clean environment. Afaik that is intended to be a security feature.
Workaround: Create a file /etc/systemd/system.conf.d/myenvironment.conf:
[Manager]
DefaultEnvironment=CONTAINER_NAME=xyz NWMODE=HOST XDG_RUNTIME_DIR=/run/user/0
systemd will set the environment variables declared in this file.
You can set up an ENTRYPOINT script that automatically creates this file before running systemd. Example:
RUN echo '#! /bin/bash \n\
echo "[Manager] \n\
DefaultEnvironment=$(while read -r Line; do echo -n "$Line" ; done < <(env) \n\
" >/etc/systemd/system.conf.d/myenvironment.conf \n\
exec /lib/systemd/systemd \n\
' >/usr/local/bin/setmyenv && chmod +x /usr/bin/setmyenv
ENTRYPOINT /usr/bin/setmyenv
Instead of creating the script within Dockerfile you can store it outside and add it with COPY:
#! /bin/bash
echo "[Manager]
DefaultEnvironment=$(while read -r Line; do echo -n "$Line" ; done < <(env)
" >/etc/systemd/system.conf.d/myenvironment.conf
exec /lib/systemd/systemd
TL;DR
Run the the command using bash, first store the docker environment variables to a file (or just pipe them two awk), extract & export the variable and finally run your main script.
ExecStart=/bin/bash -c "cat /proc/1/environ | tr '\0' '\n' > /home/env_file; export MY_ENV_VARIABLE=$(awk -F= -v key="MY_ENV_VARIABLE" '$1==key {print $2}' /home/env_file); /usr/bin/python3 /usr/bin/my_python_script.py"
Whatever #mviereck is saying is true, still I have found another solution to this problem.
My use case is to pass an environment variable to my system-d container in the Docker run command (docker run -e MY_ENV_VARIABLE="some_val") and use that in the python script that is run through the system-d unit file.
According to this post (https://forums.docker.com/t/where-are-stored-the-environment-variables/65762) the container environment variables can be found in the running process /proc/1/environ inside the container. Performing a cat does show that the environment variable MY_ENV_VARIABLE=some_val does exist, though in some mangled form.
$ cat /proc/1/environ
HOSTNAME=271fbnd986bdMY_ENV_VARIABLE=some_valcontainer=dockerLC_ALL=CDEBIAN_FRONTEND=noninteractiveHOME=/rootroot#271fb0d986bd
The main task now would be to extract MY_ENV_VARIABLE="some_val" value and pass it to the ExecStart directive in the system-d unit file.
(extraction code referenced from How to grep for value in a key-value store from plain text)
# this outputs a nice key,value pair
$ cat /proc/1/environ | tr '\0' '\n'
HOSTNAME=861f23cd1b33
MY_ENV_VARIABLE=some_val
container=docker
LC_ALL=C
DEBIAN_FRONTEND=noninteractive
HOME=/root
# we can store this in a file for use, too
$ cat /proc/1/environ | tr '\0' '\n' > /home/env_var_file
# we can then reuse the file to extract the value of interest against a key
$ awk -F= -v key="MY_ENV_VARIABLE" '$1==key {print $2}' /home/env_file
some_val
Now in the ExecStart directive in the system-d unit file we can do this:
[Service]
Type=simple
ExecStart=/bin/bash -c "cat /proc/1/environ | tr '\0' '\n' > /home/env_file; export MY_ENV_VARIABLE=$(awk -F= -v key="MY_ENV_VARIABLE" '$1==key {print $2}' /home/env_file); /usr/bin/python3 /usr/bin/my_python_script.py"

Share files between host system and docker container using specific UID

I'm trying to share files within a Docker guest using the volume sharing. In order to get the same UID, and therefore interoperability with those files, I would like to create a user in the Docker guest with the same UID as my own user.
In order to test out the idea, I wrote the following simplistic Dockerfile:
FROM phusion/baseimage
RUN touch /root/uid-$UID
Testing it with docker build -t=docktest . and then docker run docktest ls -al /root reveals that the file is simply named uid-.
Is there a means to share host environment variables with Docker during the guest build process?
While researching a solution to this problem, I have found the following article to be a great resource: https://medium.com/#mccode/understanding-how-uid-and-gid-work-in-docker-containers-c37a01d01cf
In my scripts, the solution boiled down to the following :
docker run --user $(id -u):$(id -g) -v /hostdirectory:/containerdirectory -v /etc/passwd:/etc/passwd myimage
Of course, id -u can be replaced by other means of retrieving a user's gid, such as stat -c "%u" /somepath
The environment is not shared, you could use -e, --env options to set env variables in container.
I usually use this approach when I want to have the same owner of the mapped volume: I check uid & gid of directory in container and then create a corresponding user. Here my script (setuser.sh) which creates a user for a directory:
#!/bin/bash
setuser() {
if [ -z "$1" ]; then
echo "Usage: $0 <path>"
return
fi
CURRENT_UID=`id -u`
DEST_UID=`stat -c "%u" $1`
if [ $CURRENT_UID = $DEST_UID ]; then
return
fi
DEST_GID=`stat -c "%g" $1`
if [ -e /home/$DEST_UID ]; then
return
fi
groupadd -g $DEST_GID $DEST_GID
useradd -u $DEST_UID -g $DEST_GID $DEST_UID
mkdir -p /home/$DEST_UID
chown $DEST_UID:$DEST_GID /home/$DEST_UID
}
setuser $1
And this is the wrapper script which runs commands as the user, where the directory with permissions is specified either as $USER_DIR or in /etc/user_dir
#!/bin/bash
if [ -z "$USER_DIR" ]; then
if [ -e /etc/user_dir ]; then
export USER_DIR=`head -n 1 /etc/user_dir`
fi
fi
if [ -n "$USER_DIR" ]; then
if [ ! -d "$USER_DIR" ]; then
echo "Please mount $USER_DIR before running this script"
exit 1
fi
. `dirname $BASH_SOURCE`/setuser.sh $USER_DIR
fi
if [ -n "$USER_DIR" ]; then
cd $USER_DIR
fi
if [ -e /etc/user_script ]; then
. /etc/user_script
fi
if [ $CURRENT_UID = $DEST_UID ]; then
"$#"
else
su $DEST_UID -p -c "$#"
fi
P.S. Alleo suggested different approach: to map users and groups files into container and to specify uid and gid. So your container does not depend on built-in users/groups you could use it without additional scripts.
This is not possible and will probably never be possible because of the design philosophy of keeping builds independent of machines. Issue 6822.
I slightly modified #ISanych answer:
#!/usr/bin/env bash
user_exists() {
id -u $1 > /dev/null 2>&1
}
group_exists() {
id -g $1 > /dev/null 2>&1
}
setuser() {
if [[ "$#" != 3 ]]; then
echo "Usage: $0 <path> <user> <group>"
return
fi
local dest_uid=$(stat -c "%u" $1)
local dest_gid=$(stat -c "%g" $1)
if user_exists $dest_uid; then
id -nu $dest_uid
return
fi
local dest_user=$2
local dest_group=$3
if user_exists $dest_user; then
userdel $dest_user
fi
if group_exists $dest_group; then
groupdel $dest_user
fi
groupadd -g $dest_gid $dest_group
useradd -u $dest_uid -g $dest_gid -s $DEFAULT_SHELL -d $DEFAULT_HOME -G root $dest_user
chown -R $dest_uid:$dest_gid $DEFAULT_HOME
id -nu $dest_user
}
REAL_USER=$(setuser $SRC_DIR $DEFAULT_USER $DEFAULT_GROUP)
setuser function accepts user and group names that you want to assign to uid and gid of provided directory. Then if user with such uid exists then it simply returns login corresponding to this uid, otherwise it creates user and group and returns login originally passed to function.
So you get the login of user that owns destination directory.

Resources