Share files between host system and docker container using specific UID - docker

I'm trying to share files within a Docker guest using the volume sharing. In order to get the same UID, and therefore interoperability with those files, I would like to create a user in the Docker guest with the same UID as my own user.
In order to test out the idea, I wrote the following simplistic Dockerfile:
FROM phusion/baseimage
RUN touch /root/uid-$UID
Testing it with docker build -t=docktest . and then docker run docktest ls -al /root reveals that the file is simply named uid-.
Is there a means to share host environment variables with Docker during the guest build process?

While researching a solution to this problem, I have found the following article to be a great resource: https://medium.com/#mccode/understanding-how-uid-and-gid-work-in-docker-containers-c37a01d01cf
In my scripts, the solution boiled down to the following :
docker run --user $(id -u):$(id -g) -v /hostdirectory:/containerdirectory -v /etc/passwd:/etc/passwd myimage
Of course, id -u can be replaced by other means of retrieving a user's gid, such as stat -c "%u" /somepath

The environment is not shared, you could use -e, --env options to set env variables in container.
I usually use this approach when I want to have the same owner of the mapped volume: I check uid & gid of directory in container and then create a corresponding user. Here my script (setuser.sh) which creates a user for a directory:
#!/bin/bash
setuser() {
if [ -z "$1" ]; then
echo "Usage: $0 <path>"
return
fi
CURRENT_UID=`id -u`
DEST_UID=`stat -c "%u" $1`
if [ $CURRENT_UID = $DEST_UID ]; then
return
fi
DEST_GID=`stat -c "%g" $1`
if [ -e /home/$DEST_UID ]; then
return
fi
groupadd -g $DEST_GID $DEST_GID
useradd -u $DEST_UID -g $DEST_GID $DEST_UID
mkdir -p /home/$DEST_UID
chown $DEST_UID:$DEST_GID /home/$DEST_UID
}
setuser $1
And this is the wrapper script which runs commands as the user, where the directory with permissions is specified either as $USER_DIR or in /etc/user_dir
#!/bin/bash
if [ -z "$USER_DIR" ]; then
if [ -e /etc/user_dir ]; then
export USER_DIR=`head -n 1 /etc/user_dir`
fi
fi
if [ -n "$USER_DIR" ]; then
if [ ! -d "$USER_DIR" ]; then
echo "Please mount $USER_DIR before running this script"
exit 1
fi
. `dirname $BASH_SOURCE`/setuser.sh $USER_DIR
fi
if [ -n "$USER_DIR" ]; then
cd $USER_DIR
fi
if [ -e /etc/user_script ]; then
. /etc/user_script
fi
if [ $CURRENT_UID = $DEST_UID ]; then
"$#"
else
su $DEST_UID -p -c "$#"
fi
P.S. Alleo suggested different approach: to map users and groups files into container and to specify uid and gid. So your container does not depend on built-in users/groups you could use it without additional scripts.

This is not possible and will probably never be possible because of the design philosophy of keeping builds independent of machines. Issue 6822.

I slightly modified #ISanych answer:
#!/usr/bin/env bash
user_exists() {
id -u $1 > /dev/null 2>&1
}
group_exists() {
id -g $1 > /dev/null 2>&1
}
setuser() {
if [[ "$#" != 3 ]]; then
echo "Usage: $0 <path> <user> <group>"
return
fi
local dest_uid=$(stat -c "%u" $1)
local dest_gid=$(stat -c "%g" $1)
if user_exists $dest_uid; then
id -nu $dest_uid
return
fi
local dest_user=$2
local dest_group=$3
if user_exists $dest_user; then
userdel $dest_user
fi
if group_exists $dest_group; then
groupdel $dest_user
fi
groupadd -g $dest_gid $dest_group
useradd -u $dest_uid -g $dest_gid -s $DEFAULT_SHELL -d $DEFAULT_HOME -G root $dest_user
chown -R $dest_uid:$dest_gid $DEFAULT_HOME
id -nu $dest_user
}
REAL_USER=$(setuser $SRC_DIR $DEFAULT_USER $DEFAULT_GROUP)
setuser function accepts user and group names that you want to assign to uid and gid of provided directory. Then if user with such uid exists then it simply returns login corresponding to this uid, otherwise it creates user and group and returns login originally passed to function.
So you get the login of user that owns destination directory.

Related

Dockerfile - CMD with nohup

How can I put the following into Dockerfile? I'm rebuilding a image and have modified few things.
Upon inspecting the original image, I see that it has CMD in a weird format.
"Cmd": [
"/bin/sh",
"-c",
"#(nop) CMD [\"supervisord\" \"-c\" \"/etc/supervisor.conf\"]"
],
The Entrypoint script executes this as its argument. But I'm unsure how to add this in the new Dockerfile.
I'm not sure if I can have 2 CMD and not sure how to add the nohup in a Dockerfile
ENTRYPOINT ["/usr/local/bin/docker-entrypoint.sh"]
CMD ["/bin/sh", "-c", "#(nop)"]
CMD ["supervisord","-c", "/etc/supervisor.conf"]
Here is the entrypoint.sh
#!/bin/bash
set -eo pipefail
# if command starts with an option, prepend supervisord
if [ "${1:0:1}" = '-' ]; then
set -- supervisord "$#"
fi
# Add local user;
# Either use the MARADNS_USER_ID if passed in at runtime or fallback.
USER_ID=${MARADNS_USER_ID:-9001}
echo "Starting with UID : $USER_ID"
usermod -u $USER_ID maradns
# update permissions
chown -R maradns.maradns /etc/maradns
chown -R maradns.maradns /var/cache/deadwood
# replace the UID and GID of the maradns user
MARADNS_UID=`id -u maradns`
MARADNS_GID=`id -g maradns`
cp /etc/mararc.custom /etc/mararc
sed -i -r "s/(maradns_uid\s*=\s*)([0-9]+)(.*)/\1${MARADNS_UID}\3/" /etc/mararc
sed -i -r "s/(maradns_gid\s*=\s*)([0-9]+)(.*)/\1${MARADNS_GID}\3/" /etc/mararc
# bind maradns on container host
MARADNS_ADDRESS=`ifconfig eth0 | grep 'inet addr:' | cut -d ' ' -f12 | cut -d ':' -f2`
sed -i -r "s/(ipv4_bind_addresses\s*=\s*)(.*)(.*)/\1\"${MARADNS_ADDRESS}\"\3/" /etc/mararc
# copy filebeat configuration
cp /etc/filebeat/filebeat.yml.custom /etc/filebeat/filebeat.yml
# run command
exec "$#"

how to make docker keep running in frontend and not exit so that I could see the running log output

Now I want to make a docker command run in frontend so that I could see the log output. Now I am using this command to run my docker container:
docker run -p 11110:11110 -p 11111:11111 -p 11112:11112 --name canal-server dolphinjiang/canal-server:v1.1.5
this is the Dockerfile of my project:
FROM centos:7
RUN cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
RUN echo ZONE=\"Asia/Shanghai\" > /etc/sysconfig/clock
RUN rm -rf /etc/yum.repos.d/*.repo
COPY CentOS6-Base-163.repo /etc/yum.repos.d/
RUN yum clean all
RUN groupadd -g 2500 canal; useradd -u 2501 -g canal -d /home/canal -m canal
RUN echo canal:De#2018er | chpasswd; echo root:dockerroot | chpasswd
RUN yum -y update && yum -y install wget vi openssl.x86_64 glibc.x86_64 tar tar.x86_64 inetutils-ping net-tools telnet which file
RUN yum clean all
COPY jdk-8u291-linux-x64.tar.gz /opt
RUN tar -zvxf /opt/jdk-8u291-linux-x64.tar.gz -C /opt && \
rm -rf /opt/jdk-8u291-linux-x64.tar.gz && \
chmod -R 755 /opt/jdk1.8.0_291 && \
chown -R root:root /opt/jdk1.8.0_291
RUN echo 'export JAVA_HOME=/opt/jdk1.8.0_291' >> /etc/profile
RUN echo 'export JRE_HOME=$JAVA_HOME/jre' >> /etc/profile
RUN echo 'export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH' >> /etc/profile
RUN echo 'export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH' >> /etc/profile
RUN source /etc/profile
RUN yum install kde-l10n-Chinese -y
RUN yum install glibc-common -y
RUN localedef -c -f UTF-8 -i zh_CN zh_CN.utf8
ENV JAVA_HOME /opt/jdk1.8.0_291
ENV PATH $PATH:$JAVA_HOME/bin
ENV LANG zh_CN.UTF-8
ENV LC_ALL zh_CN.UTF-8
ADD canal-server /home/canal/
RUN chmod 755 /home/canal/bin
WORKDIR /home/canal/bin
RUN chmod 777 /home/canal/bin/restart.sh
RUN chmod 777 /home/canal/bin/startup.sh
RUN chmod 777 /home/canal/bin/stop.sh
RUN chmod 777 /home/canal/bin/config.sh
CMD /home/canal/bin/config.sh
this is the config.sh:
cat > /home/canal/conf/canal.properties <<- EOF
# register ip
canal.register.ip = ${HOSTNAME}.canal-server-discovery-svc-stable.testcanal.svc.cluster.local
# canal admin config
canal.admin.manager = canal-admin-stable:8089
canal.admin.port = 11110
canal.admin.user = admin
canal.admin.passwd = 4ACFE3202A5FF5CF467898FC58AAB1D615029441
# admin auto register
canal.admin.register.auto = true
canal.admin.register.cluster =
EOF
sh /home/canal/bin/restart.sh
and this is the restart.sh:
#!/bin/bash
args=$#
case $(uname) in
Linux)
bin_abs_path=$(readlink -f $(dirname $0))
;;
*)
bin_abs_path=$(cd $(dirname $0) ||exit ; pwd)
;;
esac
sh "$bin_abs_path"/stop.sh $args
sh "$bin_abs_path"/startup.sh $args
and this is the start.sh:
#!/bin/bash
current_path=`pwd`
case "`uname`" in
Linux)
bin_abs_path=$(readlink -f $(dirname $0))
;;
*)
bin_abs_path=`cd $(dirname $0); pwd`
;;
esac
base=${bin_abs_path}/..
canal_conf=$base/conf/canal.properties
canal_local_conf=$base/conf/canal_local.properties
logback_configurationFile=$base/conf/logback.xml
export LANG=en_US.UTF-8
export BASE=$base
if [ -f $base/bin/canal.pid ] ; then
echo "found canal.pid , Please run stop.sh first ,then startup.sh" 2>&2
exit 1
fi
if [ ! -d $base/logs/canal ] ; then
mkdir -p $base/logs/canal
fi
## set java path
if [ -z "$JAVA" ] ; then
JAVA=$(which java)
fi
ALIBABA_JAVA="/usr/alibaba/java/bin/java"
TAOBAO_JAVA="/opt/taobao/java/bin/java"
if [ -z "$JAVA" ]; then
if [ -f $ALIBABA_JAVA ] ; then
JAVA=$ALIBABA_JAVA
elif [ -f $TAOBAO_JAVA ] ; then
JAVA=$TAOBAO_JAVA
else
echo "Cannot find a Java JDK. Please set either set JAVA or put java (>=1.5) in your PATH." 2>&2
exit 1
fi
fi
case "$#"
in
0 )
;;
1 )
var=$*
if [ "$var" = "local" ]; then
canal_conf=$canal_local_conf
else
if [ -f $var ] ; then
canal_conf=$var
else
echo "THE PARAMETER IS NOT CORRECT.PLEASE CHECK AGAIN."
exit
fi
fi;;
2 )
var=$1
if [ "$var" = "local" ]; then
canal_conf=$canal_local_conf
else
if [ -f $var ] ; then
canal_conf=$var
else
if [ "$1" = "debug" ]; then
DEBUG_PORT=$2
DEBUG_SUSPEND="n"
JAVA_DEBUG_OPT="-Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,address=$DEBUG_PORT,server=y,suspend=$DEBUG_SUSPEND"
fi
fi
fi;;
* )
echo "THE PARAMETERS MUST BE TWO OR LESS.PLEASE CHECK AGAIN."
exit;;
esac
str=`file -L $JAVA | grep 64-bit`
if [ -n "$str" ]; then
JAVA_OPTS="-server -Xms2048m -Xmx3072m -Xmn1024m -XX:SurvivorRatio=2 -XX:PermSize=96m -XX:MaxPermSize=256m -Xss256k -XX:-UseAdaptiveSizePolicy -XX:MaxTenuringThreshold=15 -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError"
else
JAVA_OPTS="-server -Xms1024m -Xmx1024m -XX:NewSize=256m -XX:MaxNewSize=256m -XX:MaxPermSize=128m "
fi
JAVA_OPTS=" $JAVA_OPTS -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dfile.encoding=UTF-8"
CANAL_OPTS="-DappName=otter-canal -Dlogback.configurationFile=$logback_configurationFile -Dcanal.conf=$canal_conf"
if [ -e $canal_conf -a -e $logback_configurationFile ]
then
for i in $base/lib/*;
do CLASSPATH=$i:"$CLASSPATH";
done
CLASSPATH="$base/conf:$CLASSPATH";
echo "cd to $bin_abs_path for workaround relative path"
cd $bin_abs_path
echo LOG CONFIGURATION : $logback_configurationFile
echo canal conf : $canal_conf
echo CLASSPATH :$CLASSPATH
$JAVA $JAVA_OPTS $JAVA_DEBUG_OPT $CANAL_OPTS -classpath .:$CLASSPATH com.alibaba.otter.canal.deployer.CanalLauncher 2>&1
echo $! > $base/bin/canal.pid
echo "cd to $current_path for continue"
cd $current_path
else
echo "canal conf("$canal_conf") OR log configration file($logback_configurationFile) is not exist,please create then first!"
fi
after I start the docker, it exit automaticlly, and the docker not startup, no log output. what should I do to make it run in frontend. after successs, switch to the backend. I also tried to run in deamon like this(make the container run background and not exit):
docker run -it -d -p 11110:11110 -p 11111:11111 -p 11112:11112 --name canal-server canal/canal-server:v1.1.5
the process still exit automaticlly. and docker container did not startup.
Basically, you should get the point (based on your latest comment).
Docker is based on some command, when it's done - it stops the container.
So to make it continuously running you should have command and run infinitely.
Also check this answer as well, there are more explanation
Why docker exiting with code 0
One of the easiest solution is to tail some logs.
Like,
tail -f /dev/null
Taken from here
you can use tail -f /dev/null to keep the container from stopping, try this
docker run -it -d -p 11110:11110 -p 11111:11111 -p 11112:11112 --name canal-server canal/canal-server:v1.1.5 tail -f /dev/null
see also this post

Docker stop/start resets to initial passwords

I've just started using docker by copy-pasting pre-made repos from github.
Here is the scenario and steps:
I've passed mysql/shell root password via environment variable -e and these passwords are set as expected inside entry.sh.
I then go inside container and reset shell/mysql root password to something different.
Now the main issue, each time I do docker stop + start from host, it resets passwords to the initial ones of step1.
Please suggest the changes so it retain the modified step2 passwords even I do docker start/stop.
Used entry.sh and dockerfile scripts can be checked from this github repo.
Thanks.
I just noticed that the entry.sh will always update the root password with $MYSQL_RANDOM_ROOT_PASSWORD on docker start. So assuming we already persist the /var/lib/mysql in the host, we can edit the entry.sh a bit to only update the password when /var/lib/mysql doesn't exists:
#!/bin/sh
# start apache
echo "Starting httpd"
httpd
echo "Done httpd"
# check if mysql data directory is nuked
# if so, install the db
echo "Checking /var/lib/mysql folder"
if [ ! -f /var/lib/mysql/ibdata1 ]; then
echo "Installing db"
mariadb-install-db --user=mysql --ldata=/var/lib/mysql > /dev/null
echo "Installed"
# from mysql official docker repo
if [ -z "$MYSQL_ROOT_PASSWORD" -a -z "$MYSQL_RANDOM_ROOT_PASSWORD" ]; then
echo >&2 'error: database is uninitialized and password option is not specified '
echo >&2 ' You need to specify one of MYSQL_ROOT_PASSWORD, MYSQL_RANDOM_ROOT_PASSWORD'
exit 1
fi
# random password
if [ ! -z "$MYSQL_RANDOM_ROOT_PASSWORD" ]; then
echo "Using random password"
MYSQL_ROOT_PASSWORD="$(pwgen -1 32)"
echo "GENERATED ROOT PASSWORD: $MYSQL_ROOT_PASSWORD"
echo "Done"
fi
tfile=`mktemp`
if [ ! -f "$tfile" ]; then
return 1
fi
cat << EOF > $tfile
USE mysql;
DELETE FROM user;
FLUSH PRIVILEGES;
GRANT ALL PRIVILEGES ON *.* TO 'root'#'%' IDENTIFIED BY "$MYSQL_ROOT_PASSWORD" WITH GRANT OPTION;
GRANT ALL PRIVILEGES ON *.* TO 'root'#'localhost' WITH GRANT OPTION;
UPDATE user SET password=PASSWORD("") WHERE user='root' AND host='localhost';
FLUSH PRIVILEGES;
EOF
echo "Querying user"
/usr/bin/mysqld --user=root --bootstrap --verbose=0 < $tfile
rm -f $tfile
echo "Done query"
# setting ssh root password
if [ -z "$SSH_ROOT_PASSWORD" ]; then
echo >&2 'You need to specify SSH_ROOT_PASSWORD'
exit
fi
# Set root password to root, format is 'user:password'.
echo "root:$SSH_ROOT_PASSWORD" | chpasswd
fi;
echo "Generating ssh keys"
if [ ! -f "/etc/ssh/ssh_host_rsa_key" ]; then
# generate fresh rsa key
ssh-keygen -f /etc/ssh/ssh_host_rsa_key -N '' -t rsa
fi
if [ ! -f "/etc/ssh/ssh_host_dsa_key" ]; then
# generate fresh dsa key
ssh-keygen -f /etc/ssh/ssh_host_dsa_key -N '' -t dsa
fi
#prepare run dir
if [ ! -d "/var/run/sshd" ]; then
mkdir -p /var/run/sshd
fi
ssh-keygen -A
/usr/sbin/sshd
# start mysql
# nohup mysqld_safe --skip-grant-tables --bind-address 0.0.0.0 --user mysql > /dev/null 2>&1 &
echo "Starting mariadb database"
exec /usr/bin/mysqld --user=root --bind-address=0.0.0.0
Basically we just move this block of code to the if block above it

Can't edit local files due to permission errors once I run docker-compose up

I run docker-compose up -d and then ssh into the container. I can load the site via localhost just fine but when I try to edit the source code on my local it does not let me due to permission errors. This is the ls -la output on container vs local:
Container:
Local:
My dockerfile has the chown command:
My local user is called pwm. I tried running chown -R pwm:pwm ../app from host at which point I am able to edit files but then I get laravel permission denied errors. Then I need to runchown -R www-data:www-data ../app again to fix it.
How can I fix this?
For a development environment, my go-to solution for this is to setup an entrypoint script inside the container that starts as root, changes the user inside the container to match that of the file/directory owner from a volume mount (which will be your user on the host), and then switch to that user to run the app. I've got an example of doing this along with the scripts needed to implement this in your own container in my base image repo: https://github.com/sudo-bmitch/docker-base
In there, the fix-perms script does the heavy lifting, including code like the following:
# update the uid
if [ -n "$opt_u" ]; then
OLD_UID=$(getent passwd "${opt_u}" | cut -f3 -d:)
NEW_UID=$(stat -c "%u" "$1")
if [ "$OLD_UID" != "$NEW_UID" ]; then
echo "Changing UID of $opt_u from $OLD_UID to $NEW_UID"
usermod -u "$NEW_UID" -o "$opt_u"
if [ -n "$opt_r" ]; then
find / -xdev -user "$OLD_UID" -exec chown -h "$opt_u" {} \;
fi
fi
fi
That script is run as root inside the container on startup. The last step of the entrypoints that I run will call something like:
exec gosu ${app_user} "$#"
which runs the container command as the application user as the new pid 1 executable.

How to run nodejs and nginx with dyanmic Docker Option?

I'm fairly new to node and nginx. I've a task of building a simple webserver which host dynamic contents. A very crucial part of the webserver is to take inputs from user about ports to be used , any custom domain to be used (in place of localhost) , SSL certificates etc. from installer [Its supposed to be built for docker ] but I have no idea how to execute a script such that is passes the variable entered by user ( like $SERVER_URI) to nginx.conf and node file and overwrite current data
I will suggest to create a config file and read the value from them so everything will be dynamic.
Here is how you can achieve SSL certificate and other ENV and port dynamically also docker name and image name will be get and set.
Create file docker.config which contain ports, ENV, path mapping, hosts values and links if you wish to link container. leave them blank
if you do not need them. remove host_port:container_port this entry
just for comment purpose.
docker.config
START_PORT_MAPPINGS
host_port:container_port
8080:80
END_PORT_MAPPINGS
START_PATH_MAPPINGS
/path_to_code/:/var/www/htlm/test
/path_to_nginx_config1:/etc/nginx/nginx.conf
/path_to_ssl_certs:/container_path_to_Certs
END_PATH_MAPPINGS
START_LINKING
db:db-server
END_LINKING
START_HOST_MAPPINGS
test.com:192.168.1.23
test2.com:192.168.1.23
END_HOST_MAPPINGS
START_ENV_VARS
MYSQL_ROOT_PASSWORD=1234
OTHER_ENV_VAR=value
END_ENV_VARS
create start.sh this will read the values from docker.config and will run command your docker container.
Need two arguments 1st: docker name and 2nd: image name.
function read_connfig() {
docker_name="${1}"
input="docker.config"
option_key=$(echo "${2}" | cut -d':' -f1)
config_name=$(echo "${2}" | cut -d':' -f2)
post_fix=$(echo "${2}" | cut -d':' -f3)
while IFS=$' \t\n\r' read -r line; do
if [[ $line == END_"${config_name}" ]] ; then
read_prop="no"
fi
if [[ $read_prop == "yes" ]] ; then
echo -n "${option_key}${line}${post_fix} "
fi
if [[ $line == START_"${config_name}" ]] ; then
read_prop="yes"
fi
done < "$input"
}
function get_run_configs() {
docker_name=${1}
declare -a configs=("-p :PORT_MAPPINGS:" "-v :PATH_MAPPINGS:" "--add-host=:HOST_MAPPINGS:" "-e :ENV_VARS:" "--link :LINKING:")
local run_command=""
for config in "${configs[#]}"
do
config_vals=($(read_connfig "${docker_name}" "${config}"))
if [ ! -z "${config_vals}" ];
then
for config_val in "${config_vals[#]}"
do
run_command="${run_command} ${config_val}"
done
else
echo >&2 "No config found for ${config}"
fi
done
echo "${run_command}"
}
container_name=$1;
image_name=$2
docker_command=$(get_run_configs $docker_name)
echo $docker_command
docker run --name $container_name $docker_command -dit $image_name
Resulting command will be. ./start.sh test test
docker run --name test -p host_port:container_port -p 8080:80 -v /path_to_code/:/var/www/htlm/test -v /path_to_nginx_config1:/etc/nginx/nginx.conf -v /path_to_ssl_certs:/container_path_to_Certs --add-host=test.com:192.168.1.23 --add-host=test2.com:192.168.1.23 -e MYSQL_ROOT_PASSWORD=1234 -e OTHER_ENV_VAR=value --link db:db-server -dit test

Resources