Change time zone for docker and Dockerfile - docker

I have an AWS EC2 instance that I have scheduled to open at 11:50 PM MST and close at 11:59 PM PST. I have set the timezone of the instance to MST so that I can run a cron job that executes a .sh file at 11:55 PM MST. The cron job is pretty simple: 55 23 * * * sudo bash docker run --mount type=bind,source="/home/ec2-user/environment/Project",target="/Project" myubuntu. The docker will mount to a local folder "Project" that contains a .cpp file that web scrapes data from Steam's user information page. The code within the .cpp file is very reliant on the current time/date, hence why I have gone through so much work to get everything running in MST so that everything is standard throughout. However, even with everything running on MST, when the docker container is running it is not in MST despite the dockerfile stating to run with ENV TZ="America/Salt Lake City", I have since changed it from Salt Lake City to Phoenix just to try it out but it still doesn't run the docker in MST. For example, when I run the docker at 9:22 PM MST Nov 24th, the date within the docker is 04:22 AM UTC Nov 25th. This slight date and time change is greatly affecting the code I am trying to run.
To kind of explain what the code does, Steam has a .json URL that holds about 48-62 hours worth of data in "[unix epoch time, # users logged in]". The goal is automation so I figured if I had the code cut out any data that did not match the date the code was run at, it would not be included in the data collection. So I am collecting 24 hours worth of data at a time by running the code at the very end of the day every single day. The difference in date/time between the MST time that both I and my EC2 instance are running on and the UTC time my docker is running on is causing data collection issues.
I was given the dockerfile by my professor, and it supposedly is set up to run on MST but it is not from what I can tell. I have tried to run my command within my .sh file with the included -v /etc/timezone:/etc/timezone but that does not seem to fix the timezone issue either. The dockerfile I was given is below:
# This image will be based on the ubuntu image. Build it using the
# command in the comment below. You may omit the ubuntu pull if you already
# did it. Before running the build, change into the directory containing
# this Dockerfile.
FROM ubuntu
# Set a default shell.
SHELL ["/bin/bash", "-c"]
# The timezone library stops the docker build and waits for user input to
# select a timezone. This breaks the build. To get around this,
# set up timezone information prior to installing that library. This
# Docker code does that. Composited from two sources:
# https://rtfm.co.ua/en/docker-configure-tzdata-and-timezone-during-build/
# https://serverfault.com/questions/949991/how-to-install-tzdata-on-a-ubuntu-docker-image
ARG DEBIAN_FRONTEND=noninteractive
ENV TZ="America/Phoenix"
# Install a handful of useful libraries. Note that this is generally against
# best practices -- containers should be lean, and not include 'things you
# might use'. In our case, we're using the containers for development, so
# this may be reasonable here.
RUN apt-get -y update && apt-get -y install \
apt-utils \
emacs-nox \
g++
# Copy in the files from the current folder (recursively) For our purposes,
# put them in /cs3505
COPY . /cs3505
RUN apt-get -y install wget
Is there something I, or my professor, has done wrong in the setup of the docker to cause this timezone issue? How can I go about fixing my docker so that every time it runs at 11:55 PM MST it opens up with MST as the timezone?
Edit: I do not know if this makes a difference but running cat /etc/timezone returns "United States/Mountain" and running emacs /etc/timezone shows the same thing.

This is a dockerfile I customized based on Debian, you can refer to it:
FROM debian:stable-slim
ARG ARG_TIMEZONE=Asia/Shanghai
ENV ENV_TIMEZONE ${ARG_TIMEZONE}
# install base dependence
RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections \
&& apt-get update && apt-get install -y -q \
dialog apt-utils \
locales systemd cron \
vim wget curl exuberant-ctags tree \
tzdata ntp ntpstat ntpdate \
&& apt-get clean && apt-get autoremove -y && rm -rf /var/lib/apt/lists/* \
&& localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8
# sync timezone
RUN echo '$ENV_TIMEZONE' > /etc/timezone \
&& ln -fsn /usr/share/zoneinfo/$ENV_TIMEZONE /etc/localtime \
&& dpkg-reconfigure --frontend noninteractive tzdata

Related

Error when starting custom Airflow Docker Image GROUP_OR_COMMAND

I created a custom image with the following Dockerfile:
FROM apache/airflow:2.1.1-python3.8
USER root
RUN apt-get update \
&& apt-get -y install gcc gnupg2 \
&& curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add - \
&& curl https://packages.microsoft.com/config/debian/10/prod.list > /etc/apt/sources.list.d/mssql-release.list
RUN apt-get update \
&& ACCEPT_EULA=Y apt-get -y install msodbcsql17 \
&& ACCEPT_EULA=Y apt-get -y install mssql-tools
RUN echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bashrc \
&& echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bashrc \
&& source ~/.bashrc
RUN apt-get -y install unixodbc-dev \
&& apt-get -y install python-pip \
&& pip install pyodbc
RUN echo -e “AIRFLOW_UID=$(id -u) \nAIRFLOW_GID=0” > .env
USER airflow
The image creates successfully, but when I try to run it, I get this error:
"airflow command error: the following arguments are required: GROUP_OR_COMMAND, see help above."
I have tried supplying a group ID with the --user, but I can't figure it out.
How can I start this custom Airflow Docker image?
Thanks!
First of all this line is wrong:
RUN echo -e “AIRFLOW_UID=$(id -u) \nAIRFLOW_GID=0” > .env
If you are running it with Docker Compose (I presume you took it from https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html), this is something you should run on "Host" machine, not in the image. Remove that line, it has no effect.
Secondly - it really depends what "command" you run. The "GROUP_OR_COMMAND" message you got is the output of "airflow" command. You have not copied the whole output of your command but this is a message you get when you try to run airflow without telling it what to do. When you run the image you will run by default the airflow command which has a number of subcommands that can be executed. So the "see help above" message tells you the very thing you should do - look at the help and see what subcommand you wanted to run (and possibly run it).
docker run -it apache/airflow:2.1.2
usage: airflow [-h] GROUP_OR_COMMAND ...
positional arguments:
GROUP_OR_COMMAND
Groups:
celery Celery components
config View configuration
connections Manage connections
dags Manage DAGs
db Database operations
jobs Manage jobs
kubernetes Tools to help run the KubernetesExecutor
pools Manage pools
providers Display providers
roles Manage roles
tasks Manage tasks
users Manage users
variables Manage variables
Commands:
cheat-sheet Display cheat sheet
info Show information about current Airflow and environment
kerberos Start a kerberos ticket renewer
plugins Dump information about loaded plugins
rotate-fernet-key
Rotate encrypted connection credentials and variables
scheduler Start a scheduler instance
sync-perm Update permissions for existing roles and optionally DAGs
version Show the version
webserver Start a Airflow webserver instance
optional arguments:
-h, --help show this help message and exit
airflow command error: the following arguments are required: GROUP_OR_COMMAND, see help above.
when you extend the official image, it will pass the parametor to "airflow" command which causing this problem. Check this out: https://airflow.apache.org/docs/docker-stack/entrypoint.html#entrypoint-commands

determine the time zone when building a singularity image

I am trying to build an image using singularity. in one step I have to run a R script to do so, in the recipe file I need to install R and I did using the following command:
apt-get install -y systemd systemd-sysv gdebi-core procps libssl1.1 ed wget curl libqt5webkit5 libqt5core5a libxml2-dev r-cran-xml wget libssl-dev curl libcurl4-openssl-dev libnetcdf-dev netcdf-bin libcairo2-dev libxt-dev default-jre texlive-latex-base libhdf5-dev r-base r-base-dev
curl https://download1.rstudio.org/rstudio-xenial-1.1.463-amd64.deb > /rstudio-1.1.463-amd64.deb
apt-get -y install /rstudio-1.1.463-amd64.deb
wget -O /rstudio-server-stretch-1.1.463-amd64.deb \
https://download2.rstudio.org/rstudio-server-stretch-1.1.463-amd64.deb
gdebi -n /rstudio-server-stretch-1.1.463-amd64.deb
and I run the recipe file using this command:
sudo singularity build nanos.sif Singularity.recipe
but after running it, at some point it asks me that which time zones I am located at and here is the message:
Please select the geographic area in which you live. Subsequent configuration
questions will narrow this down by presenting a list of cities, representing
the time zones in which they are located.
1. Africa 6. Asia 11. System V timezones
2. America 7. Atlantic Ocean 12. US
3. Antarctica 8. Europe 13. None of the above
4. Australia 9. Indian Ocean
5. Arctic Ocean 10. Pacific Ocean
Geographic area:
I chose one of them using name and numbers but building did not proceed. do you know how I can fix this problem?
You can set this via the TZ environment variable in %post. e.g., export TZ=UTC. You may also need to set TZ=UTC (or the desired timezone) in %environment as well.
See additional info on timezones in R and a related question: How to change the default time zone in R?
To me the following spell worked
%environment
TZ=UTC
DEBIAN_FRONTEND=noninteractive
%post
export TZ=UTC
export DEBIAN_FRONTEND=noninteractive
apt-get update
apt-get install -y .....
Replace ..... with your packages. No worries about the specific timzone. Normally it should be taken from the host system, so UTC will be overridden unless you instruct singularity otherwise at the run time.

Docker file owners and groups

I think I have a dilemma. I am trying to create a Dockerfile to reproduce a long and complicated installation process (of ROS) so that my students can get it running with less headache.
I am combining various scripts provided with manual steps that are documented. The manual steps often say to do "sudo" but I am told that doing sudo inside a Dockerfile is to be avoided. So I move those steps to before the USER command in the Dockerfile because I am told that those commands run as root. However as a result the files and directories created are owned by root and I believe subsequent steps are failing.
I have two choices I think: move the commands to after the USER command and include sudo or try to make the install scripts create directories and files of the right ownership. Of course a priori I dont know what files and directories are going to be created.
Here is my Dockerfile (actually one of many I have been experimenting with.) Also if you see any other things that need to be improved or fixed please let me know!
FROM ubuntu:16.04
# create non-root user
ENV USERNAME ros
RUN adduser --ingroup sudo --disabled-password --gecos "" --shell /bin/bash --home /home/$USERNAME $USERNAME
RUN bash -c 'echo $USERNAME:ros | chpasswd'
ENV HOME /home/$USERNAME
RUN apt-get update && apt-get install --assume-yes wget sudo && \
wget https://raw.githubusercontent.com/ROBOTIS-GIT/robotis_tools/master/install_ros_kinetic.sh && \
chmod 755 ./install_ros_kinetic.sh && \
bash ./install_ros_kinetic.sh
RUN apt-get install --assume-yes ros-kinetic-joy ros-kinetic-teleop-twist-joy ros-kinetic-teleop-twist-keyboard ros-kinetic-laser-proc ros-kinetic-rgbd-launch ros-kinetic-depthimage-to-laserscan ros-kinetic-rosserial-arduino ros-kinetic-rosserial-python ros-kinetic-rosserial-server ros-kinetic-rosserial-client ros-kinetic-rosserial-msgs ros-kinetic-amcl ros-kinetic-map-server ros-kinetic-move-base ros-kinetic-urdf ros-kinetic-xacro ros-kinetic-compressed-image-transport ros-kinetic-rqt-image-view ros-kinetic-gmapping ros-kinetic-navigation ros-kinetic-interactive-markers
USER $USERNAME
WORKDIR /home/$USERNAME
RUN cd /home/$USERNAME/catkin_ws/src/ && \
git clone https://github.com/ROBOTIS-GIT/turtlebot3_msgs.git && \
git clone https://github.com/ROBOTIS-GIT/turtlebot3.git && \
git clone https://github.com/ROBOTIS-GIT/turtlebot3_simulations.git
# add catkin env
RUN echo 'source /opt/ros/kinetic/setup.bash' >> /home/$USERNAME/.bashrc
RUN echo 'source /home/ros/catkin_ws/devel/setup.bash' >> /home/$USERNAME/.bashrc
# RUN . /home/ros/.bashrc && \
# cd /home/$USERNAME/catkin_ws && \
# catkin_make
USER $USERNAME
ENTRYPOINT /bin/bash
Would be interesting for my own information to get why sudo should be avoided in containers.
Historically we use docker to automate build, test and deploy processes in our team and always tried to write Dockerfiles as close as possible to original process.
Lets say if you build in your host some app and launch some commands with sudo, some without, we managed to create exactly the same Dockerfiles. The positive feedback from this is that you are not obligated to write readme's on how to build the code anymore - you just supply Dockerfile and whenever someone wants to repeat all steps in non-container environment, he just follows (copy/pastes) commands from the file.
So my proposal is - in Dockerfile install packages first, then switch to user and proceed with all remaining steps, using sudo when necessary. You will have all artifacts owned by the user, not root.
UPD
Got the original discussion and this one. So it sounds like you choose the best approach based on your particular case and needs.

How to update docker daylight saving schedule

We have a docker image using America/Sao_Paulo timezone.
Since the government changed the period of daylight saving time in Brazil, it looks like the docker image didn't get the update, even if the linux machine where the docker is installed was.
I think I've found something who solve this:
apt-get install -y tzdata
ln -fs /usr/share/zoneinfo/America/New_York /etc/localtime
dpkg-reconfigure --frontend noninteractive tzdata
I'm running this first at my Dockerfile.yml. Looks like it's solved.
You can set the environment variable TZ in the docker container to change the timezone.
For example:
docker run -e "TZ=America/Sao_Paulo" image_name

Succesfully created a virtualenv (using "mkproject") in Dockerfile, but can't run "workon" properly

Edit: Solved- typo
I have a Dockerfile that successfully creates a virtualenv using virtualenvwrapper (along with setting up a heap of "standard" settings/packages in our normal environment). I am using the resulting image as a "base image" for further use. All good so far. However, the following Dockerfile (based of the first image, "base_image_14.04") falls down at the last line:
FROM base_image_14.04
USER root
RUN DEBIAN_FRONTEND=noninteractive \
apt-get update && apt-get install -y \
libproj0 libproj-dev \
libgeos-c1v5 libgeos-dev \
libjpeg62 libjpeg-dev \
zlib1g zlib1g-dev \
libfreetype6 libfreetype6-dev \
libgdal20 libgdal-dev \
&& rm -rf /var/lib/apt/lists
USER webdev
RUN ["/bin/bash", "-ic", "mkproject maproxy"]
EXPOSE 80
WORKDIR $PROJECT_HOME/mapproxy
ADD ./requirements.txt .
RUN ["/bin/bash", "-ic", "workon mapproxy && pip install -r requirements.txt"]
The "mkproject mapproxy" works fine. If I comment out the last line it builds successfully and I can spin up the container and run "workon mapproxy" manually, not a problem. But when I try and build with the last line, it gives a workon error:
ERROR: Environment 'mapproxy' does not exist. Create it with 'mkvirtualenv mapproxy'.
workon is being called, but for some reason it can't find the mapproxy virtualenv.
WORKON_HOME & PROJECT_HOME both exist (defined in the parent image) and point to the correct locations (and are used successfully by "mkproject mapproxy").
So why is workon returning an error when the mapproxy virtualenv exists? The same error happens when I isolate that last line into a third Dockerfile building on the second.
Solved: It was a simple typo. mkproject maproxy instead of mapproxy. :sigh:
I am trying to build a docker image and am running into similar problems.
First question was why use a virtual env in docker? The main reason in a nutshell is to minimize effort to migrate an existing and working approach into a docker container. I will eventually use docker-compose, but I wanted to start by getting my feet wet with it all in a single docker container.
In my first attempt I installed almost everything with apt-get, including uwsgi. I installed my app "globally" with pip3. The app has command line functionality and a separate flask web app, hence the need for uwsgi. The command line functionality works, but when I make a request of the flask app uwsgi / python has a problem with locale: Fatal Python error: Py_Initialize: Unable to get the locale encoding and ImportError: No module named 'encodings
I have stripped away all my app specific additions to narrow down the problem. This is the Dockerfile I'm using:
# Docker image definition for testing
FROM ubuntu:xenial
# Create a user
RUN useradd -G sudo -ms /bin/bash tester
RUN echo 'tester:password' | chpasswd
WORKDIR /home/tester
# Skipping apt-get update to save some build time. Some are kept
# to insure they are the same as on host setup.
RUN apt-get install -y python3 python3-dev python3-pip \
virtualenv virtualenvwrapper sudo nano && \
apt-get clean -qy
# After above, can we use those installed in rest of Dockerfile?
# Yes, but not always, such as with virtualenvwrapper. What about
# virtualenv? How do you "source" the script? Doesn't appear to be
# installed, as bash complains "source needs a single parameter"
ENV VIRTUALENVWRAPPER_PYTHON /usr/bin/python3
ENV VIRTUALENVWRAPPER_VIRTUALENV /usr/bin/virtualenv
RUN ["/bin/bash", "-c", "source", "/usr/share/virtualenvwrapper/virtualenvwrapper.sh"]
# Create a virtualenv so uwsgi can find locale
# RUN mkdir /home/tester/.virtualenv && virtualenv -p`which python3` /home/bts_tools/.virtualenv/bts_tools
RUN mkvirtualenv -p`which python3` bts_tools && \
workon bts_tools && \
pip3 --disable-pip-version-check install --upgrade bts_tools
USER tester
ENTRYPOINT ["/bin/bash"]
CMD ["--login"]
The build fails on the line I try to source the virtualenvwrapper script. Bash complains source needs an argument - the file to be sourced. So I comment out the RUN lines and it builds without error. When I run the resulting container I see all the additions to the ENV that virtualenvwrapper makes (you can see all of them by executing the "set" command without any args), and the script to be sourced is there too.
So my question is why doesn't docker find them? How does the docker build process work if the results of any previous RUNs or ENVs aren't applied for subsequent use in the Dockerfile? I know some things are applied and work, for example if you apt-get nginx you can refer to /etc/nginx or alter things under that folder. You can create a user and set it's password or cd into its home folder for example. If I move the WORKDIR before the RUN useradd -G I see a warning from useradd the home folder already exists. I tried to use the "time" program to time how long it takes to do various things in the Dockerfile and docker complains it can't find 'time'.
So what exactly is going on? I have spent the last 3 days trying to figure this out. It just shouldn't be this difficult. What am I missing?
Parts of the bts_tools flask app worked when I wasn't using virtual envs. Most of the app didn't work, and the issue was this locale problem. Since everything works on the host outside of docker, and after trying to alter the PATH, PYTHONHOME, PYTHONPATH in my uwsgi start script to overcome the dreaded "locale encoding" fatal error, I decided to try to replicate the host setup as closely as possible since that didn't have the locale issue. When I have had that problem before I could run dpkg-reconfigure python3 or fix with changes to PATH or ENV settings. If you google the problem you'll see many people have difficulties with python & locale. It's almost enough reason to avoid using python!
I posted this elsewhere about locale issue, if it helps.

Resources