Suggestion on optimizing the repetitive installation of packages in new lxc - lxc

As part of each test run we need to create new ubuntu lxc and install set of 10-15 packages(using apt-get install and pip install). The installation of additional packages add 5-minutes of run time. Is it possible to have customized lxc images with those packages pre-installed. I dont want to do lxc-clone and lxc-snapshot as i want to keep the networks to which the lxc interfaces attached very dynamic.
Is this doable with lxc and should i switch to docker.
Thanks in advance

Yes, it is doable; First, you create the image and then install all the required components and then make use of the commands like fakeroot, cpio to create initramfs.gz;
Follow this (backup):
$ cd /var/lib/lxc/C1/rootfs
$> find . | fakeroot -i fakeroot.env -s fakeroot.env -- cpio -o -H newc | gzip >"/home/<user>/initramfs.gz"
If you want to create new lxc (like restore) then do the following.
Follow this (restore):
$ mkdir -p /var/lib/lxc/C2/rootfs
$ cd /var/lib/lxc/C2/rootfs
$ cp /home/user/initramfs.gz
$ gunzip initramfs.gz
$ cpio -i < initramfs
$ rm initramfs
$ cp /var/lib/lxc/C1/config /var/lib/lxc/C2/config
Now edit the copied config file to have a new UTS name "C2" and other things which is needed specific to C2;

Related

How could I Install openmodelica in my docker image?

I have set up a docker image and install ubuntu on it. Can you please tell me how can I install Openmodelica inside ubuntu to that docker image?
for example, if I want to install node.js on this docker image I could use this code:
apt install nodejs
so I need some codes like that to install open Modelica on my docker image.
p.s: my docker image is an ubuntu image.
I happened to create a Docker image for OpenModelica to debug something, so I might add it here as well. We got this questions in the OpenModelica forum as well.
While the answer of #sjoelund.se will stay up to date this one is a bit more explaining.
Dockerfile
FROM ubuntu:18.04
# Export DISPLAY, so a XServer can display OMEdit
ARG DEBIAN_FRONTEND=noninteractive
ENV DISPLAY=host.docker.internal:0.0
# Install wget, gnupg, lsb-release
RUN apt-get update \
&& apt install -y wget gnupg lsb-release
# Get the OpenModelica stable version
RUN for deb in deb deb-src; do echo "$deb http://build.openmodelica.org/apt `lsb_release -cs` stable"; done | tee /etc/apt/sources.list.d/openmodelica.list
RUN wget -q http://build.openmodelica.org/apt/openmodelica.asc -O- | apt-key add -
# Install OpenModelica
RUN apt-get update \
&& apt install -y openmodelica
# Install OpenModelica libraries (like all of them)
RUN for PKG in `apt-cache search "omlib-.*" | cut -d" " -f1`; do apt-get install -y "$PKG"; done
# Add non-root user for security
RUN useradd -m -s /bin/bash openmodelicausers
USER openmodelicausers
ENV HOME /home/openmodelicausers
ENV USER openmodelicausers
WORKDIR $HOME
# Return omc version
CMD ["omc", "--version"]
Let's build and tag it:
docker build --tag openmodelica:ubuntubionic .
How to use omc from the docker image
Let's create a small helloWorld.mo Modelica model:
model helloWorld
Real x(start=1.0, fixed=true);
equations
der(x) = 2.5*x;
end helloWorld;
and a MOS script which will simulate it, called runHelloWorld.mos
loadFile("helloWorld.mo"); getErrorString();
simulate(helloWorld); getErrorString();
Now we can make our files accessible to the docker container with the -v flag and run our small example with:
docker run \
--rm \
-v $(pwd):/home/openmodelicausers \
openmodelica:ubuntubionic \
omc runHelloWorld.mos
Note that -v needs an absolute path. I added --rm to clean up.
Using OMEdit with a GUI
I'm using Windows + Docker with WSL2. So in order to get OMEdit running I need to have a XServer installed on my Windows host system. They are not trivial to set up, but I'm using VcXsrv and so far it is working for me. On Linux this is of course much simpler.
I'm using this config to start XLaunch:
<?xml version="1.0" encoding="UTF-8"?>
<XLaunch WindowMode="MultiWindow" ClientMode="NoClient" LocalClient="False" Display="-1" LocalProgram="xcalc" RemoteProgram="xterm" RemotePassword="" PrivateKey="" RemoteHost="" RemoteUser="" XDMCPHost="" XDMCPBroadcast="False" XDMCPIndirect="False" Clipboard="True" ClipboardPrimary="True" ExtraParams="" Wgl="True" DisableAC="True" XDMCPTerminate="False"/>
But when the XServer is running you can use OMEdit in nearly the same fashion you would from a Linux OS, just mount some directory with your files and that's it:
docker run \
--rm \
-v $(pwd):/home/openmodelicausers \
openmodelica:ubuntubionic \
OMEdit
You could get some inspiration from the Dockerfiles that are used to generate the OpenModelica docker images. For example: https://github.com/OpenModelica/OpenModelicaDockerImages/tree/v1.16.2

RStudio in docker image does not deal with some librairies

I have a problem to use the docker rstudio-image rocker/rstudio proposed
on https://www.rocker-project.org/ (docker containers for R). Since I am a beginner with both docker and RStudio, I suspect the problem comes from me and does not deserve a bug report:
I open a proper terminal with 'Docker Quickstart Terminal'
where I run the image with docker run -d -p 8787:8787 -e DISABLE_AUTH=true -v <...>:/home/rstudio/<...> --name rstudio rocker/rstudio
in my browser I then get a nice RStudio instance at the address http://192.168.99.100:8787
but in this instance I can't install several packages such as xml2. I get the message:
Using PKG_CFLAGS=
Using PKG_LIBS=-lxml2
------------------------- ANTICONF ERROR ---------------------------
Configuration failed because libxml-2.0 was not found. Try installing:
* deb: libxml2-dev (Debian, Ubuntu, etc)
* rpm: libxml2-devel (Fedora, CentOS, RHEL)
* csw: libxml2_dev (Solaris)
If libxml-2.0 is already installed, check that 'pkg-config' is in your
PATH and PKG_CONFIG_PATH contains a libxml-2.0.pc file. If pkg-config
is unavailable you can set INCLUDE_DIR and LIB_DIR manually via:
R CMD INSTALL --configure-vars='INCLUDE_DIR=... LIB_DIR=...'
--------------------------------------------------------------------
ERROR: configuration failed for package ‘xml2’
* removing ‘/usr/local/lib/R/site-library/xml2’
Warning in install.packages :
installation of package ‘xml2’ had non-zero exit status
I don't know whether xml2 is on the image but the file libxml-2.0.pc does exist on my laptop in the directory /opt/local/lib/pkgconfig and pkg-config is in /opt/local/bin. So I tried linking these pkg paths when running
the image (to see what happen when I play with the image environment
in RStudio), adding options -v
/opt/local/lib/pkgconfig:/home/rstudio/lib/pkgconfig -v
/opt/local/bin:/home/rstudio/bin to the run command. But it doesn't work: for some reason
I don't see the content of lib/pkgconfig in RStudio...
Also the RStudio instance does not accept root/sudo commands so I can't
use tools such as apt-get in the RStudio terminal
so, what's the trick ?
Libraries on your laptop (the host for docker) are not available for docker containers. You should create a custom image with required libraries, create a Dockerfile like this:
FROM rocker/rstudio
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
libxml2-dev # add any additional libraries you need
CMD ["/init"]
Above I added the libxml2-dev but you can add as many libraries as you need.
Then build your image using this command (you need to execute below command in directory there you created Dockerfile):
docker build -t my_rstudio:0.1 .
Then you can start your container:
docker run -d -p 8787:8787 -e DISABLE_AUTH=true --name rstudio my_rstudio:0.1
(you can add any additional arguments like -v to above).

Change system locale inside a CentOS/RHEL without using localectl?

I'm trying to build a Docker image based on oracle/database:11.2.0.2-xe (which is based on Oracle Linux based on RHEL) and want to change the system locale in this image (using some RUN command inside a Dockerfile).
According to this guide I should use localectl set-locale <MYLOCALE> but this command is failing with Failed to create bus connection: No such file or directory message. This is a known Docker issue for commands that require SystemD to be launched.
I tried to start the SystemD anyway (using /usr/sbin/init as first process as well as using -v /sys/fs/cgroup:/sys/fs/cgroup:ro -v /run thanks to this help) but then the localectl set-locale failed with Could not get properties: Connection timed out message.
So I'm now trying to avoid the usage of localectl to change my system globale locale, how could I do this?
According to this good guide on setting locale on Linux, I should use
localedef -c -i fr_FR -f ISO-8859-15 fr_FR.ISO-8859-15
But this command failed with
cannot read character map directory `/usr/share/i18n/charmaps': No such file or directory`
This SO reply indicated one could use yum reinstall glibc-common -y to fix this and it worked.
So my final working Dockerfile is:
RUN yum reinstall glibc-common -y && \
localedef -c -i fr_FR -f ISO-8859-15 fr_FR.ISO-8859-15 && \
echo "LANG=fr_FR.ISO-8859-15" > /etc/locale.conf
ENV LANG fr_FR.ISO-8859-15

docker build using previous build caches from registry

I'm configuring a bamboo build plan to build docker images. Using AWS ECS as registry. Build plan is something like this;
pull the latest tag
docker pull xxx.dkr.ecr.eu-west-1.amazonaws.com/myimage:latest
build image with latest tag
docker build -t myimage:latest .
tag the image (necessary for ECS)
docker tag -f myimage:latest xxx.dkr.ecr.eu-west-1.amazonaws.com/myimage:latest
Push the image to the registry
docker push xx.dkr.ecr.eu-west-1.amazonaws.com/myimage:latest
Because build tasks run on different and fresh build engines/servers every time, It doesn't have local cache.
When I don't change anything at Dockerfile and execute it again(at another server), I would expect docker to use local cache(comes from docker pull) and doesn't execute each line again. But it tries to build image everytime. I was also expecting that when I change something at the bottom of the file, it's going to use cache and executes only the latest line, but I'm not sure about this.
Do I know something wrong or are there any opinions on approach?
are you considering using squid proxy
?
edit : in case you dont want to go to the official website above, here is quick setup on squid proxy (debian based)
apt-get install squid-deb-proxy
and then change the squid configuration to create a larger space by open up
/etc/squid/squid.conf
and replace #cache_dir ufs /var/spool/squid with cache_dir ufs /var/spool/
squid 10000 16 256
and there you go,, a 10.000 MB worth of cache space
and then point the proxy address in dockerfile,,
here is a example of dockerfile with squid proxy
yum and apt-get based distro:
apt-get based distro
`FROM debian
RUN apt-get update -y && apt-get install net-tools
RUN echo "Acquire::http::Proxy \"http://$( \
route -n | awk '/^0.0.0.0/ {print $2}' \
):8000\";" \ > /etc/apt/apt.conf.d/30proxy
RUN echo "Acquire::http::Proxy::ppa.launchpad.net DIRECT;" >> \
/etc/apt/apt.conf.d/30proxy
CMD ["/bin/bash"]`
yum based distro
`FROM centos:centos7
RUN yum update -y && yum install -y net-tools
RUN echo "proxy=http://$(route -n | \
awk '/^0.0.0.0/ {print $2}'):3128" >> /etc/yum.conf
RUN sed -i 's/^mirrorlist/#mirrorlist/' \
/etc/yum.repos.d/CentOS-Base.repo
RUN sed -i 's/^#baseurl/baseurl/' \
/etc/yum.repos.d/CentOS-Base.repo
RUN rm -f /etc/yum/pluginconf.d/fastestmirror.conf
RUN yum update -y
CMD ["/bin/bash"]`
lets say you install squid proxy in your aws registry,,only the first build would fetch the data from the internet the rest (another server) build should be from the squid proxy cached. .
this technique based on book docker in practice technique 57 with tittle set up a package cache for faster build
i dont think there is a cache feature in docker without any third party software.. maybe there is and i just dont know it. .im not sure,,
just correct me if im wrong. .

How do I edit a file after I shell to a Docker container?

I successfully shelled to a Docker container using:
docker exec -i -t 69f1711a205e bash
Now I need to edit file and I don't have any editors inside:
root#69f1711a205e:/# nano
bash: nano: command not found
root#69f1711a205e:/# pico
bash: pico: command not found
root#69f1711a205e:/# vi
bash: vi: command not found
root#69f1711a205e:/# vim
bash: vim: command not found
root#69f1711a205e:/# emacs
bash: emacs: command not found
root#69f1711a205e:/#
How do I edit files?
As in the comments, there's no default editor set - strange - the $EDITOR environment variable is empty. You can log in into a container with:
docker exec -it <container> bash
And run:
apt-get update
apt-get install vim
Or use the following Dockerfile:
FROM confluent/postgres-bw:0.1
RUN ["apt-get", "update"]
RUN ["apt-get", "install", "-y", "vim"]
Docker images are delivered trimmed to the bare minimum - so no editor is installed with the shipped container. That's why there's a need to install it manually.
EDIT
I also encourage you to read my post about the topic.
If you don't want to add an editor just to make a few small changes (e.g., change the Tomcat configuration), you can just use:
docker cp <container>:/path/to/file.ext .
which copies it to your local machine (to your current directory). Then edit the file locally using your favorite editor, and then do a
docker cp file.ext <container>:/path/to/file.ext
to replace the old file.
You can use cat if it's installed, which will most likely be the case if it's not a bare/raw container. It works in a pinch, and ok when copy+pasting to a proper editor locally.
cat > file
# 1. type in your content
# 2. leave a newline at end of file
# 3. ctrl-c / (better: ctrl-d)
cat file
cat will output each line on receiving a newline. Make sure to add a newline for that last line. ctrl-c sends a SIGINT for cat to exit gracefully. From the comments you see that you can also hit ctrl-d to denote end-of-file ("no more input coming").
Another option is something like infilter which injects a process into the container namespace with some ptrace magic: https://github.com/yadutaf/infilter
To keep your Docker images small, don't install unnecessary editors. You can edit the files over SSH from the Docker host to the container:
vim scp://remoteuser#containerip//path/to/document
You can use cat if installed, with the > caracter.
Here is the manipulation :
cat > file_to_edit
#1 Write or Paste you text
#2 don't forget to leave a blank line at the end of file
#3 Ctrl + C to apply configuration
Now you can see the result with the command
cat file
For common edit operations I prefer to install vi (vim-tiny), which uses only 1491 kB or nano which uses 1707 kB.
In other hand vim uses 28.9 MB.
We have to remember that in order for apt-get install to work, we have to do the update the first time, so:
apt-get update
apt-get install vim-tiny
To start the editor in CLI we need to enter vi.
You can open existing file with
cat filename.extension
and copy all the existing text on clipboard.
Then delete old file with
rm filename.extension
or rename old file with
mv old-filename.extension new-filename.extension
Create new file with
cat > new-file.extension
Then paste all text copied on clipboard, press Enter and exit with save by pressing ctrl+z. And voila no need to install any kind of editors.
Sometime you must first run the container with root:
docker exec -ti --user root <container-id> /bin/bash
Then in the container, to install Vim or something else:
apt-get install vim
I use "docker run" (not "docker exec"), and I'm in a restricted zone where we cannot install an editor. But I have an editor on the Docker host.
My workaround is: Bind mount a volume from the Docker host to the container (https://docs.docker.com/engine/reference/run/#/volume-shared-filesystems), and edit the file outside the container. It looks like this:
docker run -v /outside/dir:/container/dir
This is mostly for experimenting, and later I'd change the file when building the image.
After you shelled to the Docker container, just type:
apt-get update
apt-get install nano
You can just edit your file on host and quickly copy it into and run it inside the container. Here is my one-line shortcut to copy and run a Python file:
docker cp main.py my-container:/data/scripts/ ; docker exec -it my-container python /data/scripts/main.py
If you use Windows container and you want change any file, you can get and use Vim in Powershell console easily.
To shelled to the Windows Docker container with PowerShell:
docker exec -it <name> powershell
First install Chocolatey package manager
Invoke-WebRequest https://chocolatey.org/install.ps1 -UseBasicParsing | Invoke-Expression;
Install Vim
choco install vim
Refresh ENVIRONMENTAL VARIABLE
You can just exit and shell back to the container
Go to file location and Vim it vim file.txt
See Stack Overflow question
sed edit file in place
It would be a good option here, if:
To modify a large file, it's impossible to use cat.
Install Vim is not allowed or takes too long.
My situation is using the MySQL 5.7 image when I want to change the my.cnf file, there is no vim, vi, and Vim install takes too long (China Great Firewall). sed is provided in the image, and it's quite simple. My usage is like
sed -i /s/testtobechanged/textwanted/g filename
Use man sed or look for other tutorials for more complex usage.
It is kind of screwy, but in a pinch you can use sed or awk to make small edits or remove text. Be careful with your regex targets of course and be aware that you're likely root on your container and might have to re-adjust permissions.
For example, removing a full line that contains text matching a regex:
awk '!/targetText/' file.txt > temp && mv temp file.txt
(More)
If you can only shell into container with bin/sh (in case bin/bash doesn't work)
and apt or apt-get doesn't work in the container, check whether apk is installed by entering apk in command prompt inside the container.
If yes, you can install nano as follows:
apk add nano
then nano will work as usual
An easy way to edit a few lines would be:
echo "deb http://deb.debian.org/debian stretch main" > sources.list
You can install nano
yum install nano
You can also use a special container which will contain only the command you need: Vim. I chose python-vim. It assumes that the data you want to edit are in a data container built with the following Dockerfile:
FROM debian:jessie
ENV MY_USER_PASS my_user_pass
RUN groupadd --gid 1001 my_user
RUN useradd -ms /bin/bash --home /home/my_user \
-p $(echo "print crypt("${MY_USER_PASS:-password}", "salt")" | perl) \
--uid 1001 --gid 1001 my_user
ADD src /home/my_user/src
RUN chown -R my_user:my_user /home/my_user/src
RUN chmod u+x /home/my_user/src
CMD ["true"]
You will be able to edit your data by mounting a Docker volume (src_volume) which will be shared by your data container (src_data) and the python-vim container.
docker volume create --name src_volume
docker build -t src_data .
docker run -d -v src_volume:/home/my_user/src --name src_data_1 src_data
docker run --rm -it -v src_volume:/src fedeg/python-vim:latest
That way, you do not change your containers. You just use a special container for this work.
First login as root :
docker run -u root -ti bash
Type following commands:
apt-get update &&
apt-get install nano
docker comes up with no editors. so simply install vim, 36MB space don't kill your docker!
Make sure to update the container before trying to install the editor.
apt-get update
apt-get install nano vi

Resources