I'm currently building a Docker container that contains all the libraries needed for deployment of our app on a test machine, such as, for example, OpenCV 3.3 built with CUDA 9.
So, on a clean minimal OS install we can download the container and fire up our app in the desired environment, which is as I understand it one of the main reasons to use Docker.
So, after a while we decide to do our tests on the bare metal without the Docker file system, etc, in the way. Can we somehow replay the Dockerfile commands or image command history to run the apt-get, etc of not just the current package, but all FROM packages that are not yet installed on the raw environment?
Related
I was wondering If any one could help me to understand the difference of python virtual environment and docker container.
So I would like to have environment for each tools isolating from each other to avoid dependency conflict for example: use of different version of same dependency causing error in one of the tool because one tool need older version and other one requires newer version.
I’m tested out python venv but not sure if it’s the right one I should use for the issue I just explained or docker is something I should be using for my situation?.
Particularly for day-to-day development, prefer a virtual environment if it's practical.
Virtual environment
Docker
Works with native tools; can just run python myscript.py
Requires Docker-specific setup
Every IDE and editor works fine with it
Requires Docker-specific IDE support
Can just open() data files with no special setup
Can't access data files without Docker-specific mount setup
Immediately re-run code after editing it
Re docker build image or use Docker-specific mount setup
Uses Python installation from host
Use any single specific version of Python
Isolated Python library tree
Isolated Python library tree
Uses host version of C library dependencies
Isolated C library dependencies
A virtual environment acts like a normal Python installation in an alternate path. You don't need to do special things to make your local code or data files available; you can just run your script directly or via your IDE. The one downside is that you're limited to what your host OS's package manager makes available for Python versions and C library dependencies.
A Docker container contains the filesystem of a complete OS, including a completely isolated Python installation. It can be a good match if you need a very specific version of Python or if you need host OS dependencies that are tricky to install. It can also be a good match if you're looking for a production-oriented deployment setup that doesn't specifically depend on installing things on to the target system. But, Docker by design makes it hard to access your host files; it is not a great match for a live development environment or especially for one-off scripts that read and write host files.
The other consideration here is, if you use the standard Python packaging tools, it's straightforward to run your program in a virtual environment, and converting that to a Docker image is almost boilerplate. Starting from Docker can make it tricky to go back the other way, and I see some setups around SO that can only be run via Docker; if they were restructured to use a standard setup.cfg/requirements.txt installation setup they would not require Docker but could still be used with it.
I have a bootable iso image (live cd) with Linux system that is pretty old. That distro doesn't have remote repo (all installations are done from cdrom and separate disk with packages). I wanted to turn it into a docker image. Reading through articles google gave me, I've found several ways to do that. The first one is to mount the iso and find filesystem.squashfs - only modern distros use that way, not my case. My distro doesn't have that file available. The second approach is to call debootstrap but it requires to specify the repo for the distro with dist directory available in it. My distro doesn't have a public repo. What can I do? Is it even possible? I think that should be possible by doing a lot of things manually but how?
I faced similar problems when I had to containerize an old build server (building natively for legacy systems), eventually I succeeded. This approach describes how to containerize some old Linux distro (kernel 2.6.27 in my case), in the present Linux kernel 5 era.
General steps
if necessary: boot the old OS (or Live CD image)
login to the old system as root (or use sudo)
create a tarball from the relevant folders present in root
cd / ; tar cfvz image.tar.gz --one-file-system --exclude=/var/log --exclude=/image.tar.gz /
the selection worked in my case; review for yourself which folders to include or exclude
transfer the tarball to the Docker host (step not shown here)
and import it:
docker import image.tar.gz
the previous command will print out some hash
if convenient, tag the imported image:
docker tag <import-hash> <your-label>
Legacy problem: unsupported system calls
The imported image contains a Linux distribution snapshot. Some binaries can be executed from Docker, eg.:
docker run --rm <your-label> bin/ls
may actually work.
Some important binaries initially did not work for me, most notably bash:
docker run -it --rm <your-label> bin/bash
was failing silently. (Also, running with strace was possible but gave no clear indication.)
As #hiranchaudhuri pointed out, this is likely due to an API discrepancy between the host's kernel and the container's user space code.
In my case the problem was solved by enabling the legacy vsyscall kernel API
for Windows WSL2, this is described here https://learn.microsoft.com/en-us/windows/wsl/wsl-config
for native Linux systems of today, I guess this can be set in the boot configuration, with the kernel command-line parameter vsyscall=emulate, if the present kernel supports this option
I seriously doubt you will succeed on that.
Be aware Docker is not a full virtualization like KVM or VirtualBox. The lightweight virtualization benefits from the docker containers running on the host's Linux kernel. Which means the kernel is the same inside and outside of the container.
If you now try to install some old distro inside the container you may end up with an incompatible combination. Patching the kernel may involve upgrading glibc, and patching that may involve recompiling the rest of the OS.
I am not sure why you want to stick to the old distro, but seriously I believe you are better off with real virtualization.
I'm hoping this is an appropriate venue for my question. There's a lot of pieces to this puzzle.
I'm building a container using Docker that is destined to run on a Raspberry Pi Zero. The RPi Zero has an ARMv6 hard-float processor. The container will run a Python program that includes some dependencies that must be compiled (uses binary libraries). I am able to build and run the container on the RPi Zero itself, but building the container literally takes hours. I'm hoping to 1) speed up the process of building and 2) allow this to happen in a CI environment.
The approach I've taken in the past to build minimal Python containers that have dependencies requiring compilation is to use a multistage Docker build. I first startup a container with a full toolchain, then run pip wheel to compile all requirements into .whl files. I then copy the .whl files to the final container, install any binary libraries using the typical package manager, and then point pip install at this cache (--find-links=/wheels) for the installation of Python dependencies. This approach also works just fine on the Pi, but as I said it takes forever.
I've considered a few different approaches I could take:
Figure out how to get the Docker engine on my main dev machine (also in CI) to run and build an ARM image using qemu-arm-static while running docker build, and then somehow tag the resulting image as ARMv6 and upload it to my registry somehow. (I could just use a tag or a different repo name) I haven't honestly dug too deep into this, but my main concern is that every example I've seen of qemu-arm seems to indicate that it runs ARMv7 emulation. The RPi Zero can't actually even run many Docker containers that are made available for ARM due to this (exit 139). The arm32v6 "user" does provide working base images that run fine on the RPi Zero, which is what I'm using as the source images to build on my Pi itself.
Emulate an entire RasPi using qemu-system-arm. Again though, it looks like this emulates ARMv7, meaning the compiled wheels might not be able to run on the Pi zero.
Setup a cross-compiling toolchain for ARMv6. A few problems: I wouldn't know how to make sure pip uses that toolchain when compiling, and also I'd need to get and compile any other dependent library (even possibly all the way down to glibc?) so the header files will resolve.
It looks like this is easy to do if you want to do it for ARMv7 (which I believe the RPi 2 uses) or later, but I'm specifically using a Zero for my project, so I don't have that option.
TL;dr: How do I build binary Python wheels for ARMv6 using Docker without having to do it on a slow, single-core Raspberry Pi Zero?
I'm trying to streamline my iOS development builds and read about Docker.
If I understood it right, I could create an image that would include all the dependencies and my fellow devs could just pull it and build inside it.
Point is now, does this also work with Fastlane (which uses the Xcode cli tools I think) and "Docker for Mac"?
Also, I'm using React-Native, which seems to start a second process for bundling the JavaScript that will be included in the native build later and I read Docker only allows one process, is this a problem?
The problem with using Docker is that even if you use Docker for mac, you won't have access to macOS-based images. Docker runs in a lightweight virtual machine called xhyve - at least if you install docker via the Docker for Mac package - that runs Linux on your mac.
Essentially what this means is that your docker container is going to be limited to non-Xcode functionality. Here's what you definitely won't be able to do, at least not without a non-trivial amount of work:
Compile your app's native code
Take screenshots of your app or run your app in the Simulator
Signing the finished app with Apple's codesign
Here's things that you could potentially use your docker container for:
Building the JS code (I assume, since RN should work on Linux)
Uploading your app with iTMSTransporter (i.e. using fastlane's deliver)
Downloading/Creating certificates, provisioning profiles and push certificates (i.e. fastlane's match, cert, pem and sigh)
Working with git
All in all you're probably going to be very limited. Instead, it would be advisable to use things like Gemfile and Brewfile to list all your dependencies, and have a small setup.sh script that runs brew bundle and bundle install to install them on your colleague's machines. You can also set it up to run those during building (with Xcode's script build phases), so that no one can accidentally forget to install something that is needed for the build.
That being said, there is a fastlane docker image that is being worked on here that is also available on the Docker Hub. Note that it has only ever been tested to run the fastlane tests (that don't depend on macOS-only software), so it doesn't actually claim to run fastlane reliably.
I read Docker only allows one process
Docker allows multiple processes, it just doesn't allow more than one main process. If your main process stops everything else and the container stops with it. If you just want to use it to install dependencies so that you can run one-off commands that use them, instead of hosting a long-running service, you can always do that by using docker run:
docker run <repo/image:tag> <your_command>
Or launch an interactive shell into the container:
docker run -it <repo/image:tag> /bin/bash
I am a Windows user.
I have looked at the official Docker tutorial "Get Started". The example focus is a python app. I don't know python and I guess a Docker container can have many programs installed as an environment, not just python.
Is Docker good for testing a program I download from the internet in an isolated environment (like a sandbox in firewalls or antivirus) ?
How for example can I make a container that has an environment containing installed programs like Visual Studio, VLC player, Office, etc.?
Thanks,
Abe
Yes; you can have an isolated environment with docker. You can set your desired configurations, download from internet, install, and whatever you do in a Virtual Machine.
Yes, you can. What your container contains depends on the base image you create it FROM and packages you install inside of it.
Tips
You can build your container from an empty OS (e.g. ubuntu), configure the OS, download/install/configure/run whatever you want.
You can create a base image which derives FROM a suitable OS, then install any basic application (e.g. firefox) which you may use in a lot of containers on it. Then you should push it in a registry (e.g. Github). After that, you can use it as a base image for other containers, so your new containers have installed applications by default; no need to install them again. It reduces complexity and repetitions in Dockerfile.