I am using a Docker container to run a C++ compiled executable. The Docker container is built using the latest Linux Debian distribution, while the host is a MacOS system (MacOS 12.6, on MacBook Pro 16 Latest 2019).
Within the C++ code, I call the function __rdtscp(unsigned int *__A) including x86intrin.h for monitoring purpose. Compiling and executing the application on the MacOS host it works correctly. But if I try to run it within the Docker container, I obtain a Illegal instruction error (it is compiled on another physical Linux host, I need this: anyway, I can run the same executable on different Linux machines and also on the container generated by the same Docker image I use if executed on another host).
Looking deeper into the issue, I found that __rdtscp(unsigned int *__A) must be supported by the CPU. It should be supported by all the CPUs after 2010/2011. In fact, it seems that flag is reported within the host CPU's features (RDTSCP). The problem is that I cannot find it within the container CPU's features.
Note that using __rdtsc() it works correctly, but this is not serializable, so I want to use __rdtscp(unsigned int *__A).
Following the MacOS host output of sysctl -a | grep machdep.cpu
And this is the output of the Debian docker container of lscpu
Could you help me to figure out the reason of this difference? Is there a way to force Docker to provide the same host CPU's features?
Thank you!
Related
Are containers specific to a particular host OS? For instance, if a container is created on Windows with particular dependencies (e.g., DLL files), can it run in a setup in which the host OS is Linux? I initially assumed that a container must be specific to a particular host OS.
But the following two excerpts seem to suggest that I may not have understood the mechanics correctly. So my question is: are containers built over the docker engine so when the dependencies are included, they are relative to the docker engine and the underlying host OS does not matter?
(1) From IBM:
Containerization allows developers to create and deploy applications faster and more securely. With traditional methods, code is developed in a specific computing environment which, when transferred to a new location, often results in bugs and errors. For example, when a developer transfers code from a desktop computer to a virtual machine (VM) or from a Linux to a Windows operating system. Containerization eliminates this problem by bundling the application code together with the related configuration files, libraries, and dependencies required for it to run. This single package of software or “container” is abstracted away from the host operating system, and hence, it stands alone and becomes portable—able to run across any platform or cloud, free of issues. [https://www.ibm.com/cloud/learn/containerization]
(2) From Docker:
Does Docker run on Linux, macOS, and Windows?
You can run both Linux and Windows programs and executables in Docker containers. The Docker platform runs natively on Linux (on x86-64, ARM and many other CPU architectures) and on Windows (x86-64).
Docker Inc. builds products that let you build and run containers on Linux, Windows and macOS.
What does Docker technology add to just plain LXC?🔗
Docker technology is not a replacement for LXC. “LXC” refers to capabilities of the Linux kernel (specifically namespaces and control groups) which allow sandboxing processes from one another, and controlling their resource allocations. On top of this low-level foundation of kernel features, Docker offers a high-level tool with several powerful functionalities:
Portable deployment across machines. Docker defines a format for bundling an application and all its dependencies into a single object called a container. This container can be transferred to any Docker-enabled machine. The container can be executed there with the guarantee that the execution environment exposed to the application is the same in development, testing, and production. LXC implements process sandboxing, which is an important pre-requisite for portable deployment, but is not sufficient for portable deployment. If you sent me a copy of your application installed in a custom LXC configuration, it would almost certainly not run on my machine the way it does on yours. The app you sent me is tied to your machine’s specific configuration: networking, storage, logging, etc. Docker defines an abstraction for these machine-specific settings. The exact same Docker container can run - unchanged - on many different machines, with many different configurations.
The host OS, or precisely, the kernel provided still matters. That's why you can't run Windows containers on Linux. You can run Linux container on Windows due to Hyper-V and WSL2, and on macOS with Hypervisor, but that's it. If the provided kernel is compatible (doesn't have to be identical), usually similar version and the same architecture (remember, there are x64, ARM64, etc) or at least supported virtualization (x64 containers can run on M1, which is ARM64) then you can just run the container, no need to worry about DLLs because they're supposed to be included either in one of the base image you start with or the image you generate.
I have a bootable iso image (live cd) with Linux system that is pretty old. That distro doesn't have remote repo (all installations are done from cdrom and separate disk with packages). I wanted to turn it into a docker image. Reading through articles google gave me, I've found several ways to do that. The first one is to mount the iso and find filesystem.squashfs - only modern distros use that way, not my case. My distro doesn't have that file available. The second approach is to call debootstrap but it requires to specify the repo for the distro with dist directory available in it. My distro doesn't have a public repo. What can I do? Is it even possible? I think that should be possible by doing a lot of things manually but how?
I faced similar problems when I had to containerize an old build server (building natively for legacy systems), eventually I succeeded. This approach describes how to containerize some old Linux distro (kernel 2.6.27 in my case), in the present Linux kernel 5 era.
General steps
if necessary: boot the old OS (or Live CD image)
login to the old system as root (or use sudo)
create a tarball from the relevant folders present in root
cd / ; tar cfvz image.tar.gz --one-file-system --exclude=/var/log --exclude=/image.tar.gz /
the selection worked in my case; review for yourself which folders to include or exclude
transfer the tarball to the Docker host (step not shown here)
and import it:
docker import image.tar.gz
the previous command will print out some hash
if convenient, tag the imported image:
docker tag <import-hash> <your-label>
Legacy problem: unsupported system calls
The imported image contains a Linux distribution snapshot. Some binaries can be executed from Docker, eg.:
docker run --rm <your-label> bin/ls
may actually work.
Some important binaries initially did not work for me, most notably bash:
docker run -it --rm <your-label> bin/bash
was failing silently. (Also, running with strace was possible but gave no clear indication.)
As #hiranchaudhuri pointed out, this is likely due to an API discrepancy between the host's kernel and the container's user space code.
In my case the problem was solved by enabling the legacy vsyscall kernel API
for Windows WSL2, this is described here https://learn.microsoft.com/en-us/windows/wsl/wsl-config
for native Linux systems of today, I guess this can be set in the boot configuration, with the kernel command-line parameter vsyscall=emulate, if the present kernel supports this option
I seriously doubt you will succeed on that.
Be aware Docker is not a full virtualization like KVM or VirtualBox. The lightweight virtualization benefits from the docker containers running on the host's Linux kernel. Which means the kernel is the same inside and outside of the container.
If you now try to install some old distro inside the container you may end up with an incompatible combination. Patching the kernel may involve upgrading glibc, and patching that may involve recompiling the rest of the OS.
I am not sure why you want to stick to the old distro, but seriously I believe you are better off with real virtualization.
I'm trying to create a 32-bit docker image with Ubuntu 14.04 and, any time that I run uname, I see that it is x86_64 instead of i386. Could anyone tell me why this is happening?
docker run talex5/lucid32 uname -m
The weird thing is when I look up the architecture type a different way, it says 32-bit:
docker run i386/ubuntu:14.04 file /sbin/init
/sbin/init: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, BuildID[sha1]=c394677bccc720a3bb4f4c42a48e008ff33e39b1, stripped`
This happens consistently whenever I download different docker images that say they are 32-bit and even when I create my own docker image using debootstrap.
Thanks!
uname reports the version and OS details of the kernel, but Docker containers always use the host system's kernel, and if it's a 64-bit kernel it will report x86_64.
You should see the same results running this with a mixed 32-/64-bit OS install (in Ubuntu land installing packages like libc6:i686); with a 32-bit filesystem tree in a chroot; and in a Docker container; which are all the same case of running 32-bit binaries on a system with a 64-bit kernel.
This is possible these days, with just a simple script. You could use https://github.com/docker-32bit/ubuntu.
I have trouble understanding this concept. I know a little bit about how Docker works and what the benefits are, and while I understand running web servers, databases and development environments in containers, I don't understand the point of running an OS like Ubuntu in Docker.
Can someone explain why you would want to do that and also the benefits of an entire OS in a container?
The OS is essentially the runtime environment required to run your app. If you app is compiled to run on Linux, it relies on Linux libraries (libc, glib, and so on) that must be present in executing environment, regardless of its type. Docker makes no exception to this.
So a Ubuntu application requires a Ubuntu image in order to run correctly.
Note that Docker container does not include nor run an entire OS, but only the minimum set of libraries that allow your app to run. In particular it does never contain or execute a kernel, as it runs under the host kernel.
Docker doesn't have its own OS, it is installed on a machine and this allows it to share host operating system resources. There will be only one OS and all the containers will be using that OS.
Most of the application are meaningless without OS since it is required for IO, hardware calls etc.
Each docker container may have different packages (java, python, jboss etc), applications installed.
When I am in my container, I run
lspci | grep -i nvidia
and nothing shows.
When I run ./deviceQuery from the samples NVIDIA provides I get
no CUDA-capable device is detected
I know I have a nvidia driver on my mac. I just can't figure out how to get my docker container to realize that.
On OS X, docker is a container running inside a separate virtualbox vm which does not expose the host GPU.
You'll first need to make the graphics card available in the Virtual Box VM. I'm not sure how to do that, but this looks like it might help:
https://www.virtualbox.org/manual/ch04.html#guestadd-video
Once you've got it mounted within the VM, then you can also share it with the container.
I haven't tried this myself, but this guy says that he can run native X11 Apps on a Mac using a beta docker client called Kinematic along with socat, XQuartz, and QGIS, and he seems to imply that NVidia driver issues were thus avoided. This looks worth a try!