Enabling cgroup cpu real-time runtime in ubuntu kernel - docker

Question

I am trying to use real-time scheduling in a docker container running on Ubuntu 18.04.
I have already installed a realtime kernel following the method given here. I have selected kernel version 5.2.9 and its associated rt patch.
The output of uname -a confirms that the realtime kernel is well installed and running:
Linux myLaptop 5.2.9-rt3 #1 SMP PREEMPT RT ...
To run my container I issue the following command:
docker run --cpu-rt-runtime=95000 \
--ulimit rtprio=99 \
--ulimit memlock=102400 \
--cap-add=sys_nice \
--privileged \
-it \
myimage:latest
However, the output I got is:
docker: Error response from daemon: Your kernel does not support cgroup cpu real-time runtime.
I have seen that this can be linked to the missing CONFIG_RT_GROUP_SCHED as detailed in the issue here. Indeed if I run the script provided at this page to check the kernel compatibility with Docker I get:
- CONFIG_RT_GROUP_SCHED: missing
Which seems to confirm that Docker is using this for realtime scheduling but is not provided in the kernel, although patched to be realtime.
From there, I tried to find a solution in vain. I am not well versed in kernel configurations to know if I need to compile it with a specific option, and which one to choose, to add the missing CONFIG_RT_GROUP_SCHED.
Thanks a lot in advance for recommendations and help.

Answer 1

When talking about real-time Linux there are different approaches ranging from single kernel approaches (like PREEMPT_RT) to dual-kernel approaches (such as Xenomai). You can use real-time capable Dockers in combination with all of them (clearly the kernel of your host machine has to match) to produce real-time capable systems but the approaches differ. In your case you are mixing up two different approaches: You installed PREEMPT_RT while following a guide for control groups which are incompatible with PREEMPT_RT.
By default the Linux kernel can be compiled with different levels of preempt-ability (see e.g. Reghenzani et al. - "The real-time Linux kernel: a Survey on PREEMPT_RT"):
PREEMPT_NONE has no way of forced preemption
PREEMPT_VOLUNTARY where preemption is possible in some locations in order to reduce latency
PREEMPT where preemption can occur in any part of the kernel (excluding spinlocks and other critical sections)
These can be combined with the feature of control groups (cgroups for short) by setting CONFIG_RT_GROUP_SCHED=y during kernel compilation, which reserves a certain fraction of CPU-time for processes of a certain (user-defined) group.
PREEMPT_RT developed from PREEMPT and is a set of patches that aims at making the kernel fully preemptible, even in critical sections (PREEMPT_RT_FULL). For this purpose e.g. spinlocks are largely replaced by mutexes.
As of 2021 it is being slowly merged into the mainline and will be available to the general public without the need to patch the kernel. As stated here PREEMPT_RT currently can't be compiled with the CONFIG_RT_GROUP_SCHED and therefore can't be used with control groups (see here for a comparison). From what I have read this is due to high latency spikes, something that I have already observed with control groups by means of cyclicytests.
This means you can either compile your kernel (see the Ubuntu manual for details)
Without PREEMPT_RT but with CONFIG_RT_GROUP_SCHED (see this post for details) and follow the Docker guide on real-time with control groups as well as my post here. From my experience this has though quite high latency spikes, something not desirable for real-time system where the worst-case latency is much more important than the average latency.
With PREEMPT_RT without CONFIG_RT_GROUP_SCHED (which can also be installed from a Debian package such as this one). In this case it is sufficient to execute the Docker with the options --privileged --net=host, or the Docker-compose equivalent privileged: true network_mode: host. Then any process from inside the Docker can set real-time priorities rtprio (e.g. by calling ::pthread_setschedparam from inside the code or by using chrt from the command line).
In case you are not using the root as user inside the Docker you furthermore will have to have give yourself a name of a user that belongs to a group with real-time privileges on your host computer (see $ ulimit -r). This can be done by configuring the PAM limits (/etc/security/limits.conf file) accordingly (as described here) by copying the section of the #realtime user group and creating a new group (e.g. #some_group) or adding the user (e.g. some_user) directly:
#some_group soft rtprio 99
#some_group soft priority 99
#some_group hard rtprio 99
#some_group hard priority 99
In this context rtprio is the maximum real-time priority allowed for non-privileged processes. The hard limit is the real limit to which the soft limit can be set to. The hard limits are set by the super-user and enforce by the kernel. The user cannot raise his code to run with a higher priority than the hard limit. The soft limit on the other hand is the default value limited by the hard limit. For more information see e.g. here.
I use latter option for real-time capable robotic applications and could not observe any differences in latency between with and without the Docker. You can find a guide on how to set up PREEMPT_RT and automated scripts for building it on my Github.