I want to build a system over seL4 and I do not want to write the drivers from scratch. I know that L4linux managaged to raise an entire linux kernel, drivers included, over fiasco.OC.
Ideally I want a driver wrapper that would allow me to run linux drivers as standalone tasks over sel4.
I am willing to code much. but I want to avoid reading hardware spec sheets and rewriting drivers.
I last looked at L4 in depth many years ago.
Based on my understanding the answer to your question should be in general a no. The reasons for this are mainly in two aspects: For one is because a fully bloated linux driver needs to take care of too many aspects to integrate into the kernel subsystems. The another reason is the two kernels are different.
If the specific driver you are looking at does not heavily integrate into the kernel subsystems, it may be not a huge task for you to develop a wrapper.
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 2 years ago.
Improve this question
I want to access my NVIDIA GPUs from inside containers. Can I do this without nvidia-container-runtime?
Requiring a custom Docker runtime just to talk to one device seems very strange. There is a whole universe of PCI devices out there. Why does this one need its own runtime? For example, suppose I had both NVIDIA and AMD GPUs. Would I be unable to access both from inside one container?
I understand that nvidia-container-runtime lets me control which GPUs are visible via NVIDIA_VISIBLE_DEVICES. But I do not care about this. I am not using containers to isolate devices; I am using containers to manage CUDA/CUDNN/TensorFlow version h*ll. And if I did want to isolate devices, I would use the same mechanism as forever: By controlling access to nodes in /dev.
In short, the whole "custom runtime" design looks flawed to me.
So, questions:
What am I missing?
Can I obtain access to my NVIDIA GPUs using the stock Docker (or podman) runtime?
If not, why not?
I certainly won't be able to answer every conceivable question related to this. I will try to give a summary. Some of what I write here is based on what's documented here and here. My discussion here will also be focused on linux, and docker (not windows, not singularity, not podman, etc.). I'm also not likely to be able to address in detail questions like "why don't other PCI devices have to do this?". I'm also not trying to make my descriptions of how docker works perfectly accurate to an expert in the field.
The NVIDIA GPU driver has components that run in user space and also other components that run in kernel space. These components work together and must be in harmony. This means the kernel mode component(s) for driver XYZ.AB must be used only with user-space components from driver XYZ.AB (not any other version), and vice-versa.
Roughly speaking, docker is a mechanism to provide an isolated user-space linux presence that runs on top of, and interfaces to, the linux kernel (where all the kernel space stuff lives). The linux kernel is in the base machine (outside the container) and much/most of linux user space code is inside the container. This is one of the architectural factors that allow you to do neato things like run an ubuntu container on a RHEL kernel.
From the NVIDIA driver perspective, some of its components need to be installed inside the container and some need to be installed outside the container.
Can I obtain access to my NVIDIA GPUs using the stock Docker (or podman) runtime?
Yes, you can, and this is what people did before nvidia-docker or the nvidia-container-toolkit existed. You need to install the exact same driver in the base machine as well as in the container. Last time I checked, this works (although I don't intend to provide instructions here.) If you do this, the driver components inside the container match those outside the container, and it works.
What am I missing?
NVIDIA (and presumably others) would like a more flexible scenario. The above description means that if a container was built with any other driver version (than the one installed on your base machine) it cannot work. This is inconvenient.
The original purpose of nvidia-docker was to do the following: At container load time, install the runtime components of the driver, which are present in the base machine, into the container. This harmonizes things, and although it does not resolve every compatibility scenario, it resolves a bunch of them. With a simple rule "keep your driver on the base machine updated to the latest" it effectively resolves every compatibility scenario that might arise from a mismatched driver/CUDA runtime. (The CUDA toolkit, and anything that depends on it, like CUDNN, need only be installed in the container.)
As you point out, the nvidia-container-toolkit has picked up a variety of other, presumably useful, functionality over time.
I'm not spending a lot of time here talking about the compatibility strategy ("forward") that exists for compiled CUDA code, and the compatibility strategy ("backward") that exists when talking about a specific driver and the CUDA versions supported by that driver. I'm also not intending to provide instructions for use of the nvidia-container-toolkit, that is already documented, and many questions/answers about it already exist also.
I won't be able to respond to follow up questions like "why was it architected that way?" or "that shouldn't be necessary, why don't you do this?"
To answer my own question: No, we do not need nvidia-container-runtime.
The NVIDIA shared libraries are tightly coupled to each point release of the driver. NVIDIA likes to say "the driver has components that run in user space", but of course that is a contradiction in terms. So for any version of the driver, you need to make the corresponding release of these shared libraries accessible inside the container.
A brief word on why this is a bad design: Apart from the extra complexity, the NVIDIA shared libraries have dependencies on other shared libraries in the system, in particular C and X11. If a newer release of the NVIDIA libraries ever required features from newer C or X11 libraries, a system running those newer libraries could never host an older container. (Because the container would not be able to run the newer injected libraries.) The ability to run old containers on new systems is one of the most important features of containers, at least in some applications. I guess we have to hope that never happens.
The HPC community figured this out and made it work some time ago. Here are some old instructions for creating a portable Singularity GPU container which injects the required NVIDIA shared libraries when the container runs. You could easily follow a similar procedure to create a portable OCI or Docker GPU container.
These days, Singularity supports a --nv flag to inject the necessary shared libraries automatically. It also supports a --rocm flag for AMD GPUs. (Yes, AMD chose the same bad design.) Presumably you could combine these flags if you needed both.
All of these details are pretty well-documented in the Singularity manual.
Bottom line: If you are asking the same question I was, try Singularity.
I wish to create a platform as a service in the financial markets using Erlang/Elixir. I will provide AWS lambda-style functions in financial markets, but rather than being accessible via web/rest/http, I plan to distribute my own ARM-based hardware terminals to clients (Nvidia Jetson TX2-based or similar, so decent hardware). They will access the functions from these terminals. I want said terminals to be full nodes in the system. So they will use the actor model to message pass to my central servers, and indeed, the terminals might message pass amongst each other if terminal users decide to put their own functions online.
Is this a viable model? Could I run 1000 terminals like this? 100 000? What kinds of limitations might I start bumping into? Is Erlang message routing scalable enough to imagine such a network still being performant if we had soft-real time financial markets streaming data flowing around? (mostly from central servers to terminals, but a good proportion possible moving directly around from terminal to terminal). We could have a system where up to 100k or more different "subscription" data channel processes were available, many of them taking input and producing output every second.
Basically I'd like a canonical guide to the scalability capabilities of an Erlang system something like the above. Ideally I'd also like some guide to the security implications of such a system ie. would global routing tables or any other part of the system be compromisable by a rogue terminal user, or can edge nodes be partly "sealed off" from sensitive parts of the rest of the Erlang network?
Note that I'd want to make heavy use of ports/NIFs for high-compute processes.
I would not pursue this avenue for various reasons, all of which hark back to the sort of systems that Erlang's distribution mechanism was developed for - a set of boards on a passive backplane: "free" local bandwidth and the whole machine sits in the same security domain. The Erlang distribution protocol is probably too chatty to work well on widely spread and large networks, and it is certainly too insecure. Unless you want nodes to be able to execute :os.cmd("rm -rf /") on each other, of course.
Use the Erlang distribution protocol in your central system to your heart's content, and have these terminals talk something that's data-only-over-SSL to that system and each other. On top of that, you can quite simply build a sort of overlay network to do whatever you want.
I recommend read this carefully and i recommend divide your service to little Micro-Services too.
Another benchmark is Investigating the Scalability Limits of
Distributed Erlang.
In the Joe Armstorng's book programming Erlang, he said:
"A few years ago, when I had my research hat on, I was working with PlanetLab. I had access to the PlanetLab a network, so I installed empty Erlang servers on all the PlanetLab machines (about 450 of them).
I didn’t really know what I would do with the machines, so I just set up the server infrastructure to do something later."
Do not use External ports, use internal drivers which are written in C or C++ instead.
You will find a lot of information regarding erlang Architectures is this answer: How scalable is distributed Erlang?
Short answer is, there is a pratical limitation of nodes in a cluster, but this limitation can be breach with federations fairly easily.
EDIT 1/ Further more I would recommend to read this book : Designing for scalability with Erlang/OTP
I'm moving from windows programming (By windows programming I mean using Windows API) to Linux Programming.
For programming Windows, the option we have is Win32API (MFC is just a C++ wrapper for the same).
I want to know if there is something like Linux API (equivalent to WINAPI) that is exposed directly to the programmer? Where can I find the reference?
With my little knowledge of POSIX library I see that it wraps around part of Linux API. But what about creating GUI applications? POSIX doesn't offer that. I know there are tons of 3rd party Widget toolkits like gtk, Qt etc. But I don't want to use the libraries that encapsulates Linux API. I want to learn using the "Core Linux API".
If there are somethings that I should know, please inform. Any programmer who is familiar with both Windows & Linux programming, please map the terminologies of Linux world so that I can quickly move on.
Any resources (books,tutorials,references) are highly appreciated.
I think you're looking for something that doesn't exactly exist. Unlike the Win32 API, there is no "Linux API" for doing GUI applications. The closest you can get is the X protocol itself, which is a pretty low level way of doing GUI (it's much more detailed and archaic than Win32 GDI, for example). This is why there exist wrappers such as GTK and Qt that hide the details of the X protocol.
The X protocol is available to C programs using XLib.
What you must understand is that Linux is very bare as to what is contained within it. The "Core" Linux API is POSIX and glibc. Linux is NOT graphical by default, so there is no core graphics library. Really, Windows could be stripped down to not have graphics also and thus not have parts of the win32 API like GDI. This you must understand. Linux is very lightweight compared to Windows.
For Linux there are two main graphical toolkits, GTK and Qt. I myself prefer GTK, but I'd research both. Also note that GTK and Qt exist for Windows to, because they are just wrappers. If you go take a look at the X protocol code for say xterm, you'll see why no one tries to actually creating graphical applications on top of it.
Oh, also SDL is pretty nice, it is pretty bare, but it is nice if your just needing a framebuffer for a window. It is portable between Linux and Windows and very easy to learn. But it will only stretch so far..
Linux and win aren't quite as different as it looks.
On both systems there exists a kernel that is not graphical.
It's just that Microsoft doesn't document this kernel and publishes an API that references various different components.
On Unix, it's more transparent. There really is a (non-GUI) kernel API and it is published. Then, there are services that run on top of this, optionally, and their interfaces are published without an attempt to merge them into an imaginary layer that doesn't really exist.
So, the lowest GUI level is a the X Window System and it has a lowest level library called Xlib. There are various libraries that run on top of this one, as you have noted.
I would highly recommended looking at the QT/C++ UI framework, it's arguably the most comprehensive UI toolkit for any platform.
We're using it at work developing cross platform apps that run on windows, osx and linux.
It also runs on Nokia's smart phone Operating System Maemo which has recently been merged with Intel's Moblin Linux OS, now called MeeGo.
This is going to sound insane since you're asking about "serious" stuff like C++ and C (and the "core linux API"), but you might want to consider building in something else. For instance:
Java Swing (many people love it! Others hate it and call it obsolete)
Mono GTK# (C# or VisualBasic or whatever you want, lots of people say it's pretty cool, but they're not not that many people)
Adobe AIR (ActionScript, you might hate it)
Titanium (totally new and unproven, but getting a lot of buzz in the iPhone world, at least)
And many other possibilities, some of which let you work on multiple platforms at once.
Sorry if this answer is not at all what you're looking for. The "real" answers on Linux are "pick a toolkit," which is also no answer at all :)
Have a look at Cairo. This something roughly similar to GDI+ and is under the hood of some of of the few usable GUI programs for Linux i.e. Firefox or Eclipse (SWT). It wraps most the natsy and ancient Linux stuff for you into a nice API that runs on most Linux installations without locking you into a entire subsystems like GTK or QT.
There is also the docs for the two different desktop platforms: Gnome and KDE that might help you down that road.
I'm going to start the development of an application on a Zynq board. My task is basically to port an existing application running on a Microblaze on the dual core ARM.
What I'm wondering about is which O.S. to use on the new system, because I have no experience at all in this field.
It seems to me that there are four main approaches:
1) Petalinux (use both cores)
2) Petalinux+FreeRTOS (use both cores)
3) FreeRTOS (use only a core)
4) Baremetal (use only a core)
What my application has to do is to move a big amount of data between Ethernet and multiple custom links, so it has to serve a lot of interrupts and command a lot of DMA operations.
How much is the overhead introduced by Petalinux in the interrupt service with respect to baremetal or FreeRTOS? Do you think that, for this kind of work, is faster a single core application running without any OS or, for example, a Petalinux application that has the overhead of the OS (and of the synchronization mechanisms like semaphores or mutex)?
I know the question is not precise and quite vague, but having no experience in the field I strongly need some initial hints.
Thank you.
As you say, this is too vague to give a considered answer because it really depends on your application (when does it not). If you need all the 'stuff' that is available for Linux and boot time is not an issue then go with that. If you need actual real time behaviour, fast boot time, simplicity, and don't need anything Linux specific, then FreeRTOS might be your best choice. There is a Zynq FreeRTOS TCP project that uses the BSD style sockets interface (like Linux) here: http://www.freertos.org/FreeRTOS-Plus/FreeRTOS_Plus_TCP/TCPIP_FAT_Examples_Xilinx_Zynq.html
Usually the performance should not differ alot.
If you compile your linux with a well optimizing compiler there is a good chance to be faster compared to bare metal.
But if you need hard real time linux is not suitable for you.
There is a good whitepaper from Altera but should fit for Xilinx too:
whitepaper on real time jitter
I completely newbie in device drivers, so I hope my question is in place, but I need to develop a driver to control some equipment. I was thinking on using Linux as the host OS, but not sure if it is such good idea. I've heard some horror stories about the mess of developing device drivers under Linux. Is there a better alternative under the *Nix world? Or maybe should I check other OSes?
Linux documentation is basically non-existent (similar to other platforms). However, there are a few books which do cover enough information to get started, and the trickier kernel bits can borrowed from other drivers (yay for Open Source).
However, it is one of the easiest current platforms to develop drivers for. There are cleaner models, such as QNX, but that product is sadly near the end (and doesn't support 1/10th as much as hardware as Linux)
What type of device is the driver targetting? Many times you can avoid writing in-kernel drivers (for instance, using libusb in userspace, or the user space IO framework)