As far as I understand Docker uses memory mapped files to start from image. Since I can do this over and over again and as far as I remember start different instances of the same image in parallel, I guess docker abstracts the file system and stores changes somewhere else.
I wonder if docker can be configured (or does it by default) to run in a memory only mode without some sort of a temporary file?
Docker uses a union filesystem that allows it to work in "layers" (devicemapper, BTRFS, etc). It's doing copy-on-write so that starting new containers is cheap, and when it performs the first write, it actually creates a new layer.
When you start a container from an image, you are not using memory-mapped files to restore a frozen process (unless you built all of that into the image yourself...). Rather, you're starting a normal Unix process but inside a sandbox where it can only see its own unionfs filesystem.
Starting many copies of an image where no copy writes to disk is generally cheap and fast. But if you have a process with a long start-up time, you'll still pay that cost for every instance.
As for running Docker containers wholly in memory, you could create a RAM disk and specify that as Docker's storage volume (configurable, but typically located under /var/lib/docker).
In typical use-cases, I would not expect this to be a useful performance tweak. First, you'll spend a lot of memory holding files you won't access. The base layer of an image contains most Linux system files. If you fetch 10 packages from the Docker Hub, you'll probably hit 20G worth of images easily (after that the storage cost tends to plateau). Second, the system already manages memory and swapping pretty well (which is why a RAM disk is a performance tweak) and you get all of that applied to processes running inside a container. Third, for most of the cases where a RAM disk might help, you can use the -v flag to mount the disk as a volume on the container rather than needing to store your whole unionfs there.
Related
I am trying to benchmark I/O performance on my host and docker container using flexible IO tool with O_direct enabled in order to bypass memory caching. The result is very suspicious. docker performs almost 50 times better than my host machine which is impossible. It seems like docker is not bypassing the caching at all. even if I ran it with --privileged mode. This is the command I ran inside of a container, Any suggestions?
fio --name=seqread --rw=read --direct=1 --ioengine=libaio --bs=4k --numjobs=1 --size=10G --runtime=600 --group_reporting --output-format=json >/home/docker/docker_seqread_4k.json
(Note this isn't really a programming question so Stackoverflow is the wrong place to ask this... Maybe Super User or Serverfault would be a better choice and get faster answers?)
The result is very suspicious. docker performs almost 50 times better than my host machine which is impossible. It seems like docker is not bypassing the caching at all.
If your best case latencies are suspiciously small compared to your worst case latencies it is highly likely your suspicions are well founded and that kernel caching is still happening. Asking for O_DIRECT is a hint not an order and the filesystem can choose to ignore it and use the cache anyway (see the part about "You're asking for direct I/O to a file in a filesystem but...").
If you have the option and you're interested in disk speed, it is better to do any such test outside of a container (with all the caveats that implies). Another option when you can't/don't want to disable caching is ensure that you do I/O that is at least two to three times the size (both in terms of amount and the region being used) of RAM so the majority of I/O can't be satisfied by buffers/cache (and if you're doing write I/O then do something like end_fsync=1 too).
In summary, the filesystem being used by docker may make it impossible to accurately do what you're requesting (measure the disk speed by bypassing cache while using whatever your default docker filesystem is).
Why a Docker benchmark may give the results you expect
The Docker engine uses, by default, the OverlayFS [1][2] driver for data storage in a containers. It assembles all of the different layers from the images and makes them readable. Writing is always done to the "top" layer, which is the container storage.
When performing reads and writes to the container's filesystem, you're passing through Docker's overlay2 driver, through the OverlayFS kernel driver, through your filesystem driver (e.g. ext4) and onto your block device. Additionally, as Anon mentioned, DIRECT/O_DIRECT is just a hint, and may not be respected by any of the layers you're passing through.
Getting more accurate results
To get an accurate benchmarks within a Docker container, you should write to a volume mount or change your storage driver to one that is not overlaid, such as the Device Mapper driver or the ZFS driver.
Both the Device Mapper driver and the ZFS driver require a dedicated block device (you'll likely need a separate hard drive), so using a volume mount might be the easiest way to do this.
Use a volume mount
Use the -v options with a directory that sits on a block device on your host.
docker run -v /absolute/host/directory:/container_mount_point alpine
Use a different Docker storage driver
Note that the storage driver must be changed on the Docker daemon (dockerd) and cannot be set per container. From the documentation:
Important: When you change the storage driver, any existing images and containers become inaccessible. This is because their layers cannot be used by the new storage driver. If you revert your changes, you can access the old images and containers again, but any that you pulled or created using the new driver are then inaccessible.
With that disclaimer out of the way, you can change your storage driver by editing daemon.json and restarting dockerd.
{
"storage-driver": "devicemapper",
"storage-opts": [
"dm.directlvm_device=/dev/sd_",
"dm.thinp_percent=95",
"dm.thinp_metapercent=1",
"dm.thinp_autoextend_threshold=80",
"dm.thinp_autoextend_percent=20",
"dm.directlvm_device_force=false"
]
}
Additional container benchmark notes - kernel
If you are trying to compare different flavors of Linux, keep in mind that Docker is still running on your host machine's kernel.
I was asked this question in an interview and i m not sure of the correct answer hence I would like your suggestions.
I was asked whether we should persist production critical data inside of the docker instance or outside of it? What would be my choice and the reasons for it.
Would your answer differ incase we have a non-prod non critical data ?
Back your answers with reasons.
Most data should be managed externally to containers and container images. I tend to view data constrained to a container as temporary (intermediate|discardable) data. Otherwise, if it's being captured but it's not important to my business, why create it?
The name "container" is misleading. Containers aren't like VMs where there's a strong barrier (isolation) between VMs. When you run multiple containers on a single host, you can enumerate all their processes using ps aux on the host.
There are good arguments for maintaining separation between processes and data and running both within a single container makes it more challenging to retain this separation.
Unlike processes, files in container layers are more isolated though. Although the layers are manifest as files on the host OS, you can't simply ls a container layer's files from the host OS. This makes accessing the data in a container more complex. There's also a performance penalty for effectively running a file system atop another file system.
While it's common and trivial to move container images between machines (viz docker push and docker pull), it's less easy to move containers between machines. This isn't generally a problem for moving processes as these (config aside) are stateless and easy to move and recreate, but your data is state and you want to be able to move this data easily (for backups, recovery) and increasingly to move amongst a dynamic pool of nodes that perform processing upon it.
Less importantly but not unimportantly, it's relatively easy to perform the equivalent of a rm -rf * with Docker by removing containers (docker container rm ...) and thereby deleting the application and your data.
The two very most basic considerations you should have here:
Whenever a container gets deleted, everything in the container filesystem is lost.
It's extremely common to delete containers; it's required to change many startup options or to update a container to a newer image.
So you don't really want to keep anything "in the container" as its primary data storage: it's inaccessible from outside the container, and will get lost the next time there's a critical security update and you must delete the container.
In plain Docker, I'd suggest keeping
...in the image: your actual application (the compiled binary or its interpreted source as appropriate; this does not go in a volume)
...in the container: /tmp
...in a bind-mounted host directory: configuration files you need to push into the container at startup time; directories of log files produced by the container (things where you as an operator need to directly interact with the files)
...in either a named volume or bind-mounted host directory: persistent data the container records in the filesystem
On this last point, consider trying to avoid this layer altogether; keeping data in a database running "somewhere else" (could be another container, a cloud service like RDS, ...) simplifies things like backups and simplifies running multiple replicas of the same service. A host directory is easier to back up, but on some environments (MacOS) it's unacceptably slow.
My answers don't change here for "production" vs. "non-production" or "critical" vs. "non-critical", with limited exceptions you can justify by saying "it's okay if I lose this data" ("because it's not the master copy of it").
As far as I understand Docker it is virtualizing a system and loads a certain image along with booting it and doing some other stuff. Since I can use different OS with docker, I think it is quite far reaching in order to provide such an abstraction.
In order to speed up setting up a test environment, is it possible to freeze a docker instance in a certain state (like after initializing the database) and rerun the image from this point?
Docker is not virtualizing a system and boots it. Instead of loading its own system kernel into memory it simply creates encapsulated processes that run in the Linux kernel of the host system. That is by the way the reason why a Linux host is required.
There is no virtualization but just process/resource encapsulation. More details about the Docker architecture and its concepts you can find in the documentation.
A "freeze" would be a commit of your base image which you used to run your container. You can get back to that commit at any point in time by using the image id.
I have a Lamp Docker Image.
I want to start 500 containers of this image, how many RAM i need?
I have tracked memory usage of each new container and it nearly the same as any other container of its image.
So,if single container is using 200 MB, I can start 5 containers on Linux machine with 1 GB RAM.
My question is:
Is docker container using same memory as, for example, same Virtual Machine Image?
May be I am doing something wrong in docker configuration or docker files?
docker stats might give you the feedback you need. https://docs.docker.com/engine/reference/commandline/stats/
I don't know the exact details of the docker internals, but the general idea is that Docker tries to reuse as much as it can. So if you start five identical containers, it should run much faster than a virtual machine, because docker should only have one instance of the base image and file system which all containers refer to.
Any changes to the file system of one container will be added as a layer on top, only marking the change. The underlying image will not be changed, so the five containers can still refer to the same single base image.
The virtual machine however (i believe) will have a complete copy of the file system for each of the five instances, because it doesn't use a layered file system.
So I'm not sure how you can determine exactly how much memory you need, but this should make the concept clearer to you. You could start one container to see the 'base memory' that will be needed for one and then each new container should only add a smaller constant amount of memory and that should give you a broad idea how much you need.
I am currently reading up on Docker. From what I understand, a container which is based on an image saves only the changes. If I were to use this in a production setup, does it persist it as soon as changes are written to disk by applications running "inside" the container or does it have to be done manually?
My concern is - what if the host abruptly shuts down? Will all the changes be lost?
The theory is that there's no real difference between a Docker container and a classical VM or physical host in most situations.
If the host abruptly dies, you can loose recent data using a container as well as using a physical host:
your application may not have decided to really send the write operation to save the data on disk,
the Operating System may have decided to wait a bit before sending data to storage devices
the filesystem may not have finished the write
the data may not have been really flushed to the physical storage device.
Now by default, Docker uses AUFS (stackable filesystem) which works at the file level.
If you're writing to a file that was existing in the Docker image, AUFS will first copy this base file to the upper, writable layer (container), before writing your change. This causes a delay depending on the size of the original file. Interesting and more technical information here.
I guess that if a power cut occurs happens while this original file is being copied and before your changes have been written, then that would be one reason to get more data loss with a Docker container than with any "classical" host.
You can move your critical data to a Docker "volume", which would be a regular filesystem on the host, bind-mounted into the container. This is the recommended way to deal with important data that you want to keep across containers deployments
To mitigate the AUFS potential issue, you could tell Docker to use LVM thin provisioning block devices instead of AUFS (wipe /var/lib/dockerand start the daemon with docker -d -s devicemapper). However I don't know if this storage backend received as much testing as the default AUFS one (it works ok for me though).