Docker and file locking - docker

I wrote a simple go application and added a flock system to prevent being running twice at the same time:
import "github.com/nightlyone/lockfile"
lock, err := lockfile.New(filepath.Join(os.TempDir(), "pagerduty-read-api.lock"))
if err != nil {
panic(err)
}
if err = lock.TryLock(); err != nil {
fmt.Println("Already running.")
return
}
defer lock.Unlock()
It works well on my host. On docker, I tried to run it with volume sharing of tmp:
docker run --rm -it -v /tmp:/tmp my-go-binary
But it does not work. I suppose it's because the flock system is not ported on volume sharing.
My question: Does Docker have option to make flock working between running instance? If not, what are my other options to have the same behavior?
Thanks.

This morning I wrote a little Python test program that just writes one million consecutive integers to a file, with flock() locking, obtaining and releasing the lock once for each number appended. I started up 5 containers, each running that test program, and each writing to the same file in a docker volume.
With the locking enabled, the numbers were all written without interfering with each other, and there were exactly 5 million integers in the file. They weren't consecutive when written this way, but that's expected and consistent with flock() working.
Without locking, many of the numbers were written in a manner that indicates the numbers were running afoul of the multitasking sans locking. There were only 3,167,546 numbers in the file and there were 13,357 blank lines. That adds up to the 3,180,903 lines in the file - substantially different than the desired 5,000,000.
While a program cannot definitively prove that there will never be problems just by testing many times, to me that's a pretty convincing argument that Linux flock() works across containers.
Also, it just kinda makes sense that flock would work across containers; containers are pretty much just a shared kernel, distinct pid, distinct file (other than volumes) and distinct IP port space.
I ran my test on a Linux Mint 19.1 system with Linux kernel 4.15.0-20-generic, Docker 19.03.0 - build aeac949 and CPython 3.6.8.
Go is a cool language, but why flock() didn't appear to be working across volumes in your Go program I do not know.
HTH.

I suppose that you want to use docker volume or you may need some other docker volume plugins.
According to this article, Docker Volume File permission and locking, docker volumes only provides a way to define a volume to use by multi containers or use by a container after restarting.
In docker volume plugins, flocker may meet your requirements.
Flocker is an open-source Container Data Volume Manager for your Dockerized applications.
BTW, if you are using kubernetes, you may need to learn more about persistent volume, persistent volume claim, storage class.

I've been doing some research on this myself recently, and the issue is that nightlyone/lockfile doesn't actually use the flock syscall. Instead, the lockfile it writes is a PIDfile, a file that just contains the PID (Process IDentifier) of the file that created it.
When checking if a lock is locked, lockfile checks the PID stored in the lockfile, and if it's different to the PID of the current process, it sees it as locked.
The issue is that lockfile doesn't have any special logic to know if it's in a docker container or not, and PIDs get a little muddled when working with docker; the PID of a process when viewed from inside the container will be different from the PID of a process outside the container.
Where this often ends up is that we have two containers running your code above, and they both have PIDs of 1 within their containers. They'll each try to create a lockfile, writing what they think their PID is (1). They then both think they hold the lock - after all, their PID is the one that wrote it! So the lockfile is ignored.
My advice is to switch to a locking implementation that uses flock. I've switched to flock, and it seems to work okay.

Related

when to run or start a docker container

As far as I understood, 'docker create' creates a new container from an image, 'docker start' starts the container (conditions apply so the container doesn't stop again immediately) and 'docker run' does both (with some differences in the details, but let's ignore them for now).
What I don't understand is when to use which. Let me explain:
For a normal executable, I run it by starting a new process. When the process dies, it is gone, and I start the executable again by creating a new process. There is no other way.
Some executables become daemons and keep running, so now there are three ways: Interact with a running daemon, start a second one, or stop and restart. With very few exceptions, running a second instance of the daemon is not useful, and you'd usually interact with the running instance, restarting it only when it locks up or to change its configuration. So again, it is clear to me how to use such a program.
For docker applications, things are not as clear to me. Of course, we have daemons and non-daemons there too, but for both of them, it is not clear to me whether I should use them by creating a new container or restarting an existing container. For daemons, there is the added distinction of starting a non-running daemon and restarting a running daemon, each of which can be done with the existing or a new container. On top of that, there is a third class of applications besides fire-and-forget and daemons, namely VM-like containers that just have a file system and a shell, and contain multiple programs to run.
As far as my knowledge goes, all this is especially relevant when the application stores run-time data on a file system that is private to the container, so creating a second container would create a second file system independent of the first one (with important files possibly spread across those file systems).
Of those docker-based applications I have used, none seems to have described this in its manual, so it seems that there are best practices on when to create or re-use containers, but then I could not find anything about that on the internet either. I could only find sites that explain the mechanics of those commands, but no best practices.
When should I create a new container, and when should I restart an existing one? If "it depends", how do you find out?

Where should production critical and non-production non-critical data stored?

I was asked this question in an interview and i m not sure of the correct answer hence I would like your suggestions.
I was asked whether we should persist production critical data inside of the docker instance or outside of it? What would be my choice and the reasons for it.
Would your answer differ incase we have a non-prod non critical data ?
Back your answers with reasons.
Most data should be managed externally to containers and container images. I tend to view data constrained to a container as temporary (intermediate|discardable) data. Otherwise, if it's being captured but it's not important to my business, why create it?
The name "container" is misleading. Containers aren't like VMs where there's a strong barrier (isolation) between VMs. When you run multiple containers on a single host, you can enumerate all their processes using ps aux on the host.
There are good arguments for maintaining separation between processes and data and running both within a single container makes it more challenging to retain this separation.
Unlike processes, files in container layers are more isolated though. Although the layers are manifest as files on the host OS, you can't simply ls a container layer's files from the host OS. This makes accessing the data in a container more complex. There's also a performance penalty for effectively running a file system atop another file system.
While it's common and trivial to move container images between machines (viz docker push and docker pull), it's less easy to move containers between machines. This isn't generally a problem for moving processes as these (config aside) are stateless and easy to move and recreate, but your data is state and you want to be able to move this data easily (for backups, recovery) and increasingly to move amongst a dynamic pool of nodes that perform processing upon it.
Less importantly but not unimportantly, it's relatively easy to perform the equivalent of a rm -rf * with Docker by removing containers (docker container rm ...) and thereby deleting the application and your data.
The two very most basic considerations you should have here:
Whenever a container gets deleted, everything in the container filesystem is lost.
It's extremely common to delete containers; it's required to change many startup options or to update a container to a newer image.
So you don't really want to keep anything "in the container" as its primary data storage: it's inaccessible from outside the container, and will get lost the next time there's a critical security update and you must delete the container.
In plain Docker, I'd suggest keeping
...in the image: your actual application (the compiled binary or its interpreted source as appropriate; this does not go in a volume)
...in the container: /tmp
...in a bind-mounted host directory: configuration files you need to push into the container at startup time; directories of log files produced by the container (things where you as an operator need to directly interact with the files)
...in either a named volume or bind-mounted host directory: persistent data the container records in the filesystem
On this last point, consider trying to avoid this layer altogether; keeping data in a database running "somewhere else" (could be another container, a cloud service like RDS, ...) simplifies things like backups and simplifies running multiple replicas of the same service. A host directory is easier to back up, but on some environments (MacOS) it's unacceptably slow.
My answers don't change here for "production" vs. "non-production" or "critical" vs. "non-critical", with limited exceptions you can justify by saying "it's okay if I lose this data" ("because it's not the master copy of it").

What's the purpose of the node-modules container in wolkenkit?

That container is built when deploying the application.
Looks like its purpose is to share dependencies across modules.
It looks like it is started as a container but nothing is apparently running, a bit like an init container.
Console says it starts/stops that component when using respective wolkenkit start and wolkenkit stop command.
On startup:
On shutdown:
When you docker ps, that container cannot be found:
Can someone explain these components?
When starting a wolkenkit application, the application is boxed in a number of Docker containers, and these containers are then started along with a few other containers that provide the infrastructure, such as databases, a message queue, ...
The reason why the application is split into several Docker containers is because wolkenkit builds upon the CQRS pattern, which suggests separating the read side of an application from the application's write side, and hence there is one container for the read side, and one for the write side (actually there are a few more, but you get the picture).
Now, since you may develop on an operating system other than Linux, the wolkenkit application may run under a different operating system than when you develop it, as within Docker it's always Linux. This means that the start command can not simply copy over the node_modules folder into the containers, as they may contain binary modules, which are then not compatible (imagine installing on Windows on the host, but running on Linux within Docker).
To avoid issues here, wolkenkit runs an npm install when starting the application inside of the containers. The problem now is that if wolkenkit did this in every single container, the start would be super slow (it's not the fastest thing on earth anyway, due to all the Docker building and starting that's happening under the hood). So wolkenkit tries to optimize this as much as possible.
One concept here is to run npm install only once, inside of a container of its own. This is the node-modules container you encountered. This container is then linked as a volume to all the containers that contain the application's code. This way you only have to run npm install once, but multiple containers can use the outcome of this command.
Since this container now contains data, but no code, it only has to be there, it doesn't actually do anything. This is why it gets created, but is not run.
I hope this makes it a little bit clearer, and I was able to answer your question :-)
PS: Please note that I am one of the core developers of wolkenkit, so take my answer with a grain of salt.

Is it best practice to daemonize a process within docker?

Many best practice guides emphasize making your process a daemon and having something watch it to restart in case of failure. This made sense for a while. A specific example can be sidekiq.
bundle exec sidekiq -d
However, with Docker as I build I've found myself simply executing the command, if the process stops or exits abruptly the entire docker container poofs and a new one is automatically spun up - basically the entire point of daemonizing a process and having something watch it (All STDOUT is sent to CloudWatch / Elasticsearch for monitoring).
I feel like this also tends to re-enforce the idea of a single process in a docker container, which if you daemonize would tend to in my opinion encourage a violation of that general standard.
Is there any best practice documentation on this even if you're running only a single process within the container?
You don't daemonize a process inside a container.
The -d is usually seen in the docker run -d command, using a detached (not daemonized) mode, where the the docker container would run in the background completely detached from your current shell.
For running multiple processes in a container, the background one would be a supervisor.
See "Use of Supervisor in docker" (or the more recent docker --init).
Some relevent 12 Factor app recommendations
An app is executed in the execution environment as one or more processes
Concurrency is implemented by running additional processes (rather than threads)
Website:
https://12factor.net/
Docker was open sourced by a PAAS operator (dotCloud) so it's entirely possible the authors were influenced by this architectural recommendation. Would explain why Docker is designed to normally run a single process.
The thing to remember here is that a Docker container is not a virtual machine, although it's entirely possible to make it quack like one. In practice a docker container is a jailed process running on the host server. Container orchestration engines like Kubernetes (Mesos, Docker Swarm mode) have features that will ensure containers stay running, replacing them should the need arise.
Remember my mention of duck vocalization? :-) If you want your container to run multiple processes then it's possible to run a supervisor process that keeps everything healthy and running inside (A container dies when all processes stop)
https://docs.docker.com/engine/admin/using_supervisord/
The ultimate expression of this VM envy would be LXD from Ubuntu, here an entire set of VM services get bootstrapped within LXC containers
https://www.ubuntu.com/cloud/lxd
In conclusion is it a best practice? I think there is no clear answer. Personally I'd say no for two reasons:
I'm fixated on deploying 12 factor compliant applications, so married to the single process model
If I need to run two processes on the same set of data, then in Kubernetes I can run containers within the same POD... Means Kubernetes manages the processes (running as separate containers with a common data volume).
Clearly my reasons are implementation specific.
There are multiple run supervisors that can help you take a foreground process (or multiple ones) run them monitored and restart them on failure (or exit the container).
one is runit (http://smarden.org/runit/), which I have not used myself.
my choice is S6 (http://skarnet.org/software/s6/). someone already built a container envelope for it, named S6-overlay (https://github.com/just-containers/s6-overlay) which is what I usually use if/when I need to have a user-space process run as daemon. it also has facets to do prep work on container start, change permissions and more, in runtime.
tl;dr: I can't find a best practices document that relates directly to this for docker, but I agree with you.
The only best "Best Practices" for docker I could find was at dockers own site, which states that containers should be one process. In my mind, that means foregrounded processes as well. So basically, I've drawn the same conclusion as you. (You've probably read that too, but this is for anyone else reading this).
Honestly, I think we are still in (relatively) new territory with best practices for docker. Anecdotally, it has been a best practice in the organizations I've worked with. The number of times I've felt more satisfied with a foregrounded process has been significantly greater then the times I've said to myself "Boy, I sure wish I backgrounded that one." In fact, I don't think I've ever said that.
The only exception I can think of is when you are trying to evaluate software and need a quick and dirty way to ship infrastructure off to someone. EG: "Hey, there is this new thing called LAMP stacks I just heard of, here is a docker container that has all the components for you to play around with". Again, though, that's an outlier and I would shudder if something like that ever made it to production or even any sort of serious development environment.
Additionally, it certainly forces a micro-architecture style, which I think is ultimately a good thing.

Docker separation of concerns / services

I have a laravel project which I am using with docker. Currently I am using a single container to host all the services (apache, mySQL etc) as well as the needed dependencies (project files, git, composer etc) I need for my project.
From what I am reading the current best practice is to put each service into a separate container. So far this seems simple enough since these services are designed to run at length (apache server, mySQL server). When I spin up these 'service' containers using -d they remain running (docker ps) since their main process continuously runs.
However, when I remove all the services from my project container, then there is no main process left to continuously run. This means my container immediately exits once spun up.
I have read the 'hacks' of running other processes like tail -f /dev/null, sleep infinity, using interactive mode, installing supervisord (which I assume would end up watching no processes in such containers?) and even leaving the container to run in the foreground (taking up a terminal console...).
How do I network such a container to keep it running like the abstracted services but detached without these hacks? I cannot seem to find much information on this in the official docker docs nor can I find any examples of other projects (please link any)
EDIT: I am not talking about volumes / storage containers to store the data my project processes, but rather how I can use a container to store the project itself and its dependencies that aren't services (project files, git, composer)
when you run the container try running with the flags ...
docker run -dt ..... etc
you might even try .....
docker run -dti ..... etc
let me know if this brings any joy. has certainly worked for me on occassions.
i know you wanted to avoid hacks but if the above fails then also add ...
CMD cat
to the end of your Dockerfile - it is a hack but is the cleanest hack :)
So after reading this a few times along with Joachim Isaksson's comment, I finally get it. Tools don't need the containers to run continuously to use. Proper separation of the project files, services (mySQL, apache) and tools (git, composer) are done differently.
The project files are persisted within a data volume container. The services are networked since they expose ports. The tools live in their own containers which share the project files data volume - they are not networked. Logs, databases and other output can be persisted in different volumes.
When you wish to run one of these tools, you spin up the tool container by passing the relevant command using docker run. The tool then manipulates the data within the directory persisted within the shared volume. The containers only persist as long as the command to manipulate the data within the shared volume takes to run and then the container stops.
I don't know why this took me so long to grasp, but this is the aha moment for me.

Resources