I am planning a setup where, the docker containers are using remote volume - volume that have ssh-ed to another machine and it is reading all the time.
Lets say we have 5 containers using that remote volume. In my understanding, the docker is ssh-ed to the remote machine and constantly reading on certain directory (with about 100 files, not more than few MB).
Presumably that constant reading will put some load to the remote machine. Will that load be significant or it can be negligible? There is php-fpm and Apache2 on the remote machine, will the constant reading slow down that web server? Also, how often the volume is refreshing the files?
Sincerely.
OK after some testing:
I have created a remote volume with vieux/sshfs driver.
Created an ubuntu container with the volume mounted under certain folder.
Then tail a txt file from the container itself.
Write to that txt file form the remote machine (the one that contains the physical folder).
I have found out, if we write to the file continuously (like echo "whatever" >> thefile.txt). The changes appear all at once after few seconds, not one by one as they have been introduced. Also, if I print or list the files in the mounted directory, the response is instant. This makes me thing, that Docker is making local copy of the folder ssh-ed in the volume and refreshes it every 5 sec or so. Basically negligible load after the folder is copied once.
Also, when trying to write from the container to the mounted folder, the changes on the file are reflected almost instantly (considering any latency). Which makes me think that the daemon is propagating the write changes instantly.
In conclusion - reading a remote folder, puts negligible load to the remote machine. The plan is to use such setup in production environment, so we don't have to pull changes on two different places (prod server and machine which is sharing (local) volume between containers).
If there is anyone who can confirm my findings, that would be great.
Sincerely
Related
Synopsis. A remote instance gets connected to the Internet via satellite modem when technician visits the cabin. Technician setups the application stack via docker compose and leaves the location. The location has no internet connection and periodically loses electricity (once in a few days).
The application stack is typical, like mysql + nodejs. And it is used by "polar bears". I mean nobody, it is a monitoring app.
How to ensure that docker images will be persisted for an undefined amount of time and the compose stack survives through endless reboots?
Unfortunately there is no real easy solution.
But with a little bit of yq magic to parse docker-compose.yaml and docker save command it is possible to store the images locally to a specific location.
Then we can add startup script to import these images using docker load into the local docker cache.
I am a beginner with Docker and I have been searching for 2 days now and I do not understand which would be a better solution.
I have a docker container on a Ubuntu server. I need to copy many large video files to the Ubuntu host via FTP. Docker via cron will process the videos using ffmpeg and save the result to the Ubuntu host somehow so the files are accessible via FTP.
What is the best solution:
create a bind drive - I understand the host may change files in the bind drive
create a volume but I do not understand how may I add files to the volume
create a folder on the Ubuntu and have a cron that will copy using "docker cp" command and after a video has been processed to copy it to the host?
Thank you in advance.
Bind-mounting a host directory for this is probably the best approach, for exactly the reasons you lay out: both the host and container can directly read and write to it, but the host can't easily write to a named volume. docker cp is tricky, you note the problem of knowing when the process is completed, and anyone who can run any docker command at all can pretty trivially root the host; you don't want to give this permission to something network-facing.
If you're designing a larger-scale system, you also might consider an approach where no files are actually shared at all. The upload server sends the files (maybe via HTTP POST) to an internal storage service, then posts a message to a message queue (maybe RabbitMQ). That then retrieves the files from the storage service, does its work, uploads the result, and posts a response message. The big advantages of this approach are being able to run it on multiple systems, easily being able to scale the individual components of it, and not needing to worry about filesystem permissions. But, it's a much more involved design.
We have to deploy a dockerized web app into a closed external system(our client's server).
(our image is made of gunicorn, nginx, django-python web app)
There are few options i have already considered:
option-1) using docker registries: push image into registry, pull
from client's system run docker-compose up with pulled image
option-2) docker save/load .tar files: docker save image in local dev
environment, move .tar file into the client's system and run docker load
(.tar file) there.
Our current approach:
we want to move source code inside a docker image(if possible).
we can't make our private docker registry public --yet--(so option-1 is gone)
client servers are only accessible from their internal local network(has no connection to any other external network)
we dont want to copy all the files when we make an update(to our app), what we want to do is somehow detect diff or changes on docker image and copy/move/update only changed parts of app into client's server.(option-2 is gone too)
Is there any better way to deploy to client's server with the approach explained above?
PS: I'm currently checking "docker commit": what we could do is, docker load our base image into client's server, start container with that image and when we have an update we could just copy our changed files into that container's file system, then docker commit(in order to keep changed version of container). But the thing i don't like in that option is we would need to keep changes in our minds, then move our changed files(like updated .py or .html files) to client's server.
Thanks in advance
I am currently reading up on Docker. From what I understand, a container which is based on an image saves only the changes. If I were to use this in a production setup, does it persist it as soon as changes are written to disk by applications running "inside" the container or does it have to be done manually?
My concern is - what if the host abruptly shuts down? Will all the changes be lost?
The theory is that there's no real difference between a Docker container and a classical VM or physical host in most situations.
If the host abruptly dies, you can loose recent data using a container as well as using a physical host:
your application may not have decided to really send the write operation to save the data on disk,
the Operating System may have decided to wait a bit before sending data to storage devices
the filesystem may not have finished the write
the data may not have been really flushed to the physical storage device.
Now by default, Docker uses AUFS (stackable filesystem) which works at the file level.
If you're writing to a file that was existing in the Docker image, AUFS will first copy this base file to the upper, writable layer (container), before writing your change. This causes a delay depending on the size of the original file. Interesting and more technical information here.
I guess that if a power cut occurs happens while this original file is being copied and before your changes have been written, then that would be one reason to get more data loss with a Docker container than with any "classical" host.
You can move your critical data to a Docker "volume", which would be a regular filesystem on the host, bind-mounted into the container. This is the recommended way to deal with important data that you want to keep across containers deployments
To mitigate the AUFS potential issue, you could tell Docker to use LVM thin provisioning block devices instead of AUFS (wipe /var/lib/dockerand start the daemon with docker -d -s devicemapper). However I don't know if this storage backend received as much testing as the default AUFS one (it works ok for me though).
I am playing a couple of days with docker.io. It is awesome!
Some simple containers already run in production servers.
At the moment I stuck in how to make container backup.
Assume I have running complicated docker container with supervisor+apache+memcached+mysql inside under hi load (10k requests per second).
Is it safe to make
docker export > backup.tar
Or I have to stop all processes inside container and only after that export container to tar file?
If by "safe" you mean "consistent", then the exported archive will be filesystem consistent.
The docker exportcommand, as any other classical backup method, won't solve the problem of being application consistent.
Apache and Memcached won't be an issue, since they don't need storage to maintain any kind of state.
But backuping Mysql this way will probably make your database restart in recovery mode, if you run a container from the image generated by docker export.
In particular, if backuped when having to perform write activity (insert, updates..),
as with any other filesystem-level backup, you will loose a few transactions.
If you need your Mysql backuped datafiles to be 100% consistent and to reflect the exact state of your data, you have to either:
Stop Mysql before running docker export
Stop the whole container
Have something connecting to Mysql before running the export command, and run flush tables with read lock;. When the export completes, you'd have to run unlock tables;The backuped data files (generally under /var/lib/mysql) will be consistent.
Use the classical Mysql backup tools (mysqldump...)