I need advise, how to backup Docker/K8.
I am migrating my dev/lab environment from VMWare to Docker (which I am not 100% familiar yet) and I need to set up backup/restore procedure. It was quite simple with VMware images (copy all VM files).
I would like to backup "whole Docker desktop" or in other words, I nee to make lab environment recovery as simple as possible: e.g. "copy/untar backups to freshly installed PC" and after recovery all containers,defined networks, volumes, K8s, etc. are recovered as well. And after recovery procedure I am ready to start using "good old recovered containers".
I am aware that ideal scenario is to keep all container data in volumes and that there are defined procedures how to backup one by one:
volumes ( https://docs.docker.com/storage/volumes/#back-up-restore-or-migrate-data-volumes )
images
containers ( https://docs.docker.com/desktop/backup-and-restore/ )
K8 backup/recovery is probably somehow defined ( I am not familiar yet ).
I am wondering if there is any way how to backup "Docker Desktop" as single "tar" file for example, because until I understand deeply Docker/K8 backup procedures, I would prefer to keep backup as simple as possible.
I am using Docker/WSL2 on Windows and my most "valuable" containers are 4+ Oracle Database containers ( with different Oracle versions/flavors), but I assume number containers will grow and I do not want to update backup script each time new container is created ( this would definitely end up in missed backup one day ).
I there any "simple" recommended way I missed ?
First you need to build images of the existing containers that you are using, next docker save command can be used to save all your docker images but you need to mention each image name manually which you don’t want to do, however these docker commands can be executed by using python or shell script. This document explains clearly about how to manage your docker environment using python code. Refer this and there are many functions and example code snippets available, hope this will help you.
Related
I am new to docker and I have done some ground work on docker like how to build a image, how to create a container, what is dockerfile.yml, docker-compose.yml file does etc. I have done my practice in my local machine. I am having following questions when it is coming to real time development using docker.
1) Does each developer in a team has to do development on docker and create images in their local machine and push it to docker registry ? or developers work without docker and one person will be creating final image from the committed code?
2) For each release, do we maintain the container or an image for that release?
3) what is the best practice to maintain the database means do we incorporate the database in image and build the container or we include only application related things and build image and container, and this container will communicate to database which is in outside container or physical database server ?
Thanks in advance.
Questions like these generally do not have a definitive answer. It depends heavily on your company, your team, the tooling that is being used, the software stack, etc. The best thing to do would be to lean on the senior development resources and senior technical leadership in your organization to help shape the answers to questions like these.
Take the following answers with a silo of salt as there is no way to answer these kinds of questions definitively.
1) Depends on what is most convenient for the developers and what language you are using. Some developers find an all container workflow to be convenient, some developers find they can iterate faster with their existing IDE/CLI workflow and test against running container images last.
In most cases you will want to let CI/CD tooling take care of the builds that are intended for production.
2) Yes. You can use container tagging for this purpose.
3) Running databases in containers is possible, but unless your team is experienced with containers and container orchestration I would leave the databases on traditional bare-metal or VMs.
Containers are a fancy wrapper around a single linux process. Generally the rule of thumb is one container for one process. You should not be combining multiple things in a single container. (This story gets slightly more complicated when you go to something like Red Hat OpenShift or Kubernetes as the discussion revolves around how many containers per pod).
The setup I'm used to is that developers mostly ignore Docker, until they need it to deploy. Tools like Node's node_modules directory or Python's virtual environments can help isolate subproject from each other. Any individual developer should be able to run docker build to build an image, but won't usually need to until the final stages of testing a particular change. You should deploy a continuous integration system and that will take responsibility for testing and building a final Docker image of each commit.
You never "maintain containers". If a container goes wrong, delete it and start a new one. Your CI system should build an image for each release, and you should deploy a registry to hold these.
You should never keep the database in the same container as the application. (See the previous point about routinely deleting containers.) My experience has generally been that production databases are important and special enough to merit their own dedicated non-Docker hosts, but there's nothing actually wrong with running a database in Docker; just make sure you know how to do backups and restores and migrations and whatever else on it.
There's no technical reason you can't use Docker Compose for production, but if you wind up needing to deploy your application on more than one physical server you might find it limiting. Kubernetes is more complex but seems to be the current winner in this space; Docker Swarm has some momentum; Hashicorp Nomad is out there; or you can build a manual deployment system by hand. (Note that at least Kubernetes and Nomad are both very big on the "something changed so I'm going to delete and recreate a container" concept, and both make it extremely tricky to do live development in a quasi-production setup.)
Also note that where I say "deploy" there are public-cloud versions of all of these things (Docker Hub, CircleCI, end-to-end solutions including a registry and Kubernetes built on top of AWS or Azure or GCP) and if you're comfortable with the cost-to-effort tradeoff and the implications of using an external service in your build/deploy chain then these can help you get started faster
Our build setup is backed into a large docker container (basically a 2 GB image coming with a complete X86 linux in itself).
We have two ways to actually build: the official approach is jenkins environment (running on X86 hardware). But we also have a little "side X86 server" running RH 7. Developers can log into that RH server and kick off specific builds (using said docker images) themselves.
Those RH servers will be shut down at some point, to be replaced with IBM Power8 machines (running RH7 Little Endian for power).
I am simply wondering: is there a chance that our existing build setup and docker images simply work on Power8? Or are the fundamental technical issues that make it unlikely and not even worth trying?
You can probably use your existing build methodology and scripts close to unchanged, but you'll need to rebuild the actual images.
You can't directly run x86 binaries on Power (at a very low level, the bytes of machine code are just different). Docker doesn't contain any sort of virtualization layer; it does a bunch of setup to isolate the container from the host, but then runs the binaries in an image directly.
If your Jenkins setup has enough parameters for image names and version tags, then you should be able to run the x86 and Power setups side-by-side; you need to encode the architecture somewhere in the built image name or tag; for instance, repo.example.com/app/build:20180904-power. (I don't know that one or the other is considered better if you control all of the machinery.) If you have a private repo, you could encode it earlier in the path, winding up with image names like repo.example.com/power/build:20180904.
You'd need to double-check that everywhere that has a Docker image reference has it correctly parameterized (which is a good practice anyways). That would include any direct docker run commands; any Docker Compose or Kubernetes YAML files or similar artifacts; and the FROM line of any Dockerfiles.
Existing build setup? Not sure!
Docker images? NO, don’t even try.
Docker images are actually multiple layers which stored on filesystem through corresponding storage driver and backing filesystem(shown in the output of docker info).
If storage driver/backing filesystem has been changed, which likely be true when OS changed, older docker images could not be valid any more. Meaning they must be rebuilt for sure.
I need to containerize a JanusGraph database inside Docker, i don't know what files/directories needs to reside in volume to become persistent/writable. In order to make all the things simple and fast, can i install the entire database in a volume? Not only the data, but the entire app, all the binaries etc. I think this is a fast way to containerize some of my apps.
The janusGraph subdirectories of binaries, data, log resides inside a "janusgraph-hadoop" directory
For example: i will create a volume called /janusgraph-hadoop and run the command to install all the software inside that (it will be a volume).
This can be considered a bad practice or there are no problem in doing that?
I know, we have some JanusGraph already containerized, but they are not official, and my doubt is more general in order to containerize some apps in a more direct way without the need to research what directories need to be in volume and what not.
I will not redistribute any of this, it's just to my use.
At a technical level, nothing would stop you from launching a plain container with an attached volume and installing software there.
docker run -v my_opt:/opt -it --rm ubuntu sh
I wouldn't consider this an especially effective use of Docker. If your colleague wants to use your database installation, you have no way of giving it to them; if you leave the project for six months and come back to it, you'll have no record of how you built this setup. If you were set on this approach, you might find the networking and snapshot setups for more typical virtual machines to be better matched to it.
I am learning CDH and Docker and didn't have prior experiene in setting up both tools. After reading documentation i managed to run CDH docker in mac environment and also completed example given in quick start guid. But when next day when i started mac book again to learn something new but i didn't find my previous work which i found very strange and even couldn't see container running which seems fine to me.
What i really want to do is i don't want to loose my work even after stoping docker container. could you please guid me how do i configure docker so that i will not loose my work even after restarting docker again?
Every instance of a docker run will allocate a new filesystem, essentially starting from scratch.
If you actually want to "save" your work, then you need to volume mount (using -v docker flag) your local filesystem into the container for at least the following directories.
HDFS Data Directory
NameNode Data Directory
/home/cloudera
I think the hadoop data folders are somewhere under /var/lib/hadoop-*, by default
The better alternative for saving your workloads would be the CDH VM, where it actually has a persistent HDD associated with it.
I'm new to Docker and was wondering if it was possible (and a good idea) to develop within a docker container.
I mean create a container, execute bash, install and configure everything I need and start developping inside the container.
The container becomes then my main machine (for CLI related works).
When I'm on the go (or when I buy a new machine), I can just push the container, and pull it on my laptop.
This sort the problem of having to keep and synchronize your dotfile.
I haven't started using docker yet, so is it something realistic or to avoid (spacke disk problem and/or pull/push timing issue).
Yes. It is a good idea, with the correct set-up. You'll be running code as if it was a virtual machine.
The Dockerfile configurations to create a build system is not polished and will not expand shell variables, so pre-installing applications may be a bit tedious. On the other hand after building your own image to create new users and working environment, it won't be necessary to build it again, plus you can mount your own file system with the -v parameter of the run command, so you can have the files you are going to need both in your host and container machine. It's versatile.
> sudo docker run -t -i -v
/home/user_name/Workspace/project:/home/user_name/Workspace/myproject <container-ID>
I'll play the contrarian and say it's a bad idea. I've done work where I've tried to keep a container "long running" and have modified it, but then accidentally lost it or deleted it.
In my opinion containers aren't meant to be long running VMs. They are just meant to be instances of an image. Start it, stop it, kill it, start it again.
As Alex mentioned, it's certainly possible, but in my opinion goes against the "Docker" way.
I'd rather use VirtualBox and Vagrant to create VMs to develop in.
Docker container for development can be very handy. Depending on your stack and preferred IDE you might want to keep the editing part outside, at host, and mount the directory with the sources from host to the container instead, as per Alex's suggestion. If you do so, beware potential performance issue on macos x with boot2docker.
I would not expect much from the workflow with pushing the images to sync between dev environments. IMHO keeping Dockerfiles together with the code and synching by SCM means is more straightforward direction to start with. I also carry supporting Makefiles to build image(s) / run container(s) same place.