I wish to test code that may have trouble on file systems that do not support d_type.
In order to do so I would like to create two small xfs file systems that respectively have ftype=0 which does not support d_type, and ftype=1 which does.
I'd like to run these tests in a Docker container as that is how our testing is set up.
It looks like I might be able to take advantage of the Docker devicemapper https://docs.docker.com/storage/storagedriver/device-mapper-driver/ .
I do not necessarily control the Docker Engine, that is I don't want to rely on creating these filesystems on the underlying machine and then exposing them to my container - so I would want to be able to do this in my Dockerfile or one I am running in the container.
But maybe there are other or better ways to do this.
I'm not sure this is a complete answer but hopefully a step towards it.
I am trying to do something similar. I want to do some tests using at least xfs, tmpfs & sshfs. I was hoping to find a fuse solution to emulate xfs as well.
You could definitely put a tmpfs, fuse-sshfs and even nfs inside a docker.
I have no experience with nbd but I think you could it within docker to provide xfs via a loopback device. See this blog for example.
This might not be necessary though as you can mount an image as a partition E.g.
mount -t <fs type> -o loop file.img /mnt
Assuming this works in docker we have our solution. I haven't tried this myself yet. If you get there first please post your solution
(perhaps you did as this question is a year old).
See also Emulate a hard drive in Linux
Otherwise vagrant is probably good solution for this kind of problem.
Related
I need to containerize a JanusGraph database inside Docker, i don't know what files/directories needs to reside in volume to become persistent/writable. In order to make all the things simple and fast, can i install the entire database in a volume? Not only the data, but the entire app, all the binaries etc. I think this is a fast way to containerize some of my apps.
The janusGraph subdirectories of binaries, data, log resides inside a "janusgraph-hadoop" directory
For example: i will create a volume called /janusgraph-hadoop and run the command to install all the software inside that (it will be a volume).
This can be considered a bad practice or there are no problem in doing that?
I know, we have some JanusGraph already containerized, but they are not official, and my doubt is more general in order to containerize some apps in a more direct way without the need to research what directories need to be in volume and what not.
I will not redistribute any of this, it's just to my use.
At a technical level, nothing would stop you from launching a plain container with an attached volume and installing software there.
docker run -v my_opt:/opt -it --rm ubuntu sh
I wouldn't consider this an especially effective use of Docker. If your colleague wants to use your database installation, you have no way of giving it to them; if you leave the project for six months and come back to it, you'll have no record of how you built this setup. If you were set on this approach, you might find the networking and snapshot setups for more typical virtual machines to be better matched to it.
I adjust a docker compose project for a development case. The main requirement is to synchronize sources from host to guest containers. How exactly? If it is so clear, give me an answer :) Beneath I will tell you my story for details. Thank you.
There are a lot of cases but no one works for me. I reuse this docker-compose.yml https://github.com/wodby/docker4drupal/blob/master/docker-compose.yml
I develop on Ubuntu. A) The docker documentation proposes the sshfs plugin based approach - https://docs.docker.com/storage/volumes/#create-a-volume-using-a-volume-driver. But what host do I have to use? I see that guest containers connect to the host. B) Docker-sync approach seems weird https://github.com/EugenMayer/docker-sync/wiki/2.1-sync-commands . C) I've read the docker documentation once more and 've found a type of mounting bind. But do not see any good use cases. Internet uses the C variant docker-sync.
I also can not use docker compose long format for volumes https://docs.docker.com/compose/compose-file/compose-file-v2/#long-syntax.
The base project uses docker-compose format v2. I see that higher version of formats exist. Why do they still use this?
Thank you.
There are a lot of methods. The proper way is the docker volume driver for development needs.
To understand this better you need be introduced with:
Docker volumes
Configure bind propagation
This a small patch from my docker-compose.yml
- type: bind
source: "${PROJECT_PATH}"
target: /var/www/html
bind:
propagation: shared
It works for me.
I happen to use docker in a questionable way for a specific purpose:
I have a container with a few development tools and their configurations. These are typically tools found in operating systems that are configured with dotconf files usually found in home directories (like tmux, vim, git, programming languages...). This is normally handled by configuring an OS with tools and dotfiles.
But with the setup becoming more complex over the years, properly setting up a new OS is becoming very hard. On the other hand, moving a container around machines is very simple. So for that precise case, I happen to use docker in a way that goes against the "docker way". But that is really comfortable to me.
That being said, I also want to synchronize the container's filesystem with another container (which, in my understanding is definitely not the "docker way", but yet.) I want to run 2 instances of the same image, on different machines. Then I want to synchronize their read write layer that is on top of the image. So when a file is created, deleted or modified on one, this is replicated on the other.
I was thinking of using rsync or unison to do that. But I don't know how the overlay2 driver works. Are the directories in /var/lib/docker/overlay2/<container-id> the actual containers filesystem layer? Or should it be mounted? I saw some people mount their containers filesystem on the host with the device mapper driver fairly easily. Would that make sense with overlay2?
I think your best option here is to use a bind mount. This changes your initial design a bit - but it will likely be the cleanest, and easiest to implement.
First things first - you'll want to ensure that any files that you want to have synced are in a specific folder; so rather than rsyncing the entire underlying filesystem - you'll just do, for example /app/my_files inside your container and set your application to read/write from there.
Now - create your directory and setup the rsync between your machines; let's say it's at /rsync
Lastly - run your containers and use a bind-mount; which if you're just bringing up a container would look like: docker run -d -v /rsync:/app/my_files my_image
After reading this page: https://docs.docker.com/engine/userguide/storagedriver/overlayfs-driver/ and experimenting with image / container creation and deletion, I have found out a few things.
When creating an image, the folder /var/lib/docker/overlay2 is populated with a new folder, called with what seems to be a hash (let's call it 123), and is itself populated with the image content / filesystem.
Then when creating the container from this same image, two more folders get created inside /var/lib/docker/overlay2, also named with what seems to be another hash, with one of them having -init at the end. Let's call them 456 and 456-init. They seem to contain the container layer.
When looking at the output of docker inspect <container-name>, the GraphDriver section has some info about how docker uses the overlay2. The lower dir contains the init container dir + the image dir as in: /var/lib/docker/overlay2/456-init/diff:/var/lib/docker/overlay2/123/diff. I don't fully understand how that works, but I understand that I am not interested in the lower dir since it should be the image dir in read only mode. And that is something I already have on all hosts and thus do not need to sync.
Then in my understanding the upper dir in overlay2 is the read write layer that the container uses on top of the image layer. In the GraphDriver this is found to be /var/lib/docker/overlay2/456/diff. That is the directory that gets the changes made inside the container. I could not find the documentation so I experimented a bit and found out that this upper dir never changed during the life of the container. I stopped and started it and the upper dir stayed the same. Then when removing the container this folder is deleted. And when creating the container again, a new folder with a different name is created.
So it looks like what I need to sync is this upper dir, which can be found with docker inspect. I'll experiment a bit more with that.
I'm new to Docker and was wondering if it was possible (and a good idea) to develop within a docker container.
I mean create a container, execute bash, install and configure everything I need and start developping inside the container.
The container becomes then my main machine (for CLI related works).
When I'm on the go (or when I buy a new machine), I can just push the container, and pull it on my laptop.
This sort the problem of having to keep and synchronize your dotfile.
I haven't started using docker yet, so is it something realistic or to avoid (spacke disk problem and/or pull/push timing issue).
Yes. It is a good idea, with the correct set-up. You'll be running code as if it was a virtual machine.
The Dockerfile configurations to create a build system is not polished and will not expand shell variables, so pre-installing applications may be a bit tedious. On the other hand after building your own image to create new users and working environment, it won't be necessary to build it again, plus you can mount your own file system with the -v parameter of the run command, so you can have the files you are going to need both in your host and container machine. It's versatile.
> sudo docker run -t -i -v
/home/user_name/Workspace/project:/home/user_name/Workspace/myproject <container-ID>
I'll play the contrarian and say it's a bad idea. I've done work where I've tried to keep a container "long running" and have modified it, but then accidentally lost it or deleted it.
In my opinion containers aren't meant to be long running VMs. They are just meant to be instances of an image. Start it, stop it, kill it, start it again.
As Alex mentioned, it's certainly possible, but in my opinion goes against the "Docker" way.
I'd rather use VirtualBox and Vagrant to create VMs to develop in.
Docker container for development can be very handy. Depending on your stack and preferred IDE you might want to keep the editing part outside, at host, and mount the directory with the sources from host to the container instead, as per Alex's suggestion. If you do so, beware potential performance issue on macos x with boot2docker.
I would not expect much from the workflow with pushing the images to sync between dev environments. IMHO keeping Dockerfiles together with the code and synching by SCM means is more straightforward direction to start with. I also carry supporting Makefiles to build image(s) / run container(s) same place.
I have two hosts and docker is installed in each.
As we know, each docker stores the images in local /var/lib/docker directory.
So If I want to use some image, such as ubuntu, I must execute the docker pull to download from internet in each host.
I think it's slow.
Can I store the images in a shared disk array? Then have some host pull the image once, allowing every host, with access to the shared disk, to use the image directly.
Is it possible or good practice? Why docker is not designed like this?
It may need to hack the docker's source code to implement this.
Have you looked at this article
Dockerizing an Apt-Cacher-ng Service
http://docs.docker.com/examples/apt-cacher-ng/
extract
This container makes the second download of any package almost instant.
At least one node will be very fast, and I think it should possible to tell the second node to use the cache of the first node.
Edit : you can run your own registry, with a command similar to
sudo docker run -p 5000:5000 registry
see
https://github.com/docker/docker-registry
What you are trying to do is not supposed to work as explained by cpuguy83 at this github/docker issue.
Indeed:
The underlying storage driver would need to synchronize access.
Sharing /var/lib/docker is far not enough and won't work!
According to the doc.docker.com/registry:
You should use the Registry if you want to:
tightly control where your images are being stored
fully own your images distribution pipeline
integrate image storage and distribution tightly into your in-house development workflow
So I guess that this is the (/your) best option to work this out (but I guess that you got that info -- I just add it here to update the details).
Good luck!
Update in 2016.1.25 docker mirror feature is deprecated
Therefore this answer is not applicable now, leave for reference
Old info
What you need is the mirror mode for docker registry, see https://docs.docker.com/v1.6/articles/registry_mirror/
It is supported directly from docker-registry
Surely you can use public mirror service locally.