I'm working on a project with multiple collaborators; to share code and compute environment, we've setup a github repository which includes a Dockerfile and docker-compose.yml file. I can work on code and my collaborators can just pull the repository, run docker-compose up and have access to my jupyter notebooks in the same environment that I develop them.
The only problem with this is that, because we are working at different sites, the data that we are computing over is in different locations. So on my end, I want my docker-compose.yml to include:
volumes:
- /mnt/shared/data:/data
while my collaborators need it to say something like
volumes:
- /Volumes/storage/data:/data
I get that one way to do this would be to use an environment variable; in the docker-compose.yml file:
volumes:
- "$DATA_PATH":/data
This forces them to run something like:
DATA_PATH=/Volumes/storage/data docker-compose up
As a solution, this isn't necessarily a problem, but it feels clunky to me, and fails to be self-documenting in the repository. I can wrap docker-compose in a shell script (a potential solution to almost any problem), but this also feels clunky. I can't help but suspect that there's a better solution here. Does docker-compose allow for this kind of functionality? Is there a best-practices way of accomplishing this? If not, I'm curious if anyone knows what the motivation behind excluding this functionality might be and/or why it isn't considered a good idea.
Thanks in advance.
You are extremely close. What I would add is, that you have a host specific .env file, see Environment variables in Compose, on each computer, in the same folder as the docker-compose.yml, with
DATA_PATH=/mnt/shared/data
or whatever value for DATA_PATH you like. Just add that .env to your .gitignore, so that every host keeps his own config off the repository and that's it.
Related
I'd like to build and operate containers only with docker-compose. Is it possible to do it with using all features of the Dockerfile, but without using it?
IMHO not, but I'm not sure if it is true.
For example: are there any replacement of Dockerfile's RUN,ADD,COPY commands in the docker-compose.yaml? I can't find.
In general, no: the Dockerfile says how to build an image, and the docker-compose.yml says how to run it. There are various combinations of things that it's impossible to do in one place or the other.
In some cases you can simulate things the Dockerfile might do with docker-compose.yml directives (you can use volumes: to mount content into a running container, which looks similar to COPYing the content into an image) but they're not the same and you generally can't use exclusively one, unless you can describe your application using only prebuilt images.
I'd tend to recommend the Dockerfile COPY and CMD directives over trying to use only the docker-compose.yml equivalents. Environment variables are often runtime configuration ("what is the host name of the database server?") and those it makes more sense to specify in the docker-compose.yml or other runtime configuration.
I've checked on SO but couldn't find a exhaustive answer.
My docker-composer.yml defines few things including volumes
app:
volumes:
- "./:/app"
...
If I use docker run to instance the image, then I will need to specify again the same volumes specified in docker-compose.yml.
docker run -v "./:/app"
That might be desirable for some use cases, but in general having the same definition specified in 2 different places is not really maintainable (or obvious for future devs). I'd like to avoid defining the same config in different locations (one for docker-compose and one as arguments for docker run).
Can it be stated that if configuring volumes (or others parameters) inside docker-compose.yml then, in order to have them, the image should be run via docker-compose up rather than docker run -v redundant:volume:specification?
Note: I am asking about best practices more than personal opinions.
You should think of the docker-compose.yml as not unlike a very specialized shell script that runs docker run for you. It's not a bad idea to try to minimize the number of require mounts and options (for example, don't bind-mount code over the code in your image) but it's also not especially a best practice to say "this is only runnable via this docker-compose.yml file".
Also consider that there are other ways to run a container, with different syntaxes. Both Kubernetes and Hashicorp's Nomad have very different syntaxes, and can't reuse the docker-compose.yml. If you're running the image in a different context, you'll basically have to rewrite this configuration anyways.
In limited scopes – "for this development project, in this environment, in this specific repository" – it's reasonable enough to say "the standard way to run this with standard options is via docker-compose up", but it's still possible to run the image a different way if you need to.
In general one should rely on docker-compose, once starting to use it, since just relying on docker <cmd> might miss some configuration and give unexpected results (especially if freshly landing on the project and not having confidence with it).
Executing the images with docker run will lead to the following disadvantages:
having to remember adding eventual parameters at each run, that are instead implicit into with docker-compose
even when remembering, or having a bash script calling docker run with the right parameters, future changes to these parameters will need to be reflected in two different places. This is not very maintainable and error prone.
eventual other correlated images will not run and one has to remember to run them manually; or add them into a script, ending again with definition in two different places.
However, for a broader view considering other runners (k8s) check David Maze's answer.
We are trying to run two apps via docker-compose. These apps are (obviously) in separate folders, each of them having their own docker-compose.yml . On the filesystem it looks like this:
dir/app1/
-...
-docker-compose.yml
dir/app2/
-...
-docker-compose.yml
Now we need a way to compose these guys together for they have some nitty-gritty integration via http.
The issue with default docker-compose behaviour is that if treats all relative paths with respect to folder it is being run at. So if you go to dir from the example above and run
docker-compose up -f app1/docker-compose.yml -f app2/docker-compose.yml
you'll end up being out of luck if any of your docker-compose.yml's uses relative paths to env files or whatever.
Here's the list of ways that actually work, but have their drawbacks:
Run those apps separately, and use networks.
It is described in full at Communication between multiple docker-compose projects
I've tested that just now, and it works. Drawbacks:
you have to mention network in docker-compose.yml and push that to repository some day, rendering entire app being un-runnable without the app that publishes the network.
you have to come up with some clever way for those apps to actually wait for each other
2 Use absolute paths. Well, it is just bad and does not need any elaboration.
3 Expose the ports you need on host machine and make them talk to host without knowing a thing about each other. That is too, obviously, meh.
So, the question is: how can one manage the task with just docker-compose ?
Thanks to everyone for your feedback. Within our team we have agreed to the following solution:
Use networks & override
Long story short, your original docker-compose.yml's should not change a bit. All you have to do is to make docker-compose.override.yml near it, which publishes the network and hooks your services into it.
So, whoever wants to have a standalone app runs
docker-compose -f docker-compose.yml up
But when you need to run apps side-by-side and communicating with each other, you should go with
docker-compose -f docker-compose.yml -f docker-compose.override.yml up
I've built a simple Docker Compose project as a development environment. I have PHP-FPM, Nginx, MongoDB, and Code containers.
Now I want to automate the process and deploy to production.
The docker-compose.yml can be extended and can define multiple environments. See https://docs.docker.com/compose/extends/ for more information.
However, there are Dockerfiles for my containers. And for the dev environment are needed more packages than in production.
The main question is should I use separate dockerfiles for dev and prod and manage them in docker-compose.yml and production.yml ?
Separate dockerfiles are easy approach but there is code duplication.
The other solution is to use environment variables and somehow handle them from bash script (maybe as entrypoint ?).
I am searching for other ideas.
According to the official docs:
... you’ll probably want to define a separate Compose file, say
production.yml, which specifies production-appropriate configuration.
Note: The extends keyword is useful for maintaining multiple Compose
files which re-use common services without having to manually copy and
paste.
In docker-compose version >= 1.5.0 you can use environment variables, may be this suits you?
If the packages needed for development aren't too heavy (i.e. the image size isn't significally bigger) you could just create Dockerfiles that include all the components and then decide whether to activate them based on the value of an environment variable in the entrypoint.
That way you would could have the main docker-compose.yml providing the production environment while development.yml would just add the correct environment variable value where needed.
In this situation it might be worth considering using an "onbuild" image to handle the commonalities among environments, then using separate images to handle the specifics. Some official images have onbuild versions, e.g., Node. Or you can create your own.
I am trying to create local development environment using Docker Compose. I started using this example https://github.com/b00giZm/docker-compose-nodejs-examples/tree/master/03-express-gulp-watch and it is working like a charm. No problem there.
However the structure in that example is too simple and doesn't fit to my needs. I am planning to run my application with coreos on production, so I need a bunch of other config files also. This is roughly how I CHANGED the example above:
application
app
bin
public
routes
views
app.js
gulpfile.js
package.json
vm
coreos (production configs here)
docker (development configs here)
app
Dockerfile
docker-compose.yml
Dockerfile for actual application in there, because I would like to use separate dockerfiles for production and development use.
I also changed my docker-compose.yml to this:
web:
build: app
volumes:
- "../../app:/src/app"
ports:
- "3030:3000"
- "35729:35729"
After this "docker-compose build" goes ok, but "docker-compose up" doesn't. I get an error saying, that gulpfile cant be found. In my logic this is because of volume mounts, they don't work with parent directories as I assume.
Any idea what I am doing wrong? Or I you have working example for this situation, please share it.
Your are probably hitting the issue of using volumes too early and trying to access them in cascading docker images.
See this:
https://github.com/docker/docker/issues/3639
dnephin was right. Removing old containers did the trick. Thanks!