I am not an advance user so please bear with me.
I am building a docker image using docker-compose -f mydocker-compose-file.yml ... on my machine.
The image then been pushed to a remote docker registry.
Then from a remote server I pull down this image.
To run this image; I have to copy mydocker-compose-file.yml from my machine to remote server and then run docker-compose -f mydocker-compose-file.yml up -d.
I find this very inefficient as why I need the same YAML file to run the docker image (should I?).
Is there a way to just spin up the container without this file from remote machine?
As of compose 1.24 along with the 18.09 release of docker (you'll need at least that client version on the remote host), you can run docker commands to a remote host over SSH.
# all docker commands in this shell will not talk to the remote host
export DOCKER_HOST=ssh://user#host
# you can verify that with docker info to see which engine you're talking to
docker info
# and now run your docker-compose up command locally to start/stop containers
docker-compose up -d
With previous versions, you could configure TLS certificates to allow specific clients to connect to the docker API over a network connection. See these docs for more details.
Note, if you have host volumes, the variables and paths will be expanded to your laptop directories, but the host mounts will happen on the remote server where those directories may not exist. This is a good situation to switch to named volumes.
Everything you can do with Docker Compose, you can do with plain docker commands.
Depending on how exactly you're interacting with the remote server, your tooling might have native ways to do this. One specific example I'm familiar with is the Ansible docker_container module. If you're already using a tool like Ansible, Chef, or Salt, you can probably use a tool like this to do the same thing your docker-compose.yml file does.
But otherwise there's more or less a direct translation between a docker-compose.yml file
version: '3'
services:
foo:
image: me/foo:20190510.01
ports: ['8080:8080']
and a command line
docker run -d --name foo -p 8080:8080 me/foo:20190510.01
My experience has been that the docker run commands quickly become unwieldy and you want to record them in a file; and once they're in a file, you start to wish they were in a more structured format, even if you need an auxiliary tool to run them; which brings you back to copying around the docker-compose.yml file. I think that's pretty routine. (Something needs to tell the server what to run.)
Related
TLDR: When using docker compose, I can simply recreate a container by changing its configuration and/or image in the docker-compose.yml file along with running docker-compose up. Is there any generic equivalent for recreating a container (to apply changes) which was created by a bare docker create/run command?
Elaborating a bit:
The associated docker compose documentation states:
If there are existing containers for a service, and the service’s configuration or image was changed after the container’s creation, docker-compose up picks up the changes by stopping and recreating the containers (preserving mounted volumes).
I'm having troubles to understand which underlaying steps are actually performed during this recreation, as e.g. the docker (without compose) documentation doesn't really seem to use the recreate term at all.
Is it safe to simply run docker container rm xy and then docker container create/run (along with passing the full and modified configuration)? Or is docker compose actually doing more under the hood?
I already found answers about applying specific configuration changes like e.g. this one about port mappings, but I'm still wondering whether there is a more general answer to this.
I'm having troubles to understand which underlaying steps are actually performed during this recreation, as e.g. the docker (without compose) documentation doesn't really seem to use the recreate term at all.
docker-compose is a high level tool; it performs in a single operation what would require multiple commands using the docker cli. When docker-compose says, "docker-compose up picks up the changes by stopping and recreating the containers", it means it is doing the equivalent of:
docker stop <somecontainer>
docker rm <somecontainer>
docker run ...
(Where ... represents whatever configuration is implied by the service definition in your docker-compose.yaml).
Let's say it recognizes a change in container1 it does (not really, working via API):
docker compose rm -fs container1
docker compose create (--build) container1
docker compose start container1
What is partially close to (depending on your compose-config):
docker rm -f projectname_container1
(docker build --flags)
docker create --allDozensOfAttributes projectname_container1
docker start projectname_container1
docker network connect (--flags) projectname_networkname projectname_container1
and maybe more..
so i would advise to use the docker compose commands for single services instead of docker cli if suitable..
The issue is that the variables and settings are not exposed through any docker apis. It may be possible by way of connecting directly to the docker socket, parsing the variables, and then stopping/removing the container and recreating it.
This would be prone to all kinds of errors and would require lots of debugging to get these values.
What I do is to simply store my docker commands in a shell script. You can just save the command you need to run into a text file, name it .sh, set the -x on the file, then run it. Then when you stop/delete the container, you can just rerun the shell script.
Another thing you can do would be to replace the docker command with a function (in something like your ~/.bashrc) that stores the arguments to a text file and rechecks that text file with a passed argument (like "recreate" followed by a name). However, I'm more a fan of doing docker containers in their own shell scripts as its more portable.
I am a little bit confused when using docker and docker-compose:
With dockerfile I can build, run, and push the docker application image to docker hub in order to allow other people to download and run it on their local computers.
With Docker-compose I can build, run and push the service image ( eg: redis, cassandra, etc)
My concern:
I got an application folder with the following files:
- main.go # the main app in Golang
- Dockerfile # container definition file
- Docker-compose.yml # contains all the services ( Redis and Cassandra)
which command should I use to build and push my entire app and its services on the docker hub? docker or docker-compose?
One useful conceptual way to think about this is that the docker-compose.yml file specifies a set of docker build and docker run commands. You cannot separately push the docker-compose.yml file to Docker Hub or other registries, and in a typical Compose setup there are references a number of standard images you don't need to push yourself (the standard Docker Hub redis and cassandra images will be fine no matter where you run them).
This means you need to do two things:
Push the application image (but not its public-image dependencies) to a registry; and
Publish the docker-compose.yml file or another way to run the combined application.
You can use docker-compose push, but in a CI environment it's probably a little more straightforward to use docker build and docker push. Mechanically they're not any different.
Make sure your image contains everything that's needed to run your application, up to external dependencies. The ideal is to be able to docker run your application container, with --net, -e, and -p settings to configure it, but without separately providing the application code or a command. In a docker-compose.yml file see if you can run it with only ports:, environment:, and image: (and build:). Prefer a CMD in the Dockerfile to a command: in docker-compose.yml. If you're bind-mounting host code into the container (unlikely for Go and other compiled languages) delete those volumes:.
A typical sequence for deploying things using Compose might look like:
here$ docker build -t me/myapp:20200525 .
here$ docker push me/myapp:20200525
here$ scp docker-compose.yml there:
here$ ssh there
there$ MYAPP_TAG=20200525 docker-compose up -d
Note that the only thing we directly copied to the target system is the docker-compose.yml that specifies how to run the image; we have not copied any application code or other dependencies, since that is all encapsulated in the image.
I'm new to docker and trying to understand what's best for my project (a webapp).
So far, I understand that I can either :
use docker-compose up -d to start a container defined by a set of rule in a docker-compose.yaml
build an image from a dockerfile and then create a container from this image
If I understand correctly, docker-compose up -d allows me (via volumes) to mount files (e.g my application) into the container. If i build an image however, I am able to embed my application natively in it (with a Dockerfile and COPY instruction).
Is my understanding correct ? How should I choose between those 2 choices ?
Docker Compose is simply a convenience wrapper around the docker command.
Everything you can do in docker compose, you can do plainly with running docker.
For example, these docker commands:
$ docker build -t temp .
$ docker run -i -p 3000:80 -v $PWD/public:/docroot/ temp
are similar to having this docker compose file:
version: '3'
services:
web:
build: .
image: temp
ports: ["3000:80"]
volumes:
- ./public:/docroot
and running:
$ docker-compose up web
Although docker compose advantages are most obvious when using multiple containers, it can also be used to start a single container.
My advice to you is: Start without docker compose, to understand how to build a simple image, and how to run it using the docker command line. When you feel comfortable with it, take a look at docker compose.
As for the best practice in regards to copying files to the container, or mounting them - the answer is both, and here is why:
When you are in development mode, you do not want to build the image on every code change. This is where the volume mount comes into play. However, your final docker image should contain your code so it can be deployed anywhere else. After all, this is why we use containers right? This is where the COPY comes into play.
Finally, remember that when you mount a volume to the container, it will "shadow" the contents of that folder in the container - this is how using both mount and COPY actually works as you expect it to work.
Docker-compose is just a container orchestrator.
I just provides you a simple way to create multiple related containers. The relationship between containers can be volumes, networks, start order, environment variables, etc.
In background, docker-compose uses plain docker. So, anything you can do using docker-compose (mounting volumes, using custom networks, scaling) can be done using docker commands (but of course is harder).
I'm pretty new to docker and docker compose. I have managed to build my image and push it to Docker Hub. The app I built is simple and consists of 2 images php7-apache and mysql offical images. All declared in docker-compose.yml.
I informed my team to pull the image I built from my Docker Hub repository using docker pull ... and start it using docker run -d .... But when we run docker ps in the production server, only 1 process is running but no MySQL.
Usually, when I run locally using docker-compose up I get this in the terminal:
Creating network "myntrelease_default" with the default driver
Creating myntrelease_mysql_1
Creating myntrelease_laravel_1
I can then access the MySql using docker-compose exec mysql bash and tweak some tables there. So far so good.
Question is how can I use docker-compose.yml in the production server when it's not available because its in the image itself?
Short answer: Yes, you need the docker-compose.yml in the production environment.
Explanation: Every image is independent. Since your image is independent of MySQL image (at least that's what I understand from your questions), and docker-compose.yml defines the relationship between the two (eg. how MySQL is accessible in the php7-apache image), then you definitely need the docker-compose.yml in production. Even if you only have a single image its usually good to use docker-compose.yml so that settings and configuration like volume mounts, ports etc. can be clearly defined.
I have my local docker machine and a remote docker machine, on the cloud. My docker-compose app has a webcontainer with this config:
web:
container_name: web
restart: always
build: ./web
expose:
- "8000"
links:
- postgres:postgres
volumes:
- /usr/src/app/static
- ./data:/usr/src/app/data
env_file: .env
command: /usr/local/bin/gunicorn --workers 4 --timeout 120 --bind :8000 app:app
The important part is that second volume. I have this local folder called data with some 10GB of data in it. I made it a volume in the first place because otherwise building the container takes forever. Now that the app is production-ready, I'd like to deploy it. One problem: now my remote web container has an empty data folder mounted in it. So how do I move data from my local machine into a container on a remote docker machine? Where do I even move it to?
It seems like there are two tools for this:
docker cp which doesn't seem like it will work for remote docker machines
docker-machine scp which seems made for this, right?
I'm almost positive I need to use the second of these, but since I don't quite understand how docker machine works or where it keeps its data, I'm not sure what destination path to use:
$ dm scp -r /Users/alex/Documents/Project/data remote-machine:/usr/src/app/data
fails with error message:
scp: /usr/src/app/data: No such file or directory
Where should I be scp'ing this data in order to have it mount properly on my remote web container?
Local path vs. in-container path
Assuming you will use the same model remotely that you used locally, keep in mind that the path /usr/src/app/data is the path inside the container. When you are copying the files from one system to another, you just need to copy them from the current system to the remote system, then put them in a path where docker-compose knows how to find them, to mount into a new container.
So all you have to do is copy them from here to there, and use the same path relative to docker-compose.yml. It only knows your external volume as ./data, so if you put the directory in the same place (from docker-compose's perspective), everything should work the same.
How to copy the files
As for how to do the copy, these are just files, so it doesn't matter. scp -r should work, or make a zipfile, copy that, unzip into the correct place, etc. There are a ton of ways to copy files, so pick whatever is simplest for your case.
What exactly needs to be copied?
In the comments you expressed confusion about local vs. remote operations in docker-machine, and what else you needed to copy. Here's a bit more full of an explanation:
On your local system (which I'm assuming is your own PC or laptop), you have docker-machine installed, and you've been using that for all of this development. Completely separate from that is your new cloud instance where you would like to deploy.
To run what you have locally already, up on your cloud instance, the cloud instance will need to have the following.
The docker-compose.yml file.
As long as you plan to use docker-compose to run this, that must be available.
Your .env file.
Since you are using an environment file in this setup, it must be available or docker-compose can't make use of it.
Your web image.
You have a build parameter for this container, but not an image parameter. So currently the only thing you can do is docker-compose build web which will locally generate an image, which docker-compose then knows how to run.
Another option is to add an image parameter, with a repository:tag, such as myuser/myapp_web:1.0, and push that up to Docker Hub. Then, on your cloud instance, the image can be retrieved from Docker Hub instead of building it locally.
In that case, you can add an image parameter to the web container in docker-compose.yml, then build it and push it up.
docker-compose build web
docker-compose push web
Then on the cloud instance, you can fetch it:
docker-compose pull web
docker-compose will know to use that image because of the image parameter in docker-compose.yml (which is also present on the cloud server).
Ref: Creating a new repository on Docker Hub
Which of these options is preferable depends on how you want to manage things. Either one would work, but the "local build" option would require you copy any required source files to your cloud instance too (anything that is used during the build process).
I don't see in your question where the postgres container comes from. If you are also custom-building this one, then the same goes as for web. If you are using a public image for this, then you shouldn't need to copy anything; docker-compose will know how to fetch it, i.e. you can do this:
docker-compose pull postgres
What about docker cp and docker-machine scp?
You mentioned docker cp and docker-machine scp in your question.
As you already determined, docker cp is not a solution here. That command is for copying files between a container and the host filesystem. It has nothing to do with copying over a network.
As far as I know, docker-machine scp is to copy files between your local host and a docker-machine-managed VM. To copy files to your cloud instance you can likely use a more generic tool like scp or sftp more easily.
Not sure as of which docker version, but contrary to the statements in the question and the #Dan_Lowe answer this works fine:
docker cp ./data container:/usr/src/app/
docker cp is a normal part of the API, so it works like any other command, even remotely.