I'm creating an instance template under GCP's Cloud Engine section.
Usually when deploying a docker image, there's a docker file that includes some startup scripts after specifying the base image to pull and build, but I can't see where I can either submit a docker file, or enter startup scripts.
I can see a field for startup scripts for the Cloud Compute instance, but that's different from the scripts passed on for the Docker's startup.
Are these perhaps to be filled in under "Command", "Command arguments", or "Environment Variables"?
For clarification, this is someone else's image of a dockerfile I pulled from Google Images. The part I wish to add is "rectangled" in red, the RUN commands, but not these exact commands.
In my case, I would like to add something like
RUN python /pythonscript.py
If I understood well, you are trying to create a Docker image not a compute instance image.
Compute instance can run a docker image that you already builded and pushed to either gcr or any other repository.
Try to build your docker image normaly, push it in a docker repo then use it.
You can run a startup script directly in the Docker container by using a ‘command’ section. If you need to install something after starting a container for example Apache, you should use a Docker image that has Apache.
If you need to run some other arguments, like creating environment variables, here you can find the full list of flags when creating a container image on VM instance.
Related
I'm having difficulties understanding docker. No matter how many tutorials I watch, guides I read, for me docker-compose is like being able to define multiple Dockerfiles, ie multiple containers. I can define environment variables in both, ports, commands, base images.
I read in other questions/discussions that Dockerfile defines how to build an image, and docker-compose is how to run an image, but I don't understand that. I can build docker containers without having to have a Dockerfile.
It's mainly for local development though. Does Dockerfile have an important role when deploying to AWS for example (where it's probably coming out of the box for example for EC2)?
So the reason why I can work locally with docker-compose only is because the base image is my computer (sorting out the task Dockerfile is supposed to do)?
Think about how you'd run some program, without Docker involved. Usually it's two steps:
Install it using a package manager like apt-get or brew, or build it from source
Run it, without needing any of its source code locally
In plain Docker without Compose, similarly, you have the same two steps:
docker pull a prebuilt image with the software, or docker build it from source
docker run it, without needing any of its source code locally
I'd aim to have a Dockerfile that creates an immutable copy of your image, with all of its source code and library dependencies as part of the image. The ideal is that you can docker run your image without -v options to inject source code or providing the command at the docker run command line.
The reality is that there are a lot of moving parts: you probably need to docker network create a network to get containers to communicate with each other, and use docker run -e environment variables to specify host names and database credentials, and launch multiple containers together, and so on. And that's where Compose comes in: instead of running a series of very long docker commands, you can put all of the details you need in a docker-compose.yml file, check it in, and run docker-compose up to get all of those parts put together.
So, do:
Use Compose to start multiple containers together
Use Compose to write down complex runtime options like port mappings or environment variables with host names and credentials
Use Compose to build your image and start a container from it with a single command
Build your application code and a standard CMD to run it into your Dockerfile.
I am new to docker, and have downloaded the tfx image using
docker pull tensorflow/tfx
However, I am unable to find anywhere how to successfully launch a container for the same.
here's a naive attempt
You can use docker image ls to get a list of locally-built Docker images. Note that an "image" is a template for a VM.
To instantiate the VM and shell into it, you would use a command like docker run -t --entrypoint bash tensorflow/tfx. This spins up a temporary VM based on the tensorflow/tfx image.
By default, Docker assumes you want the latest version of that image stored on your local machine, i.e. tensorflow/tfx:latest in the list. If you want to change it, you can reference a specific image version by name or hash, e.g. docker run -t --entrypoint bash tensorflow/tfx:1.0.0 or docker run -t --entrypoint bash fe507176d0e6. I typically use the docker image ls command first and cut & paste the hash, so my notes can be specific about which build I'm referencing even if I later edit the relevant Dockerfile.
Also note that changes you make inside that VM will not be saved. Once you exit the bash shell, it goes away. The shell is useful for checking the state & file structure of a constructed image. If you want to edit the image itself, use a Dockerfile. Each line of a Dockerfile creates a new image when the Dockerfile is compiled. If you know that something went wrong between lines 5 and 10 of the Dockerfile, you can potentially shell into each of those images in turn (with the docker run command I gave above) to see what went wrong. Kinda tedious, but it works.
Also note that docker run is not equivalent to running a TFX pipeline. For the latter, you want to look into the TFX CLI commands or otherwise compile the pipeline - and probably upload it to an external Kubeflow server.
Also note that the Docker image is just a starting point for one piece of your TFX pipeline. A full pipeline will require you to specify the components you want, a more-complete Dockerfile, and more. That's a huge topic, and IMO, the existing documentation leaves a lot to be desired. The Dockerfile you create describes the image which will be distributed to each of the workers which process the full pipeline. It's the place to specify dependencies, necessary files, and other custom setup for the machine. Most ML-relevant concerns are handled in other files.
When you are inside a docker container, is there anyway to obtain the "build history" (i.e. the original Dockerfile, or rather list of commands in the original Dockerfile that was used to build that container)?
The reason is that for tracking and version control purposes, it might be useful to indicate what/how the environment was configured when the process was run.
Thanks.
You can do it with
docker history
command. But not inside the container. Container itself does not have Docker and container itself does not hold its own history. To run that command you need to be in the host and not in the container.
docker history documentation
have a great explanation on how to use that command.
docker label is a good way to add additional metadata to your docker images.
Check this for more info.
You can get this data using docker inspect. But these commands can be run from outside the container, to run it from the inside you need to make use of docker remote api's as explained here in this answer.
You can also retrieve details of docker image using docker history through this remote api.
If you want just few details about images like version, etc. Then put those data as environment variable while building the image so that you can refer it later inside your running docker container.
I have a very simple system consisting of two containers, and I can successfully orchestrate them on my local machine with docker compose. I would like to put this system in a single VM in the cloud and allow others to easily do the same.
Because my preferred cloud provider provides easy access to a container OS, I would like to fit this system in a single container for easy distribution and deployment. I don't think I'm doing anything to violate the difficulties here, so I was hoping to use a Docker-in-Docker setup and make a single composite image that runs docker compose to bring up my two containers, just like on my local machine.
But, when I try to add
RUN docker pull my/image1
RUN docker pull my/image2
to the composite Dockerfile that extends the Docker image, those commands fail upon build because the Docker daemon is not running.
What I'm trying to accomplish here is to pull the two sub-images into my composite image at build time to minimize startup time of the composite image. Is there a way to do that?
There is a way to do this, but it is probably a bad idea.
Use docker-machine to create a docker-machine instance.
Use docker-machine env to get the credentials for your newly created docker-machine instance. These will be a couple of environment variables.
Add something like ARG DOCKER_HOST="tcp://172.16.62.130:2376" for each of the credentials created in the previous step. Put it in your Dockerfile before the first RUN docker ....
After the last ARG .. but before the first RUN docker ... put in some ENV DOCKER_HOST=${DOCKER_HOST} for all credential variables.
This should enable the docker pull to work, but it does not really solve your problem because the pull happens on the docker-machine and does not get captured in the docker image.
To get your desired effect you would need to additionally have
RUN docker save ... to export the pulled image to a tar archive file on the image.
Then you would have to add corresponding logic to docker load ... import the tar archive file.
The bottom line is that you can do this, but you probably should not. I don't think it will save you any time. It will probably cost you time.
My goal is to use Docker to create a mail setup running postfix + dovecot, fully configured and ready to go (on Ubuntu 14.04), so I could easily deploy on several servers. As far as I understand Docker, the process to do this is:
Spin up a new container (docker run -it ubuntu bash).
Install and configure postfix and dovecot.
If I need to shut down and take a break, I can exit the shell and return to the container via docker start <id> followed by docker attach <id>.
(here's where things get fuzzy for me)
At this point, is it better to export the image to a file, import on another server, and run it? How do I make sure the container will automatically start postfix, dovecot, and other services upon running it? I also don't quite understand the difference between using a Dockerfile to automate installations vs just installing it manually and exporting the image.
Configure multiple docker images using Dockerfiles
Each docker container should run only one service. So one container for postfix, one for another service etc. You can have your running containers communicate with each other
Build those images
Push those images to a registry so that you can easily pull them on different servers and have the same setup.
Pull those images on your different servers.
You can pass ENV variables when you start a container to configure it.
You should not install something directly inside a running container.
This defeat the pupose of having a reproducible setup with Docker.
Your step #2 should be a RUN entry inside a Dockerfile, that is then used to run docker build to create an image.
This image could then be used to start and stop running containers as needed.
See the Dockerfile RUN entry documentation. This is usually used with apt-get install to install needed components.
The ENTRYPOINT in the Dockerfile should be set to start your services.
In general it is recommended to have just one process in each image.