I have a very simple system consisting of two containers, and I can successfully orchestrate them on my local machine with docker compose. I would like to put this system in a single VM in the cloud and allow others to easily do the same.
Because my preferred cloud provider provides easy access to a container OS, I would like to fit this system in a single container for easy distribution and deployment. I don't think I'm doing anything to violate the difficulties here, so I was hoping to use a Docker-in-Docker setup and make a single composite image that runs docker compose to bring up my two containers, just like on my local machine.
But, when I try to add
RUN docker pull my/image1
RUN docker pull my/image2
to the composite Dockerfile that extends the Docker image, those commands fail upon build because the Docker daemon is not running.
What I'm trying to accomplish here is to pull the two sub-images into my composite image at build time to minimize startup time of the composite image. Is there a way to do that?
There is a way to do this, but it is probably a bad idea.
Use docker-machine to create a docker-machine instance.
Use docker-machine env to get the credentials for your newly created docker-machine instance. These will be a couple of environment variables.
Add something like ARG DOCKER_HOST="tcp://172.16.62.130:2376" for each of the credentials created in the previous step. Put it in your Dockerfile before the first RUN docker ....
After the last ARG .. but before the first RUN docker ... put in some ENV DOCKER_HOST=${DOCKER_HOST} for all credential variables.
This should enable the docker pull to work, but it does not really solve your problem because the pull happens on the docker-machine and does not get captured in the docker image.
To get your desired effect you would need to additionally have
RUN docker save ... to export the pulled image to a tar archive file on the image.
Then you would have to add corresponding logic to docker load ... import the tar archive file.
The bottom line is that you can do this, but you probably should not. I don't think it will save you any time. It will probably cost you time.
Related
I'm having difficulties understanding docker. No matter how many tutorials I watch, guides I read, for me docker-compose is like being able to define multiple Dockerfiles, ie multiple containers. I can define environment variables in both, ports, commands, base images.
I read in other questions/discussions that Dockerfile defines how to build an image, and docker-compose is how to run an image, but I don't understand that. I can build docker containers without having to have a Dockerfile.
It's mainly for local development though. Does Dockerfile have an important role when deploying to AWS for example (where it's probably coming out of the box for example for EC2)?
So the reason why I can work locally with docker-compose only is because the base image is my computer (sorting out the task Dockerfile is supposed to do)?
Think about how you'd run some program, without Docker involved. Usually it's two steps:
Install it using a package manager like apt-get or brew, or build it from source
Run it, without needing any of its source code locally
In plain Docker without Compose, similarly, you have the same two steps:
docker pull a prebuilt image with the software, or docker build it from source
docker run it, without needing any of its source code locally
I'd aim to have a Dockerfile that creates an immutable copy of your image, with all of its source code and library dependencies as part of the image. The ideal is that you can docker run your image without -v options to inject source code or providing the command at the docker run command line.
The reality is that there are a lot of moving parts: you probably need to docker network create a network to get containers to communicate with each other, and use docker run -e environment variables to specify host names and database credentials, and launch multiple containers together, and so on. And that's where Compose comes in: instead of running a series of very long docker commands, you can put all of the details you need in a docker-compose.yml file, check it in, and run docker-compose up to get all of those parts put together.
So, do:
Use Compose to start multiple containers together
Use Compose to write down complex runtime options like port mappings or environment variables with host names and credentials
Use Compose to build your image and start a container from it with a single command
Build your application code and a standard CMD to run it into your Dockerfile.
I'm trying to deploy a docker image that is an asp.net core (.NET6) WebApi to ssh server.
I know the command for transferring the image file is:
docker save <my_image_namee> | ssh -C user#address docker load
Is it possible to execute this command within the Dockerfile right after building the image?
A Dockerfile can never run commands on the host, push its own image, save its output to a file, or anything else. The only thing it's possible to do in a Dockerfile is specify the commands needed to build the image within its isolated environment.
So, for example, in your Dockerfile you can't specify the image name that will be used (or its tag), or forcibly push something to a registry, or the complex docker save | ssh sequence you show. There's no option in the Dockerfile to do it.
You must run this as two separate commands using pretty much the syntax you show as your second command. If the two systems are on a shared network, a better approach would be to set up a registry server of some sort and docker push the image there; the docker save...docker load sequence isn't usually a preferred option, unless the two systems are on physically isolated networks. Whatever you need to do after you build the image, you could also consider asking your continuous-integration system to do it for you to avoid the manual step.
I'm pretty new to using Docker. I'm needing to deploy a NiFi instance through my employer, but the internal service we need to use requires a Dockerfile, not an image.
The service we're using requires the Dockerfile because each time the repository we're using is updated, the service is pointed to the Dockerfile and initiates the build process from it, then runs/operates the container.
I've already set up the NiFi flow to how it needs to operate, I'm just unsure of how to get a Dockerfile from an already existing container (or if that is even possible?)
I was looking into this myself, apparently there is no real way to do it, but you can inspect the docker container and pretty much get all the commands used to create the container except the OS used which is easy to find, you can spawn a bash into the container and do something like sudo uname -a, which you can just take and make your own docker image with. Usually you can find it on github, though.
docker inspect <image>
or you can do it through the docker desktop UI
You can use the Dockerfile that is in NiFi source code, see in this directory: https://github.com/apache/nifi/tree/main/nifi-docker/dockerhub
I am new to docker, and have downloaded the tfx image using
docker pull tensorflow/tfx
However, I am unable to find anywhere how to successfully launch a container for the same.
here's a naive attempt
You can use docker image ls to get a list of locally-built Docker images. Note that an "image" is a template for a VM.
To instantiate the VM and shell into it, you would use a command like docker run -t --entrypoint bash tensorflow/tfx. This spins up a temporary VM based on the tensorflow/tfx image.
By default, Docker assumes you want the latest version of that image stored on your local machine, i.e. tensorflow/tfx:latest in the list. If you want to change it, you can reference a specific image version by name or hash, e.g. docker run -t --entrypoint bash tensorflow/tfx:1.0.0 or docker run -t --entrypoint bash fe507176d0e6. I typically use the docker image ls command first and cut & paste the hash, so my notes can be specific about which build I'm referencing even if I later edit the relevant Dockerfile.
Also note that changes you make inside that VM will not be saved. Once you exit the bash shell, it goes away. The shell is useful for checking the state & file structure of a constructed image. If you want to edit the image itself, use a Dockerfile. Each line of a Dockerfile creates a new image when the Dockerfile is compiled. If you know that something went wrong between lines 5 and 10 of the Dockerfile, you can potentially shell into each of those images in turn (with the docker run command I gave above) to see what went wrong. Kinda tedious, but it works.
Also note that docker run is not equivalent to running a TFX pipeline. For the latter, you want to look into the TFX CLI commands or otherwise compile the pipeline - and probably upload it to an external Kubeflow server.
Also note that the Docker image is just a starting point for one piece of your TFX pipeline. A full pipeline will require you to specify the components you want, a more-complete Dockerfile, and more. That's a huge topic, and IMO, the existing documentation leaves a lot to be desired. The Dockerfile you create describes the image which will be distributed to each of the workers which process the full pipeline. It's the place to specify dependencies, necessary files, and other custom setup for the machine. Most ML-relevant concerns are handled in other files.
I'm creating an instance template under GCP's Cloud Engine section.
Usually when deploying a docker image, there's a docker file that includes some startup scripts after specifying the base image to pull and build, but I can't see where I can either submit a docker file, or enter startup scripts.
I can see a field for startup scripts for the Cloud Compute instance, but that's different from the scripts passed on for the Docker's startup.
Are these perhaps to be filled in under "Command", "Command arguments", or "Environment Variables"?
For clarification, this is someone else's image of a dockerfile I pulled from Google Images. The part I wish to add is "rectangled" in red, the RUN commands, but not these exact commands.
In my case, I would like to add something like
RUN python /pythonscript.py
If I understood well, you are trying to create a Docker image not a compute instance image.
Compute instance can run a docker image that you already builded and pushed to either gcr or any other repository.
Try to build your docker image normaly, push it in a docker repo then use it.
You can run a startup script directly in the Docker container by using a ‘command’ section. If you need to install something after starting a container for example Apache, you should use a Docker image that has Apache.
If you need to run some other arguments, like creating environment variables, here you can find the full list of flags when creating a container image on VM instance.