My query is similar to this Execute Command on host during docker build but I need my container to be running when I execute the command.
Background - I'm trying to create a base image for the database part of an application, using the mysql:8.0 image. The installation instructions for the product require me to run a DDL script to create the database (Done, by copying .sql file to the entrypoint directory), but the second step involves running a java based application which reads various config files to insert the required data into the running database. I would like this second step to be captured in the dockerfile somehow so I can then build a new base image containing the tables and the initial data.
Things I've thought of:
Install java and copy the quite large config tool to the container
and EXEC the appropriate command, but I want to avoid installing
java into the database container and certainly the subsequent image
if I can.
I could run the config tool on the host manually and
connect to the running container but my understanding is that this
would only apply to the running container - I couldn't get this into
a new image? It needs to be done from the dockerfile for docker build
to work.
I suspect docker just isn't designed for this.
Related
I'm having difficulties understanding docker. No matter how many tutorials I watch, guides I read, for me docker-compose is like being able to define multiple Dockerfiles, ie multiple containers. I can define environment variables in both, ports, commands, base images.
I read in other questions/discussions that Dockerfile defines how to build an image, and docker-compose is how to run an image, but I don't understand that. I can build docker containers without having to have a Dockerfile.
It's mainly for local development though. Does Dockerfile have an important role when deploying to AWS for example (where it's probably coming out of the box for example for EC2)?
So the reason why I can work locally with docker-compose only is because the base image is my computer (sorting out the task Dockerfile is supposed to do)?
Think about how you'd run some program, without Docker involved. Usually it's two steps:
Install it using a package manager like apt-get or brew, or build it from source
Run it, without needing any of its source code locally
In plain Docker without Compose, similarly, you have the same two steps:
docker pull a prebuilt image with the software, or docker build it from source
docker run it, without needing any of its source code locally
I'd aim to have a Dockerfile that creates an immutable copy of your image, with all of its source code and library dependencies as part of the image. The ideal is that you can docker run your image without -v options to inject source code or providing the command at the docker run command line.
The reality is that there are a lot of moving parts: you probably need to docker network create a network to get containers to communicate with each other, and use docker run -e environment variables to specify host names and database credentials, and launch multiple containers together, and so on. And that's where Compose comes in: instead of running a series of very long docker commands, you can put all of the details you need in a docker-compose.yml file, check it in, and run docker-compose up to get all of those parts put together.
So, do:
Use Compose to start multiple containers together
Use Compose to write down complex runtime options like port mappings or environment variables with host names and credentials
Use Compose to build your image and start a container from it with a single command
Build your application code and a standard CMD to run it into your Dockerfile.
I'm trying to deploy a docker image that is an asp.net core (.NET6) WebApi to ssh server.
I know the command for transferring the image file is:
docker save <my_image_namee> | ssh -C user#address docker load
Is it possible to execute this command within the Dockerfile right after building the image?
A Dockerfile can never run commands on the host, push its own image, save its output to a file, or anything else. The only thing it's possible to do in a Dockerfile is specify the commands needed to build the image within its isolated environment.
So, for example, in your Dockerfile you can't specify the image name that will be used (or its tag), or forcibly push something to a registry, or the complex docker save | ssh sequence you show. There's no option in the Dockerfile to do it.
You must run this as two separate commands using pretty much the syntax you show as your second command. If the two systems are on a shared network, a better approach would be to set up a registry server of some sort and docker push the image there; the docker save...docker load sequence isn't usually a preferred option, unless the two systems are on physically isolated networks. Whatever you need to do after you build the image, you could also consider asking your continuous-integration system to do it for you to avoid the manual step.
I am building a Dockerfile where I pull a postgresq client. What I would like to do is to connect to the db and execute some sql commands (all defined in Dockerfile and with no interaction). If I execute something like this below, I am able to connect:
docker run -it --rm jbergknoff/postgresql-client postgresql://username:password#10.1.0.173:5432/db
but I do not know how to replicate this inside the Dockerfile:
FROM jbergknoff/postgresql-client
RUN postgresql://username:password#10.1.0.173:5432/db // error
where the RUN command gives me error.
The reason is very clear, Postgres service is not running at this stage I mean during the build time.
RUN postgresql://username:password#10.1.0.173:5432/db // error
So this will not work as you are expecting.
This is a common problem with such docker image or custom Docker, So I will suggest using offical image so you will not bother with custom entrypoint script and all you just need to copy your SQL script to /docker-entrypoint-initdb.d during build time or can mount to this path at runtime.
FROM postgres
COPY my_initdb.sql /docker-entrypoint-initdb.d/
That's it, the offical image will take care of this script and will run the script when the Postgres service is up and running.
If you want To make this working, you must place such command at entrypoint where the Postgres service is running and capable to handle the connection.
Initialization scripts
If you would like to do additional initialization in an image derived
from this one, add one or more *.sql, *.sql.gz, or *.sh scripts under
/docker-entrypoint-initdb.d (creating the directory if necessary).
After the entrypoint calls initdb to create the default postgres user
and database, it will run any *.sql files, run any executable *.sh
scripts, and source any non-executable *.sh scripts found in that
directory to do further initialization before starting the service.
I have two Dockerfiles, one for a database, and one for a web server. The web server's Dockerfile has a RUN statement which requires a connection to the database container. The web server is unable to resolve the database's IP then errors out. But if I comment out the RUN line, then manually run it inside the container, it successfully resolves the database. Should the web server be able to resolve the database during its build process?
# Web server
FROM tomcat:9.0.26-jdk13-openjdk-oracle
# The database container cannot be resolved when myscript runs. "Unable to connect to the database." is thrown.
RUN myscript
CMD catalina.sh run
# But if I comment out the RUN line then connect to web server container and run myscript, the database container is resolved
docker exec ... bash
# This works
./myscript
I ran into the same problem on database migrations and NuGet pushes. You may want to run something similar on your db like migrations, initial/test data and so on. It could be solved in two ways:
Move your DB operations to the ENTRYPOINT so that they're executed at runtime (where the DB container is up and reachable).
Build your image using docker build instead of something like docker-compose up --build because docker build has a switch called --network. So you could create a network in your compose file, bring the DB up with docker-compose up -d db-container and then access them during the build with docker build --network db-container-network -t your-image .
I'd prefer #1 over #2 if possible because
it's simpler: the network is only present in docker-compose file, not on multiple places
you can specify relations usind depends_on and make sure that they're respected properly without taking manually care of it
But depending on the action you want to execute, you need to take care that it's not executed multiple times because it's running on every start and not just during build (when the cache got purged by file changes).
However, I'd consider this as best practice anyway when running such automated DB operations to expect that they may executed more than one and should create the expected result anyway (e.g. by checking if the migration version or change is present).
We're trying to build a Docker stack that includes our complete application: a Postgres database and at least one web application.
When the stack is started, we expect the application to be immediately working - there should not be any delay due to database setup or data import. So the database schema (DDL) and the initial data have to be imported when the image is created.
This could be achieved by a RUN command in the dockerfile, for example
RUN psql.exe -f initalize.sql -h myhost -d mydatabase -U myuser
RUN data-import.exe myhost mydatabase myuser
However, AFAIU this would execute data-import.exe inside the Postgres container, which can only work if the Postgres container is a Windows container. Our production uses a Linux Postgres distribution, so this is not a good idea. We need the image to be a Linux Postgres container.
So the natural solution is to execute data-import.exe on the host, like this:
When we run docker build, a Linux Postgres container is started.
RUN psql.exe ... runs some SQL commands inside the Postgres container.
Now, our data-import.exe is executed on the host. Its Postgres client connects to the database in the container and imports the data.
When the data import is done, the data is committed to the image, and docker builds an image which contains the Postgres database together with the imported data.
Is there such a command? If not, how can we implement this scenario in docker?
Use the correct tool, a dockerfile is not a hammer for everything.
Obviously you come from a state where you had postgres up before using some import-tool. Now you can mimic that strategy by firing up a postgres container (without dockerfile, just docker/kubernetes). Then run the import-tool, stop the postgres-container, and make a snapshot of the result using "docker commit". The committed image will be used for the next stages of your deployment.
In Docker generally the application data is stored separately from images and containers; for instance you'd frequently use a docker run -v option to store data in a host directory or Docker volume so that it would outlive its container. You wouldn't generally try to bake data into an image, both for scale reasons and because any changes will be lost when a container exits.
As you've described the higher-level problem, I might distribute something like a "test kit" that included a docker-compose.yml and a base data directory. Your Docker Compose file would use a stock PostgreSQL container with data:
postgres:
image: postgres:10.5
volumes:
- './postgres:/var/lib/postgresql/data'
To answer the specific question you asked, docker build steps only run individual commands within Docker container space; they can't run arbitrary host commands, read filesystem content outside of the tree containing the Dockerfile, or write any sort of host filesystem content outside the container.