I'm new to docker and I'm starting of building, deploying, and maintaining telemetry like services (grafana, prometheus, ...). One thing I've come accross is that I have a need to start up grafana with some default/preconfigured settings (dashboard, users, org, datasources, ...). Grafana allows some startup configuration in its config file but not with all its features (users, org, ...). Outside of (if I weren't using) docker I use a ansible script to configure the not supported parts of grafana. However, when I build my custom grafana image (with allowed startup config) and later start a grafana container of that image is there a way to specify "post-start" commands or steps in docker file? I image it to be something like every time a container of my image is deployed some steps are issues to configure that container.
Any suggestions? Would I still need to use ansible or other tools like this to manage it?
This is trickier than it sounds. Continuing to use Ansible to configure it post-startup is probably a good compromise between being straightforward, code you already have, and using standard Docker tooling and images.
If this is for a test environment, one possibility is to keep a reference copy of Grafana's config and data directories. You'd have to distribute these separately from the Docker images.
mkdir grafana
docker run \
-v $PWD/grafana/config:/etc/grafana \
-v $PWD/grafana/data:/var/lib/grafana \
... \
grafana/grafana
...
tar cvzf grafana.tar.gz grafana
Once you have the tar file, you can restart the system from a known configuration:
tar xvzf grafana.tar.gz
docker run \
-v $PWD/grafana/config:/etc/grafana \
-v $PWD/grafana/data:/var/lib/grafana \
... \
grafana/grafana
Several of the standard Docker Hub database images have the ability to do first-time configuration, via an entrypoint script; I'll refer to the mysql image's entrypoint script here. The basic technique involves:
Determine whether the command given to start the container is to actually start the server, and if this is the first startup.
Start the server, as a background process, recording its pid.
Wait for the server to become available.
Actually do the first-time initialization.
Stop the server that got launched as a background process.
Go on to exec "$#" as normal to launch the server "for real".
The basic constraint here is that you want the server process to be the only thing running in the container once everything is done. That means commands like docker stop will directly signal the server, and if the server fails, it's the main container process so that will cause the container to exit. Once the entrypoint script has replaced itself with the server as the main container process (by execimg it), you can't do any more post-startup work. That leads to the sequence of starting a temporary copy of the server to do initialization work.
Once you've done this initialization work once the relevant content is usually stored in persisted data directories or external databases.
SO questions have a common shortcut of starting a server process in the background, and then using something like tail -f /dev/null as the actual main container process. This means that docker stop will signal the tail process, but not tell the server that it's about to shut down; it also means that if the server does fail, since the tail process is still running, the container won't exit. I'd discourage this shortcut.
Related
When running a Docker Compose project, it would be nice to be able to open an app with certain parameters on the host operating system (on which docker-compose up is being invoked). This would be legitimately useful when running web apps. For example, I would love to have Docker Compose automatically open a browser on the host with location of http://localhost:8080, when I run docker-compose run, rather than manually opening a browser and entering the http://localhost:8080. Just the way we see in Minikube (e.g when running minikube service web-deployment).
I am aware there are parameters to use in docker-compose.yml to pass commands to run in containers, like command and entrypoint, but I don't know if that is possible for applications on the host OS.
Compose can do a pretty limited set of things. It can build Docker images, without any ordering constraints, and it can start (presumably long-running) Docker containers, with very limited ordering constraints. It can create a couple of associated Docker objects like networks and named volumes. That's literally all it can do, though; it cannot do larger-scale orchestration task ("run this migration container to completion, then run this application") or launch non-Docker tasks.
You might use some host-based tool to manage this instead. Even a shell script would be enough; possibly something like
#!/bin/sh
# start the container stack
# (assumes the caller has permission to do this)
docker-compose up -d
# wait for the service to be ready
while ! curl --fail --silent --head http://localhost:8080; do
sleep 1
done
# open the browser window
open http://localhost:8080
I used to run a long training process on a remote server with GPU capabilities. Now my work schedule changes, so I can't have my computer connected to a network all the time till I finish the process. I found that nohup is the solution for me. But I don't know how to keep invoke the process correctly related my situation.
I use ssh to connect to the remote server.
I have to use docker to access to GPU.
Then I start the process in the docker.
If I start the process with nohup in docker, I can't really leave docker, right. So, do I use nohup at each step?
Edit:
I need the terminal output of the process at step 3, because I need that information to carry out the rest of the work. Consider, step 3 is training a neural network. So, the training log tells me the accuracy of different models at different iterations. I use that information to do the testing.
Following #David Maze's suggestion, I did this (a slightly different approach as I was not familiar with docker a whole lot)
Logged in to the remote server.
Configured the docker script to have remote workdir.
...
WORKDIR /workspace
...
After building the docker container, run docker with mount option to mount the local project to docker workdir. When running docker, I used nohup. Since I don't need interactive mode I ignored the -it flag.
nohup docker run --gpus all -v $(pwd)/path-to-project-root:/workspace/ docker-image:tag bash -c "command1; command2" > project.out 2>&1 &
To test this, I logged out from the server and see the content of project.out later. It contained the expected output.
In my Dockerfile I am trying to install multiple services and want to have them all start up automatically when I launch the container.
One among the services is mysql and when I launch the container I don't see the mysql service starting up. When I try to start manually, I get the error:
Failed to get D-Bus connection: Operation not permitted
Dockerfile:
FROM centos:7
RUN yum -y install mariadb mariadb-server
COPY start.sh start.sh
CMD ["/bin/bash", "start.sh"]
My start.sh file:
service mariadb start
Docker build:
docker build --tag="pbellamk/mariadb" .
Docker run:
docker run -it -d --privileged=true pbellamk/mariadb bash
I have checked the centos:systemd image and that doesn't help too. How do I launch the container with the services started using systemctl/service commands.
When you do docker run with bash as the command, the init system (e.g. SystemD) doesn’t get started (nor does your start script, since the command you pass overrides the CMD in the Dockerfile). Try to change the command you use to /sbin/init, start the container in daemon mode with -d, and then look around in a shell using docker exec -it <container id> sh.
Docker is designed around the idea of a single service/process per container. Although it definitely supports running multiple processes in a container and in no way stops you from doing that, you will run into areas eventually where multiple services in a container doesn't quite map to what Docker or external tools expect. Things like moving to scaling of services, or using Docker swarm across hosts only support the concept of one service per container.
Docker Compose allows you to compose multiple containers into a single definition, which means you can use more of the standard, prebuilt containers (httpd, mariadb) rather than building your own. Compose definitions map to Docker Swarm services fairly easily. Also look at Kubernetes and Marathon/Mesos for managing groups of containers as a service.
Process management in Docker
It's possible to run systemd in a container but it requires --privileged access to the host and the /sys/fs/cgroup volume mounted so may not be the best fit for most use cases.
The s6-overlay project provides a more docker friendly process management system using s6.
It's fairly rare you actually need ssh access into a container, but if that's a hard requirement then you are going to be stuck building your own containers and using a process manager.
You can avoid running a systemd daemon inside a docker container altogether. You can even avoid to write a special start.sh script - that is another benefit when using the docker-systemctl-replacement script.
The docker systemctl.py can parse the normal *.service files to know how to start and stop services. You can register it as the CMD of an image in which case it will look for all the systemctl-enabled services - those will be started and stopped in the correct order.
The current testsuite includes testcases for the LAMP stack including centos, so it should run fine specifically in your setup.
I found this project:
https://github.com/defn/docker-systemd
which can be used to create an image based on the stock ubuntu image but with systemd and multiuser mode.
My use case is the first one mentioned in its Readme. I use it to test the installer script of my application that is installed as a systemd service. The installer creates a systemd service then enables and starts it. I need CI tests for the installer. The test should create the installer, install the application on an ubuntu, and connect to the service from outside.
Without systemd the installer would fail, and it would be much more difficult to write the test with vagrant. So, there are valid use cases for systemd in docker.
I really enjoy the concept of having a cluster of docker machines available to execute docker services. I also like the additional features not available to singular docker containers (such as docker secret).
But I really have no need for long-standing services. My use case is to simply execute a bash script to use the docker swarm to take in an arbitrary number of finite commands, and execute each as a running docker container on the same docker image, while using the secrets loaded up with docker swarm's secrets.
Can I do this?
I do not want to have this container be "long running". I want it to run, and then exit with the output when the bash script loaded into the container is finished.
You can apply the ideas presented in "One-shot containers on Docker Swarm" from alex ellis.
You still neeeds to create a service, but with the right restart policy.
For instance, for a quick web server:
docker service create --restart-condition=none --name crawler1 -e url=http://blog.alexellis.io -d crawl_site alexellis2/href-counter
(--restart-condition, not --restart-policy, as commented by ethergeist)
So by setting a restart condition of 0, the container will be scheduled somewhere in the swarm as a (task). The container will execute and then when ready - it will exit.
If the container fails to start for a valid reason then the restart policy will mean the application code never executes. It would also be ideal if we could immediately return the exit code (if non-zero) and the accompanying log output, too.
For the last part, use his tool: alexellis/jaas.
Run your first one-shot container:
# jaas -rm -image alexellis2/cows:latest
The -rm flag removes the Swarm service that was used to run your container.
The exit code from your container will also be available, you can check it with echo $?.
I'm dockerizing some of our services. For our dev environment, I'd like to make things as easy as possible for our developers and so I'm writing some scripts to manage the dockerized components. I want developers to be able to start and stop these services just as if they were non-dockerized. I don't want them to have to worry about creating and running the container vs stopping and starting and already-created container. I was thinking that this could be handled using Fig. To create the container (if it doesn't already exist) and start the service, I'd use fig up --no-recreate. To stop the service, I'd use fig stop.
I'd also like to ensure that developers are running containers built using the latest images. In other words, something would check to see if there was a later version of the image in our Docker registry. If so, this image would be downloaded and run to create a new container from that image. At the moment it seems like I'd have to use docker commands to list the contents of the registry (docker search) and compare that to existing local containers (docker ps -a) with the addition of some greping and awking or use the Docker API to achieve the same thing.
Any persistent data will be written to mounted volumes so the data should survive the creation of a new container.
This seems like it might be a common pattern so I'm wondering whether anyone else has given these sorts of scenarios any thought.
This is what I've decided to do for now for our Neo4j Docker image:
I've written a shell script around docker run that accepts command-line arguments for the port, database persistence directory on the host, log file persistence directory on the host. It executes a docker run command that looks like:
docker run --rm -it -p ${port}:7474 -v ${graphdir}:/var/lib/neo4j/data/graph.db -v ${logdir}:/var/log/neo4j my/neo4j
By default port is 7474, graphdir is $PWD/graph.db and logdir is $PWD/log.
--rm removes the container on exit, however the database and logs are maintained on the host's file system. So no containers are left around.
-it allows the container and the Neo4j service running within it to receive signals so that the service can be gracefully shut down (the Neo4j server gracefully shuts down on SIGINT) and the container exited by hitting ^C or sending it a SIGINT if the developer puts this in the background. No need for separate start/stop commands.
Although I certainly wouldn't do this in production, I think this fine for a dev environment.
I am not familiar with fig but your scenario seems good.
Usually, I prefer to kill/delete + run my container instead of playing with start/stop though. That way, if there is a new image available, Docker will use it. This work only for stateless services. As you are using Volumes for persistent data, you could do something like this.
Regarding the image update, what about running docker pull <image> every N minutes and checking the "Status" that the command returns? If it is up to date, then do nothing, otherwise, kill/rerun the container.