I used to run a long training process on a remote server with GPU capabilities. Now my work schedule changes, so I can't have my computer connected to a network all the time till I finish the process. I found that nohup is the solution for me. But I don't know how to keep invoke the process correctly related my situation.
I use ssh to connect to the remote server.
I have to use docker to access to GPU.
Then I start the process in the docker.
If I start the process with nohup in docker, I can't really leave docker, right. So, do I use nohup at each step?
Edit:
I need the terminal output of the process at step 3, because I need that information to carry out the rest of the work. Consider, step 3 is training a neural network. So, the training log tells me the accuracy of different models at different iterations. I use that information to do the testing.
Following #David Maze's suggestion, I did this (a slightly different approach as I was not familiar with docker a whole lot)
Logged in to the remote server.
Configured the docker script to have remote workdir.
...
WORKDIR /workspace
...
After building the docker container, run docker with mount option to mount the local project to docker workdir. When running docker, I used nohup. Since I don't need interactive mode I ignored the -it flag.
nohup docker run --gpus all -v $(pwd)/path-to-project-root:/workspace/ docker-image:tag bash -c "command1; command2" > project.out 2>&1 &
To test this, I logged out from the server and see the content of project.out later. It contained the expected output.
Related
TL;DR
"Reading/writing to stdin/stdout of Entrypoint executable process on running container (ex. Docker, podman) via socket/http"
My question is in regards to executable containers and how to spin/wrap them up as servers, and connect to them remotely (socket/curl/http).
I have an executable container that is not a server, just a standalone executable ENTRYPOINT command. (Example: wkhtmltopdf). I would love to use an image like that as a container, but not make it a part of my own container/Dockerfile (i.e. not to modify/maintain it).
As I understand if I run this image as a container, it will immediately die as there is no input/ on this process and it does not act as a server/listening for connections.
A lot of tutorials state that you can run containers via docker run ... command.
If I am on Windows inside WSL2 distro I am not necessarily going to have docker installed locally. But I believe that we can connect to other containers that are on the same network (default bridge network).
But how would one spin-up/expose executable container as a server and connect to it's stdin/stdout?
I want for container to listen on port (direct to executable stdin), process the request with said executable, and obtain the results from stdout/curl.
What I think can work:
Spinning process as a server (Golang example)
Do I need to wrap every executable that I want to expose BUT not make it as a part of my own Dockerfile or is there a common pattern/tool I can use for such a scenario.
P.S. wkhtmltopdf executable example I gave also comes with python and node included. (I am thinking about some sort of one-liner solution akin to docker run --rm -v ... X ... python -m SimpleHTTPServer ... )
This answer just describes the situation on Linux.
I haven't tested this but I think you could combine systemd socket activation
[Socket]
ListenStream=127.0.0.1:9999
Accept=yes
(see example https://unix.stackexchange.com/questions/551000/systemd-socket-activation-stdin)
with netcat (see example
https://unix.stackexchange.com/questions/332163/netcat-send-text-to-echo-service-read-reply-then-exit)
Use the -i flag, i.e., podman run -i ... or docker run -i to pass stdin in to the container.
So, is there a point in the command "start"? like in "docker start -i albineContainer".
If I do this, I can't really do anything with the albine inside the container, I would have to do a run and create another container with the "-it" command and "sh" after (or "/bin/bash", don't remember it correctly right now).
Is that how it will go most of the times? delete and rebuilt containers and do the command "-it" if you want to do stuff in them? or would it more depend on the Dockerfile, how you define the cmd.
New to Docker in general and trying to understand the basics on how to use it. Thanks for the help.
Running docker run/exec with -it means you run the docker container and attach an interactive terminal to it.
Note that you can also run docker applications without attaching to them, and they will still run in the background.
Docker allows you to run a program (which can be bash, but does not have to be) in an isolated environment.
For example, try running the jenkins docker image: https://hub.docker.com/_/jenkins.
this will create a container, without you having attach to it, and you would still be able to use it.
You can also attach to an existing, running container by using docker exec -it [container_name] bash.
You can also use docker logs to peek at the stdout of a certain docker container, without actually attaching to its shell interactively.
You almost never use docker start. It's only possible to use it in two unusual circumstances:
If you've created a container with docker create, then docker start will run the process you named there. (But it's much more common to use docker run to do both things together.)
If you've stopped a container with docker stop, docker start will run its process again. (But typically you'll want to docker rm the container once you've stopped it.)
Your question and other comments hint at using an interactive shell in an unmodified Alpine container. Neither is a typical practice. Usually you'll take some complete application and its dependencies and package it into an image, and docker run will run that complete packaged application. Tutorials like Docker's Build and run your image go through this workflow in reasonable detail.
My general day-to-day workflow involves building and testing a program outside of Docker. Once I believe it works, then I run docker build and docker run, and docker rm the container once I'm done. I rarely run docker exec: it is a useful debugging tool but not the standard way to interact with a process. docker start isn't something I really ever run.
I'm new to docker and I'm starting of building, deploying, and maintaining telemetry like services (grafana, prometheus, ...). One thing I've come accross is that I have a need to start up grafana with some default/preconfigured settings (dashboard, users, org, datasources, ...). Grafana allows some startup configuration in its config file but not with all its features (users, org, ...). Outside of (if I weren't using) docker I use a ansible script to configure the not supported parts of grafana. However, when I build my custom grafana image (with allowed startup config) and later start a grafana container of that image is there a way to specify "post-start" commands or steps in docker file? I image it to be something like every time a container of my image is deployed some steps are issues to configure that container.
Any suggestions? Would I still need to use ansible or other tools like this to manage it?
This is trickier than it sounds. Continuing to use Ansible to configure it post-startup is probably a good compromise between being straightforward, code you already have, and using standard Docker tooling and images.
If this is for a test environment, one possibility is to keep a reference copy of Grafana's config and data directories. You'd have to distribute these separately from the Docker images.
mkdir grafana
docker run \
-v $PWD/grafana/config:/etc/grafana \
-v $PWD/grafana/data:/var/lib/grafana \
... \
grafana/grafana
...
tar cvzf grafana.tar.gz grafana
Once you have the tar file, you can restart the system from a known configuration:
tar xvzf grafana.tar.gz
docker run \
-v $PWD/grafana/config:/etc/grafana \
-v $PWD/grafana/data:/var/lib/grafana \
... \
grafana/grafana
Several of the standard Docker Hub database images have the ability to do first-time configuration, via an entrypoint script; I'll refer to the mysql image's entrypoint script here. The basic technique involves:
Determine whether the command given to start the container is to actually start the server, and if this is the first startup.
Start the server, as a background process, recording its pid.
Wait for the server to become available.
Actually do the first-time initialization.
Stop the server that got launched as a background process.
Go on to exec "$#" as normal to launch the server "for real".
The basic constraint here is that you want the server process to be the only thing running in the container once everything is done. That means commands like docker stop will directly signal the server, and if the server fails, it's the main container process so that will cause the container to exit. Once the entrypoint script has replaced itself with the server as the main container process (by execimg it), you can't do any more post-startup work. That leads to the sequence of starting a temporary copy of the server to do initialization work.
Once you've done this initialization work once the relevant content is usually stored in persisted data directories or external databases.
SO questions have a common shortcut of starting a server process in the background, and then using something like tail -f /dev/null as the actual main container process. This means that docker stop will signal the tail process, but not tell the server that it's about to shut down; it also means that if the server does fail, since the tail process is still running, the container won't exit. I'd discourage this shortcut.
I am creating a docker container with Ubuntu:16.04 image using python docker package. I am passing tty as True and detach as True to the client.containers.run() function. The container starts with /sbin/init process. The container is created successfully. But the problem is, the login prompt on my host machine is replaced with the container login prompt on my host machine console. As a result, I am not able to the login on the machine on the console. SSH connection to the machine work fine.
This happens even when I run my python script after connecting SSH to the machine. I tried different options like setting tty to False, setting stdout to False, setting the environment variable TERM to xterm in the container, but nothing help.
It would be really great if someone can suggest a solution for this problem.
My script is very simple:
import docker
client = docker.from_env()
container = client.containers.run('ubuntu:16.04', '/sbin/init', privileged=True,
detach=True, tty=True, stdin_open=True, stdout=False, stderr=False,
environment=['TERM=xterm'])
I am not using any dockerfile.
I have been able to figure out that this problem happens when I start container in privileged mode. If I do this, the /sbin/init process launches /sbin/agetty processes which causes /dev/tty to be attached to the container. I need to figure out a way to start /sbin/init in such a way that it does not create /sbin/agetty processes.
/sbin/init in Ubuntu is a service called systemd. If you look at the linked page it does a ton of things – configures various kernel parameters, mounts filesystems, configures the network, launches getty process, .... Many of these things require changing host-global settings, and if you launch a container with --privileged you're allowing systemd to do that.
I'd give two key recommendations on this command:
Don't run systemd in Docker. If you really need a multi-process init system, supervisord is popular, but prefer single-process containers. If you know you need some init(8) (process ID 1 has some responsibilities) then tini is another popular option.
Don't directly run bare Linux distribution images. Whatever software you're trying to run, it's almost assuredly not in an alpine or ubuntu image. Build a custom image that has the software you need and run that; you should set up its CMD correctly so that you can docker run the image without any manual setup.
Also remember that the ability to run any Docker command at all implies unrestricted root-level access over the host. You're seeing some of that here where a --privileged container is taking over the host's console; it's also very very easy to read and edit files like the host's /etc/shadow and /etc/sudoers. There's nothing technically wrong with the kind of script you're showing, but you need to be extremely careful with standard security concerns.
Right now, for my workflow (personal project, but for sharing with friends I would like to simplfy it), I have the steps:
1. docker run -it -p 3456:3456 -p 39499:39499 maccam912/tahoe-node /bin/bash
2. In docker, run the script tahoe-run.sh which will either do a first-time configuration, or just run the tahoe service if it is not the first run.
My question is how I would simplify this. A few specific points below:
I would like to tell docker to just run the container and let it do its thing in the background. if I changed it by replacing the -it with -d I understand that would get me closer, but it never seems to leave anything running when I do a docker ps the same way the above workflow does, and then me detaching with CTRL+P, CTRL+Q.
Do I need to -p those ports? I EXPOSE them in the Dockerfile. Also, since they are being mapped directly to the same port in the container, do I need to write each twice? Those ports are what the container is using AND what the outside world is expecting to be able to use to connect to the tahoe "server".
How could I get this working so that instead of running /bin/bash I run /tahoe-run.sh right away?
My perfect command would look something like docker run -d maccam912/tahoe-node /tahoe-run.sh if the port mapping stuff is not necessary. The end goal of this would be to let the container run in the background, with tahoe-run.sh kicking off a server that just runs "forever".
Let me know if I can clarify anything or if there is a simpler way to do this. Feel free to take a look at my Dockerfile and make any suggestions if it would simplify things there, but this project is intentionally following ease of use (hopefully getting things down to 1 command) over separation of duties, as is recommended.
I do believe I figured it out:
The problem with having just -d specified is that as soon as the command given returns/exits, the container stops running. My solution was to end the last line with a &, leaving the script hanging and not exiting.
As for the ports, the EXPOSE in the Dockerfile does nothing more than make those ports available to the host machine, not to remote machines like a server would need. the -p is still necessary. The doubling of the ports (in:out) is needed because specifying only one will map that port to a random high numbered port in the container.
Please correct me if I'm wrong on any of this, but in this case the best solution was to modify my script, and will be to use docker run -d -p 3456:3456 -p 39499:39499 maccam912/tahoe-node /tahoe-run.sh and the new script will leave it hanging and the container running.