I run machine learning model in docker container. The model will go through train , eval , test and interactive 4 process. I use following ways to start docker:
sudo docker run --runtime=nvidia --name tensorflow_bert_dst -it -p 8888:8888 tensorflow/tensorflow:1.13.2-gpu-py3-jupyter
The process of training is so long that I close the terminal and let docker running in the server. Several hours later, I want to use following method check the result and interactive with the model.
docker attach tensorflow_bert_dst
But the docker hang and do not show any result, how to enter the running container and see the last time result and interactive with my model?
You can access the logs of docker containers via:
docker logs -f <containerid/name>
Please note this command shows the logs of the main process of the container (the process that you started the container with and this process should log to stdout/stderr.
So if you start your container like:
docker run --name mylongrunningcontainer <image name> <long running command that outputs to stdout/stderr>
then you can check the output anytime with:
docker logs -f mylongrunningcontainer
Related
I understand that docker start starts a container and docker run creates and starts a container given an image. But naively at first I just ran the command docker start image_name and it just outputs image_name to the console and no container is created and started.
Does docker start image_name do anything besides echo the name to the console? The doc is not very illustrative. If so, what a bad way to fail, better would have been to tell me, that that is not a container but an image and I should first create a container, but maybe I'm missing some useful action which docker start image_name does?
What happens when I run docker start image_name?
$ docker start --help
Usage: docker start [OPTIONS] CONTAINER [CONTAINER...]
Start one or more stopped containers
it just outputs image_name to the console and no container is created and started
a, --attach Attach STDOUT/STDERR and forward signals
The default is not to attach.
Does docker start image_name do anything besides echo the name to the console?
Yes it Start(-s) one or more stopped containers.
I'm missing some useful action which docker docker start image_name does?
It restarts the stopped container.
$ docker run --name work alpine sh -c 'echo important_work ; sleep 10 ; echo success'
important_work
success
$ docker start -a work
important_work
success
Most probably your container that you tested does nothing or expects interactive session to work, and without input it just exits.
I use following command to build web server
docker run --name webapp -p 8080:4000 mypyweb
When it stopped and I want to restart, I always use:
sudo docker start webapp && sudo docker exec -it webapp bash
But I can't see the server state as the first time:
Digest: sha256:e61b45be29f72fb119ec9f10ca660c3c54c6748cb0e02a412119fae3c8364ecd
Status: Downloaded newer image for ericgoebelbecker/stackify-tutorial:1.00
* Running on http://0.0.0.0:4000/ (Press CTRL+C to quit)
How can I see the state instead of interacting with the shell?
When you use docker run, the default behavior is to run the container detached. This runs in the background and is detached from your shell's stdin/out.
To run the container in the foreground and connected to stdin/out:
docker run --interactive --tty --publish=8080:4000 mypyweb
To docker start a container, similarly:
docker start --interactive --attach [CONTAINER]
NB --attach rather than -tty
You may list (all add --all) running containers:
docker container ls
E.g. I ran Nginx:
CONTAINER ID IMAGE PORTS NAMES
7cc4b4e1cfd6 nginx 0.0.0.0:8888->80/tcp nostalgic_thompson
NB You may use the NAME or any uniquely identifiable subset of the ID to reference the container
Then:
docker stop nostalgic_thompson
docker start --interative --attach 7cc4
You may check the container's logs (when running detached or from another shell) by grabbing the container's ID or NAMES
docker logs nostalgic_thompson
docker logs 7cc4
HTH!
Using docker exec is causing the shell to attach to the container. If you are comparing the behavior of docker run versus docker start, they behave differently, and it is confusing. Try this:
$ sudo docker start -a webapp
the -a flag tells docker to attach stdout/stderr and forward signals.
There are some other switches you can use with the start command (and a huge number for the run command). You can run docker [command] --help to get a summary of the options.
One other command that you might want to use is logs which will show the console output logs for a running container:
$ docker ps
[find the container ID]
$ docker logs [container ID]
If you think your container's misbehaving, it's often not wrong to just delete it and create a new one.
docker rm webapp
docker run --name webapp -p 8080:4000 mypyweb
Containers occasionally have more involved startup sequences and these can assume they're generally starting from a clean slate. It should also be extremely routine to delete and recreate a container; it's required for some basic tasks like upgrading the image underneath a container to a newer version or changing published ports or environment variables.
docker exec probably shouldn't be part of your core workflow, any more than you'd open a shell to interact with your Web browser. I generally don't tend to docker stop containers, except to immediately docker rm them.
Whenever I am using
docker run -i -t ae8c587afa40 /bin/bash command
it always creates a new container with a new ID. Can someone help me on how to save that particular container so that whenever I exit from that container my data doesn't get lost?
You can run the container in background by using the below option:
docker run -d --hostname=quickstart.cloudera --privileged=true -t -i -p 7180:7180 -p 8888:8888 -p 10000:10000 -p 8020:8020 -p 9092:9092 -p 9093:9093 -p 9393:9393 -p 9394:9394 -p 24042:24042 4239cd2958c6 /usr/bin/docker-quickstart
The -d option will help to run the container even if you exit from the terminal on which container was started. New container id will be created. The id present in the docker run command is the image id.
Can someone help me on how to save that particular container so that whenever I exit from that container my data doesn't get lost?
You should not rely on containers re usability. Instead of executing manually commands inside a container, build your own custom image including these commands. By doing this, you do not need to worry about starting a specific container cause all containers of that image will include the commands that you have defined when building the image.
Another advantage is that you can easily share your image via its Dockerfile or by pushing it to a repository.
I suggest that you read the docs to understand docker's concept.
A docker container goes into the stopped state when you exit from the container.
You can view all docker containers (running and stopped) by using docker ps -a.
In this listing you will see your old container.
If you need to restart it you can do this:
docker start -i <container_id>
This will restart the stopped container.
See - https://docs.docker.com/engine/reference/commandline/start/ for further details on the docker start command.
Note: Having data inside a docker container is not a good idea. You should use volumes or bind mounts to save your data. A good idea is to treat containers as being ephemeral and immutable.
Edit:
To Detach from a container without exiting the shell/process running in the container use the Key sequence Control + P Control + Q.
See https://docs.docker.com/engine/reference/commandline/attach/
Related to
docker container started in Detached mode stopped after process execution
https://serverfault.com/questions/661909/the-right-way-to-keep-docker-container-started-when-it-used-for-periodic-tasks
I do understand the difference between docker run and create + start, but don't understand how the actual containers created in these two ways differ.
Say I create and run a container with
docker run -dit debian:testing-slim
and then stop it. The created container can later be started with
docker start silly_docker_name
and it'll run in the background, because the entry command for the image is bash.
But when a container is first created
docker create --name silly_name debian:testing-slim
and then started with
docker start silly_name
then it'll exit immediately. Why isn't bash started, or how come it exits in this case?
The difference for a container process that is a shell (like bash in your debian example) is that a shell started without a terminal+interactive "mode" exits without doing anything.
You can test this by changing the command of a create'd container to something that doesn't require a terminal:
$ docker create --name thedate debian date
Now if I run thedate container, each time I run it it outputs the date (in the logs) and exits. docker logs thedate will show this; one entry for each run.
To be explicit, your docker run command has flags -dit: detached, interactive (connect STDIN), and tty are all enabled.
If you want a similar approach with create & start, then you need to allocate a tty for the created container:
$ docker create -it --name ashell debian
Now if I start it, I ask to attach/interactively to it and I get the same behavior as run:
$ docker start -ai ashell
root#6e44e2ae8817:/#
NOTE: [25 Jan 2018] Edited to add the -i flag on create as a commenter noted that as originally written this did not work, as the container metadata did not have stdin connected at the create stage
I'm new to docker.
I have an image that I want to run, but I want docker to see if that image is already running from another terminal...if it is running I don't want it to load another one...
is this something that can be done with docker?
if it helps, I'm running the docker with a privileged mode.
I've tried to search for singleton docker or something like that, but no luck.
updates-
1.working from ubuntu.
My scenario- from terminal X I run docker run Image_a
from terminal Y I run docker run Image_a
when trying to run from terminal Y, I want docker to check if there is already a docker running with Image_a, and the answer is true - I want docker not to run in terminal Y
You can use the following docker command to get all containers that running from specific image:
docker ps --filter ancestor="imagename:tag"
Example:
docker ps --filter ancestor="drone/drone:0.5"
Example Output:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3fb00087d4c1 drone/drone:0.5 "/drone agent" 6 days ago Up 26 minutes 8000/tcp drone_drone-agent_1
This approach uses docker api and docker daemon, so it doesnt matter if the run command executed in background or other terminal.
Aother approach:
If you have a single container form a single image:
Try naming your containers, You cant have 2 containers with the same name:
docker run --name uniquecontainer Image_a
Next time you run the above command you will get an error. Btw consider using -d so you dont have to switch terminals.
docker run -d --name uniquecontainer Image_a