how to bring up failed container - docker

have a container that failed after a long setup and i want to log in (exec bash) at that point instead of executing the slow setup again. Is there any way?
The container is a left over from a docker build process, it is still the FROM ... AS builder stage.
if i try to start it, it will fail right away.
$ docker start -ai 3d35a7f7a7b4
/bin/sh: mvn: command not found
trying to exec anything right away doesn't work either
$ docker start 3d35a7f7a7b4 & docker exec 3d35a7f7a7b4 -it /bin/sh
[1] 403273
3d35a7f7a7b4
unable to upgrade to tcp, received 500
[1]+ Done docker start 3d35a7f7a7b4
more info:
$ docker inspect 3d35a7f7a7b4
[
{
"Id": "3d35a7f7a7b4018ebbbd9aa59356714d7fed291a43752cbcb86dd852c946cc1e",
"Created": "2022-07-06T23:56:37.001004587Z",
"Path": "/bin/sh",
"Args": [
"-c",
"mvn --version"
],
"State": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 127,
"Error": "",
"StartedAt": "2022-07-07T00:02:35.755444447Z",
"FinishedAt": "2022-07-07T00:02:35.75741167Z"
},
"Image": "sha256:4819e2469963fdf531ec5bce5401b7ae7d28cd403528c0109512b5170ef61752",
...

this is not an optimal answer. Here just for documentation (and for people to vote up if it is the best one can do with docker)
docker run can be used on the image of the stopped container, and you can pass the CMD parameter right away. But any other peculiarity of the stopped container will also have to be repeated. e.g. network.
for the example on the question:
host$ docker run -it sha256:4819e2469963fdf531ec5bce5401b7ae7d28cd403528c0109512b5170ef61752 /bin/bash
container# _

Related

In docker inspect, what do "StartedAt" and "FinishedAt" mean?

This is the result of running docker inspect on a running container:
$ docker inspect some_container | jq .[0].State
{
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 16086,
"ExitCode": 0,
"Error": "",
"StartedAt": "2021-09-16T02:36:12.036585245Z",
"FinishedAt": "2021-09-16T04:36:10.87103895+02:00"
}
Nobody was logged into that system at the times listed in the StartedAt and FinishedAt entries, and it doesn't seem like the container was restarted after a crash:
$ docker inspect lxonlinedlservice_rabbitmq_1 | grep RestartCount
"RestartCount": 0,
What do the StartedAt and FinishedAt entries mean?
From github
startedAt - Time at which previous execution of the container started
finishedAt - Time at which the container last terminated
You mentioned crash. Maybe container started after a crash at 2021-09-16T02:36:12.036585245Z and at 2021-09-16T04:36:10.87103895+02:00 there was another crash?
Or, might it me that docker host where the container runs was rebooted?
Suggest also to check that your clock is synced using ntp. Check this docker best practice.
To get the exact uptime for a container: docker inspect -f '{{ .State.StartedAt }}' CONTAINER_ID
StartedAt: when you started your image or container
FinishedAt: when you stopped your image or container
(from this answer https://stackoverflow.com/a/28203469/500902)

not able to run startup command in container created via docker-engine's go sdk api

I am trying to create a container with docker engine's go sdk api. I need to run a command on startup in the container, which I pass as a parameter to Client.ContainerCreate() api. I tried passing that command in different ways, but everytime found some issue. Below is the code I use:
resp, err := cli.ContainerCreate(ctx, &container.Config{
Image: "hyperledger/fabric-ca",
Cmd: []string{"/bin/sh"," fabric-ca-server start -b admin:adminpw"},
Env: []string{"FABRIC_CA_HOME=/etc/hyperledger/fabric-ca-server", "FABRIC_CA_SERVER_CA_NAME=ca.example.com"}, }, nil, nil, "ca.example.com")
if err != nil {
fmt.Println(" failed to create container, err:", err)
} else {
fmt.Println(" Container ID :", resp.ID, "warning:", resp.Warnings, "err:", err)
}
if err := cli.ContainerStart(ctx, resp.ID, types.ContainerStartOptions{}); err != nil {
fmt.Println("failed to start container, err:", err)
}
1)
If I don't provide Cmd parameter, container is created.
Container ID : 02edc80d6545ca2c8089a191ba9174070e1527dc027191e0d7686bff23a9f39d warning: [] err: <nil>
vignesh#vignesh-ThinkPad-E470 ~ $ docker container list
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
02edc80d6545 hyperledger/fabric-ca "/bin/sh -c 'fabric-…" 7 minutes ago Up 7 minutes 7054/tcp ca.example.com
2)
If I provide Cmd parameter as Cmd: []string{"/bin/sh"," fabric-ca-server start -b admin:adminpw"}, ContainerCreate() returns a container Id, but docker container list doesn't show any container and docker container inspect shows status as exited.
Container ID : c2f7bbc54e09665b0797eeaea43723f3fddf4538db8bf4327362b2535b9a088b warning: [] err: <nil>
vignesh#vignesh-ThinkPad-E470 ~ $ docker container list
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
vignesh#vignesh-ThinkPad-E470 ~ $ docker container inspect c2f7bbc54e09665b0797eeaea43723f3fddf4538db8bf4327362b2535b9a088b
[
{
"Id": "c2f7bbc54e09665b0797eeaea43723f3fddf4538db8bf4327362b2535b9a088b",
"Created": "2019-02-05T20:05:11.360875853Z",
"Path": "/bin/sh",
"Args": [
" fabric-ca-server start -b admin:adminpw"
],
"State": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 127,
"Error": "",
"StartedAt": "2019-02-05T20:05:12.448079587Z",
"FinishedAt": "2019-02-05T20:05:12.644957269Z"
},
3) If I provide Cmd parameter as Cmd: []string{"sh -c"," fabric-ca-server start -b admin:adminpw"}, ContainerCreate() returns a container Id, but ContainerStart() gives "exec: \"sh -c\": executable file not found in $PATH" error. docker container list doesn't show any container.
Container ID : d2752eb14267ccc170121d28ea9c51f2cd99227eba3d53b4430bd4b7eeec4787 warning: [] err: <nil>
failed to start container, err: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"sh -c\": executable file not found in $PATH": unknown
vignesh#vignesh-ThinkPad-E470 ~ $ docker container list
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
vignesh#vignesh-ThinkPad-E470 ~ $ docker container inspect d2752eb14267ccc170121d28ea9c51f2cd99227eba3d53b4430bd4b7eeec4787
[
{
"Id": "d2752eb14267ccc170121d28ea9c51f2cd99227eba3d53b4430bd4b7eeec4787",
"Created": "2019-02-06T17:02:13.989788091Z",
"Path": "sh -c",
"Args": [
" fabric-ca-server start -b admin:adminpw"
],
"State": {
"Status": "created",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 127,
"Error": "OCI runtime create failed: container_linux.go:348: starting container process caused \"exec: \\\"sh -c\\\": executable file not found in $PATH\": unknown",
"StartedAt": "0001-01-01T00:00:00Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
4) If I provide Cmd parameter as Cmd: []string{"fabric-ca-server start -b admin:adminpw"}, ContainerCreate() returns a container Id, but ContainerStart() gives \"exec: \\"fabric-ca-server start -b admin:adminpw\\": executable file not found in $PATH\": unknown" error. docker container list doesn't show any container.
Container ID : d81d4b8f5ae964ec8ef805671a8e4233b41ea363ad890da0218c0ef586d7a72c warning: [] err: <nil>
failed to start container, err: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"fabric-ca-server start -b admin:adminpw\": executable file not found in $PATH": unknown
vignesh#vignesh-ThinkPad-E470 ~ $ docker container list
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
vignesh#vignesh-ThinkPad-E470 ~ $ docker container inspect d81d4b8f5ae964ec8ef805671a8e4233b41ea363ad890da0218c0ef586d7a72c
[
{
"Id": "d81d4b8f5ae964ec8ef805671a8e4233b41ea363ad890da0218c0ef586d7a72c",
"Created": "2019-02-06T17:12:10.443261734Z",
"Path": "fabric-ca-server start -b admin:adminpw",
"Args": [],
"State": {
"Status": "created",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 127,
"Error": "OCI runtime create failed: container_linux.go:348: starting container process caused \"exec: \\\"fabric-ca-server start -b admin:adminpw\\\": executable file not found in $PATH\": unknown",
"StartedAt": "0001-01-01T00:00:00Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
There is a .yaml file to create the same container via docker-compose command where sh -c 'fabric-ca-server start -b admin:adminpw' is passed as startup command and the container is created.
I am trying to create that container via go code and facing issues.
As I am not seeing any issue with the command in .yaml file, I think command is fine. I am not able to figure out what I am missing. Kindly help :)
Had posted this question at github and got the answer there. https://github.com/moby/moby/issues/38687

Marathon Docker Tasks Failing

I have setup Marathon and Mesos on two of my machines.
I can successfully schedule commands from the marathon web console, but when I try to schedule a job involving docker images I immediately get job failed. Plus I get no stderr or stdout files.
Example Running a normal command:
Marathon job conf:
{
"id": "testecho",
"cmd": "echo hello; sleep 10",
"cpus": 1,
"mem": 128,
"disk": 0,
"instances": 1
}
On mesos I see that the tasks have succeeded. I have the stderr and stdout files like normal.
But now if I run a simple docker image task:
Marathon job conf:
{
"id": "/ubuntu",
"cmd": "date -u +%T",
"cpus": 0.5,
"mem": 512,
"disk": 0,
"instances": 1,
"container": {
"type": "DOCKER",
"volumes": [],
"docker": {
"image": "libmesos/ubuntu",
"network": null,
"portMappings": null,
"privileged": false,
"parameters": [],
"forcePullImage": false
}
},
"portDefinitions": [
{
"port": 10001,
"protocol": "tcp",
"labels": {}
}
]
}
On mesos, I see that it has instantly failed:
And I have no stderr or stdout files:
I also notice that on both my machines, when I run:
docker ps -a
I see nothing on both the machines. So that would mean that the docker jobs were not even launched
What could be affecting docker deployment?
The one reason I can think of is that the user that marathon uses to launch tasks not have access to docker? How do I test this?
I noticed that when I run the command:
sudo cat /etc/passwd
I see a user zookeeper. Maybe this is the user that doesn't have access to docker?
But when i do:
su zookeeper
I don't change user profiles
After going through a few tutorials I found the answer from the following tutorial: http://frankhinek.com/deploy-docker-containers-on-mesos-0-20/
I had to enable Docker Containerizer on my mesos-slaves
Set the --containerizers=docker,mesos" command line parameter:
echo "docker,mesos" | sudo tee /etc/mesos-slave/containerizers
Increase the executor timeout to 5 minutes1: (i guess this is optional)
echo "5mins" | sudo tee /etc/mesos-slave/executor_registration_timeout
Restart the Mesos Slave:
sudo service mesos-slave restart

marathon docker jobs hanged in deployment state

Hi I have been successfull so far with simple jobs in marathon but it stuck when i have tried deploying a deocker job in mesos through marathon framework.
I am using a json file as below to deploy a docker job:
{
"id": "pga-docker",
"cpus": 0.2,
"mem": 1024.0,
"instances": 1,
"container": {
"type": "DOCKER",
"docker": {
"image": "pga",
"network": "BRIDGE",
"portMappings": [
{ "containerPort": 80, "hostPort": 6565, "servicePort": 0, "protocol": "tcp" }
]
}
}
}
My pga docker image have no problem when run as container, but through marathon its just not working. Its staying in the deploying state forever.
I am using the below command line:
curl -X POST http://10.141.141.10:8080/v2/apps -d #basic-3.json -H "Content-type: application/json"
But when I run the same image from marathon UI, its working. To run from marathon I used "docker run --publish 6060:80 --name test --rm pga" in the cmd field of the UI new job page.
Any one have idea why this is hanged in the command line approach?
This is what i have found during some trial and error with the json file.
I found that when we run docker image in local system, if we have mentioned an entry point or a cmd then that will execute while running the container. But this is not same for mesos/marathon. my observation is that if I explicitly mentioned cmd in the deployment json then its working fine.
"cmd":"sh pga-setup.sh"
I will love to know if anyone faced a similar issue an solved it by another way.

Why does docker stop immediately after starting and how to prevent it from stopping?

I am trying to start a docker container using the following POST request:
Content-Type: application/json
{
"Hostname":"",
"Domainname": "",
"User":"",
"Memory":0,
"MemorySwap":0,
"CpuShares": 512,
"Cpuset": "0,1",
"AttachStdin":true,
"AttachStdout":true,
"AttachStderr":true,
"PortSpecs":6002,
"Tty":false,
"OpenStdin":false,
"StdinOnce":false,
"Env":null,
"Cmd":
[
"python",
"app.py"
],
"Image":"jobinar/smile_webapp",
"Volumes":{
"/tmp": {}
},
"WorkingDir":"",
"NetworkDisabled": false,
"ExposedPorts":{
"5000/tcp": {}
}
}
However, the container immediately stops after starting. How do I configure my request to prevent it from exiting?
I would appreciate a POST request which does this instead of the command-line way.
EDIT: I get a 201 CREATED response with the id of the created container and I can see that the container is created by running by using the docker ps -a command.
If you have upgraded you docker version you habe to delete /var/lib/docker/network on ubuntu

Resources