I'm having issues pulling from a private docker repo when I add a marathon application. I've tarred my ~/.docker folder (including the docker.config file which contains my login information) and distributed that along to my mesos slaves as /etc/docker.tar.gz (I'm using docker 1.6.2).
I've then added a new marathon app with:
dcos marathon add app marathon.json
My marathon.json is as follows:
{
"id": "api",
"cpus": 1,
"mem": 1024,
"instances": 1,
"container": {
"type": "DOCKER",
"docker": {
"image": "company/api",
}
},
"args": ["java", "-jar", "api.jar"],
"uris": [
"file:///etc/docker.tar.gz"
]
}
The marathon app never starts, however. In my slave logs I've found the following line:
Container x for executor y of framework z failed to start: Failed to 'docker pull company/api': exit status = exited with status 1 stderr = time="2015-11-12T00:03:57Z" level=fatal msg="Error: image company/api:latest not found"
How can I get this to pull correctly?
Related
I am building and deploying an application via Docker and ECS Fargate. I have my entrypoint command defined in the ECS Task definition. Upon pushing the image into a private ECR repository, I am getting this error when ECS Fargate attempts to deploy the docker image. Any advice would be helpful. Below is the dockerfile, Task Definition, and the error.
Dockerfile
FROM centos:7
COPY /src/main/build/application.zip /tmp/application.zip
COPY /src/main/residual-container-setup/application/init.sh /tmp/init.sh
#Environment variables and Entry point being defined via task definition
Task Definition
{
"containerDefinitions": [
{
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/application",
"awslogs-region": "us-east-2",
"awslogs-stream-prefix": "ecs"
}
},
"entryPoint": [
"[\"/bin/sh -c\"",
"\"/tmp/init.sh\"]"
],
"portMappings": [
{
"hostPort": 9003,
"protocol": "tcp",
"containerPort": 9003
}
],
"cpu": 0,
"environment": [
{
"name": "HOST",
"value": "dev.application.com"
},
{
"name": "REST_PORT",
"value": "8003"
}
],
"mountPoints": [],
"volumesFrom": [],
"image": "xxxxxxxxxxxx.dkr.ecr.us-east-2.amazonaws.com/application:latest",
"essential": true,
"name": "application"
}
]
Error
container_linux.go:380: starting container process caused: exec: "[\"/bin/sh -c\"": stat ["/bin/sh -c": no such file or directory
I attempted running the container locally with the following command: `docker run -it $docker_image /bin/sh
I was unable to even exec into the container. I believe I may need to install additionally in the image to get this to work. Any advice would be helpful.
Update
I have updated the dockerfile to update the permissions on the init script using the following command: chmod +x /tmp/init.sh
I have also update the task definition entrypoint attribute to ["/bin/sh", "-c", "/tmp/init.sh"]
After making these changes I am now being presented with the following:
container_linux.go:380: starting container process caused: exec: "-c": executable file not found in $PATH
Your entrypoint is defined wrongly.
The way you did it Linux thinks the path to the binary is "/bin/sh -c". If you check the container image I'm pretty sure you do not find that file either.
"container": {
"type": "MESOS",
"docker": {
"image": "redis",
"forcePullImage": false
}
}
The above example has the container type as Mesos..but again specifying "docker" image... for using universal container or mesos container, do we need to install docker?
because, when i try to run a sample in mesos with type "mesos" container, i am getting error like this:
unsupported container image:DOCKER.
I have not installed docker.
I am using Mesos1.1 version
See https://mesosphere.github.io/marathon/docs/native-docker.html#mesos-containerizer-and-universal-container-runtime for a valid example on how to run a Docker image with the Mesos UCR.
{
"id": "mesos-docker",
"container": {
"docker": {
"image": "mesosphere/inky"
},
"type": "MESOS"
},
"args": ["hello"],
"cpus": 0.2,
"mem": 16.0,
"instances": 1
}
You'll need Marathon >= 1.3.0 and Mesos >= 1.0 for that.
I have setup Marathon and Mesos on two of my machines.
I can successfully schedule commands from the marathon web console, but when I try to schedule a job involving docker images I immediately get job failed. Plus I get no stderr or stdout files.
Example Running a normal command:
Marathon job conf:
{
"id": "testecho",
"cmd": "echo hello; sleep 10",
"cpus": 1,
"mem": 128,
"disk": 0,
"instances": 1
}
On mesos I see that the tasks have succeeded. I have the stderr and stdout files like normal.
But now if I run a simple docker image task:
Marathon job conf:
{
"id": "/ubuntu",
"cmd": "date -u +%T",
"cpus": 0.5,
"mem": 512,
"disk": 0,
"instances": 1,
"container": {
"type": "DOCKER",
"volumes": [],
"docker": {
"image": "libmesos/ubuntu",
"network": null,
"portMappings": null,
"privileged": false,
"parameters": [],
"forcePullImage": false
}
},
"portDefinitions": [
{
"port": 10001,
"protocol": "tcp",
"labels": {}
}
]
}
On mesos, I see that it has instantly failed:
And I have no stderr or stdout files:
I also notice that on both my machines, when I run:
docker ps -a
I see nothing on both the machines. So that would mean that the docker jobs were not even launched
What could be affecting docker deployment?
The one reason I can think of is that the user that marathon uses to launch tasks not have access to docker? How do I test this?
I noticed that when I run the command:
sudo cat /etc/passwd
I see a user zookeeper. Maybe this is the user that doesn't have access to docker?
But when i do:
su zookeeper
I don't change user profiles
After going through a few tutorials I found the answer from the following tutorial: http://frankhinek.com/deploy-docker-containers-on-mesos-0-20/
I had to enable Docker Containerizer on my mesos-slaves
Set the --containerizers=docker,mesos" command line parameter:
echo "docker,mesos" | sudo tee /etc/mesos-slave/containerizers
Increase the executor timeout to 5 minutes1: (i guess this is optional)
echo "5mins" | sudo tee /etc/mesos-slave/executor_registration_timeout
Restart the Mesos Slave:
sudo service mesos-slave restart
Hi I have been successfull so far with simple jobs in marathon but it stuck when i have tried deploying a deocker job in mesos through marathon framework.
I am using a json file as below to deploy a docker job:
{
"id": "pga-docker",
"cpus": 0.2,
"mem": 1024.0,
"instances": 1,
"container": {
"type": "DOCKER",
"docker": {
"image": "pga",
"network": "BRIDGE",
"portMappings": [
{ "containerPort": 80, "hostPort": 6565, "servicePort": 0, "protocol": "tcp" }
]
}
}
}
My pga docker image have no problem when run as container, but through marathon its just not working. Its staying in the deploying state forever.
I am using the below command line:
curl -X POST http://10.141.141.10:8080/v2/apps -d #basic-3.json -H "Content-type: application/json"
But when I run the same image from marathon UI, its working. To run from marathon I used "docker run --publish 6060:80 --name test --rm pga" in the cmd field of the UI new job page.
Any one have idea why this is hanged in the command line approach?
This is what i have found during some trial and error with the json file.
I found that when we run docker image in local system, if we have mentioned an entry point or a cmd then that will execute while running the container. But this is not same for mesos/marathon. my observation is that if I explicitly mentioned cmd in the deployment json then its working fine.
"cmd":"sh pga-setup.sh"
I will love to know if anyone faced a similar issue an solved it by another way.
I have a private Docker registry that is accessible at https://docker.somedomain.com (over standard port 443 not 5000). My infrastructure includes a set up of Mesosphere, which have docker containerizer enabled. I'm am trying to deploy a specific container to a Mesos slave via Marathon; however, this always fails with Mesos failing the task almost immediately with no data in stderr and stdout of that sandbox.
I tried deploying from an image from the standard Docker Registry and it appears to work fine. I'm having trouble figuring out what is wrong. My private Docker registry does not require password authentication (turned off for debugging this), AND if I shell into the Meso's slave instance, and sudo su as root, I can run a 'docker pull docker.somedomain.com/services/myapp' successfully every time.
Here is my Marathon post data for starting the task:
{
"id": "myapp",
"cpus": 0.5,
"mem": 64.0,
"instances": 1,
"container": {
"type": "DOCKER",
"docker": {
"image": "docker.somedomain.com/services/myapp:2",
"network": "BRIDGE",
"portMappings": [
{ "containerPort": 7000, "hostPort": 0, "servicePort": 0, "protocol": "tcp" }
]
},
"volumes": [
{
"containerPath": "application.yml",
"hostPath": "/var/myapp/application.yml",
"mode": "RO"
}
]
},
"healthChecks": [
{
"protocol": "HTTP",
"portIndex": 0,
"path": "/",
"gracePeriodSeconds": 5,
"intervalSeconds": 20,
"maxConsecutiveFailures": 3
}
]
}
I've been stuck on this for almost a day now, everything I've tried seems to be yielding the same result. Any insights on this would be much appreciated.
My versions:
Mesos: 0.22.1
Marathon: 0.8.2
Docker: 1.6.2
So this turns out to be an issue with volumes
"volumes": [
{
"containerPath": "/application.yml",
"hostPath": "/var/myapp/application.yml",
"mode": "RO"
}
]
Using the root path of the container of the root path may be legal in docker, but Mesos appears not to handle this behavior. Modifying the containerPath to a non-root path resolves this, i.e
"volumes": [
{
"containerPath": "/var",
"hostPath": "/var/myapp",
"mode": "RW"
}
]
If it is a problem between Marathon and the registry, the answer should be in the http logs of your registry. If Marathon connects, there will be an entry. And the Mesos master log should contain a clue as well.
It doesn't really sound like a problem between Marathon and Registry though. Are you sure you have 'docker,mesos' in /etc/mesos-slave/containerizers?
Did you --despite having no authentification-- try to follow Using a Private Docker Repository?
To supply credentials to pull from a private repository, add a .dockercfg to the uris field of your app. The $HOME environment variable will then be set to the same value as $MESOS_SANDBOX so Docker can automatically pick up the config file.