I have Mesos cluster consists of 3 CentOS6.5 machines.
ZooKeeper and Mesos-Master is running on one of the machines and Mesos-Slave is running on each machine.
Also, Marathon is running on master node.
Then, I am trying to run docker containers on Marathon, following this instruction by Mesosphere.
job.json is like as follows,
{
"container": {
"type": "DOCKER",
"docker": {
"image": "libmesos/ubuntu"
}
},
"id": "ubuntu",
"instances": 1,
"cpus": 0.5,
"mem": 512,
"uris": [],
"cmd": "date -u +%T"
}
Then I run following command,
curl -X POST -H "Accept: application/json" -H "Content-Type: application/json" <master-hostname>:8080/v2/apps -d#job.json
Then on Marathon Web UI, I can see the Docker container is "Deploying" status even after long time.
And on Mesos-Master Web UI, I can see the Task is in "STAGING" status after long time.
On Sandbox pane, I can see the stdout and the command seems to completed successfly. No problem.
stderr is like this,
I0416 19:19:49.254998 29178 exec.cpp:132] Version: 0.22.0
I0416 19:19:49.257824 29193 exec.cpp:206] Executor registered on slave 20150416-160950-109643786-5050-30728-S0
stdout is like this,
Registered executor on master-hostname
10:19:49
But I expect the container(TASK) to finish after completed the command.
Is it possible?
If possible, how to do that way?
Thank you.
The task will finish (you should be able to see in the Mesos completed tasks) but the container will be restarted by Marathon. Marathon is for long-running apps.
If you don't want your application to be running continuously, you should take a look at another framework like Chronos.
Marathon is for long running processes. Even if you remove the containers, marathon will try to restart these. One more thing that I observed is that Marathon tries to launch containers and continue to do that till you are left with no memory and CPU. When you are out of resources your task will go in stage state.
Related
i'm new to this docker service creation please help me if i'm wrong , i executed this command
docker service create --name "testApp" -p 8000:8000 testApp
Its taking too long time to execute (stuck at overall progress 0 out of 1 tasks) so that my shell is getting timeout .
anything wrong am i doing here??
You can inspect the service task(s) to find out what's happening. The following command will show all the tasks for your testApp service.
docker service ps testApp
The CURRENT STATE and ERROR columns might give you some information. You can get the full status details by inspecting a specific task using docker inspect $TASK_ID, where $TASK_ID is the ID of the one of the tasks from the service. The Status section will show information like this.
"Status": {
"Timestamp": "2018-07-28T10:45:03.4277984Z",
"State": "running",
"Message": "started",
"ContainerStatus": {
"ContainerID": "d85be58fc6f84e28ee42c578a65923237e3b60c1c82b18382c39d13a69b10b84",
"PID": 40335,
"ExitCode": 0
},
"PortStatus": {}
}
Usually, the information in there is enough to diagnose basic problems with creating a service.
I have a 1/1 master/slave setup with the slave having 8gb ram 8 cpus. I am trying to use marathon to deploy a docker container with 1gb mem and 1 cpu but it just hangs on waiting
I believe this is usually caused by marathon not getting the resources it wants for the task
when I look at my logs I see
Sending 1 offers to framework
8bb1a298-cc23-426e-ad43-d440a2a560c4-0000 (marathon) at
scheduler-d4a993b4-69ea-4ac3-9e98-b54afe1e790b#127.0.0.1:52016 I0127
23:07:37.396546 2471 master.cpp:3297] Processing DECLINE call for
offers: [ 5271fcb3-4d77-4b12-af85-d94fd9172514-O127 ] for framework
8bb1a298-cc23-426e-ad43-d440a2a560c4-0000 (marathon) at
scheduler-d4a993b4-69ea-4ac3-9e98-b54afe1e790b#127.0.0.1:52016 I0127
23:07:37.396917 2466 hierarchical.cpp:744] Recovered cpus():6;
mem():5968; disk():156020; ports():[31000-31056, 31058-32000]
(total: cpus():8; mem():6992; disk():156020;
ports():[31000-32000], allocated: cpus():2; mem():1024;
ports(*):[31057-31057]) on slave
8bb1a298-cc23-426e-ad43-d440a2a560c4-S0 from framework
8bb1a298-cc23-426e-ad43-d440a2a560c4-0000
so it looks like marathon is declining the offer it gets? the next line in the logs say that mesos is reclaiming the offered resources and what its reclaiming looks like its plenty for my task?
any ideas on how to trouble shoot this further?
edit: so got to dig into this a bit further and found the marathon logs.
Basically the deployment works if we do not enter any information for port mapping in the marathon docker section. The docker container deploys successfully and I can ping it successfully from its host but I cannot access it from elsewhere.
if we set the container port as 8081 (which is what the docker container exposes are its application listens on) we get further in the deployment process but the app within the container fails to build with error
Error: listen EADDRINUSE :::8081
at Object.exports._errnoException (util.js:856:11)
at exports._exceptionWithHostPort (util.js:879:20)
at Server._listen2 (net.js:1234:14)
at listen (net.js:1270:10)
at Server.listen (net.js:1366:5)
at EventEmitter.listen (/usr/src/app/node_modules/express/lib/application.js:617:24)
at Object. (/usr/src/app/index.js:16:18)
at Module._compile (module.js:425:26)
at Object.Module._extensions..js (module.js:432:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:313:12)
at Function.Module.runMain (module.js:457:10)
at startup (node.js:138:18)
at node.js:974:3
So I think we are further along than we were but we are still having some port issues. I dont know why the container would build successfully on its own and with marathon with no port settings but not with marathon with port settings
There are few things to check:
On you slave: ps aux | grep sbin/mesos-slave should contain something like:
--containerizers=docker,mesos --executor_registration_timeout=5mins
Again on slave check that there's a Docker Daemon running:
ps aux | grep "docker daemon"
Make sure you've configured Docker network (in Marathon) as BRIDGE. With HOST mode you might get in collision with ports already used on host. This will allow mapping slave:32001 -> docker:8080.
...
"network": "BRIDGE",
"portMappings": [
{
"containerPort": 8080,
"hostPort": $PORT0,
"protocol": "tcp"
}
],
...
When the task starts in Marathon you'll see the app ID like myapp.a72db5b0-ca16-11e5-ba5f-fea9945fabaf. Use Mesos CLI (pip install mesos.cli mesos.interface) to fetch the logs. There's a command similar to Unix's tail for fetching stdout logs (-f follow logs):
mesos tail -f -i myapp.a72db5b0-ca16-11e5-ba5f-fea9945fabaf
and stderr:
mesos tail -f -i myapp.a72db5b0-ca16-11e5-ba5f-fea9945fabaf stderr
-i allows you to get logs from inactive tasks (in case that the task is crashing quickly). If you don't catch the ID in Marathon, use mesos ps -i.
In case that the task is not starting, there's either not enough resources or some problem with Marathon. Navigate your browser to http://{marathon URI:8080]/logging and increase verbosity for task allocation. Then check Marathon logs.
We are created a wordpress container using mesos-Marathon, we allocated 0.1 CPU and 64mb RAM.
When we check the docker stats, we observed that memory allocations we differed with what we are allocated in marathon,
Is there any way to update memory usage limit for Docker container, can we set up any default limits for all containers at demon level.(By Mesos / Docker demon level)
We try do load test on WordPress site, container got killed for just 500 connections, we try to do load test using JMeter.
Thanks in Advance
Docker doesn't have a memory option for your docker daemon yet. As to what is the default memory limit for containers you can only set limits at runtime (not after runtime) with the following options:
-m, --memory="" Memory limit
--memory-swap="" Total memory (memory + swap), '-1' to disable swap
As per this
I also see that there's still in issue open here. Make sure you are using Mesos (0.22.1) or later.
How about creating your containers with something like this Marathon request?
curl -X POST -H "Content-Type: application/json" http://<marathon-server>:8080/v2/apps -d#helloworld.json
helloworld.json:
{
"id": "helloworld",
"container": {
"docker": {
"image": "ubuntu:14.04"
},
"type": "DOCKER",
"volumes": []
},
"cmd": "while true; do echo hello world; sleep 1; done",
"cpus": 0.1,
"mem": 96.0, # Update the memory here.
"instances": 1
}
Hi I have been successfull so far with simple jobs in marathon but it stuck when i have tried deploying a deocker job in mesos through marathon framework.
I am using a json file as below to deploy a docker job:
{
"id": "pga-docker",
"cpus": 0.2,
"mem": 1024.0,
"instances": 1,
"container": {
"type": "DOCKER",
"docker": {
"image": "pga",
"network": "BRIDGE",
"portMappings": [
{ "containerPort": 80, "hostPort": 6565, "servicePort": 0, "protocol": "tcp" }
]
}
}
}
My pga docker image have no problem when run as container, but through marathon its just not working. Its staying in the deploying state forever.
I am using the below command line:
curl -X POST http://10.141.141.10:8080/v2/apps -d #basic-3.json -H "Content-type: application/json"
But when I run the same image from marathon UI, its working. To run from marathon I used "docker run --publish 6060:80 --name test --rm pga" in the cmd field of the UI new job page.
Any one have idea why this is hanged in the command line approach?
This is what i have found during some trial and error with the json file.
I found that when we run docker image in local system, if we have mentioned an entry point or a cmd then that will execute while running the container. But this is not same for mesos/marathon. my observation is that if I explicitly mentioned cmd in the deployment json then its working fine.
"cmd":"sh pga-setup.sh"
I will love to know if anyone faced a similar issue an solved it by another way.
when I run a docker container as a marathon job, it creates a docker container in the active mesos slave system. when suspend or destroy the docker job what I expect that marathon should delete the docker container as its no longer required. But the container does not get deleted. I have to delete them manually every time marathon restart a docker container job.
is there any way to delete these unwanted containers automatically?
Edit:
Adding json file for initiating a marathon job
{
"id": "pga-docker",
"cmd":"sh pga-setup.sh",
"cpus": 0.5,
"mem": 1024.0,
"container": {
"type": "DOCKER",
"docker": {
"image": "pga:test",
"parameters": [
{ "key": "env", "value": "SERVER_HOST=value" },
{ "key": "env", "value": "SERVER_PORT=value" }
],
"network": "BRIDGE",
"portMappings": [
{ "containerPort": 80, "hostPort": 0}
]
}
}
}
Marathon will restart a docker container which failed so that you have the number of instances you requested. It could be that you see stopped/failed containers which were not cleaned up by Mesos. This could be related to the fact that Mesos delays container cleanup until GC.
see https://issues.apache.org/jira/browse/MESOS-1656
It is the behavior of Marathon, because it is meant for long running services, as soon the task is completed, Marathon assumes it has been terminated in that host and immediately it will assign a new instance for running the application. If you need one of task you can use Chronos, so it makes the task to run only one time. I have written a script to do this automatically for marathon.
start=$1
end=$2
for (( c=$start; c<=$end; c++ ))
do
echo "deleting:$c"
sleep 10
var=$(curl -X GET http://localhost:8080/v2/apps/docker-app-$c | grep "startedAt")
echo "$var"
if [[ $var == *"startedAt"* ]]
then
curl -X DELETE http://localhost:8080/v2/apps/docker-app-$c
echo "going to delete"
else
echo "application not started yet"
fi
sleep 1
done
echo "Completed!"