Is is possible to run docker-in-docker on AWS batch?
I have tried the approach of mounting the docker socket via the container properties:
container_properties = <<CONTAINER_PROPERTIES
{
"command": ["docker", "run", "my container"],
"image": "docker/compose",
"jobRoleArn": "my-role",
"memory": 2000,
"vcpus": 1,
"privileged": true,
"mountPoints": [
{
"sourceVolume": "/var/run/docker.sock",
"containerPath": "/var/run/docker.sock",
"readOnly": false
}
]
}
However running this batch job in a SPOT compute environment with default configuration yields a job that immediately transitions to FAILED status with the status transition reason:
Status reason
Unknown volume '/var/run/docker.sock'.
The solution is that both volumes and mountPoints must be defined. For example the following container properties work:
{
"command": ["docker", "run", "<my container>"],
"image": "docker/compose",
"jobRoleArn": "my-role",
"memory": 2000,
"vcpus": 1,
"privileged": false,
"volumes": [
{
"host": {
"sourcePath": "/var/run/docker.sock"
},
"name": "dockersock"
}
],
"mountPoints": [
{
"sourceVolume": "dockersock",
"containerPath": "/var/run/docker.sock",
"readOnly": false
}
]
Access to your private ECR images works fine from the inner docker, however the authentication to ECR from the outer docker does not carry over, so you need to reauthenticate with
aws ecr get-login-password \
--region <region> \
| docker login \
--username AWS \
--password-stdin <aws_account_id>.dkr.ecr.<region>.amazonaws.com
before you run your privately hosted docker container.
Turns out privileged is not even required which is nice.
Related
In an existing swarm, I created a service via a docker-compose yaml file using the 'docker stack' command.
When I check the service via 'docker service ls' command, the new service shows up on the list. it shows "0/1" in the REPLICAS column
When I check the service using the command below, it shows 'Running' as the Desired State
docker service ps --no-trunc (service id)
When I check if there is already a corresponding container for the service, I can see none
When I try to access the service via the browser, it seems to be not started.
What is difficult is I cannot see any logs to find the cause of why this is happening
docker service logs (service id)
I figured it may just be slow to start but I waited for about half an hour and it was still in that state. Not sure how can I find out the cause of this without any logs. Can anyone help me on this?
EDIT: Below is the result when I did a docker inspect of the service task
[
{
"ID": "wt2tdoz64j5wmci4gr3q3io2e",
"Version": {
"Index": 3407514
},
"CreatedAt": "2020-08-25T00:58:13.012900717Z",
"UpdatedAt": "2020-08-25T00:58:13.012900717Z",
"Labels": {},
"Spec": {
"ContainerSpec": {
"Image": "my-ui-image:1.8.006",
"Labels": {
"com.docker.stack.namespace": "myservice-stack"
},
"Env": [
"BACKEND_HOSTNAME=somewebsite.com",
"BACKEND_PORT=3421"
],
"Privileges": {
"CredentialSpec": null,
"SELinuxContext": null
},
"Hosts": [
"10.152.30.18 somewebsite.com"
],
"Isolation": "default"
},
"Resources": {},
"Placement": {},
"Networks": [
{
"Target": "lt87emwtgbeztof5k2r1z2v27",
"Aliases": [
"myui_poc2"
]
}
],
"ForceUpdate": 0
},
"ServiceID": "nbskoeofakkgxlgj3utgn45c5",
"Slot": 1,
"Status": {
"Timestamp": "2020-08-25T00:58:13.012883476Z",
"State": "new",
"Message": "created",
"PortStatus": {}
},
"DesiredState": "running"
}
]
If you store your images in private registry then you must be logged in by command docker login and deploy your services by docker stack deploy -c docker-compose.yml your_service --with-registry-auth.
From the docker service ps ... output, you will see a column with the task id. You can get further details of the state of that task by inspecting the task id:
docker inspect $taskid
My guess is that your app is not redirecting it's output to stdout and that's why you don't get any output when doing "docker service logs...".
I would start by looking at this: https://docs.docker.com/config/containers/logging/
How you redirect the apps output to stdout will depend on what language your app is developed in.
I have the following packer config file:
{
"builders":[
{
"type": "docker",
"image": "ubuntu:18.04",
"commit": true
}
],
"post-processors": [
[
{
"type": "shell-local",
"inline": ["$(aws ecr get-login --no-include-email --region us-east-2)"]
},
{
"type": "docker-tag",
"repository": "localhost/my_image",
"tag": "latest"
},
{
"type": "docker-tag",
"repository": "123456789.dkr.ecr.us-east-2.amazonaws.com/my_image",
"tag": "latest"
},
"docker-push"
]
]
}
This gives me the following error
==> docker: Running post-processor: shell-local
==> docker (shell-local): Running local shell script: /var/folders/zh/wsr6wlx11v9703__rn7f3b080000gn/T/packer-shell756682313
==> docker (shell-local): WARNING! Using --password via the CLI is insecure. Use --password-stdin.
docker (shell-local): Login Succeeded
==> docker: Running post-processor: docker-tag
Build 'docker' errored: 1 error(s) occurred:
* Post-processor failed: Unknown artifact type:
Can only tag from Docker builder artifacts.
It works if I remove the shell-local post-processor.
It also doesn't matter what kind of command I execute in the shell-local post-processor.
I tried to add "keep_input_artifact": true to the shell-local post-processor but this did not change anything.
How can I execute a shell-local post-processor before a docker-tag / docker-push post-processor?
I figured it out. I have to put the shell-local post-processor in a separate list, i.e. I have to add another list to the list of post-processors, like so:
"post-processors": [
[
{
"type": "shell-local",
"inline": ["$(aws ecr get-login --no-include-email --region us-east-2)"]
}
],
[
{
"type": "docker-tag",
"repository": "localhost/my_image",
"tag": "latest"
},
{
"type": "docker-tag",
"repository": "123456789.dkr.ecr.us-east-2.amazonaws.com/my_image",
"tag": "latest"
},
"docker-push"
]
]
}
I'm running ECS tasks, and recently the service cpu hit 100% and went down.
I waited for the instance to settle down and sshed-in.
I was looking for logs, but it seemed docker container restarted and logs are all gone (logs when the cpu was high)
Next time, how do I make sure I can see the logs at least to diagnose the problem?
I have the following, hoping to see some logs somewhere (mounted in the host machine)
"mountPoints": [
{
"readOnly": null,
"containerPath": "/var/log/uwsgi",
"sourceVolume": "logs"
}
],
But there's no /var/log/uwsgi in the host machine.
And I probably need syslog and stuff..
As far you current configuration logs totally depend on the path that you define in the volume section.
"mountPoints": [
{
"readOnly": null,
"containerPath": "/var/log/uwsgi",
"sourceVolume": "logs"
}
],
the souces path defined in volume logs logs not /var/log/uwsgi, so you are mounting
/var/log/uwsgi (container path) -> logs volume (host path). you find these logs in path define in logs volume. but better to set something like
{
"readOnly": null,
"containerPath": "/var/log/uwsgi",
"sourceVolume": "volume_name"
}
then volume config
"volumes": [
{
"name": "logs",
"host": {
"sourcePath": "/home/ec2-user/logs"
}
}
]
From documentation
In the task definition volumes section, define a bind mount with name
and sourcePath values.
"volumes": [
{
"name": "webdata",
"host": {
"sourcePath": "/ecs/webdata"
}
}
]
In the containerDefinitions section, define a container with
mountPoints values that reference the name of the defined bind mount
and the containerPath value to mount the bind mount at on the
container.
"containerDefinitions": [
{
"name": "web",
"image": "nginx",
"cpu": 99,
"memory": 100,
"portMappings": [
{
"containerPort": 80,
"hostPort": 80
}
],
"essential": true,
"mountPoints": [
{
"sourceVolume": "webdata",
"containerPath": "/usr/share/nginx/html"
}
]
}
]
bind-mounts-ECS
Now if I come to my suggestion I will go for AWS log driver.
Working in AWS, the best approach is to push all logs to CW, but AWS log driver only pushes container stdout and stderr logs to CW.
Using AWS log driver you do not need to worry about instance and container, you will log in CW and you can stream these logs to ELK as well.
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "awslogs-wordpress",
"awslogs-region": "us-west-2",
"awslogs-stream-prefix": "awslogs-example"
}
}
using_awslogs
I'm new to DC/OS and I have been really struggling trying to deploy a POD. I have tried the simple examples provided in the documentation
but the deployments remain stuck in the deploying stage. There are plenty of resources available so that is not the issue.
I have 3 containers that I need to exist within a virtual network (queue, PDI, API). I have included my definition file that starts with a single container deployment and once I can successfully deploy I will add 2 additional containers to the definition. I have been looking at this example but have been unsuccessful.
I have successfully deployed the containers one at a time through Jenkins. All 3 images have been published and exist in the docker registry (Jfrog). I have included an example of my marathon.json for one of those successful deployments. I would appreciate any feedback that can help. The service is stuck in a deployed stage so I'm unable to drill down and see the logs via the command line or UI.
containers.image = pdi-queue
artifactory server = repos.pdi.com:5010/pdi-queue
1 Container POD Definition - (Error: Stuck in Deployment Stage)
{
"id":"/pdi-queue",
"containers":[
{
"name":"simple-docker",
"resources":{
"cpus":1,
"mem":128,
"disk":0,
"gpus":0
},
"image":{
"kind":"DOCKER",
"id":"repos.pdi.com:5010/pdi-queue",
"portMappings":[
{
"hostPort": 0,
"containerPort": 15672,
"protocol": "tcp",
"servicePort": 15672
}
]
},
"endpoints":[
{
"name":"web",
"containerPort":80,
"protocol":[
"http"
]
}
],
"healthCheck":{
"http":{
"endpoint":"web",
"path":"/"
}
}
}
],
"networks":[
{
"mode":"container",
"name":"dcos"
}
]
}
Marathon.json - (No Error: Successful deployment)
{
"id": "/pdi-queue",
"backoffFactor": 1.15,
"backoffSeconds": 1,
"container": {
"portMappings": [
{"containerPort": 15672, "hostPort": 0, "protocol": "tcp", "servicePort": 15672, "name": "health"},
{"containerPort": 5672, "hostPort": 0, "protocol": "tcp", "servicePort": 5672, "name": "queue"}
],
"type": "DOCKER",
"volumes": [],
"docker": {
"image": "repos.pdi.com:5010/pdi-queue",
"forcePullImage": true,
"privileged": false,
"parameters": []
}
},
"cpus": 0.1,
"disk": 0,
"healthChecks": [
{
"gracePeriodSeconds": 300,
"intervalSeconds": 60,
"maxConsecutiveFailures": 3,
"portIndex": 0,
"timeoutSeconds": 20,
"delaySeconds": 15,
"protocol": "MESOS_HTTP",
"path": "/"
}
],
"instances": 1,
"maxLaunchDelaySeconds": 3600,
"mem": 512,
"gpus": 0,
"networks": [
{
"mode": "container/bridge"
}
],
"requirePorts": false,
"upgradeStrategy": {
"maximumOverCapacity": 1,
"minimumHealthCapacity": 1
},
"killSelection": "YOUNGEST_FIRST",
"unreachableStrategy": {
"inactiveAfterSeconds": 300,
"expungeAfterSeconds": 600
},
"fetch": [],
"constraints": [],
"labels": {
"traefik.frontend.redirect.entryPoint": "https",
"traefik.frontend.redirect.permanent": "true",
"traefik.enable": "true"
}
}
I may not know the answer to the issues you are running into but I think I may be able to share some pointers to help debug this.
First of all, if you are unable to view logs from the DC/OS UI, you can also go to <cluster_url>/mesos and find the simple_docker task under Completed Tasks . It would show up as TASK_FAILED. Click on the Sandbox link on the right and then check stderr and stdout files for the task. There might be some clues there as to why it failed.
Another place to look can be to note the Agent IP from the Mesos UI where the task failed. SSH into the node and run sudo journalctl -u dcos-mesos-slave to see agent logs and try to find the logs corresponding to the failing task
One difference between the running the application as a Pod and a the App definition you shared is that your app definition is using DOCKER as the containerizer for the task while Pods use MESOS containerizer.
I noticed that you are using a private docker registry for your docker images. One possibility is that if your private registry's certificate is not trusted by Mesos but docker is configured already to trust it:
<copy the certificate(s) to /var/lib/dcos/pki/tls/certs>
cd /var/lib/dcos/pki/tls/certs
for file in *.crt; do ln -s \"$file\" \"$(openssl x509 -hash -noout -in \"$file\")\".0; done
This would need to be done on each agent node.
If its not a certificate issue, it could be docker registry credential issues. If the docker registry you are using requires authentication then you can specify docker credential at install time (assuming advanced install method) using : https://docs.mesosphere.com/1.11/installing/production/advanced-configuration/configuration-reference/#cluster-docker-credentials
I have a private Docker registry that is accessible at https://docker.somedomain.com (over standard port 443 not 5000). My infrastructure includes a set up of Mesosphere, which have docker containerizer enabled. I'm am trying to deploy a specific container to a Mesos slave via Marathon; however, this always fails with Mesos failing the task almost immediately with no data in stderr and stdout of that sandbox.
I tried deploying from an image from the standard Docker Registry and it appears to work fine. I'm having trouble figuring out what is wrong. My private Docker registry does not require password authentication (turned off for debugging this), AND if I shell into the Meso's slave instance, and sudo su as root, I can run a 'docker pull docker.somedomain.com/services/myapp' successfully every time.
Here is my Marathon post data for starting the task:
{
"id": "myapp",
"cpus": 0.5,
"mem": 64.0,
"instances": 1,
"container": {
"type": "DOCKER",
"docker": {
"image": "docker.somedomain.com/services/myapp:2",
"network": "BRIDGE",
"portMappings": [
{ "containerPort": 7000, "hostPort": 0, "servicePort": 0, "protocol": "tcp" }
]
},
"volumes": [
{
"containerPath": "application.yml",
"hostPath": "/var/myapp/application.yml",
"mode": "RO"
}
]
},
"healthChecks": [
{
"protocol": "HTTP",
"portIndex": 0,
"path": "/",
"gracePeriodSeconds": 5,
"intervalSeconds": 20,
"maxConsecutiveFailures": 3
}
]
}
I've been stuck on this for almost a day now, everything I've tried seems to be yielding the same result. Any insights on this would be much appreciated.
My versions:
Mesos: 0.22.1
Marathon: 0.8.2
Docker: 1.6.2
So this turns out to be an issue with volumes
"volumes": [
{
"containerPath": "/application.yml",
"hostPath": "/var/myapp/application.yml",
"mode": "RO"
}
]
Using the root path of the container of the root path may be legal in docker, but Mesos appears not to handle this behavior. Modifying the containerPath to a non-root path resolves this, i.e
"volumes": [
{
"containerPath": "/var",
"hostPath": "/var/myapp",
"mode": "RW"
}
]
If it is a problem between Marathon and the registry, the answer should be in the http logs of your registry. If Marathon connects, there will be an entry. And the Mesos master log should contain a clue as well.
It doesn't really sound like a problem between Marathon and Registry though. Are you sure you have 'docker,mesos' in /etc/mesos-slave/containerizers?
Did you --despite having no authentification-- try to follow Using a Private Docker Repository?
To supply credentials to pull from a private repository, add a .dockercfg to the uris field of your app. The $HOME environment variable will then be set to the same value as $MESOS_SANDBOX so Docker can automatically pick up the config file.