I tried pushing a docker image of Eclipse theia to cf, however unable to start it (or rather connect to it). The image exposes port 3000 with EXPOSE 3000. The app works and running it locally opens the default theia home screen
On CF, Sufficient disk and memory are given.
When the default port health check is set, cf hangs at starting app.
Creating app theia-docker...
Mapping routes...
Staging app and tracing logs...
Cell 15fcfa4a-a364-4dc2-ab6b-349f5196bd80 creating container for instance bd4b9e65-946f-485a-9de1-5c7fc8d4ad01
Cell 15fcfa4a-a364-4dc2-ab6b-349f5196bd80 successfully created container for instance bd4b9e65-946f-485a-9de1-5c7fc8d4ad01
Staging...
Staging process started ...
Staging process finished
Exit status 0
Staging Complete
Cell 15fcfa4a-a364-4dc2-ab6b-349f5196bd80 stopping instance bd4b9e65-946f-485a-9de1-5c7fc8d4ad01
Cell 15fcfa4a-a364-4dc2-ab6b-349f5196bd80 destroying container for instance bd4b9e65-946f-485a-9de1-5c7fc8d4ad01
Cell 15fcfa4a-a364-4dc2-ab6b-349f5196bd80 successfully destroyed container for instance bd4b9e65-946f-485a-9de1-5c7fc8d4ad01
It eventually comes to FAILED
cf logs would show:
2021-06-12T14:37:25.40+0530 [APP/PROC/WEB/0] OUT root INFO Deploy plugins list took: 161.7 ms
2021-06-12T14:38:24.77+0530 [HEALTH/0] ERR Failed to make TCP connection to port 2375: connection refused; Failed to make TCP connection to port 2376: connection refused
2021-06-12T14:38:24.77+0530 [CELL/0] ERR Failed after 1m0.303s: readiness health check never passed.
Why is it taking the wrong PORT number?
If I try setting the port in the env variable as cf set-env PORT 3000, I would get
FAILED
Server error, status code: 400, error code: 100001, message: The app is invalid: environment_variables cannot set PORT
I then set the health check is set to process. Of course, this would start successfully (failure or not). Checking the logs it can be seen that the app has started successfully. When I ssh into the app (cf ssh theia-docker) I am able to curl the application as localhost:3000 and returns the HTML of the homepage.
~ % cf ssh theia-docker
bash-5.0$ curl localhost:3000
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="apple-mobile-web-app-capable" content="yes">
<script type="text/javascript" src="./bundle.js" charset="utf-8"></script>
</head>
<body>
<div class="theia-preload"></div>
</body>
</html>bash-5.0$
However, when I try to connect to the app via the application URL I get the error:
502 Bad Gateway: Registered endpoint failed to handle the request.
The reason I see for this is that the base image I have used for this is based on docker:dind and it seems like in the base image ports 2375 and 2376 are exposed.
Why does CF pick the ports exposed in the base image rather than the one exposed in the docker image that is created? Shouldn't the port in the current image take precedence?
Changing the route mappings helped.
The following steps helped:
Get app guid
~ % cf app theia-docker --guid
8032eea6-d146-4d27-9b17-c7331852b59b
Add the required port
cf curl /v2/apps/8032eea6-d146-4d27-9b17-c7331852b59b -X PUT -d '{"ports": [3000]}'
{
"metadata": {
"guid": "8032eea6-d146-4d27-9b17-c7331852b59b",
"url": "/v2/apps/8032eea6-d146-4d27-9b17-c7331852b59b",
"created_at": "2021-06-12T09:04:51Z",
"updated_at": "2021-06-12T17:58:03Z"
},
"entity": {
"name": "theia-docker",
"production": false,
.
.
.
"ports": [
3000,
2375,
2376
],
.
.
.
Get the routes attached to the app
~ % cf curl /v2/apps/8032eea6-d146-4d27-9b17-c7331852b59b/routes
{
"total_results": 1,
"total_pages": 1,
"prev_url": null,
"next_url": null,
"resources": [
{
"metadata": {
"guid": "21f89763-baab-456d-8151-aad383a3c28f",
.
.
.
Use the route-guid to find route_mappings:
cf curl /v2/routes/21f89763-baab-456d-8151-aad383a3c28f/route_mappings
{
"total_results": 1,
"total_pages": 1,
"prev_url": null,
"next_url": null,
"resources": [
{
"metadata": {
"guid": "33bde252-ad3e-49b4-91df-78543ac452b4",
"url": "/v2/route_mappings/33bde252-ad3e-49b4-91df-78543ac452b4",
"created_at": "2021-06-12T09:04:51Z",
"updated_at": "2021-06-12T09:04:51Z"
},
"entity": {
"app_port": null,
"app_guid": "8032eea6-d146-4d27-9b17-c7331852b59b",
"route_guid": "21f89763-baab-456d-8151-aad383a3c28f",
"app_url": "/v2/apps/8032eea6-d146-4d27-9b17-c7331852b59b",
"route_url": "/v2/routes/21f89763-baab-456d-8151-aad383a3c28f"
}
}
]
}
Update the route-mapping using the app_guid, route_guid and app_port:
~% cf curl /v2/route_mappings -X POST -d '{"app_guid":"8032eea6-d146-4d27-9b17-c7331852b59b","route_guid":"21f89763-baab-456d-8151-aad383a3c28f", "app_port":3000}'
{
"metadata": {
"guid": "a62a2ea6-859f-48cc-aa33-a8d6583081da",
"url": "/v2/route_mappings/a62a2ea6-859f-48cc-aa33-a8d6583081da",
"created_at": "2021-06-12T18:02:19Z",
"updated_at": "2021-06-12T18:02:19Z"
},
"entity": {
"app_port": 3000,
"app_guid": "8032eea6-d146-4d27-9b17-c7331852b59b",
"route_guid": "21f89763-baab-456d-8151-aad383a3c28f",
"app_url": "/v2/apps/8032eea6-d146-4d27-9b17-c7331852b59b",
"route_url": "/v2/routes/21f89763-baab-456d-8151-aad383a3c28f"
}
}
List the route mappings again:
~ % cf curl /v2/routes/21f89763-baab-456d-8151-aad383a3c28f/route_mappings
{
"total_results": 2,
"total_pages": 1,
"prev_url": null,
"next_url": null,
"resources": [
{
"metadata": {
"guid": "33bde252-ad3e-49b4-91df-78543ac452b4",
"url": "/v2/route_mappings/33bde252-ad3e-49b4-91df-78543ac452b4",
"created_at": "2021-06-12T09:04:51Z",
"updated_at": "2021-06-12T09:04:51Z"
},
"entity": {
"app_port": null,
"app_guid": "8032eea6-d146-4d27-9b17-c7331852b59b",
"route_guid": "21f89763-baab-456d-8151-aad383a3c28f",
"app_url": "/v2/apps/8032eea6-d146-4d27-9b17-c7331852b59b",
"route_url": "/v2/routes/21f89763-baab-456d-8151-aad383a3c28f"
}
},
{
"metadata": {
"guid": "a62a2ea6-859f-48cc-aa33-a8d6583081da",
"url": "/v2/route_mappings/a62a2ea6-859f-48cc-aa33-a8d6583081da",
"created_at": "2021-06-12T18:02:19Z",
"updated_at": "2021-06-12T18:02:19Z"
},
"entity": {
"app_port": 3000,
"app_guid": "8032eea6-d146-4d27-9b17-c7331852b59b",
"route_guid": "21f89763-baab-456d-8151-aad383a3c28f",
"app_url": "/v2/apps/8032eea6-d146-4d27-9b17-c7331852b59b",
"route_url": "/v2/routes/21f89763-baab-456d-8151-aad383a3c28f"
}
}
]
}
You will find the new route mapping that was created. Delete the unwanted one.
~ % cf curl /v2/route_mappings/33bde252-ad3e-49b4-91df-78543ac452b4 -X DELETE
That's about it. Any better solutions are welcome :). (Not involving to maintain the Dockerfile of the base image of course)
Related
I just started to use FIWARE. I downloaded the latest version on the website (v2) using docker-compose on a PopOs distro.
I'm using Postman to make requests (create the entities and subscriptions) and a Laravel application to listen the notification from the FIWARE subscriptions. But for some reason, today, when I started the docker service and start to send requests: the FIWARE notifications suddenly stopped to work.
When I access the subscriptions endpoint FIWARE returns:
"notification": {
"timesSent": 1,
"lastNotification": "2021-09-02T01:19:39.000Z",
"attrs": [],
"onlyChangedAttrs": false,
"attrsFormat": "keyValues",
"http": {
"url": "http://localhost:8000/api/notifications"
},
"lastFailure": "2021-09-02T01:19:39.000Z",
"lastFailureReason": "Couldn't connect to server"
}
FIWARE can't comunicate, but if I make a POST request using Postman for that endpoint (http://localhost:8000/api/notifications) it returns 200.
There's some aditional configuration between the FIWARE docker container and the local machine? Or I'm doing something wrong?
This is my entity:
// http://{{orion}}/v2/subscription
{
"id": "movie",
"type": "movie",
"name": {
"type": "text",
"value": "movie name"
},
"gender": {
"type": "text",
"value": "drama"
}
}
This is how I'm doing the subscription:
// http://{{orion}}/v2/subscriptions
{
"description": "Notify me about any movie of gender drama",
"subject": {
"entities": [{"idPattern": ".*","type": "movie"}],
"condition": {
"attrs": ["gender"],
"expression": {
"q": "gender==drama"
}
}
},
"notification": {
"http": {
"url": "http://127.0.0.1:8000/api/notifications"
}
}
}
If you are using Docker, then you need to consider what http://localhost:8000/api/notifications actually means. localhost will mean the localhost as experienced by the Orion container itself. Generally Orion listens on 1026 and there is nothing listening on 8000 within a dockerized Orion, therefore your subscription fails.
If you have another micro-service running within the same docker network and in a separate container you must use the hostname of that container (or an alias or defined IP) to describe the notification URL, not localhost.
So for example in the following tutorial where a subscription payload is displayed on screen:
curl -iX POST \
--url 'http://localhost:1026/v2/subscriptions' \
--header 'content-type: application/json' \
--data '{
"description": "Notify me of all product price changes",
"subject": {
"entities": [{"idPattern": ".*", "type": "Product"}],
"condition": {
"attrs": [ "price" ]
}
},
"notification": {
"http": {
"url": "http://tutorial:3000/subscription/price-change"
}
}
}'
refers to a container which is called tutorial within the docker network
tutorial:
image: fiware/tutorials.context-provider
hostname: tutorial
container_name: fiware-tutorial
depends_on:
- orion
networks:
default:
aliases:
- iot-sensors
- context-provider
expose:
- 3000
As it happens the tutorial container is also exposing its internal port 3000 to the localhost of the machine it is running on so it can be viewed by a user, but Orion can only access it via the hostname on the docker network.
In an existing swarm, I created a service via a docker-compose yaml file using the 'docker stack' command.
When I check the service via 'docker service ls' command, the new service shows up on the list. it shows "0/1" in the REPLICAS column
When I check the service using the command below, it shows 'Running' as the Desired State
docker service ps --no-trunc (service id)
When I check if there is already a corresponding container for the service, I can see none
When I try to access the service via the browser, it seems to be not started.
What is difficult is I cannot see any logs to find the cause of why this is happening
docker service logs (service id)
I figured it may just be slow to start but I waited for about half an hour and it was still in that state. Not sure how can I find out the cause of this without any logs. Can anyone help me on this?
EDIT: Below is the result when I did a docker inspect of the service task
[
{
"ID": "wt2tdoz64j5wmci4gr3q3io2e",
"Version": {
"Index": 3407514
},
"CreatedAt": "2020-08-25T00:58:13.012900717Z",
"UpdatedAt": "2020-08-25T00:58:13.012900717Z",
"Labels": {},
"Spec": {
"ContainerSpec": {
"Image": "my-ui-image:1.8.006",
"Labels": {
"com.docker.stack.namespace": "myservice-stack"
},
"Env": [
"BACKEND_HOSTNAME=somewebsite.com",
"BACKEND_PORT=3421"
],
"Privileges": {
"CredentialSpec": null,
"SELinuxContext": null
},
"Hosts": [
"10.152.30.18 somewebsite.com"
],
"Isolation": "default"
},
"Resources": {},
"Placement": {},
"Networks": [
{
"Target": "lt87emwtgbeztof5k2r1z2v27",
"Aliases": [
"myui_poc2"
]
}
],
"ForceUpdate": 0
},
"ServiceID": "nbskoeofakkgxlgj3utgn45c5",
"Slot": 1,
"Status": {
"Timestamp": "2020-08-25T00:58:13.012883476Z",
"State": "new",
"Message": "created",
"PortStatus": {}
},
"DesiredState": "running"
}
]
If you store your images in private registry then you must be logged in by command docker login and deploy your services by docker stack deploy -c docker-compose.yml your_service --with-registry-auth.
From the docker service ps ... output, you will see a column with the task id. You can get further details of the state of that task by inspecting the task id:
docker inspect $taskid
My guess is that your app is not redirecting it's output to stdout and that's why you don't get any output when doing "docker service logs...".
I would start by looking at this: https://docs.docker.com/config/containers/logging/
How you redirect the apps output to stdout will depend on what language your app is developed in.
I am currently posting a docker service with the MESOS + Marathon combination.
This means that the IP address of the docker is constantly changing.
For example, if you put mongodb on marathon, you would use the following code.
port can specify the port that is coming into the host. After a day, the service will automatically shut down and run and the IP will change.
So, when I was looking for a method called mesos dns, when I was studying the docker command, I learned how to find the ip of the service with the alias name by specifying the network alias in the docker.
I thought it would be easier to access without using mesos dns by using this method.
However, in marathon, docker service is executed in json format like below.
I was asked because I do not know how to specify the docker network alias option or the keyword or method.
{
"id": "mongodbTest",
"instances": 1,
"cpus": 2,
"mem": 2048.0,
"container": {
"type": "DOCKER",
"docker": {
"image": "mongo:latest",
"network": "BRIDGE",
"portMappings": [
{
"containerPort": 27017,
"hostPort": 0,
"servicePort": 0,
"protocol": "tcp"
}
]
},
"volumes": [
{
"containerPath": "/etc/mesos-mg",
"hostPath": "/var/data/mesos-mg",
"mode": "RW"
}
]
}
}
I'm new to DC/OS and I have been really struggling trying to deploy a POD. I have tried the simple examples provided in the documentation
but the deployments remain stuck in the deploying stage. There are plenty of resources available so that is not the issue.
I have 3 containers that I need to exist within a virtual network (queue, PDI, API). I have included my definition file that starts with a single container deployment and once I can successfully deploy I will add 2 additional containers to the definition. I have been looking at this example but have been unsuccessful.
I have successfully deployed the containers one at a time through Jenkins. All 3 images have been published and exist in the docker registry (Jfrog). I have included an example of my marathon.json for one of those successful deployments. I would appreciate any feedback that can help. The service is stuck in a deployed stage so I'm unable to drill down and see the logs via the command line or UI.
containers.image = pdi-queue
artifactory server = repos.pdi.com:5010/pdi-queue
1 Container POD Definition - (Error: Stuck in Deployment Stage)
{
"id":"/pdi-queue",
"containers":[
{
"name":"simple-docker",
"resources":{
"cpus":1,
"mem":128,
"disk":0,
"gpus":0
},
"image":{
"kind":"DOCKER",
"id":"repos.pdi.com:5010/pdi-queue",
"portMappings":[
{
"hostPort": 0,
"containerPort": 15672,
"protocol": "tcp",
"servicePort": 15672
}
]
},
"endpoints":[
{
"name":"web",
"containerPort":80,
"protocol":[
"http"
]
}
],
"healthCheck":{
"http":{
"endpoint":"web",
"path":"/"
}
}
}
],
"networks":[
{
"mode":"container",
"name":"dcos"
}
]
}
Marathon.json - (No Error: Successful deployment)
{
"id": "/pdi-queue",
"backoffFactor": 1.15,
"backoffSeconds": 1,
"container": {
"portMappings": [
{"containerPort": 15672, "hostPort": 0, "protocol": "tcp", "servicePort": 15672, "name": "health"},
{"containerPort": 5672, "hostPort": 0, "protocol": "tcp", "servicePort": 5672, "name": "queue"}
],
"type": "DOCKER",
"volumes": [],
"docker": {
"image": "repos.pdi.com:5010/pdi-queue",
"forcePullImage": true,
"privileged": false,
"parameters": []
}
},
"cpus": 0.1,
"disk": 0,
"healthChecks": [
{
"gracePeriodSeconds": 300,
"intervalSeconds": 60,
"maxConsecutiveFailures": 3,
"portIndex": 0,
"timeoutSeconds": 20,
"delaySeconds": 15,
"protocol": "MESOS_HTTP",
"path": "/"
}
],
"instances": 1,
"maxLaunchDelaySeconds": 3600,
"mem": 512,
"gpus": 0,
"networks": [
{
"mode": "container/bridge"
}
],
"requirePorts": false,
"upgradeStrategy": {
"maximumOverCapacity": 1,
"minimumHealthCapacity": 1
},
"killSelection": "YOUNGEST_FIRST",
"unreachableStrategy": {
"inactiveAfterSeconds": 300,
"expungeAfterSeconds": 600
},
"fetch": [],
"constraints": [],
"labels": {
"traefik.frontend.redirect.entryPoint": "https",
"traefik.frontend.redirect.permanent": "true",
"traefik.enable": "true"
}
}
I may not know the answer to the issues you are running into but I think I may be able to share some pointers to help debug this.
First of all, if you are unable to view logs from the DC/OS UI, you can also go to <cluster_url>/mesos and find the simple_docker task under Completed Tasks . It would show up as TASK_FAILED. Click on the Sandbox link on the right and then check stderr and stdout files for the task. There might be some clues there as to why it failed.
Another place to look can be to note the Agent IP from the Mesos UI where the task failed. SSH into the node and run sudo journalctl -u dcos-mesos-slave to see agent logs and try to find the logs corresponding to the failing task
One difference between the running the application as a Pod and a the App definition you shared is that your app definition is using DOCKER as the containerizer for the task while Pods use MESOS containerizer.
I noticed that you are using a private docker registry for your docker images. One possibility is that if your private registry's certificate is not trusted by Mesos but docker is configured already to trust it:
<copy the certificate(s) to /var/lib/dcos/pki/tls/certs>
cd /var/lib/dcos/pki/tls/certs
for file in *.crt; do ln -s \"$file\" \"$(openssl x509 -hash -noout -in \"$file\")\".0; done
This would need to be done on each agent node.
If its not a certificate issue, it could be docker registry credential issues. If the docker registry you are using requires authentication then you can specify docker credential at install time (assuming advanced install method) using : https://docs.mesosphere.com/1.11/installing/production/advanced-configuration/configuration-reference/#cluster-docker-credentials
I have a private Docker registry that is accessible at https://docker.somedomain.com (over standard port 443 not 5000). My infrastructure includes a set up of Mesosphere, which have docker containerizer enabled. I'm am trying to deploy a specific container to a Mesos slave via Marathon; however, this always fails with Mesos failing the task almost immediately with no data in stderr and stdout of that sandbox.
I tried deploying from an image from the standard Docker Registry and it appears to work fine. I'm having trouble figuring out what is wrong. My private Docker registry does not require password authentication (turned off for debugging this), AND if I shell into the Meso's slave instance, and sudo su as root, I can run a 'docker pull docker.somedomain.com/services/myapp' successfully every time.
Here is my Marathon post data for starting the task:
{
"id": "myapp",
"cpus": 0.5,
"mem": 64.0,
"instances": 1,
"container": {
"type": "DOCKER",
"docker": {
"image": "docker.somedomain.com/services/myapp:2",
"network": "BRIDGE",
"portMappings": [
{ "containerPort": 7000, "hostPort": 0, "servicePort": 0, "protocol": "tcp" }
]
},
"volumes": [
{
"containerPath": "application.yml",
"hostPath": "/var/myapp/application.yml",
"mode": "RO"
}
]
},
"healthChecks": [
{
"protocol": "HTTP",
"portIndex": 0,
"path": "/",
"gracePeriodSeconds": 5,
"intervalSeconds": 20,
"maxConsecutiveFailures": 3
}
]
}
I've been stuck on this for almost a day now, everything I've tried seems to be yielding the same result. Any insights on this would be much appreciated.
My versions:
Mesos: 0.22.1
Marathon: 0.8.2
Docker: 1.6.2
So this turns out to be an issue with volumes
"volumes": [
{
"containerPath": "/application.yml",
"hostPath": "/var/myapp/application.yml",
"mode": "RO"
}
]
Using the root path of the container of the root path may be legal in docker, but Mesos appears not to handle this behavior. Modifying the containerPath to a non-root path resolves this, i.e
"volumes": [
{
"containerPath": "/var",
"hostPath": "/var/myapp",
"mode": "RW"
}
]
If it is a problem between Marathon and the registry, the answer should be in the http logs of your registry. If Marathon connects, there will be an entry. And the Mesos master log should contain a clue as well.
It doesn't really sound like a problem between Marathon and Registry though. Are you sure you have 'docker,mesos' in /etc/mesos-slave/containerizers?
Did you --despite having no authentification-- try to follow Using a Private Docker Repository?
To supply credentials to pull from a private repository, add a .dockercfg to the uris field of your app. The $HOME environment variable will then be set to the same value as $MESOS_SANDBOX so Docker can automatically pick up the config file.