We've been exploring marathon to deploy into a docker cluster. In the application architecture, we have a postgresql database which the application server need to access.
In the development stage, we relied on fig to create links between dockers and then use the environment variables imposed by the docker to connect to destination (app server to postgresql)
Yet, in Marathon we could not find a similar approach, we tried to use dependencies but that did not work, below is our Marathon.json file
{
"id": "/project",
"groups": [
{
"id": "apps",
"apps": [
{
"id": "app",
"mem": 1024,
"env": {
"APP_HOME": "/var/lib/app",
"GIT_BRANCH": "release/2.0.0",
"SETTING_FILE": "development",
"BROKER_URL": "redis://redis_1:6379/0"
},
"dependencies": ["database", "caching", "messaging"],
"container": {
"type": "DOCKER",
"docker": {
"image": "xxx/aok:app"
}
},
"volumes": [
{
"containerPath": "/var/lib/app",
"hostPath": ".",
"mode": "RW"
}
]
},
{
"id": "celery",
"mem": 1024,
"env": {
"APP_HOME": "/var/lib/app",
"GIT_BRANCH": "release/2.0.0",
"SETTING_FILE": "development",
"BROKER_URL": "redis://redis_1:6379/0"
},
"container": {
"type": "DOCKER",
"docker": {
"image": "xxx/aok:celery"
}
},
"volumes": [
{
"containerPath": "/var/lib/app",
"hostPath": ".",
"mode": "RW"
}
]
},
{
"id": "celeryhb",
"mem": 1024,
"env": {
"APP_HOME": "/var/lib/app",
"GIT_BRANCH": "release/2.0.0",
"SETTING_FILE": "development",
"BROKER_URL": "redis://redis_1:6379/0"
},
"container": {
"type": "DOCKER",
"docker": {
"image": "xxx/aok:celeryhb"
}
},
"volumes": [
{
"containerPath": "/var/lib/app",
"hostPath": ".",
"mode": "RW"
}
]
}
]
},
{
"id": "database",
"apps": [
{
"id": "pg",
"mem": 1024,
"container": {
"type": "DOCKER",
"docker": {
"image": "mughrabi/aok:pg"
},
"volumes": [
{
"containerPath": "/var/lib/postgresql/data",
"hostPath": "/tmp/aok-postgres-data",
"mode": "RW"
}
]
}
}
]
},
{
"id": "caching",
"apps": [
{
"id": "redis",
"mem": 1024,
"container": {
"type": "DOCKER",
"docker": {
"image": "redis"
}
}
}
]
},
{
"id": "messaging",
"apps": [
{
"id": "rabbitmq",
"mem": 1024,
"container": {
"type": "DOCKER",
"docker": {
"image": "rabbitmq"
}
}
}
]
}
]
}
Can someone please advise?
Look into using something like Consul https://www.consul.io/ or etcd https://github.com/coreos/etcd
Related
I am trying to add below JAVA_OPTIONS in deployconfig in OpenshiftContainer but is throwing syntax error .Could anyone help me how to add parameters in OpenshiftContainer please
JAVA_OPTIONS
-Djavax.net.ssl.trustStore={KEYSTORE_PATH}/cacerts.ts,
-Djavax.net.ssl.trustStorePassword=changeit,
Djavax.net.ssl.keyStore=${KEYSTORE_PATH}/keystore.pkcs12-Djavax.net.ssl.keyStorePassword=${KEYSTORE_PASS}
-Djava.awt.headless=true,
deploymentConfig as json:
{
"apiVersion": "apps.openshift.io/v1",
"kind": "DeploymentConfig",
"metadata": {
"labels": {
"app": "${APP_NAME}"
},
"name": "${APP_NAME}"
},
"spec": {
"replicas": 1,
"selector": {
"app": "${APP_NAME}",
"deploymentconfig": "${APP_NAME}"
},
"strategy": null,
"template": {
"metadata": {
"labels": {
"app": "${APP_NAME}",
"deploymentconfig": "${APP_NAME}"
}
},
"spec": {
"containers": [
{
"env": [
{
"name": "SPRING_PROFILE",
"value": "migration"
},
{
"name": "JAVA_MAIN_CLASS",
"value": "com.agcs.Application"
},
{
"name": "JAVA_OPTIONS",
"value":"-Djavax.net.ssl.trustStore={KEYSTORE_PATH}/cacerts.ts",
"-Djavax.net.ssl.trustStorePassword=changeit",
-Djavax.net.ssl.keyStore=${KEYSTORE_PATH}/keystore.pkcs12
-Djavax.net.ssl.keyStorePassword=${KEYSTORE_PASS}
-Djava.awt.headless=true,
},
{
"name": "MONGO_AUTH_DB",
"valueFrom": {
"secretKeyRef": {
"key": "spring.data.mongodb.authentication-database",
"name": "mongodb-secret"
}
}
},
],
"image": "${IMAGE_NAME}",
"imagePullPolicy": "Always",
"name": "${APP_NAME}",
"ports": [
{
"containerPort": 8103,
"protocol": "TCP"
}
],
"resources": {
"limits": {
"cpu": "500m",
"memory": "1Gi"
},
"requests": {
"cpu": "500m",
"memory": "500Mi"
}
},
"volumeMounts":[
{
"name": "secret-volume",
"mountPath": "/mnt/secrets",
"readOnly": true
}
]
}
],
"volumes": [
{
"name": "secret-volume",
"secret": {
"secretName": "keystore-new"
}
}
]
}
}
}
}
{
"name": "JAVA_OPTIONS",
"value":"-Djavax.net.ssl.trustStore={KEYSTORE_PATH}/cacerts.ts",
"-Djavax.net.ssl.trustStorePassword=changeit",
-Djavax.net.ssl.keyStore=${KEYSTORE_PATH}/keystore.pkcs12
-Djavax.net.ssl.keyStorePassword=${KEYSTORE_PASS}
-Djava.awt.headless=true,
},
This is invalid json, as the key value can only have one value, while you have provided multiple comma separated strings.
JAVA_OPTIONS isn't a standard environment variable, so we don't know how it's processed but maybe this will work?
{
"name": "JAVA_OPTIONS",
"value":"-Djavax.net.ssl.trustStore={KEYSTORE_PATH}/cacerts.ts -Djavax.net.ssl.trustStorePassword=changeit -Djavax.net.ssl.keyStore=${KEYSTORE_PATH}/keystore.pkcs12 -Djavax.net.ssl.keyStorePassword=${KEYSTORE_PASS} -Djava.awt.headless=true"
},
But there's still probably an issue, because it seems like {KEYSTORE_PATH} is supposed to be a variable. That's not defined or expanded in this file. For a first attempt, probably just hardcode the values of all these variables.
For secrets (such as passwords) you can hardcode some value for initial testing, but please use OpenShift Secrets for formal testing and the actual deployment.
I'm building a tool that can be used for Module deployment to IoTEdge based on a configured set or properties in a CMDB.
The tool is able to generate a Deployment manifest and is using the .NET SDK - specifically - RegistryManager.ApplyConfigurationContentOnDeviceAsync to apply the deployment.
The serialized string from ConfigurationContent is as follows. When this template is applied, there is no error in the SDK, however in the portal for the Edge it shows the status as 400 -- The deployment configuration is malformed or invalid.
This same template (just the ModulesContent) when applied using Single Device deployment in VSCode, works fine. So, not sure what is causing the template deployment to fail through RegistryManager. Unable to figure out the malformed content in the template. Appreciate any guidance.
{
"ModulesContent": {
"$edgeAgent": {
"properties.desired": {
"schemaVersion": "1.0",
"runtime": {
"type": "docker",
"settings": {
"minDockerVersion": "v1.25",
"loggingOptions": "",
"registryCredentials": {
"paddycontainers": {
"username": "XXXXXXcontainers",
"password": "EdgeHUBT9QuW=x2v3Z37jAZUREIOTg11uoT9Y",
"address": "XXXXXXcontainers.azurecr.io"
}
}
}
},
"systemModules": {
"edgeAgent": {
"type": "docker",
"settings": {
"image": "mcr.microsoft.com/azureiotedge-agent:1.0.9.2",
"createOptions": "{}"
}
},
"edgeHub": {
"type": "docker",
"status": "running",
"restartPolicy": "always",
"settings": {
"image": "mcr.microsoft.com/azureiotedge-hub:1.0.9.2",
"createOptions": "{\"HostConfig\":{\"ExposedPorts\":{\"9600/tcp\":{},\"5671/tcp\":{},\"8883/tcp\":{}}}}"
},
"env": {
"experimentalfeatures__enabled": {
"value": true
},
"experimentalfeatures__enableMetrics": {
"value": true
}
}
}
},
"modules": {
"PySendModule1": {
"version": "1.0",
"type": "docker",
"status": "running",
"restartPolicy": "always",
"settings": {
"image": "paddycontainers.azurecr.io/pysendmodule:0.0.1-amd64.debug"
}
},
"PySendModule2": {
"version": "1.0",
"type": "docker",
"status": "running",
"restartPolicy": "always",
"settings": {
"image": "paddycontainers.azurecr.io/pysendmodule:0.0.1-amd64.debug"
}
},
"PySendModule3": {
"version": "1.0",
"type": "docker",
"status": "running",
"restartPolicy": "always",
"settings": {
"image": "paddycontainers.azurecr.io/pysendmodule:0.0.1-amd64.debug"
}
},
"PySendModule4": {
"version": "1.0",
"type": "docker",
"status": "running",
"restartPolicy": "always",
"settings": {
"image": "paddycontainers.azurecr.io/pysendmodule:0.0.1-amd64.debug"
}
},
"PySendModule5": {
"version": "1.0",
"type": "docker",
"status": "running",
"restartPolicy": "always",
"settings": {
"image": "paddycontainers.azurecr.io/pysendmodule:0.0.1-amd64.debug"
}
},
"PySendModule6": {
"version": "1.0",
"type": "docker",
"status": "running",
"restartPolicy": "always",
"settings": {
"image": "paddycontainers.azurecr.io/pysendmodule:0.0.1-amd64.debug"
}
},
"SimulatedTemperatureSensor1": {
"version": "1.0",
"type": "docker",
"status": "running",
"restartPolicy": "always",
"settings": {
"image": "mcr.microsoft.com/azureiotedge-simulated-temperature-sensor:1.0",
"createOptions": "{}"
}
},
"metricscollector1": {
"version": "1.0",
"type": "docker",
"status": "running",
"restartPolicy": "always",
"settings": {
"image": "veyalla/metricscollector:0.0.4-amd64",
"createOptions": "{}"
},
"env": {
"AzMonWorkspaceId": {
"value": "81nsaux72-0fec-8818-a258-71hshs7jas9"
},
"AzMonWorkspaceKey": {
"value": "81jsajswkjsdkahialkasdmacjklaja=="
}
}
}
}
}
},
"$edgeHub": {
"properties.desired": {
"schemaVersion": "1.0",
"routes": {},
"storeAndForwardConfiguration": {
"timeToLiveSecs": 7200
}
}
},
"PySendModule1": {
"properties.desired": {
"camera360": "172.1.1.1",
"fontSize": "10",
"message": 300,
"format": "Json",
"target": "AzureLogAnalytics",
"endpoints": {
"edgeHub": "http://edgeHub:9600/metrics"
}
}
},
"PySendModule2": {
"properties.desired": {
"camera360": "172.1.1.2",
"fontSize": "10",
"message": 300,
"format": "Json",
"target": "AzureLogAnalytics",
"endpoints": {
"edgeHub": "http://edgeHub:9600/metrics"
}
}
},
"PySendModule3": {
"properties.desired": {
"camera360": "172.1.1.3",
"fontSize": "10",
"message": 300,
"format": "Json",
"target": "AzureLogAnalytics",
"endpoints": {
"edgeHub": "http://edgeHub:9600/metrics"
}
}
},
"PySendModule4": {
"properties.desired": {
"camera360": "172.1.1.4",
"fontSize": "10",
"message": 300,
"format": "Json",
"target": "AzureLogAnalytics",
"endpoints": {
"edgeHub": "http://edgeHub:9600/metrics"
}
}
},
"PySendModule5": {
"properties.desired": {
"camera360": "172.1.1.5",
"fontSize": "10",
"message": 300,
"format": "Json",
"target": "AzureLogAnalytics",
"endpoints": {
"edgeHub": "http://edgeHub:9600/metrics"
}
}
},
"PySendModule6": {
"properties.desired": {
"camera360": "172.1.1.6",
"fontSize": "10",
"message": 300,
"format": "Json",
"target": "AzureLogAnalytics",
"endpoints": {
"edgeHub": "http://edgeHub:9600/metrics"
}
}
}
},
"DeviceContent": null
}
I don't exactly know the difference and why this works. I had a version of this tool being able to successfully deploy. I was using Newtonsoft for JSON parsing. I had switched to System.Text.Json in the CSharp SDK and that seems to have broken the JSON into a malformed state. I switched back to Newtonsoft and my deployments are good.
I'm deploying my app in azure container instance (Container Group).
I've 3 docker containers
web_api
redis
neo4j
I'm able to access the database by using localhost:7474 as hostname, but can't access redis by using localhost as the hostname.
This is the same problem I'm facing when I run containers locally using docker run command.
NOTE: I can't use docker-compose as my intention is to use ACI.
azuredeploy.json
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"containerGroupName": {
"type": "string",
"defaultValue": "devCG",
"metadata": {
"description": ""
}
}
},
"variables": {
"name_web": "web-api",
"image_web": "dev.azurecr.io/web-api:89",
"name_redis": "redis",
"image_redis": "redis:5.0.9",
"name_neo4j": "neo4j",
"image_neo4j": "neo4j:3.5.6"
},
"resources": [
{
"name": "[parameters('containerGroupName')]",
"type": "Microsoft.ContainerInstance/containerGroups",
"apiVersion": "2019-12-01",
"location": "[resourceGroup().location]",
"properties": {
"containers": [
{
"name": "[variables('name_web')]",
"properties": {
"image": "[variables('image_web')]",
"resources": {
"requests": {
"cpu": 1,
"memoryInGb": 0.5
}
},
"ports": [
{
"port": 80
},
{
"port": 8080
}
]
}
},
{
"name": "[variables('name_redis')]",
"properties": {
"image": "[variables('image_redis')]",
"resources": {
"requests": {
"cpu": 0.5,
"memoryInGb": 0.2
}
}
}
},
{
"name": "[variables('name_neo4j')]",
"properties": {
"image": "[variables('image_neo4j')]",
"resources": {
"requests": {
"cpu": 0.5,
"memoryInGb": 0.2
}
},
"ports": [
{
"port": 7474
}
]
}
}
],
"imageRegistryCredentials": [
{
"server": "dev.azurecr.io",
"username": "dev",
"password": "********************"
}
],
"restartPolicy": "Always",
"osType": "Linux",
"volumes": [
{
"name": "devfs",
"azureFile": {
"shareName": "dev",
"readOnly": "false",
"storageAccountName": "devfs",
"storageAccountKey": "*****************************"
}
}
],
"ipAddress": {
"type": "Public",
"ports": [
{
"protocol": "tcp",
"port": 80
}
],
"dnsNameLabel": "dev"
}
}
}
],
"outputs": {
"containerIPv4Address": {
"type": "string",
"value": "[reference(resourceId('Microsoft.ContainerInstance/containerGroups/', parameters('containerGroupName'))).ipAddress.ip]"
}
}
}
Accessing redis over local host requires special tweaks in the redis config.
Have a look at
https://github.com/docker-library/redis/issues/45
And
https://github.com/luin/ioredis/issues/763
The recommandation is to connect over the redis hostname, you can nap this local up to redis hostname
Trying to start up rabbitmq in K8s while attaching a configmap gives me the following error:
/usr/local/bin/docker-entrypoint.sh: line 367: rabbitmq-plugins: command not found
/usr/local/bin/docker-entrypoint.sh: line 405: exec: rabbitmq-server: not found
Exactly the same setup is working fine with docker-compose, so I am a bit lost. Using rabbitmq:3.8.3
Here is a snippet from my deployment:
"template": {
"metadata": {
"creationTimestamp": null,
"labels": {
"app": "rabbitmq"
}
},
"spec": {
"volumes": [
{
"name": "rabbitmq-configuration",
"configMap": {
"name": "rabbitmq-configuration",
"defaultMode": 420
}
}
],
"containers": [
{
"name": "rabbitmq",
"image": "rabbitmq:3.8.3",
"ports": [
{
"containerPort": 5672,
"protocol": "TCP"
}
],
"env": [
{
"name": "RABBITMQ_DEFAULT_USER",
"value": "guest"
},
{
"name": "RABBITMQ_DEFAULT_PASS",
"value": "guest"
},
{
"name": "RABBITMQ_ENABLED_PLUGINS_FILE",
"value": "/opt/enabled_plugins"
}
],
"resources": {},
"volumeMounts": [
{
"name": "rabbitmq-configuration",
"mountPath": "/opt/"
}
],
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"imagePullPolicy": "IfNotPresent"
}
],
"restartPolicy": "Always",
"terminationGracePeriodSeconds": 30,
"dnsPolicy": "ClusterFirst",
"securityContext": {},
"schedulerName": "default-scheduler"
}
},
And here is the configuration:
{
"kind": "ConfigMap",
"apiVersion": "v1",
"metadata": {
"name": "rabbitmq-configuration",
"namespace": "e360",
"selfLink": "/api/v1/namespaces/default/configmaps/rabbitmq-configuration",
"uid": "28071976-98f6-11ea-86b2-0244a03303e1",
"resourceVersion": "1034540",
"creationTimestamp": "2020-05-18T10:55:58Z"
},
"data": {
"enabled_plugins": "[rabbitmq_management].\n"
}
}
That's because you're monting a volume in /opt, which is the rabbitmq home path.
So, the entrypoint script cannot find any of the rabbitmq binaries.
You can see the rabbitmq Dockerfile here
I am trying to deploy my application by using Gitlab-CI through pushing the docker images on Azure container and from there deploying the images on azure kubernetes service. these all process is happening automatically through GitlabCI. but i'm facing challenge in deployment section. i can able to see the services, pods is running status also tiller is deployed on kubernetes but it is throwing the below error This is deployment YAML which i took from kubernetes
{
"kind": "Deployment",
"apiVersion": "extensions/v1beta1",
"metadata": {
"name": "review-37-in-cust-iosa7i",
"namespace": "XYZ",
"selfLink": "/apis/extensions/v1beta1/namespaces/XYZ/deployments/review-37-in-cust-iosa7i",
"uid": "9f5f7fff-9d65-11e9-8ceb-0e7a6fb80992",
"resourceVersion": "7143337",
"generation": 1,
"creationTimestamp": "2019-07-03T07:39:00Z",
"labels": {
"app": "review-37-in-cust-iosa7i",
"chart": "auto-deploy-app-0.2.9",
"heritage": "Tiller",
"release": "review-37-in-cust-iosa7i",
"tier": "web",
"track": "stable"
},
"annotations": {
"deployment.kubernetes.io/revision": "1"
}
},
"spec": {
"replicas": 1,
"selector": {
"matchLabels": {
"app": "review-37-in-cust-iosa7i",
"release": "review-37-in-cust-iosa7i",
"tier": "web",
"track": "stable"
}
},
"template": {
"metadata": {
"creationTimestamp": null,
"labels": {
"app": "review-37-in-cust-iosa7i",
"release": "review-37-in-cust-iosa7i",
"tier": "web",
"track": "stable"
},
"annotations": {
"checksum/application-secrets": ""
}
},
"spec": {
"containers": [
{
"name": "auto-deploy-app",
"image": "stratuscentcrdeve.azurecr.io/XYZ/dev/37-in-customer-group-customer-form-when-admin-opens-up-the-poli:65d2e2bc554242c584d5c6480e172690659ef98b",
"ports": [
{
"name": "web",
"containerPort": 5000,
"protocol": "TCP"
}
],
"env": [
{
"name": "DATABASE_URL",
"value": "postgres://user:testing-password#review-37-in-cust-iosa7i-postgres:5432/review-37-in-cust-iosa7i"
}
],
"resources": {},
"livenessProbe": {
"httpGet": {
"path": "/",
"port": 5000,
"scheme": "HTTP"
},
"initialDelaySeconds": 15,
"timeoutSeconds": 15,
"periodSeconds": 10,
"successThreshold": 1,
"failureThreshold": 3
},
"readinessProbe": {
"httpGet": {
"path": "/",
"port": 5000,
"scheme": "HTTP"
},
"initialDelaySeconds": 5,
"timeoutSeconds": 3,
"periodSeconds": 10,
"successThreshold": 1,
"failureThreshold": 3
},
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"imagePullPolicy": "IfNotPresent"
}
],
"restartPolicy": "Always",
"terminationGracePeriodSeconds": 30,
"dnsPolicy": "ClusterFirst",
"securityContext": {},
"imagePullSecrets": [
{
"name": "gitlab-registry"
}
],
"schedulerName": "default-scheduler"
}
},
"strategy": {
"type": "RollingUpdate",
"rollingUpdate": {
"maxUnavailable": 1,
"maxSurge": 1
}
},
"revisionHistoryLimit": 2147483647,
"progressDeadlineSeconds": 2147483647
},
"status": {
"observedGeneration": 1,
"replicas": 1,
"updatedReplicas": 1,
"unavailableReplicas": 1,
"conditions": [
{
"type": "Available",
"status": "True",
"lastUpdateTime": "2019-07-03T07:39:00Z",
"lastTransitionTime": "2019-07-03T07:39:00Z",
"reason": "MinimumReplicasAvailable",
"message": "Deployment has minimum availability."
}
]
}
}
Please comment if any additional info is required?