Docker Swarm Windows Server 2019 host linux container with Hyper-V/LCOW - docker

I am experimenting with a Docker single-node Swarm on a Windows 2019 host with the Mirantis container with Hyper-V and LCOW and would like to run a alpine/linux container.
I've been able to deploy the linux container via the standard 'docker' command, but am not able to do it with Docker Swarm. When I try to create a linux based service the error no suitable node is given.
PS C:\Windows\system32>docker service create --replicas 1 --name helloworld alpine ping docker.com
overall progress: 0 out of 1 tasks
1/1: no suitable node (unsupported platform on 1 node)
Presumably this is because the reported capabilities of the node does not include linux even though running Linux containers works via the 'docker' command.
Is there a way to configure what capabilities a node possess?
PS C:\Windows\system32> docker info
Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
cluster: Manage Mirantis Container Cloud clusters (Mirantis Inc., v1.9.0)
registry: Manage Docker registries (Docker Inc., 0.1.0)
Server:
Containers: 2
Running: 1
Paused: 0
Stopped: 1
Images: 21
Server Version: 20.10.9
Storage Driver: windowsfilter (windows) lcow (linux)
Windows:
LCOW:
Logging Driver: etwlogs
Plugins:
Volume: local
Network: ics internal l2bridge l2tunnel nat null overlay private transparent
Log: awslogs etwlogs fluentd gcplogs gelf json-file local logentries splunk syslog
Swarm: active
NodeID: xxxxxx
Is Manager: true
ClusterID: xxxxx
Managers: 1
Nodes: 1
Default Address Pool: 10.0.0.0/8
SubnetSize: 24
Data Path Port: 4789
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: xxx
Manager Addresses:
xxx
Default Isolation: process
Kernel Version: 10.0 17763 (17763.1.amd64fre.rs5_release.180914-1434)
Operating System: Windows Server 2019 Standard Version 1809 (OS Build 17763.3046)
OSType: windows
Architecture: x86_64
CPUs: 4
Total Memory: 32GiB
Name: xxxxx
ID: xxxxx
Docker Root Dir: C:\ProgramData\docker
Debug Mode: false
Username: fazleskhan
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
The node info
PS C:\Windows\system32>docker node inspect px
[
{
"ID": "xxx",
"Version": {
"Index": 8
},
"CreatedAt": "2022-07-07T21:06:17.0196727Z",
"UpdatedAt": "2022-07-07T21:06:17.5509223Z",
"Spec": {
"Labels": {},
"Role": "manager",
"Availability": "active"
},
"Description": {
"Hostname": "xxx",
"Platform": {
"Architecture": "x86_64",
"OS": "windows"
},
"Resources": {
"NanoCPUs": 4000000000,
"MemoryBytes": 34358669312
},
"Engine": {
"EngineVersion": "20.10.9",
"Plugins": [
{
"Type": "Log",
"Name": "awslogs"
},
{
"Type": "Log",
"Name": "etwlogs"
},
{
"Type": "Log",
"Name": "fluentd"
},
{
"Type": "Log",
"Name": "gcplogs"
},
{
"Type": "Log",
"Name": "gelf"
},
{
"Type": "Log",
"Name": "json-file"
},
{
"Type": "Log",
"Name": "local"
},
{
"Type": "Log",
"Name": "logentries"
},
{
"Type": "Log",
"Name": "splunk"
},
{
"Type": "Log",
"Name": "syslog"
},
{
"Type": "Network",
"Name": "ics"
},
{
"Type": "Network",
"Name": "internal"
},
{
"Type": "Network",
"Name": "l2bridge"
},
{
"Type": "Network",
"Name": "l2tunnel"
},
{
"Type": "Network",
"Name": "nat"
},
{
"Type": "Network",
"Name": "null"
},
{
"Type": "Network",
"Name": "overlay"
},
{
"Type": "Network",
"Name": "private"
},
{
"Type": "Network",
"Name": "transparent"
},
{
"Type": "Volume",
"Name": "local"
}
]
},
"TLSInfo": { xxx }
},
"Status": {
"State": "ready",
"Addr": "xxx"
},
"ManagerStatus": {
"Leader": true,
"Reachability": "reachable",
"Addr": "xxx"
}
}
]
My notes for getting things running as a reference
https://github.com/fazleskhan/docker-deep-dive/blob/master/Intsalling%20DockerEE%20Windows%20Server%202019.md

Related

"Cloud Run error: Container failed to start. Failed to start and then listen on the port defined by the PORT environment variable."

I'm trying to build a container image that I will later use to update the code inside of a virtual machine. The docker image works fine as I can build and run it inside of my terminal. However, I keep getting an error when I try to deploy it to cloud run: "Cloud Run error: Container failed to start. Failed to start and then listen on the port defined by the PORT environment variable." How can I fix this error?
The build log contains this:
Deploying container to Cloud Run service [SERVICE] in project [PROJECT_ID] region [REGION]
Deploying...
Creating Revision.......................................................................................................................................................................failed
Deployment failed
ERROR: (gcloud.run.deploy) Cloud Run error: Container failed to start. Failed to start and then listen on the port defined by the PORT environment variable. Logs for this revision might contain more information.
The revision log contains this:
{
"protoPayload": {
"#type": "type.googleapis.com/google.cloud.audit.AuditLog",
"status": {
"code": 9,
"message": "Ready condition status changed to False for Revision {REVISION_NAME} with message: Cloud Run error: Container failed to start. Failed to start and then listen on the port defined by the PORT environment variable. Logs for this revision might contain more information.\n\nLogs URL:{URL_LINK}"
},
"serviceName": "run.googleapis.com",
"resourceName": "{REVISION_NAME}",
"response": {
"metadata": {
"name": "{REVISION_NAME}",
"namespace": "{NAMESPACE}",
"selfLink": "{SELFLINK}",
"uid": "{UID}",
"resourceVersion": "{RESOURCEVER}",
"generation": 1,
"creationTimestamp": "{TIMESTAMP}",
"labels": {
"serving.knative.dev/route": "{SERVICE}",
"serving.knative.dev/configuration": "{SERVICE}",
"serving.knative.dev/configurationGeneration": "15",
"serving.knative.dev/service": "{SERVICE}",
"serving.knative.dev/serviceUid": "{SERVICE_UID}",
"cloud.googleapis.com/location": "{REGION}"
},
"annotations": {
"run.googleapis.com/client-name": "gcloud",
"serving.knative.dev/creator": "{NAMESPACE}#cloudbuild.gserviceaccount.com",
"client.knative.dev/user-image": "gcr.io/{PROJECT_ID}/{IMAGE}",
"run.googleapis.com/client-version": "357.0.0",
"autoscaling.knative.dev/maxScale": "100"
},
"ownerReferences": [
{
"kind": "Configuration",
"name": "{SERVICE}",
"uid": "{UID}",
"apiVersion": "serving.knative.dev/v1",
"controller": true,
"blockOwnerDeletion": true
}
]
},
"apiVersion": "serving.knative.dev/v1",
"kind": "Revision",
"spec": {
"containerConcurrency": 80,
"timeoutSeconds": 300,
"serviceAccountName": "{NAMESPACE}-compute#developer.gserviceaccount.com",
"containers": [
{
"image": "gcr.io/{PROJECT_ID}/{IMAGE}",
"ports": [
{
"name": "h2c",
"containerPort": 8080
}
],
"resources": {
"limits": {
"cpu": "1000m",
"memory": "512Mi"
}
}
}
]
},
"status": {
"observedGeneration": 1,
"conditions": [
{
"type": "Ready",
"status": "False",
"reason": "HealthCheckContainerError",
"message": "Cloud Run error: Container failed to start. Failed to start and then listen on the port defined by the PORT environment variable. Logs for this revision might contain more information.\n\nLogs URL:{LOG_LINK}",
"lastTransitionTime": "{TIME}"
},
{
"type": "Active",
"status": "Unknown",
"reason": "Reserve",
"lastTransitionTime": "{TIME}",
"severity": "Info"
},
{
"type": "ContainerHealthy",
"status": "False",
"reason": "HealthCheckContainerError",
"message": "Cloud Run error: Container failed to start. Failed to start and then listen on the port defined by the PORT environment variable. Logs for this revision might contain more information.\n\nLogs URL:{LOG_LINK}",
"lastTransitionTime": "{TIME}"
},
{
"type": "ResourcesAvailable",
"status": "True",
"lastTransitionTime": "{TIME}"
},
{
"type": "Retry",
"status": "True",
"reason": "ImmediateRetry",
"message": "System will retry after 0:00:00 from lastTransitionTime for attempt 0.",
"lastTransitionTime": "{TIME}",
"severity": "Info"
}
],
"logUrl": "{LOG_LINK}",
"imageDigest": "gcr.io/{PROJECT_ID}/{IMAGE_SHA}"
},
"#type": "type.googleapis.com/google.cloud.run.v1.Revision"
}
},
"insertId": "{ID}",
"resource": {
"type": "cloud_run_revision",
"labels": {
"location": "{REGION}",
"configuration_name": "{SERVICE}",
"service_name": "{SERVICE}",
"project_id": "{PROJECT_ID}",
"revision_name": "{REVISION_NAME}"
}
},
"timestamp": "{TIME}",
"severity": "ERROR",
"logName": "projects/{PROJECT_ID}/logs/cloudaudit.googleapis.com%2Fsystem_event",
"receiveTimestamp": "{TIME}"
}
This is my cloudbuild.yaml:
steps:
- name: 'gcr.io/cloud-builders/docker'
args: ['build', '-t', 'gcr.io/PROJECT_ID/IMAGE', '.']
- name: 'gcr.io/cloud-builders/docker'
args: ['push', 'gcr.io/PROJECT_ID/IMAGE']
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
entrypoint: gcloud
args: ['run', 'deploy', 'SERVICE-NAME', '--image', 'gcr.io/PROJECT_ID/IMAGE', '--region', 'REGION', '--port', '8080']
images:
- gcr.io/PROJECT_ID/IMAGE
This is my Dockerfile:
FROM python:3.9.7-slim-buster
WORKDIR /app
COPY . .
CMD [ "python3", "hello.py" ]
This is the code in hello.py:
print("Hello World")
When Cloud Run starts your container, a health check is sent to the container. Your container is not responding to the health check. Therefore, Cloud Run determines that your service is failing.
Cloud Run requires that a container provide service/process/program that listens for and responds to HTTP requests.
Your hello.py file only prints a message to stdout. Your program does not start a process to listen for requests.
A very simple example that converts your example into a working program:
import os
from flask import Flask
app = Flask(__name__)
#app.route('/')
def home():
return "Hello world"
if __name__ == '__main__':
app.run(debug=True, host='0.0.0.0', port=int(os.environ.get('PORT', 8080)))
Note: You will need to add a file requirements.txt to your build to include Flask. Create requirements.txt in the same location as Dockerfile.
requirements.txt:
Flask==2.0.1

Overlay driver not being applied when creating docker swarm network

I'm working with docker swarm and trying to create a network with the overlay driver.
Whenever I create the network, the driver is not attached.
If I try and attach a service to the network, the process just hangs infinitely.
If I create a service without attaching it to the network, it works right away.
pi#node3:~ $ docker network ls
NETWORK ID NAME DRIVER SCOPE
a1cc2e1f4f2b bridge bridge local
83597f713bcf docker_gwbridge bridge local
277f1166485e host host local
fs2vvjeuejxc ingress overlay swarm
5d0ce08c744c none null local
pi#node3:~ $ docker network create --driver overlay test
4bfkahhkhrblod2t79yd83vws
pi#node3:~ $ docker network ls
NETWORK ID NAME DRIVER SCOPE
a1cc2e1f4f2b bridge bridge local
83597f713bcf docker_gwbridge bridge local
277f1166485e host host local
fs2vvjeuejxc ingress overlay swarm
5d0ce08c744c none null local
4bfkahhkhrbl test swarm
I can't figure out why it's not adding the driver. I have a suspicion it has something to do with the ingress network settings, but I'm stuck as for troubleshooting here.
Relevant Info
Swarm:
pi#node3:~ $ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
ygcte2diochpbgu7bqtw41k70 node1 Ready Active 20.10.7
xbllxgfa35937rmvdi8mi8dlb node2 Ready Active 20.10.7
tvw4b53w6g3qv2k3919dg3a81 * node3 Ready Active Leader 20.10.7
Manager node:
pi#node3:~ $ docker node inspect node3
[
{
"ID": "tvw4b53w6g3qv2k3919dg3a81",
"Version": {
"Index": 165
},
"CreatedAt": "2021-07-10T16:41:23.043334654Z",
"UpdatedAt": "2021-07-11T00:27:25.807737662Z",
"Spec": {
"Labels": {},
"Role": "manager",
"Availability": "active"
},
"Description": {
"Hostname": "node3",
"Platform": {
"Architecture": "armv7l",
"OS": "linux"
},
"Resources": {
"NanoCPUs": 4000000000,
"MemoryBytes": 969105408
},
"Engine": {
"EngineVersion": "20.10.7",
"Plugins": [
{
"Type": "Log",
"Name": "awslogs"
},
{
"Type": "Log",
"Name": "fluentd"
},
{
"Type": "Log",
"Name": "gcplogs"
},
{
"Type": "Log",
"Name": "gelf"
},
{
"Type": "Log",
"Name": "journald"
},
{
"Type": "Log",
"Name": "json-file"
},
{
"Type": "Log",
"Name": "local"
},
{
"Type": "Log",
"Name": "logentries"
},
{
"Type": "Log",
"Name": "splunk"
},
{
"Type": "Log",
"Name": "syslog"
},
{
"Type": "Network",
"Name": "bridge"
},
{
"Type": "Network",
"Name": "host"
},
{
"Type": "Network",
"Name": "ipvlan"
},
{
"Type": "Network",
"Name": "macvlan"
},
{
"Type": "Network",
"Name": "null"
},
{
"Type": "Network",
"Name": "overlay"
},
{
"Type": "Volume",
"Name": "local"
}
]
},
"TLSInfo": {
"TrustRoot": "-----BEGIN CERTIFICATE-----\nMIIBajCCARCgAwIBAgIUFIx3NAw+jgaasNXCoi+QP4GxaOQwCgYIKoZIzj0EAwIw\nEzERMA8GA1UEAxMIc3dhcm0tY2EwHhcNMjEwNzEwMTYyMjAwWhcNNDEwNzA1MTYy\nMjAwWjATMREwDwYDVQQDEwhzd2FybS1jYTBZMBMGByqGSM49AgEGCCqGSM49AwEH\nA0IABKyunnrZtfkOO+Cc/MX/qbyJjG12ee8es0IHB1HXF2MhqSfYOeUuBlTvuHuB\nxl8s8eQ4IMfjP0w5LYJNqypZp0KjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMB\nAf8EBTADAQH/MB0GA1UdDgQWBBRq6yBEIFv03tQqBkohCh4A+mIZdTAKBggqhkjO\nPQQDAgNIADBFAiA5kKgC2WxcOMyfrmFr8fU6w1Mo1mq5GMKA4owTB7pcEQIhALZi\n9AH0vVyR+7NmmR7LfPO65CIJ9UVuPZBXRZ6pcmzX\n-----END CERTIFICATE-----\n",
"CertIssuerSubject": "MBMxETAPBgNVBAMTCHN3YXJtLWNh",
"CertIssuerPublicKey": "MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAErK6eetm1+Q474Jz8xf+pvImMbXZ57x6zQgcHUdcXYyGpJ9g55S4GVO+4e4HGXyzx5Dggx+M/TDktgk2rKlmnQg=="
}
},
"Status": {
"State": "ready",
"Addr": "0.0.0.0"
},
"ManagerStatus": {
"Leader": true,
"Reachability": "reachable",
"Addr": "10.0.0.93:2377"
}
}
Ingress network:
pi#node3:~ $ docker network inspect ingress
[
{
"Name": "ingress",
"Id": "fs2vvjeuejxcjxqivenb76kgj",
"Created": "2021-07-10T17:24:14.228552858-07:00",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.10.0.0/24",
"Gateway": "10.10.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": true,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"ingress-sbox": {
"Name": "ingress-endpoint",
"EndpointID": "34003d042d395b90328ed90c8133505a6bec6df90065c5b47b47ee3853545c91",
"MacAddress": "02:42:0a:0a:00:02",
"IPv4Address": "10.10.0.2/24",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "4096"
},
"Labels": {},
"Peers": [
{
"Name": "e2f4d4e6ba20",
"IP": "10.0.0.93"
},
{
"Name": "de3d98ce0f8d",
"IP": "10.0.0.25"
},
{
"Name": "b61722e30756",
"IP": "10.0.0.12"
}
]
}
]
Docker version:
pi#node3:~ $ docker --version
Docker version 20.10.7, build f0df350
Docker info:
pi#node3:~ $ docker info
Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 5
Server Version: 20.10.7
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: active
NodeID: tvw4b53w6g3qv2k3919dg3a81
Is Manager: true
ClusterID: 4vf16jdlegf3ctys5k6wumcfc
Managers: 1
Nodes: 3
Default Address Pool: 10.10.0.0/24
SubnetSize: 24
Data Path Port: 4789
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 10.0.0.93
Manager Addresses:
10.0.0.93:2377
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: d71fcd7d8303cbf684402823e425e9dd2e99285d
runc version: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
init version: de40ad0
Security Options:
seccomp
Profile: default
Kernel Version: 5.10.17-v7+
Operating System: Raspbian GNU/Linux 10 (buster)
OSType: linux
Architecture: armv7l
CPUs: 4
Total Memory: 924.2MiB
Name: node3
ID: A67O:SIT4:QOMH:SILY:WHAY:KSGQ:VWMF:QVEJ:VCOZ:KW32:PZRV:ZD4B
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory TCP limit support
WARNING: No oom kill disable support
WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
What I've tried:
Removing all the nodes and creating a new swarm
Removing the ingress network and creating a new one following the instructions here
Tried to go through the walkthrough here but can't get past Create the Services 2.
Rebooted all the nodes
Any advice or pointing in the right direction would be much appreciated! I've been stuck here for 48 hours.
Solved!
The issue ended up being:
The nodes were all on 10.0.0.x
I set the --default-addr-pool 10.10.0.0/24 when initializing the swarm
Any network I tried to create using the --driver overlay would end up without any subnet or gateway information.
How I resolved the issue:
I was able to solve it using the --subnet flag when creating a custom network.
pi#node3:~ $ docker network create --driver overlay --subnet 10.10.10.0/24 test
pi#node3:~ $ docker network ls
NETWORK ID NAME DRIVER SCOPE
55ab64773261 bridge bridge local
ce1a0f497e9d docker_gwbridge bridge local
7c85cac72cf8 host host local
o7iew29j70nl ingress overlay swarm
ca5fc5682911 none null local
plezwc8zahpl test overlay swarm
pi#node3:~ $ docker network inspect test
[
{
"Name": "test",
"Id": "plezwc8zahpl9gs8hbv64bbo3",
"Created": "2021-07-16T17:30:28.773110478Z",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.10.10.0/24",
"Gateway": "10.10.10.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": null,
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "4097"
},
"Labels": null
}
]

Filebeat 7.10.1 add_docker_metadata adds only container.id

I'm using filebeat 7.10.1 installed on host system (not docker container), running as service by root
according to https://www.elastic.co/guide/en/beats/filebeat/current/add-docker-metadata.html
and https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-container.html
filebeat config, filebeat.yml:
filebeat.inputs:
- type: container
enabled: true
paths:
- '/var/lib/docker/containers/*/*.log'
processors:
- add_docker_metadata: ~
setup.template.settings:
index.number_of_shards: 1
#index.codec: best_compression
#_source.enabled: false
setup.kibana:
output.logstash:
hosts: ["<logstash_host>:5044"]
started container:
docker run --rm -d -l my-label --label com.example.foo=bar -p 80:80 nginx
filebeat get logs and successfully send them to endpoint (in my case to logstash, which resend to elasticsearch), but generated json by filebeat contains only container.id without container.name, container.labels and container.image
it looks like (copy-paste from kibana):
{
"_index": "logstash-2021.02.10",
"_type": "_doc",
"_id": "s4a4i3cB8j0XLXFVuyMm",
"_version": 1,
"_score": null,
"_source": {
"#version": "1",
"ecs": {
"version": "1.6.0"
},
"#timestamp": "2021-02-10T11:33:54.000Z",
"host": {
"name": "<some_host>"
},
"input": {
"type": "container"
},
"tags": [
"beats_input_codec_plain_applied"
],
"log": {
.....
},
"stream": "stdout",
"container": {
"id": "15facae2115ea57c9c99c13df815427669e21053791c7ddd4cd0c8caf1fbdf8c-json.log"
},
"agent": {
"version": "7.10.1",
"ephemeral_id": "adebf164-0b0d-450f-9a50-11138e519a27",
"id": "0925282e-319e-49e0-952e-dc06ba2e0c43",
"name": "<some_host>",
"type": "filebeat",
"hostname": "<some_host>"
}
},
"fields": {
"log.timestamp": [
"2021-02-10T11:33:54.000Z"
],
"#timestamp": [
"2021-02-10T11:33:54.000Z"
]
},
"highlight": {
"log.logger_name": [
"#kibana-highlighted-field#gw_nginx#/kibana-highlighted-field#"
]
},
"sort": [
1612956834000
]
}
what am I doing wrong? How to configure filebeat for send container.name, container.labels, container.image?
So after looking on filebeat-debug and paths on filesystem - issue closed
Reason: symlink /var/lib/docker -> /data/docker produces unexpected behavior
Solution:
filebeat.inputs:
- type: container
enabled: true
paths:
- '/data/docker/containers/*/*.log' #use realpath
processors:
- add_docker_metadata:
match_source_index: 3 #subfolder for extract container id from path

Does ECS task definition support volume mapping syntax?

docker-compose spec support volume mapping syntax under services, for example:
version: '2'
volumes:
jenkins_home:
external: true
services:
jenkins:
build:
context: .
args:
DOCKER_GID: ${DOCKER_GID}
DOCKER_VERSION: ${DOCKER_VERSION}
DOCKER_COMPOSE: ${DOCKER_COMPOSE}
volumes:
- jenkins_home:/var/jenkins_home
- /var/run/docker.sock:/var/run/docker.sock
ports:
- "8080:8080"
Following "AWSTemplateFormatVersion": "2010-09-09", the corresponding ECS task definition has volume syntax un-readable(with MountPoints and Volumes), as shown below:
"EcsTaskDefinition": {
"Type": "AWS::ECS::TaskDefinition",
"Properties": {
"ContainerDefinitions": [
{
"Name": "jenkins",
"Image": "xyzaccount/jenkins:ecs",
"Memory": 995,
"PortMappings": [ { "ContainerPort": 8080, "HostPort": 8080 } ],
"MountPoints": [
{
"SourceVolume": "docker",
"ContainerPath": "/var/run/docker.sock"
},
{
"SourceVolume": "jenkins_home",
"ContainerPath": "/var/jenkins_home"
}
]
}
],
"Volumes": [
{
"Name": "jenkins_home",
"Host": { "SourcePath": "/ecs/jenkins_home" }
},
{
"Name": "docker",
"Host": { "SourcePath": "/var/run/docker.sock" }
}
]
}
}
Does ECS task definition syntax of CloudFormation (now) support volume mapping syntax? similar to docker-compose....
Yes, of course, ECS support docker socket mounting, but the syntax is bit different. Add DOCKER_HOST environment variable in the task definition and source path should start with //.
"volumes": [
{
"name": "docker",
"host": {
"sourcePath": "//var/run/docker.sock"
}
}
]
The // worked in case of AWS ecs.
Also, you need to add DOCKER_HOST environment variable in your task definition.
"environment": [
{
"name": "DOCKER_HOST",
"value": "unix:///var/run/docker.sock"
}
]

How to set up Cassandra Docker cluster in Marathon with BRIDGE network?

I have a production DC/OS(v1.8.4) cluster and I am trying to setup a Cassandra cluster inside it. I use Marathon(v1.3.0) to deploy Cassandra nodes. I use the official Docker image of Cassandra and more specifically the 2.2.3 version.
First Case: Deploy Cassandra using HOST mode network - Everything OK
In this case, I first deploy a node that I call cassasndra-seed and it attaches to a physical host with IP 10.32.0.6. From the stdout log of Marathon for this service I can see that "Node /10.32.0.6 state jump to normal" and that listen_address and broadcast_address are set to 10.32.0.6. If I check the mesos-dns records using "_cassandra-seed._tcp.marathon.mesos SRV" in a master node I can see that the IP that resolves for this service is 10.32.0.6. The node is fully functional and I manage to create a test database.
{
"id": "/cassandra-seed",
"cpus": 1.5,
"mem": 8192,
"disk": 0,
"instances": 1,
"container": {
"type": "DOCKER",
"docker": {
"image": "cassandra:2.2.3",
"network": "HOST",
"ports": [7199,7000,7001,9160,9042],
"requirePorts": true,
"privileged": true
}
},
"constraints": [ ["hostname","UNIQUE"] ],
"env": { "CASSANDRA_CLUSTER_NAME": "democluster" }
}
Now I add one more node of cassandra using a separate deployment and providing 10.32.0.6 as seed (set "CASSANDRA_SEEDS": "10.32.0.6" in the env section of the deployment JSON). The new node gets the IP of another physical host (same pattern as before) and manages to gossip with the seed node. Thus, we have a functioning Cassandra cluster.
{
"id": "/cassandra",
"cpus": 1.5,
"mem": 8192,
"disk": 0,
"instances": 1,
"container": {
"type": "DOCKER",
"docker": {
"image": "cassandra:2.2.3",
"network": "HOST",
"ports": [7199,7000,7001,9160,9042],
"requirePorts": true,
"privileged": true
}
},
"constraints": [ ["hostname","UNIQUE"] ],
"env": {
"CASSANDRA_CLUSTER_NAME": "democluster",
"CASSANDRA_SEEDS": "10.32.0.6"
}
}
Second Case: Deploy Cassandra using BRIDGE mode network - Houston we have a problem
In this case, I also deploy a first cassandra-seed node and it attaches to a physical host with IP 10.32.0.6. However, now at the stdout log of the service in Marathon I can see that "Node /172.17.0.2 state jump to normal" and that listen_address and broadcast_address are set to 172.17.0.2. 172.17.0.2 is the IP of the docker container (found using docker inspect). However, if I check the mesos-dns records using "_cassandra-seed._tcp.marathon.mesos SRV" in a master node I can see that the IP that resolves for this service is 10.32.0.6. The node is fully functional and I manage to create a test database.
{
"id": "/cassandra-seed",
"cpus": 1.5,
"mem": 8192,
"disk": 0,
"instances": 1,
"container": {
"type": "DOCKER",
"docker": {
"image": "cassandra:2.2.3",
"network": "BRIDGE",
"portMappings": [
{"containerPort": 7000, "hostPort": 7000, "servicePort": 0 },
{"containerPort": 7001, "hostPort": 7001, "servicePort": 0 },
{"containerPort": 7199, "hostPort": 7199, "servicePort": 0 },
{"containerPort": 9042, "hostPort": 9042, "servicePort": 0 },
{"containerPort": 9160, "hostPort": 9160, "servicePort": 0 },
],
"privileged": true,
}
},
"constraints": [ [ "hostname", "UNIQUE" ] ],
"env": {"CASSANDRA_CLUSTER_NAME": "democluster"}
}
Now I add one more node of cassandra using a separate deployment and providing 10.32.0.6 as seed. The new node attaches to another host and gets the IP of his container (Node /172.17.0.2 state jump to normal). The result is that the new node cannot gossip with the seed.
{
"id": "/cassandra",
"cpus": 1.5,
"mem": 8192,
"disk": 0,
"instances": 1,
"container": {
"type": "DOCKER",
"docker": {
"image": "cassandra:2.2.3",
"network": "BRIDGE",
"portMappings": [
{"containerPort": 7000, "hostPort": 7000, "servicePort": 0 },
{"containerPort": 7001, "hostPort": 7001, "servicePort": 0 },
{"containerPort": 7199, "hostPort": 7199, "servicePort": 0 },
{"containerPort": 9042, "hostPort": 9042, "servicePort": 0 },
{"containerPort": 9160, "hostPort": 9160, "servicePort": 0 },
],
"privileged": true,
}
},
"constraints": [ [ "hostname", "UNIQUE" ] ],
"env": {
"CASSANDRA_CLUSTER_NAME": "democluster",
"CASSANDRA_SEEDS": "10.32.0.6"
}
}
The question is how could I make the two nodes gossip in the second case? Which is the IP that I should provide as seed to the second node in order to find the first one? The 172.17.0.2 is the docker container IP and cannot be reached by the second node. For example, could cassandra instance in the seed node get the IP of the physical host just like in the host network mode?
Thank you in advance!
When forming a cassandra cluster in bridge network mode below settinhs should be taken care.
1. Set below values to host Ip (not container ip)
Seeds : public_ip
Broadcast_address : public_ip
Broadcast_rpc_address : public_ip
Set listen_address to container Ip
Listen_address : 172.17.x.x
3 . Set rpc_address to 0.0.0.0 (don't use localhost)
This way we can actually form a Cassandra cluster using bridge network.
Give it a try. Make sure required ports should be accessible from outside world.

Resources