I want to send logs from one rancher service (e.g. my_service) to another rancher service running the ELK stack with the syslog driver
I am setting up my stack via a docker-compose as follows more or less:
elk-custom:
# image: elk-custom
build:
context: .
dockerfile: Dockerfile-elk
ports:
- 5601:5601
- 9200:9200
- 5044:5044
- 5151:5151
- 5152:5152
my_service:
image: some_image_from_my_local_registry
depends_on:
- elk-custom
logging:
driver: syslog
options:
syslog-address: "tcp://elk-custom:514"
However, on the stack dashboard, for my_service I get:
my_service (Expected state running but got error: Error response from daemon: failed to initialize logging driver: dial tcp: lookup elk-custom on 10.0.2.3:53: server misbehaving)
Is there anything additional needed to make the specific logging (elk-custom) service discoverable?
a few things are going on there that are problematic. First, if you are doing build, you have to use either a git or HTTP remote url or s3 based context.
Please see the docs: http://rancher.com/docs/rancher/v1.6/en/cattle/rancher-compose/#builds
Typically, you build an image and deploy it as a service. Builds are more of a development bit and less used on a production side of things.
The next thing is that in a multi-host setup, you will have trouble routing on the Rancher network. I would recommend deploying a Logstash collector on all nodes with the syslog ingress in host networking mode. Then you can point the logging driver to the localhost syslog target. Each of the logstash collectors would then forward to either another logstash for filtering or Elastic cluster.
Related
how can we run docker commands inside container with docker-compose?
Simply I want to get IP of some other network container.
I am running three container va-server, db and api-server. All the containers are in same docker-network
Here I am providing docker-compose file below:
version: "2.3"
services:
va-server:
container_name: va_server
image: nitinroxx/facesense:amd64_2022.11.28 #facesense:alpha
runtime: nvidia
restart: always
mem_limit: 4G
networks:
- perimeter-network
db:
container_name: mongodb
image: mongo:latest
ports:
- "27017:27017"
restart: always
volumes:
- ./facesense_db:/data/db
command: [--auth]
networks:
- perimeter-network
api-server:
container_name: api_server
image: nitinroxx/facesense:api_amd64_2022.11.28
ports:
- "80:80"
- "465:465"
restart: always
networks:
- perimeter-network
networks:
perimeter-network:
driver: bridge
ipam:
config:
- gateway: 10.16.239.1
subnet: 10.16.239.0/24
I have install docker inside the container which giving me below permission error:
docker.errors.dockerexception: error while fetching server api version: ('connection aborted.', permissionerror(13, 'permission denied')
...inside [a] container [...] I want to get IP of some other network container....
Docker provides an internal DNS service that can resolve container names to their Docker-internal IP addresses. From one of the containers you show, you could look up a host name like db to get the container's IP address; but in practice, this is a totally normal DNS name and all but the lowest-level networking interfaces can use those directly.
This does require that all of the containers involved be on the same Docker network. Normally Compose sets this up automatically for you; in the file you show I might delete the networks: blocks and container_name: overrides in the name of simplicity. Also see Networking in Compose in the Docker documentation.
In short:
You can probably use the Compose service names va-server, db, and api-server as host names without specifically knowing their IP addresses.
This probably means you never need to know the container IP addresses at all (they're usually unusable from outside Docker).
If you do need an IP address from inside a container, a DNS lookup can find it.
You can't usually run docker commands from inside containers. You can't do this safely without making it possible for the container to take over the whole host. There are usually better patterns that don't tie you to the Docker stack specifically.
I have a Docker Swarm web application that can deploy fine and can be reached from outside by a browser.
But it was not showing the client IP address to the HTTP service.
So I decided to add a Traefik service to the Docker Compose file to expose the client IP to the HTTP service.
I use a mode: host and a driver: overlay for this reason.
The complete configuration is described in two Docker Compose files that I run in sequence.
First I run the docker stack deploy --compose-file docker-compose-dev.yml common command on the file:
version: "3.9"
services:
traefik:
image: traefik:v2.5
networks:
common:
ports:
- target: 80
published: 80
mode: host
- target: 443
published: 443
mode: host
command:
- "--providers.docker.endpoint=unix:///var/run/docker.sock"
- "--providers.docker.swarmMode=true"
- "--providers.docker.exposedbydefault=false"
- "--providers.docker.network=common"
- "--entrypoints.web.address=:80"
# Set a debug level custom log file
- "--log.level=DEBUG"
- "--log.filePath=/var/log/traefik.log"
- "--accessLog.filePath=/var/log/access.log"
# Enable the Traefik dashboard
- "--api.dashboard=true"
deploy:
placement:
constraints:
- node.role == manager
labels:
# Expose the Traefik dashboard
- "traefik.enable=true"
- "traefik.http.routers.dashboard.service=api#internal"
- "traefik.http.services.traefik.loadbalancer.server.port=888" # A port number required by Docker Swarm but not being used in fact
- "traefik.http.routers.dashboard.rule=Host(`traefik.learnintouch.com`)"
- "traefik.http.routers.traefik.entrypoints=web"
# Basic HTTP authentication to secure the dashboard access
- "traefik.http.routers.traefik.middlewares=traefik-auth"
- "traefik.http.middlewares.traefik-auth.basicauth.users=stephane:$$apr1$$m72sBfSg$$7.NRvy75AZXAMtH3C2YTz/"
volumes:
# So that Traefik can listen to the Docker events
- "/var/run/docker.sock:/var/run/docker.sock:ro"
- "~/dev/docker/projects/common/volumes/logs/traefik.service.log:/var/log/traefik.log"
- "~/dev/docker/projects/common/volumes/logs/traefik.access.log:/var/log/access.log"
networks:
common:
name: common
driver: overlay
Then I run the docker stack deploy --compose-file docker-compose.yml www_learnintouch command on the file:
version: "3.9"
services:
www:
image: localhost:5000/www.learnintouch
networks:
common:
volumes:
- "~/dev/docker/projects/learnintouch/volumes/www.learnintouch/account/data:/usr/local/learnintouch/www/learnintouch.com/account/data"
- "~/dev/docker/projects/learnintouch/volumes/www.learnintouch/account/backup:/usr/local/learnintouch/www/learnintouch.com/account/backup"
- "~/dev/docker/projects/learnintouch/volumes/engine:/usr/local/learnintouch/engine"
- "~/dev/docker/projects/common/volumes/letsencrypt/certbot/conf/live/thalasoft.com:/usr/local/learnintouch/letsencrypt"
- "~/dev/docker/projects/common/volumes/logs:/usr/local/apache/logs"
- "~/dev/docker/projects/common/volumes/logs:/usr/local/learnintouch/logs"
user: "${CURRENT_UID}:${CURRENT_GID}"
deploy:
replicas: 1
restart_policy:
condition: any
delay: 5s
max_attempts: 3
window: 10s
labels:
- "traefik.enable=true"
- "traefik.http.routers.www.rule=Host(`dev.learnintouch.com`)"
- "traefik.http.routers.www.entrypoints=web"
- "traefik.http.services.www.loadbalancer.server.port=80"
healthcheck:
test: curl --fail http://127.0.0.1:80/engine/ping.php || exit 1
interval: 10s
timeout: 3s
retries: 3
networks:
common:
external: true
name: common
Here are the networks:
stephane#stephane-pc:~$ docker network ls
NETWORK ID NAME DRIVER SCOPE
6beaf0c3a518 bridge bridge local
ouffqdmdesuy common overlay swarm
17et43c5tuf0 docker-registry_default overlay swarm
1ae825c8c821 docker_gwbridge bridge local
7e6b4b7733ca host host local
2ui8s1yomngt ingress overlay swarm
460aad21ada9 none null local
tc846a14ftz5 verdaccio overlay swarm
The docker ps command shows that all containers are healthy.
But a request to http://dev.learnintouch.com/ responds with a Bad Gateway error MOST of the times except when rarely it does not and the web application displays fine.
As a side note, I would like any unhealthy service to be restarted and seen by Traefik. Just like Docker Swarm restarts unhealthy services I would like Traefik to restart unhealthy services too.
The service log:
{"level":"debug","msg":"Configuration received from provider docker: {\"http\":{\"routers\":{\"dashboard\":{\"service\":\"api#internal\",\"rule\":\"Host(`traefik.learnintouch.com`)\"},\"nodejs\":{\"entryPoints\":[\"web\"],\"service\":\"nodejs\",\"rule\":\"Host(`dev.learnintouch.com`)\"},\"traefik\":{\"entryPoints\":[\"web\"],\"middlewares\":[\"traefik-auth\"],\"service\":\"traefik\",\"rule\":\"Host(`common-reverse-proxy`)\"},\"www\":{\"entryPoints\":[\"web\"],\"service\":\"www\",\"rule\":\"Host(`dev.learnintouch.com`)\"}},\"services\":{\"nodejs\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.14.17:9001\"}],\"passHostHeader\":true}},\"traefik\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.14.8:888\"}],\"passHostHeader\":true}},\"www\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.14.18:80\"}],\"passHostHeader\":true}}},\"middlewares\":{\"traefik-auth\":{\"basicAuth\":{\"users\":[\"stephane:$apr1$m72sBfSg$7.NRvy75AZXAMtH3C2YTz/\"]}}}},\"tcp\":{},\"udp\":{}}","providerName":"docker","time":"2021-07-04T10:25:01Z"}
{"level":"info","msg":"Skipping same configuration","providerName":"docker","time":"2021-07-04T10:25:01Z"}
I also tried to have Docker Swarm doing the load balancing with adding the - "traefik.docker.lbswarm=true" property to my service but the Bad Gateway error remained.
I also restarted the Swarm manager:
docker swarm leave --force
docker swarm init
but the Bad Gateway error remained.
I also added the two labels:
- "traefik.backend.loadbalancer.sticky=true"
- "traefik.backend.loadbalancer.stickiness=true"
but the Bad Gateway error remained.
It feels like Traefik hits the web service before that one has a chance to be ready. Would there be any way to tell Traefik to wait a given amount of seconds before hitting the web service ?
UPDATE: I could find, not a solution, but a workaround the issue, by splitting the first above common stack file into two files, with one dedicated to the traefik stack. I could then start the 3 stacks in the following order: common, www_learnintouch and traefik
The important thing was to start the traefik stack after the others. If I then had to remove and start again the www_learnintouch stack for example, then I had to follow this by removing and starting again the traefik stack.
Also, if I removed the container of www_learnintouch with a docker rm -f CONTAINER_ID command then I also needed to remove and start again the traefik stack.
I have multiple stacks running in my swarm. The stacks are created from compose files. For illustrative purposes let's say I have the following:
Stack 1
- Nginx container
Stack 2
- Web App Container
- Database Container
- Cache Container
- SMTP Container
Stack 3
- Web App Container
- Database Container
When I deploy these stacks docker creates an overlay network so I have 3 independent overlay networks for each stack.
What I would like to do is make it so the Web App Container in Stack 2 & 3 have access to the Stack 1 overlay network so that I can proxy_pass incoming connections without having to expose the port to the internet.
The docker website only seems to have an explanation for the legacy swarm networking: https://docs.docker.com/compose/networking/
That page says for stacks networking to refer to this page: https://docs.docker.com/engine/reference/commandline/stack_deploy/
I cannot see any information about networking on that page so i am a bit stuck.
Inside the web app service definition I have tried adding:
networks:
- nginx_default
This is the network name as shown by running docker network ls but I get an error message that this network is not defined.
What is the right way to get my web app containers and my nginx container on the same private network?
My issue was I needed to declare the network as external. Here is working sample config for stack 2:
version: "3.8"
networks:
stack2:
nginx:
external: true
services:
db:
networks:
stack2:
image: mysql:5.7
webapp:
networks:
stack2:
nginx:
image: webapp
This will connect webapp to the nginx network but the database wont be exposed to it.
I have a docker project that uses the MAC address for hardware license enforcement. I cannot change this, the ISV uses a hardware fingerprint that includes the MAC address.
I am trying to create a macvlan network, that will use the physical adapter and get an IP address from my network DHCP server, alternatively I will assign a static IP address manually.
I must be able to set the MAC address manually such that it does not dynamically change and invalidate my license key.
Based on Docker docs the mac_address setting is deprecated, at least in v3 schema, but seems to be honored in v2 schemas.
I have a config that builds, using vanilla LSIO Nginx as test, but fails to run with an error stating that the MAC address cannot be assigned.
version: "2.1"
services:
nginx:
image: linuxserver/nginx
container_name: nginx_macvlan
environment:
- TZ=Americas/Los_Angeles
volumes:
- .mount:/config
ports:
- 80:80
- 443:443
restart: unless-stopped
mac_address: b7-48-d5-a6-d1-99
networks:
nginx_vlan:
ipv4_address: 192.168.1.10
networks:
nginx_vlan:
driver: macvlan
ipam:
driver: default
config:
- subnet: 192.168.1.0/24
PS C:\Users\piete\source\TestMacVlan> cd "c:\Users\piete\source\TestMacVlan"
PS C:\Users\piete\source\TestMacVlan> docker-compose -f "docker-compose-macvlan.yml" up -d --build
Creating network "testmacvlan_nginx_vlan" with driver "macvlan"
Creating nginx_macvlan ... error
ERROR: for nginx_macvlan Cannot start service nginx: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:432: running prestart hook 0 caused \\\"error running hook: exit status 1, stdout: , stderr: time=\\\\\\\"2020-05-16T02:46:50Z\\\\\\\" level=fatal msg=\\\\\\\"failed to add interface veth2b7c9ef to sandbox: error setting interface \\\\\\\\\\\\\\\"veth2b7c9ef\\\\\\\\\\\\\\\" MAC to \\\\\\\\\\\\\\\"b7:48:d5:a6:d1:99\\\\\\\\\\\\\\\": cannot assign requested address\\\\\\\"\\\\n\\\"\"": unknown
ERROR: for nginx Cannot start service nginx: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:432: running prestart hook 0 caused \\\"error running hook: exit status 1, stdout: , stderr: time=\\\\\\\"2020-05-16T02:46:50Z\\\\\\\" level=fatal msg=\\\\\\\"failed to add interface veth2b7c9ef to sandbox: error setting interface \\\\\\\\\\\\\\\"veth2b7c9ef\\\\\\\\\\\\\\\" MAC to \\\\\\\\\\\\\\\"b7:48:d5:a6:d1:99\\\\\\\\\\\\\\\": cannot assign requested address\\\\\\\"\\\\n\\\"\"": unknown
ERROR: Encountered errors while bringing up the project.
PS C:\Users\piete\source\TestMacVlan>
I am testing on Win10 using Docker for Windows.
PS C:\Users\piete\source\TestMacVlan> docker --version
Docker version 19.03.1, build 5b38d82a-
PS C:\Users\piete\source\TestMacVlan> [Environment]::OSVersion
Platform ServicePack Version VersionString
-------- ----------- ------- -------------
Win32NT 10.0.18363.0 Microsoft Windows NT 10.0.18363.0
How do I use macvlan in compose and set a MAC and IP or use DHCP for IP?
I got it working on ubuntu 18 lts in a Hyper-V container.
You have to edit the Hyper-V guest network adapter to allow "enable mac address spoofing", this is under the advanced options.
When using compose, the version can't be greater than ~v2.1, when using current v3.7+ versions you'll get a gateway is unexpected error.
On Linux the host does not get traffic routed to the macvlan, so containers need to be on the same macvlan if they need to talk to each other.
There can only be one macvlan per subnet range, or one gateway per range, not sure what the cause is.
I could not get it working on Docker for Windows, specifically I do not know how to specify the parent adapter name. I tried the actual adapter name, did not work, "eth0" works for creating the macvlan, but no traffic flows. I don't know if it is because the adapter name is wrong, or something else.
I could not get network infrastructure DHCP working using macvlan, maybe this will require creating bridges on the host.
Here is working compose file running two nginx instances on two specific IP's with two specific MAC addresses, tested on Ubuntu 18.04 LTS running on Hyper-V. I have not yet tested bare metal.
version: "2.1"
services:
nginx_10:
image: linuxserver/nginx
container_name: nginx_macvlan_10
environment:
- TZ=Americas/Los_Angeles
ports:
- 80:80
- 443:443
restart: unless-stopped
mac_address: 02:42:c0:a8:84:22
networks:
nginx_vlan:
ipv4_address: 192.168.1.10
nginx_45:
image: linuxserver/nginx
container_name: nginx_macvlan_45
environment:
- TZ=Americas/Los_Angeles
ports:
- 80:80
- 443:443
restart: unless-stopped
mac_address: 02:42:c0:a8:84:23
networks:
nginx_vlan:
ipv4_address: 192.168.1.45
networks:
nginx_vlan:
driver: macvlan
driver_opts:
parent: eth0
ipam:
driver: default
config:
- subnet: 192.168.1.0/24
gateway: 192.168.1.1
# docker-compose --file docker-compose-macvlan-ubuntu-multi.yml up --detach
I'd still like to know:
How to get this working with docker compose schema v3+.
How to get it working on Docker for Windows.
How to get DHCP working.
IPAM configuration (gateway, ip_range, aux_addresses) is now supported by docker-compose v1.27.0+.
https://github.com/docker/compose/issues/6569#issuecomment-709195165
Releases 1.27.0+ have merged v2/v3 file formats, so you should be able
to use ipam anywhere now.
https://github.com/docker/compose/releases/tag/1.27.0
Merge 2.x and 3.x compose formats and align with COMPOSE_SPEC schema
More info: https://blog.jjhayes.net/wp/2020/10/28/using-a-macvlan-network-in-docker-compose/
I want to send logs from one container running my_service to another running the ELK stack with the syslog driver (so I will need the logstash-input-syslog plugin installed).
I am tweaking this elk image (and tagging it as elk-custom) via the following Dockerfile-elk
(using port 514 because this seems to be the default port)
FROM sebp/elk
WORKDIR /opt/logstash/bin
RUN ./logstash-plugin install logstash-input-syslog
EXPOSE 514
Running my services via a docker-compose as follows more or less:
elk-custom:
# image: elk-custom
build:
context: .
dockerfile: Dockerfile-elk
ports:
- 5601:5601
- 9200:9200
- 5044:5044
- 514:514
my_service:
image: some_image_from_my_local_registry
depends_on:
- elk-custom
logging:
driver: syslog
options:
syslog-address: "tcp://elk-custom:514"
However:
ERROR: for b4cd17dc1142_namespace_my_service_1 Cannot start service
my_service: failed to initialize logging driver: dial tcp: lookup
elk-custom on 10.14.1.31:53: server misbehaving
ERROR: for api Cannot start service my_service: failed to initialize
logging driver: dial tcp: lookup elk-custom on 10.14.1.31:53: server
misbehaving ERROR: Encountered errors while bringing up the project.
Any suggestions?
UPDATE: Apparently nothing seems to be listening on port 514, cause from within the container, the command netstat -a shows nothing on this port....no idea why...
You need to use tcp://127.0.0.1:514 instead of tcp://elk-custom:514. Reason being this address is being used by docker and not by the container. That is why elk-custom is not reachable.
So this will only work when you map the port (which you have done) and the elk-service is started first (which you have done) and the IP is reachable from the docker host, for which you would use tcp://127.0.0.1:514