Docker Compose - give access to docker.sock before running (Telegraf) - docker

UPDATE:
I've tried implementing the accepted answer from here Telegraf can not connect to Docker sock like this in my docker compose file:
telegraf3:
image: telegraf
user: telegraf:$$(stat -c '%g' /var/run/docker.sock)
volumes:
- ./telegraf/telegraf3.conf:/etc/telegraf/telegraf.conf
- /var/run/docker.sock:/var/run/docker.sock
I am getting this error:
Error response from daemon: unable to find group $(stat -c '%g' /var/run/docker.sock): no matching entries in group file
How can i fix this issue? :)
Background:
I'm trying to run Telegraf (https://github.com/influxdata/telegraf) with the docker input. I'm running Telegraf via Docker Compose, and i've configured it roughly like this:
telegraf3:
image: telegraf
volumes:
- ./telegraf/telegraf3.conf:/etc/telegraf/telegraf.conf
- /var/run/docker.sock:/var/run/docker.sock:rw
env_file:
- ./telegraf/telegraf.env
depends_on:
- influxdb
The telegraf configuration uses a docker input plugin to interact with the docker.sock. It doesn't work, i get a permission related error:
test-grafana-telegraf3-1 | 2022-12-29T12:09:10Z E! [inputs.docker] Error in plugin: Got permission denied while trying to connect to the Docker daemon socket at unix:///docker.sock: Get "http://%2Fdocker.sock/v1.24/info": dial unix /docker.sock: connect: permission denied
Basically, the entrypoint.sh script runs telegraf (the application) with the telegraf user, which can't access the docker.sock
There's a fix for this issue described here: Telegraf can not connect to Docker sock
Issue
As i am using docker compose, i would like this fix to be defined in the compose file, and not be dependent on me starting the container with docker run.
I've tried this:
telegraf3:
image: telegraf
volumes:
- ./telegraf/telegraf3.conf:/etc/telegraf/telegraf.conf
- /var/run/docker.sock:/var/run/docker.sock:rw
env_file:
- ./telegraf/telegraf.env
depends_on:
- influxdb
command: ["bash -c -u telegraf $$(stat -c '%g' /var/run/docker.sock)", "/entrypoint.sh"]
But then i get this error:
test-grafana-telegraf3-1 | setpriv: failed to execute bash -c -u telegraf $(stat -c '%g' /var/run/docker.sock): No such file or directory
test-grafana-telegraf3-1 exited with code 127

Related

Docker-in-Docker issues with connecting to internal container network (Anchore Engine)

I am having issues when trying to connect to a docker-compose network from inside of a container. These are the files I am working with. The whole thing runs when I ./run.sh.
Dockerfile:
FROM docker/compose:latest
WORKDIR .
# EXPOSE 8228
RUN apk update
RUN apk add py-pip
RUN apk add jq
RUN pip install anchorecli
COPY dockertest.sh ./dockertest.sh
COPY docker-compose.yaml docker-compose.yaml
CMD ["./dockertest.sh"]
docker-compose.yaml
services:
# The primary API endpoint service
engine-api:
image: anchore/anchore-engine:v0.6.0
depends_on:
- anchore-db
- engine-catalog
#volumes:
#- ./config-engine.yaml:/config/config.yaml:z
ports:
- "8228:8228"
..................
## A NUMBER OF OTHER CONTAINERS THAT ANCHORE-ENGINE USES ##
..................
networks:
default:
external:
name: anchore-net
dockertest.sh
echo "------------- INSTALL ANCHORE CLI ---------------------"
engineid=`docker ps | grep engine-api | cut -f 1 -d ' '`
engine_ip=`docker inspect $engineid | jq -r '.[0].NetworkSettings.Networks."cws-anchore-net".IPAddress'`
export ANCHORE_CLI_URL=http://$engine_ip:8228/v1
export ANCHORE_CLI_USER='user'
export ANCHORE_CLI_PASS='pass'
echo "System status"
anchore-cli --debug system status #This line throws error (see below)
run.sh:
#!/bin/bash
docker build . -t anchore-runner
docker network create anchore-net
docker-compose up -d
docker run --network="anchore-net" -v //var/run/docker.sock:/var/run/docker.sock anchore-runner
#docker network rm anchore-net
Error Message:
System status
INFO:anchorecli.clients.apiexternal:As Account = None
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 172.19.0.6:8228
Error: could not access anchore service (user=user url=http://172.19.0.6:8228/v1): HTTPConnectionPool(host='172.19.0.6', port=8228): Max retries exceeded with url: /v1
(Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',))
Steps:
run.sh builds container image and creates network anchore-net
the container has an entrypoint script, which does multiple things
firstly, it brings up the docker-compose network as detached FROM inside the container
secondly, nstalls anchore-cli so I can run commands against container network
lastly, attempts to get a system status of the anchore-engine (d.c network) but thats where I am running into HTTP request connection issues.
I am dynamically getting the IP of the api endpoint container of anchore-engine and setting the URL of the request to do that. I have also tried passing those variables from command line such as:
anchore-cli --u user --p pass --url http://$engine_ip/8228/v1 system status but that throws the same error.
For those of you who took the time to read through this, I highly appreciate any input you can give me as to where the issue may be lying. Thank you very much.

Elastic search TestContainers Timed out waiting for URL to be accessible in Docker

Local env:
MacOS 10.14.6
Docker Desktop 2.0.1.2
Docker Engine 19.03.2
Compose Engine 1.24.1
Test containers 1.12.1
I'm using Elastic search in an app, and I want to be able to use TestContainers in my integration tests. Sample code in a Play Framework app that uses ElasticSearch testcontainer:
#BeforeAll
public static void setup() {
private static final ElasticsearchContainer ES = new ElasticsearchContainer();
ES.start();
}
This works when testing locally, but I want to be able to run this inside a Docker container to run on my CI server. I'm getting this exception when running the tests inside the Docker container:
[warn] o.t.u.RegistryAuthLocator - Failure when attempting to lookup auth config (dockerImageName: alpine:3.5, configFile: /root/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /root/.docker/config.json (No such file or directory)
[warn] o.t.u.RegistryAuthLocator - Failure when attempting to lookup auth config (dockerImageName: quay.io/testcontainers/ryuk:0.2.3, configFile: /root/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /root/.docker/config.json (No such file or directory)
?? Checking the system...
? Docker version should be at least 1.6.0
? Docker environment should have more than 2GB free disk space
[warn] o.t.u.RegistryAuthLocator - Failure when attempting to lookup auth config (dockerImageName: docker.elastic.co/elasticsearch/elasticsearch:7.1.1, configFile: /root/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /root/.docker/config.json (No such file or directory)
[error] d.e.c.1.1] - Could not start container
org.testcontainers.containers.ContainerLaunchException: Timed out waiting for URL to be accessible (http://172.17.0.1:32911/ should return HTTP [200])
at org.testcontainers.containers.wait.strategy.HttpWaitStrategy.waitUntilReady(HttpWaitStrategy.java:197)
at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:35)
at org.testcontainers.containers.GenericContainer.waitUntilContainerStarted(GenericContainer.java:675)
at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:332)
at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:285)
at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:283)
at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:272)
at controllers.HomeControllerTest.setup(HomeControllerTest.java:56)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
I've read the instructions here: https://www.testcontainers.org/supported_docker_environment/continuous_integration/dind_patterns/
So my docker-compose.yml looks like (note: I've been testing with another ES container as seen commented out below, but I have not been using it with this test)($INSTANCE is a random 16 char string for a particular test run):
version: '3'
services:
# elasticsearch:
# container_name: elasticsearch_${INSTANCE}
# image: docker.elastic.co/elasticsearch/elasticsearch:6.7.2
# ports:
# - 9200:9200
# - 9300:9300
# command: elasticsearch -E transport.host=0.0.0.0
# logging:
# driver: 'none'
# environment:
# ES_JAVA_OPTS: "-Xms750m -Xmx750m"
mainapp:
container_name: mainapp_${INSTANCE}
image: test_image:${INSTANCE}
stop_signal: SIGKILL
stdin_open: true
tty: true
working_dir: $PWD
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- $PWD:$PWD
environment:
ES_JAVA_OPTS: "-Xms1G -Xmx1G"
command: /bin/bash /projectfolder/build/tests/wrapper.sh
I've also tried running my tests with this command but received the same error:
docker run -it --rm -v $PWD:$PWD -w $PWD -v /var/run/docker.sock:/var/run/docker.sock test_image:68F75D8FD4C7003772C7E52B87B774F5 /bin/bash /testproject/build/tests/wrapper.sh
I tried creating a postgres container the same way inside my testing container and had no issues. I've also tried making a GenericContainer with the Elasticsearch image with no luck.
I don't think this is a connection issue because if I run curl 172.17.0.1:{port printed to test console} from inside my test container, I do get a valid elastic search response with status code 200, so it almost seems like its timing out trying to connect even though the connection is there.
Thanks.

Filebeat not running using docker-compose: setting 'filebeat.prospectors' has been removed

I'm trying to launch filebeat using docker-compose (I intend to add other services later on) but every time I execute the docker-compose.yml file, the filebeat service always ends up with the following error:
filebeat_1 | 2019-08-01T14:01:02.750Z ERROR instance/beat.go:877 Exiting: 1 error: setting 'filebeat.prospectors' has been removed
filebeat_1 | Exiting: 1 error: setting 'filebeat.prospectors' has been removed
I discovered the error by accessing the docker-compose logs.
My docker-compose file is as simple as it can be at the moment. It simply calls a filebeat Dockerfile and launches the service immediately after.
Next to my Dockerfile for filebeat I have a simple config file (filebeat.yml), which is copied to the container, replacing the default filebeat.yml.
If I execute the Dockerfile using the docker command, the filebeat instance works just fine: it uses my config file and identifies the "output.json" file as well.
I'm currently using version 7.2 of filebeat and I know that the "filebeat.prospectors" isn't being used. I also know for sure that this specific configuration isn't coming from my filebeat.yml file (you'll find it below).
It seems that, when using docker-compose, the container is accessing another configuration file instead of the one that is being copied to the container, by the Dockerfile, but so far I haven't been able to figure it out how, why and how can I fix it...
Here's my docker-compose.yml file:
version: "3.7"
services:
filebeat:
build: "./filebeat"
command: filebeat -e -strict.perms=false
The filebeat.yml file:
filebeat.inputs:
- paths:
- '/usr/share/filebeat/*.json'
fields_under_root: true
fields:
tags: ['json']
output:
logstash:
hosts: ['localhost:5044']
The Dockerfile file:
FROM docker.elastic.co/beats/filebeat:7.2.0
COPY filebeat.yml /usr/share/filebeat/filebeat.yml
COPY output.json /usr/share/filebeat/output.json
USER root
RUN chown root:filebeat /usr/share/filebeat/filebeat.yml
RUN mkdir /usr/share/filebeat/dockerlogs
USER filebeat
The output I'm expecting should be similar to the following, which comes from the successful executions I'm getting when I'm executing it as a single container.
The ERROR is expected because I don't have logstash configured at the moment.
INFO crawler/crawler.go:72 Loading Inputs: 1
INFO log/input.go:148 Configured paths: [/usr/share/filebeat/*.json]
INFO input/input.go:114 Starting input of type: log; ID: 2772412032856660548
INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 1
INFO log/harvester.go:253 Harvester started for file: /usr/share/filebeat/output.json
INFO pipeline/output.go:95 Connecting to backoff(async(tcp://localhost:5044))
ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://localhost:5044)): dial tcp [::1]:5044: connect: cannot assign requested address
INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://localhost:5044)) with 1 reconnect attempt(s)
ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://localhost:5044)): dial tcp [::1]:5044: connect: cannot assign requested address
INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://localhost:5044)) with 2 reconnect attempt(s)
I managed to figure out what the problem was.
I needed to map the location of the config file and logs directory in the docker-compose file, using the volumes tag:
version: "3.7"
services:
filebeat:
build: "./filebeat"
command: filebeat -e -strict.perms=false
volumes:
- ./filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml
- ./filebeat/logs:/usr/share/filebeat/dockerlogs
Finally I just had to execute the docker-compose command and everything start working properly:
docker-compose -f docker-compose.yml up -d

Manifest not found on docker composer even if the tag exists

I have this YML file
docker-compose-testing.yml
with docker compose configuration:
version: '3'
services:
nginx_testing:
image: MY_SERVER_IP:5000/lens/nginx_testing:${VERSION}
volumes:
- certs:/etc/letsencrypt
- certs-data:/data/letsencrypt
ports:
- 80:80
- 443:443
depends_on:
- ws_server
- translator
- auth
ws_server:
image: MY_SERVER_IP:5000/lens/ws_server:${VERSION}
worker:
image: MY_SERVER_IP:5000/lens/worker:${VERSION}
depends_on:
- ws_server
translator:
image: MY_SERVER_IP:5000/lens/translator:${VERSION}
auth:
image: MY_SERVER_IP:5000/lens/auth:${VERSION}
volumes:
- auth-data:/usr/src/app/data
volumes:
certs:
certs-data:
auth-data:
Normally, I use this command to apply the above configuration:
export VERSION=578d8de && envsubst < docker-compose-testing.yml | docker-compose -f - pull && envsubst < docker-compose-testing.yml | docker-compose -f - -p PROJECT_NAME up -d --no-build --scale worker=5
Now, when I execute this command (above) the console show this error:
Pulling translator (MY_SERVER_IP:5000/lens/translator:578d8de)...
ERROR: manifest for MY_SERVER_IP:5000/lens/translator:578d8de not found
The response for one similar question below says that the tag do not exists:
Error response from daemon: manifest for ibmblockchain/fabric-peer:latest not found
But when I list the images with command:
docker images | grep 578d8de
Console show this output proving that the tag exists:
MY_SERVER_IP:5000/lens/auth 578d8de 8103c4d63870 2 hours ago 195MB
MY_SERVER_IP:5000/lens/nginx_testing 578d8de 578d8dead150 4 hours ago 235MB
MY_SERVER_IP:5000/lens/translator 578d8de e9eb25fa0aef 5 hours ago 185MB
MY_SERVER_IP:5000/lens/ws_server 578d8de 92b1d1a4cee9 5 hours ago 177MB
MY_SERVER_IP:5000/lens/worker 578d8de 22a935deba5c 7 days ago 175MB
Some extra details:
The server (MY_SERVER_IP) has a docker registry listening on 5000 port.
The image with version tag 578d8de was uploaded to the server not with registry, but with the "docker save" and "scp" command on the dev machine, and "docker load" on the server.
Any idea why this error is occurring?
This error is happenning when I send image with scp command to server and load the image to docker.
How I don't use docker push to the registry, the image don't exists in registry.
So when docker-compose pull execute, don't find image on registry and dispatch the error.

Connection to docker container failing because of postgis port issue

My docker container is able to successfully build but when I enter the command docker-compose build, the following error is returned:
Starting docker_etl_1 ...
Starting 1e5f56853e10_1e5f56853e10_1e5f56853e10_docker_postgis_1 ...
Starting 1e5f56853e10_1e5f56853e10_1e5f56853e10_docker_postgis_1
Starting 1e5f56853e10_1e5f56853e10_1e5f56853e10_docker_postgis_1 ... error
ERROR: for 1e5f56853e10_1e5f56853e10_1e5f56853e10_docker_postgis_1 Cannot start service postgis: driver failed programming external connectivity on endpoint 1e5f56853e10_1e5f56853e10_1e5f56853e10_docker_postgis_1 (91464afbee8bf7212061797ec0f4c017a56cc3c30c9bdaf513127a6e6a4a5a52): Error starting userland prStarting docker_etl_1 ... done
ERROR: for postgis Cannot start service postgis: driver failed programming external connectivity on endpoint 1e5f56853e10_1e5f56853e10_1e5f56853e10_docker_postgis_1 (91464afbee8bf7212061797ec0f4c017a56cc3c30c9bdaf513127a6e6a4a5a52): Error starting userland proxy: Bind for 0.0.0.0:5432 failed: port is already allocated
Here is my docker-compose.yaml
version: '2'
services:
postgis:
build: ./postgis
volumes:
- ../src/main/sql:/sql
ports:
- "5432:5432"
etl:
build: ./etl
volumes:
- ..:/national-voter-file
entrypoint:
- python3
- /national-voter-file/load/loader.py
and here is the Dockerfile:
FROM mdillon/postgis:9.5
ENV POSTGRES_DB VOTER
RUN mkdir /sql
COPY ./dockerResources/z-init-db.sh /docker-entrypoint-initdb.d/
EXPOSE 5432
Docker ps -a returns:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
da74ad97b95c docker_postgis "docker-entrypoint..." About a minute ago Created docker_postgis_1
5872c6e55fe2 docker_etl "python3 /national..." About a minute ago Exited (2) About a minute ago docker_etl_1
However, when I try rm $(docker ps -qa) I get the following error:
rm: da74ad97b95c: No such file or directory
rm: 5872c6e55fe2: No such file or directory
I don't believe I have another container running so I'm confused by the message Bind for 0.0.0.0:5432 failed: port is already allocated
Is it possible that you ran the same docker-compose earlier, which probably failed or at least failed to clean up the services?
Try running docker ps -a to check if any stopped containers exist. It is possible that the stopped containers are hogging the port. If so, just clear them out using docker rm $(docker ps -qa)

Resources