I'm trying to convert logs and ingest to signalfx as metrics. I'm using the fluent-bit internal http plugin to send to a locally installed signalfx agent. Installation of both fluent-bit and signalfx is successfull. i'm able to see the docker containers.
But while sending the logs from fluent-bit to the exposed signalfx endpoint 8095. it is not able to find the host.
Error:
[net] could not connect to localhost:8095
signalfx-processor | [2022/09/01 11:47:59] [debug] [net] could not connect to localhost:8095
signalfx-processor | [2022/09/01 11:47:59] [debug] [task] destroy task=0x7fa02603d690 (task_id=2)
signalfx-processor | [2022/09/01 11:47:59] [debug] [upstream] connection #-1 failed to localhost:8095
signalfx-processor | [2022/09/01 11:47:59] [debug] [upstream] connection #-1 failed to localhost:8095
signalfx-processor | [2022/09/01 11:47:59] [debug] [task] task_id=1 reached retry-attempts limit 1/1
signalfx-processor | [2022/09/01 11:47:59] [error] [output:http:http.0] no upstream connections available to localhost:8095
signalfx-processor | [2022/09/01 11:47:59] [error] [output:http:http.0] no upstream connections available to localhost:8095
could not connect to localhost:8095
docker compose file
version: '3.7'
services:
signalfx-agent:
image: quay.io/signalfx/signalfx-agent:5
ports:
- "8095:8095"
- "9080:9080"
volumes:
- /:/hostfs:ro
- ./etc/signalfx:/etc/signalfx:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
restart: always
fluent-bit:
image: fluent/fluent-bit:1.9-debug
user: root
depends_on:
- signalfx-agent
volumes:
- $PWD/fluentd/nginx.log:/nginx-access.log
- $PWD/fluentd:/fluent-bit/etc
container_name: signalfx-processor
restart: always
I hope this is something to do with docker network setup. Any help will be really helpful.
Thanks in advance
Related
I'm working on Mac with Docker Desktop. I'm trying to run wurstmeister/kafka from docker compose and connect a producer to it.
This is my docker-compose.yml:
version: '3.8'
services:
zookeeper:
container_name: zookeeper
image: zookeeper:3.7.0
ports:
- "2181:2181"
environment:
ZOO_MY_ID: 1
ZOO_SERVERS: server.1=zookeeper:2888:3888;2181
restart: on-failure
kafka:
container_name: kafka
image: wurstmeister/kafka:2.13-2.7.0
ports:
- "9092:9092"
environment:
KAFKA_LISTENERS: INTERNAL://kafka:19092,EXTERNAL://localhost:9092
KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:19092,EXTERNAL://localhost:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
KAFKA_BROKER_ID: 1
restart: on-failure
depends_on:
- zookeeper
Then I have producer connecting to localhost:9092 and sending a simple message. The producer works fine - tested with another kafka image confluentinc/cp-kafka:6.2.0.
When I try to use producer with wurstmeister/kafka I'm getting a lot of this errors:
22:07:13.421 [kafka-producer-network-thread | simple-producer] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=simple-producer] Initialize connection to node localhost:9092 (id: -1 rack: null) for sending metadata request
22:07:13.421 [kafka-producer-network-thread | simple-producer] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=simple-producer] Initiating connection to node localhost:9092 (id: -1 rack: null) using address localhost/127.0.0.1
22:07:13.421 [kafka-producer-network-thread | simple-producer] DEBUG org.apache.kafka.common.network.Selector - [Producer clientId=simple-producer] Created socket with SO_RCVBUF = 326640, SO_SNDBUF = 146988, SO_TIMEOUT = 0 to node -1
22:07:13.421 [kafka-producer-network-thread | simple-producer] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=simple-producer] Completed connection to node -1. Fetching API versions.
22:07:13.421 [kafka-producer-network-thread | simple-producer] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=simple-producer] Initiating API versions fetch from node -1.
22:07:13.422 [kafka-producer-network-thread | simple-producer] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=simple-producer] Sending API_VERSIONS request with header RequestHeader(apiKey=API_VERSIONS, apiVersion=3, clientId=simple-producer, correlationId=20) and timeout 30000 to node -1: {client_software_name=apache-kafka-java,client_software_version=2.7.0,_tagged_fields={}}
22:07:13.423 [kafka-producer-network-thread | simple-producer] DEBUG org.apache.kafka.common.network.Selector - [Producer clientId=simple-producer] Connection with localhost/127.0.0.1 disconnected
java.io.EOFException: null
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:97)
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:447)
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:397)
at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576)
at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:561)
at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:325)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:240)
at java.base/java.lang.Thread.run(Thread.java:829)
22:07:13.424 [kafka-producer-network-thread | simple-producer] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=simple-producer] Node -1 disconnected.
22:07:13.424 [kafka-producer-network-thread | simple-producer] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=simple-producer] Bootstrap broker localhost:9092 (id: -1 rack: null) disconnected
Why this happens? What is the cause of this error? And how can make it work?
EDIT: added kafka container logs
Kafka container last logs below and no new logs added when I try to connect producer:
[2021-09-09 21:07:28,227] INFO Kafka version: 2.7.0 (org.apache.kafka.common.utils.AppInfoParser)
[2021-09-09 21:07:28,230] INFO Kafka commitId: 448719dc99a19793 (org.apache.kafka.common.utils.AppInfoParser)
[2021-09-09 21:07:28,231] INFO Kafka startTimeMs: 1631221648211 (org.apache.kafka.common.utils.AppInfoParser)
[2021-09-09 21:07:28,238] INFO [KafkaServer id=1] started (kafka.server.KafkaServer)
[2021-09-09 21:07:28,365] INFO [broker-1-to-controller-send-thread]: Recorded new controller, from now on will use broker 1 (kafka.server.BrokerToControllerRequestThread)
I'm trying to test a Symfony3 web application with Behat/Mink and Selenium2Driver so that I can test Javascript functionallity too.
The application runs in a docker container, so I added a new docker container for selenium-hub and chrome as described here:
# docker-compose.yml
version: '3.5' # Docker Engine release 17.12.0+
networks:
servicesnet:
driver: bridge
services:
apache:
build:
context: './apache2'
container_name: apache-service
ports:
- "80:80"
- "443:443"
tty: true
networks:
- servicesnet
volumes:
- ${HOST_APACHE_CONFIG}:/etc/apache2
- ${HOST_PAGES_PATH}:/var/www/localhost/htdocs
selenium-hub:
image: selenium/hub:4.0.0-alpha-6-20200730
container_name: selenium-hub
ports:
- "4444:4444"
networks:
- servicesnet
chrome:
image: selenium/node-chrome:4.0.0-alpha-6-20200730
volumes:
- /dev/shm:/dev/shm
depends_on:
- selenium-hub
environment:
- HUB_HOST=selenium-hub
networks:
- servicesnet
When I run docker-compose up it outputs for the new containers:
chrome | 2020-08-12 07:36:19,917 INFO Included extra file "/etc/supervisor/conf.d/selenium.conf" during parsing
chrome | 2020-08-12 07:36:19,918 INFO supervisord started with pid 7
selenium-hub | 2020-08-12 07:36:19,297 INFO Included extra file "/etc/supervisor/conf.d/selenium-grid-hub.conf" during parsing
selenium-hub | 2020-08-12 07:36:19,298 INFO supervisord started with pid 7
selenium-hub | 2020-08-12 07:36:20,301 INFO spawned: 'selenium-grid-hub' with pid 10
selenium-hub | Starting Selenium Grid Hub...
selenium-hub | 2020-08-12 07:36:20,311 INFO success: selenium-grid-hub entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
selenium-hub | 07:36:20.588 INFO [LoggingOptions.getTracer] - Using OpenTelemetry for tracing
selenium-hub | 07:36:20.589 INFO [LoggingOptions.createTracer] - Using OpenTelemetry for tracing
selenium-hub | 07:36:20.607 INFO [EventBusOptions.createBus] - Creating event bus: org.openqa.selenium.events.zeromq.ZeroMqEventBus
selenium-hub | 07:36:20.638 INFO [BoundZmqEventBus.<init>] - XPUB binding to [binding to tcp://*:4442, advertising as tcp://172.28.0.3:4442], XSUB binding to [binding to tcp://*:4443, advertising as tcp://172.28.0.3:4443]
selenium-hub | 07:36:20.676 INFO [UnboundZmqEventBus.<init>] - Connecting to tcp://172.28.0.3:4442 and tcp://172.28.0.3:4443
selenium-hub | 07:36:20.680 INFO [UnboundZmqEventBus.<init>] - Sockets created
selenium-hub | 07:36:20.681 INFO [UnboundZmqEventBus.lambda$new$2] - Bus started
chrome | 2020-08-12 07:36:21,136 INFO success: xvfb entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
chrome | 2020-08-12 07:36:21,136 INFO success: fluxbox entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
chrome | 2020-08-12 07:36:21,136 INFO success: vnc entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
chrome | 2020-08-12 07:36:21,137 INFO success: selenium-node entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
selenium-hub | 07:36:21.308 INFO [Hub.execute] - Started Selenium hub 4.0.0-alpha-6 (revision 5f43a29cfc): http://172.28.0.3:4444
chrome | 07:36:21.774 INFO [LoggingOptions.getTracer] - Using OpenTelemetry for tracing
chrome | 07:36:21.775 INFO [LoggingOptions.createTracer] - Using OpenTelemetry for tracing
chrome | 07:36:21.791 INFO [EventBusOptions.createBus] - Creating event bus: org.openqa.selenium.events.zeromq.ZeroMqEventBus
chrome | 07:36:21.829 INFO [UnboundZmqEventBus.<init>] - Connecting to tcp://selenium-hub:4442 and tcp://selenium-hub:4443
chrome | 07:36:21.857 INFO [UnboundZmqEventBus.<init>] - Sockets created
chrome | 07:36:21.859 INFO [UnboundZmqEventBus.lambda$new$2] - Bus started
chrome | 07:36:22.121 INFO [NodeServer.execute] - Reporting self as: http://172.28.0.5:5555
chrome | 07:36:22.175 INFO [NodeOptions.report] - Adding Chrome for {"browserName": "chrome"} 8 times
chrome | 07:36:22.298 INFO [NodeServer.execute] - Started Selenium node 4.0.0-alpha-6 (revision 5f43a29cfc): http://172.28.0.5:5555
chrome | 07:36:22.302 INFO [NodeServer.execute] - Starting registration process for node id ff0154a7-ed4b-438a-887c-0a7f3a988cb4
selenium-hub | 07:36:22.355 INFO [LocalDistributor.refresh] - Creating a new remote node for http://172.28.0.5:5555
selenium-hub | 07:36:22.763 INFO [LocalDistributor.add] - Added node ff0154a7-ed4b-438a-887c-0a7f3a988cb4.
selenium-hub | 07:36:22.770 INFO [Host.lambda$new$0] - Changing status of node ff0154a7-ed4b-438a-887c-0a7f3a988cb4 from DOWN to UP. Reason: http://172.28.0.5:5555 is ok
chrome | 07:36:22.774 INFO [NodeServer.lambda$execute$0] - Node has been added
Then I have the next method for every test:
<?php
namespace Tests\AppBundle\Controller;
use Behat\Mink\Driver\Selenium2Driver;
use Behat\Mink\Mink;
use Behat\Mink\Session;
use Symfony\Bundle\FrameworkBundle\Client;
use Symfony\Bundle\FrameworkBundle\Test\WebTestCase;
abstract class BaseControllerTest extends WebTestCase
{
/**
* #var Client
*/
protected $client;
/**
* #var Session
*/
protected $session;
public function visitUri($uri)
{
$this->client = static::createClient();
$pass = $this->client->getKernel()->getContainer()->getParameter('http_basic_auth_pass');
$host = 'localhost'; // I've tried several things here (like 172.28.0.5:5555)
$driver = new Selenium2Driver('chrome');
$mink = new Mink(array(
'chrome' => new Session($driver)
));
$driver->setTimeouts(['page load' => 900000]);
$mink->setDefaultSessionName('chrome');
$this->session = $mink->getSession();
$this->session->visit('http://user:' . $pass . '#' . $host . $uri);
}
}
And I call this method from a specific test:
public function testClickOnSearch()
{
$this->visitUri(/mi-custom-uri);
$page = $this->session->getPage();
$this->session->wait(
200000,
"typeof jQuery !== 'undefined'"
);
$page->findButton('Buton text')->click();
$this->assertContains('my-custom-uri-2', $this->session->getCurrentUrl());
}
but I never get the session started. If I go to http://localhost:4444/wd/hub/session/url I see this error message:
"org.openqa.selenium.NoSuchSessionException: Unable to find session with ID: url\nBuild info: version: '4.0.0-alpha-6', revision: '5f43a29cfc'\nSystem info: host: 'fca78c7f81e6', ip: '172.28.0.3', os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.0-42-generic', java.version: '1.8.0_252'\nDriver info: driver.version: unknown"
And executing the test, after 200 seconds this error is thrown:
PHP Fatal error: Call to a member function click() on null
I'm sure something is missing but don't know what. Any idea?
This error message...
org.openqa.selenium.NoSuchSessionException: Unable to find session with ID: url\n
Build info: version: '4.0.0-alpha-6', revision: '5f43a29cfc'\n
System info: host: 'fca78c7f81e6', ip: '172.28.0.3', os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.0-42-generic', java.version: '1.8.0_252'\n
Driver info: driver.version: unknown
...implies that the ChromeDriver was unable to initiate/spawn a new Browsing Context i.e. Chrome Browser session which gets reflected in the logs as:
Driver info: driver.version: unknown
Hence moving forward you see the error:
PHP Fatal error: Call to a member function click() on null
and the most probhable cause is the incompatibility between the version of the binaries you are using.
Solution
Ensure that:
ChromeDriver is updated to current ChromeDriver v84.0 level.
Chrome is updated to current Chrome Version 84.0 level. (as per ChromeDriver v84.0 release notes)
If your base Web Client version is too old, then uninstall it and install a recent GA and released version of Web Client.
Take a System Reboot.
Always invoke driver.quit() within tearDown(){} method to close & destroy the WebDriver and Web Client instances gracefully.
References
You can find a couple of relevant detailed discussions in:
org.openqa.selenium.NoSuchSessionException: no such session error in Selenium automation tests using ChromeDriver Chrome with Java
Found you using selenium inside docker container. Than you can try environment variable SE_NODE_SESSION_TIMEOUT=999999
Good Day,
I wanted to test the config store which is built using spring boot. The instruction given to me is run the project using docker-compose.yml files. I'm new with this,I've tired to execute but while running those commands on iMAC terminal I'm facing the following exception.
platform-config-store | 2018-03-05 11:55:12.167 INFO 1 --- [ main] org.apache.zookeeper.ZooKeeper : Initiating client connection, connectString=localhost:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState#22bbbe6
platform-config-store | 2018-03-05 11:55:12.286 INFO 1 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
platform-config-store | 2018-03-05 11:55:12.314 WARN 1 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
platform-config-store | java.net.ConnectException: Connection refused
platform-config-store | at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_144]
platform-config-store | at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[na:1.8.0_144]
platform-config-store | at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) ~[zookeeper-3.4.6.jar!/:3.4.6-1569965]
platform-config-store | at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) ~[zookeeper-3.4.6.jar!/:3.4.6-1569965]
platform-config-store |
platform-config-store | 2018-03-05 11:55:13.422 INFO 1 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
platform-config-store | 2018-03-05 11:55:13.424 WARN 1 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
I've googled this problem and on some posts it was mentioned that zookeeper client server is not available that's why this error is occurring. So for this I've configured the zookeeper local instance on my machine and made changes in docker-compose.yml file. Instead of getting the image from docker, I tried to get it from local machine. It didn't work and faced the same issue.
Also some of them posted that this related to the firewall. I've verified and firewall's turned off.
Following is the docker-compose file I'm executing.
docker-compose.yml
version: "3.0"
services:
zookeeper:
container_name: zookeeper
image: docker.*****.net/zookeeper
#image: zookeeper // tired to connect with local zookeeper instance
ports:
- 2181:2181
postgres:
container_name: postgres
image: postgres
ports:
- 5432:5432
environment:
- POSTGRES_PASSWORD=p3rmission
redis:
container_name: redis
image: redis
ports:
- 6379:6379
Could anyone please guide me, what I'm missing here. Help will be appreciated. Thanks
I'm trying to setup the consul server and connect an agent to it for 2 or 3 days already. I'm using docker-compose.
But after performing a join operation, agent gets a message "Agent not live or unreachable".
Here are the logs:
root#e33a6127103f:/app# consul agent -join 10.1.30.91 -data-dir=/tmp/consul
==> Starting Consul agent...
==> Joining cluster...
Join completed. Synced with 1 initial agents
==> Consul agent running!
Version: 'v1.0.1'
Node ID: '0e1adf74-462d-45a4-1927-95ed123f1526'
Node name: 'e33a6127103f'
Datacenter: 'dc1' (Segment: '')
Server: false (Bootstrap: false)
Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, DNS: 8600)
Cluster Addr: 172.17.0.2 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
==> Log data will now stream in as it occurs:
2017/12/06 10:44:43 [INFO] serf: EventMemberJoin: e33a6127103f 172.17.0.2
2017/12/06 10:44:43 [INFO] agent: Started DNS server 127.0.0.1:8600 (udp)
2017/12/06 10:44:43 [INFO] agent: Started DNS server 127.0.0.1:8600 (tcp)
2017/12/06 10:44:43 [INFO] agent: Started HTTP server on 127.0.0.1:8500 (tcp)
2017/12/06 10:44:43 [INFO] agent: (LAN) joining: [10.1.30.91]
2017/12/06 10:44:43 [INFO] serf: EventMemberJoin: consul1 172.19.0.2 2017/12/06 10:44:43 [INFO] consul: adding server consul1 (Addr: tcp/172.19.0.2:8300) (DC: dc1)
2017/12/06 10:44:43 [INFO] agent: (LAN) joined: 1 Err: <nil>
2017/12/06 10:44:43 [INFO] agent: started state syncer
2017/12/06 10:44:43 [WARN] manager: No servers available
2017/12/06 10:44:43 [ERR] agent: failed to sync remote state: No known Consul servers
2017/12/06 10:44:54 [INFO] memberlist: Suspect consul1 has failed, no acks received
2017/12/06 10:44:55 [ERR] consul: "Catalog.NodeServices" RPC failed to server 172.19.0.2:8300: rpc error getting client: failed to get conn: dial tcp <nil>->172.19.0.2:8300: i/o timeout
2017/12/06 10:44:55 [ERR] agent: failed to sync remote state: rpc error getting client: failed to get conn: dial tcp <nil>->172.19.0.2:8300: i/o timeout
2017/12/06 10:44:58 [INFO] memberlist: Marking consul1 as failed, suspect timeout reached (0 peer confirmations)
2017/12/06 10:44:58 [INFO] serf: EventMemberFailed: consul1 172.19.0.2
2017/12/06 10:44:58 [INFO] consul: removing server consul1 (Addr: tcp/172.19.0.2:8300) (DC: dc1)
2017/12/06 10:45:05 [INFO] memberlist: Suspect consul1 has failed, no acks received
2017/12/06 10:45:06 [WARN] manager: No servers available
2017/12/06 10:45:06 [ERR] agent: Coordinate update error: No known Consul servers
2017/12/06 10:45:12 [WARN] manager: No servers available
2017/12/06 10:45:12 [ERR] agent: failed to sync remote state: No known Consul servers
2017/12/06 10:45:13 [INFO] serf: attempting reconnect to consul1 172.19.0.2:8301
2017/12/06 10:45:28 [WARN] manager: No servers available
2017/12/06 10:45:28 [ERR] agent: failed to sync remote state: No known Consul servers
2017/12/06 10:45:32 [WARN] manager: No servers available . `
My settings are:
docker-compose SERVER:
consul1:
image: "consul.1.0.1"
container_name: "consul1"
hostname: "consul1"
volumes:
- ./consul/config:/config/
ports:
- "8400:8400"
- "8500:8500"
- "8600:53"
- "8300:8300"
- "8301:8301"
command: "agent -config-dir=/config -ui -server -bootstrap-expect 1"
Help please solve the problem.
I think you using wrong ip-address of consul-server
"consul agent -join 10.1.30.91 -data-dir=/tmp/consul"
10.1.30.91 this is not docker container ip it might be your host address/virtualbox.
Get consul-container ip and use that to join in consul-agent command.
For more info about how consul and agent works follow the link
https://dzone.com/articles/service-discovery-with-docker-and-consul-part-1
Try to get the right IP address by executing this command:
docker inspect <container id> | grep "IPAddress"
Where the is the container ID of the consul server.
Than use the obtained address instead of "10.1.30.91" in the command
consul agent -join <IP ADDRESS CONSUL SERVER> -data-dir=/tmp/consul
I'm trying to get a consul server cluster up and running. I have 3 dockerized consul servers running, but I can't access the Web UI, the HTTP API nor the DNS.
$ docker logs net-sci_discovery-service_consul_1
==> WARNING: Expect Mode enabled, expecting 3 servers
==> Starting Consul agent...
==> Consul agent running!
Version: 'v0.8.5'
Node ID: 'ccd38897-6047-f8b6-be1c-2aa0022a1483'
Node name: 'consul1'
Datacenter: 'dc1'
Server: true (bootstrap: false)
Client Addr: 127.0.0.1 (HTTP: 8500, HTTPS: -1, DNS: 8600)
Cluster Addr: 172.20.0.2 (LAN: 8301, WAN: 8302)
Gossip encrypt: false, RPC-TLS: false, TLS-Incoming: false
==> Log data will now stream in as it occurs:
2017/07/07 23:24:07 [INFO] raft: Initial configuration (index=0): []
2017/07/07 23:24:07 [INFO] raft: Node at 172.20.0.2:8300 [Follower] entering Follower state (Leader: "")
2017/07/07 23:24:07 [INFO] serf: EventMemberJoin: consul1 172.20.0.2
2017/07/07 23:24:07 [INFO] consul: Adding LAN server consul1 (Addr: tcp/172.20.0.2:8300) (DC: dc1)
2017/07/07 23:24:07 [INFO] serf: EventMemberJoin: consul1.dc1 172.20.0.2
2017/07/07 23:24:07 [INFO] consul: Handled member-join event for server "consul1.dc1" in area "wan"
2017/07/07 23:24:07 [INFO] agent: Started DNS server 127.0.0.1:8600 (tcp)
2017/07/07 23:24:07 [INFO] agent: Started DNS server 127.0.0.1:8600 (udp)
2017/07/07 23:24:07 [INFO] agent: Started HTTP server on 127.0.0.1:8500
2017/07/07 23:24:09 [INFO] serf: EventMemberJoin: consul2 172.20.0.3
2017/07/07 23:24:09 [INFO] consul: Adding LAN server consul2 (Addr: tcp/172.20.0.3:8300) (DC: dc1)
2017/07/07 23:24:09 [INFO] serf: EventMemberJoin: consul2.dc1 172.20.0.3
2017/07/07 23:24:09 [INFO] consul: Handled member-join event for server "consul2.dc1" in area "wan"
2017/07/07 23:24:10 [INFO] serf: EventMemberJoin: consul3 172.20.0.4
2017/07/07 23:24:10 [INFO] consul: Adding LAN server consul3 (Addr: tcp/172.20.0.4:8300) (DC: dc1)
2017/07/07 23:24:10 [INFO] consul: Found expected number of peers, attempting bootstrap: 172.20.0.2:8300,172.20.0.3:8300,172.20.0.4:8300
2017/07/07 23:24:10 [INFO] serf: EventMemberJoin: consul3.dc1 172.20.0.4
2017/07/07 23:24:10 [INFO] consul: Handled member-join event for server "consul3.dc1" in area "wan"
2017/07/07 23:24:14 [ERR] agent: failed to sync remote state: No cluster leader
2017/07/07 23:24:17 [WARN] raft: Heartbeat timeout from "" reached, starting election
2017/07/07 23:24:17 [INFO] raft: Node at 172.20.0.2:8300 [Candidate] entering Candidate state in term 2
2017/07/07 23:24:17 [INFO] raft: Election won. Tally: 2
2017/07/07 23:24:17 [INFO] raft: Node at 172.20.0.2:8300 [Leader] entering Leader state
2017/07/07 23:24:17 [INFO] raft: Added peer 172.20.0.3:8300, starting replication
2017/07/07 23:24:17 [INFO] raft: Added peer 172.20.0.4:8300, starting replication
2017/07/07 23:24:17 [INFO] consul: cluster leadership acquired
2017/07/07 23:24:17 [INFO] consul: New leader elected: consul1
2017/07/07 23:24:17 [WARN] raft: AppendEntries to {Voter 172.20.0.3:8300 172.20.0.3:8300} rejected, sending older logs (next: 1)
2017/07/07 23:24:17 [WARN] raft: AppendEntries to {Voter 172.20.0.4:8300 172.20.0.4:8300} rejected, sending older logs (next: 1)
2017/07/07 23:24:17 [INFO] raft: pipelining replication to peer {Voter 172.20.0.3:8300 172.20.0.3:8300}
2017/07/07 23:24:17 [INFO] raft: pipelining replication to peer {Voter 172.20.0.4:8300 172.20.0.4:8300}
2017/07/07 23:24:18 [INFO] consul: member 'consul1' joined, marking health alive
2017/07/07 23:24:18 [INFO] consul: member 'consul2' joined, marking health alive
2017/07/07 23:24:18 [INFO] consul: member 'consul3' joined, marking health alive
2017/07/07 23:24:20 [INFO] agent: Synced service 'consul'
2017/07/07 23:24:20 [INFO] agent: Synced service 'messaging-service-kafka'
2017/07/07 23:24:20 [INFO] agent: Synced service 'messaging-service-zookeeper'
$ curl http://127.0.0.1:8500/v1/catalog/service/consul
curl: (52) Empty reply from server
dig #127.0.0.1 -p 8600 consul.service.consul
; <<>> DiG 9.8.3-P1 <<>> #127.0.0.1 -p 8600 consul.service.consul
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached
$ dig #127.0.0.1 -p 8600 messaging-service-kafka.service.consul
; <<>> DiG 9.8.3-P1 <<>> #127.0.0.1 -p 8600 messaging-service-kafka.service.consul
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached
I can't get my services to register via the HTTP API either; those shown above are registered using a config script when the container launches.
Here's my docker-compose.yml:
version: '2'
services:
consul1:
image: "consul:latest"
container_name: "net-sci_discovery-service_consul_1"
hostname: "consul1"
ports:
- "8400:8400"
- "8500:8500"
- "8600:8600"
volumes:
- ./etc/consul.d:/etc/consul.d
command: "agent -server -ui -bootstrap-expect 3 -config-dir=/etc/consul.d -bind=0.0.0.0"
consul2:
image: "consul:latest"
container_name: "net-sci_discovery-service_consul_2"
hostname: "consul2"
command: "agent -server -join=consul1"
links:
- "consul1"
consul3:
image: "consul:latest"
container_name: "net-sci_discovery-service_consul_3"
hostname: "consul3"
command: "agent -server -join=consul1"
links:
- "consul1"
I'm relatively new to both docker and consul. I've had a look around the web and the above options are my understanding of what is required. Any suggestions on the way forward would be very welcome.
Edit:
Result of docker container ps -all:
$ docker container ps --all
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e0a1c3bba165 consul:latest "docker-entrypoint..." 38 seconds ago Up 36 seconds 8300-8302/tcp, 8500/tcp, 8301-8302/udp, 8600/tcp, 8600/udp net-sci_discovery-service_consul_3
7f05555e81e0 consul:latest "docker-entrypoint..." 38 seconds ago Up 36 seconds 8300-8302/tcp, 8500/tcp, 8301-8302/udp, 8600/tcp, 8600/udp net-sci_discovery-service_consul_2
9e2dedaa224b consul:latest "docker-entrypoint..." 39 seconds ago Up 38 seconds 0.0.0.0:8400->8400/tcp, 8301-8302/udp, 0.0.0.0:8500->8500/tcp, 8300-8302/tcp, 8600/udp, 0.0.0.0:8600->8600/tcp net-sci_discovery-service_consul_1
27b34c5dacb7 messagingservice_kafka "start-kafka.sh" 3 hours ago Up 3 hours 0.0.0.0:9092->9092/tcp net-sci_messaging-service_kafka
0389797b0b8f wurstmeister/zookeeper "/bin/sh -c '/usr/..." 3 hours ago Up 3 hours 22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp net-sci_messaging-service_zookeeper
Edit:
Updated docker-compose.yml to include long format for ports:
version: '3.2'
services:
consul1:
image: "consul:latest"
container_name: "net-sci_discovery-service_consul_1"
hostname: "consul1"
ports:
- target: 8400
published: 8400
mode: host
- target: 8500
published: 8500
mode: host
- target: 8600
published: 8600
mode: host
volumes:
- ./etc/consul.d:/etc/consul.d
command: "agent -server -ui -bootstrap-expect 3 -config-dir=/etc/consul.d -bind=0.0.0.0 -client=127.0.0.1"
consul2:
image: "consul:latest"
container_name: "net-sci_discovery-service_consul_2"
hostname: "consul2"
command: "agent -server -join=consul1"
links:
- "consul1"
consul3:
image: "consul:latest"
container_name: "net-sci_discovery-service_consul_3"
hostname: "consul3"
command: "agent -server -join=consul1"
links:
- "consul1"
From the Consul Web Gui page, make sure you have launched an agent with the -ui parameter.
The UI is available at the /ui path on the same port as the HTTP API.
By default this is http://localhost:8500/ui
I do see 8500 mapped to your host on broadcast (0.0.0.0).
Check also (as in this answer) if the client_addr can help (at least for testing)