I'm trying to make a rest request through restTemplate between Spring Boot microservices in docker, but I get an error.
docker-compose.yml:
api:
image: api-service
container_name: api-service
restart: always
depends_on:
- product
ports:
- 8081:8080
links:
- product:product
environment:
- SERVICE_PORT_PRODUCT=8083
product:
image: product-service
container_name: product-service
restart: always
ports:
- 8083:8080
Exception log:
ERROR 1 --- [nio-8080-exec-1] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.web.client.ResourceAccessException: I/O error on POST request for "http://product:8083/api/products/": Connection refused (Connection refused); nested exception is java.net.ConnectException: Connection refused (Connection refused)] with root cause
java.net.ConnectException: Connection refused (Connection refused)
looks correct:
POST request for "http://product:8083/api/products/"
why does not it work?
As documented in the official docs for Networking in Compose
Networked service-to-service communication use the CONTAINER_PORT
Thus when you want to do a request from one container to the other, you need use the container port and not the host port.
The request should be to: http://product:8080/api/products/ from the api container to the product container.
Related
I am trying to run Activiti Cloud example in docker.
I am using this official tutorial:
https://activiti.gitbook.io/activiti-7-developers-guide/getting-started/getting-started-activiti-cloud/getting-started-docker-compose
All the prior steps succeeded till I try
"To start work, execute getKeycloakToken hruser in Postman Keycloak collection. Then run startProcess in rb-my-app Postman collection."
I succeeded to execute getKeycloakToken hruser in Postman Keycloak collection.
I failed to run startProcess in rb-my-app Postman collection.
I got 404 when I execute getModels in Postman modeling collection.
I got 500 when I execute startProcess in Postman rb collection.
The major error message in logs:
example-runtime-bundle | 2022-03-08 19:08:30.970 WARN [rb,,] 7 --- [nio-8080-exec-1] o.keycloak.adapters.KeycloakDeployment : Failed to load URLs from http://127.0.0.1.nip.io/auth/realms/activiti/.well-known/openid-configuration
example-runtime-bundle |
example-runtime-bundle | java.net.ConnectException: Connection refused (Connection refused)
The related configurations:
in docker-compose.yml:
keycloak:
container_name: keycloak
image: activiti/activiti-keycloak
volumes:
- ./activiti-realm.json:/opt/jboss/keycloak/activiti-realm.json
restart: unless-stopped
depends_on:
- nginx
example-runtime-bundle:
container_name: example-runtime-bundle
image: activiti/example-runtime-bundle:${VERSION}
environment:
# JAVA_OPTS: "-Xdebug -Xrunjdwp:server=y,transport=dt_socket,address=8000,suspend=n -noverify"
SPRING_JMX_ENABLED: "false"
ACT_KEYCLOAK_URL: "http://${DOCKER_IP}/auth"
SPRING_RABBITMQ_HOST: "rabbitmq"
SERVER_SERVLET_CONTEXT_PATH: /rb
SPRING_DATASOURCE_URL: jdbc:postgresql://activiti-postgres:5432/activitidb
SPRING_DATASOURCE_USERNAME: activiti
SPRING_DATASOURCE_PASSWORD: mypassword
SPRING_JPA_DATABASE_PLATFORM: org.hibernate.dialect.PostgreSQLDialect
SPRING_JPA_GENERATE_DDL: "true"
SPRING_JPA_HIBERNATE_DDL_AUTO: update
# ACTIVITI_SECURITY_POLICIES_0_NAME: "HR Group restricted to SimpleProcess and ConnectorProcess"
# ACTIVITI_SECURITY_POLICIES_0_GROUPS: "hr"
# ACTIVITI_SECURITY_POLICIES_0_ACCESS: "WRITE"
# ACTIVITI_SECURITY_POLICIES_0_SERVICENAME: "rb-my-app"
# ACTIVITI_SECURITY_POLICIES_0_KEYS: "SimpleProcess,ConnectorProcess,fixSystemFailure,twoTaskProcess"
# ACTIVITI_SECURITY_POLICIES_1_NAME: "testgroup not restricted at all"
# ACTIVITI_SECURITY_POLICIES_1_GROUPS: "testgroup"
# ACTIVITI_SECURITY_POLICIES_1_ACCESS: "WRITE"
# ACTIVITI_SECURITY_POLICIES_1_SERVICENAME: "rb-my-app"
# ACTIVITI_SECURITY_POLICIES_1_KEYS: "*"
restart: unless-stopped
depends_on:
- nginx
- keycloak
- rabbitmq
- activiti-postgres
in .env:
DOCKER_IP=127.0.0.1.nip.io
VERSION=7.1.0-M13
KEYCLOAK_REALM=activiti
KEYCLOAK_RESOURCE=activiti
Full logs output is here:
https://gist.github.com/chang4tech/affd504809249733ee1f553da1d03763
What am I supposed to do to debug/detect the problem and eliminate the errors?
Thanks.
I'm trying to load balance a simple Nodejs app with 3 instances using docker-compose & nginx. This configuration works on my local machine (windows laptop) but doesn't seem to work on EC2 server.
nginx.conf
http {
upstream all {
server nodeapp1:4100;
server nodeapp2:4200;
server nodeapp3:4300;
}
server {
listen 8080;
location / {
proxy_pass http://all/;
}
}
}
events { }
docker-compose.yml
version: '3'
services:
lb:
image: nginx
volumes:
- ./nginxproxy/nginx.conf:/etc/nginx/nginx.conf
ports:
- "3000:8080"
nodeapp1:
image: nodeapp
environment:
- PORT=4100
ports:
- "4100:4100"
nodeapp2:
image: nodeapp
environment:
- PORT=4200
ports:
- "4200:4200"
nodeapp3:
image: nodeapp
environment:
- PORT=4300
ports:
- "4300:4300"
I'm new to docker. I'm surprised why this works locally but does not work on EC2 instance. The load balancer was able to resolve the url correctly but it still says connection refused.
Error:
2022/02/28 20:00:22 [error] 33#33: *9 connect() failed (111: Connection refused) while
connecting to upstream, client: 62.113.237.40, server: , request: "GET / HTTP/1.1",
upstream: "http://172.121.0.5:4100/", host: "18.121.121.23:3000"
For me service name or ip address not worked, only work put the gateway IP of network, for default bridge is 172.17.0.1.
In the servers put the (gateway ip):(port of container) and with this haproxy connects with success.
My example of custom network with fixed ips and gateway:
---- nginx config
upstream loadbalancer {
server 172.17.0.1:8001 weight=5;
server 172.17.0.1:8002 weight=5;
}
----- haproxy config similar
backend be_pe_8545
mode http
balance roundrobin
server p1 172.20.0.254:18545 check inter 10s
server p2 172.20.0.254:28545 check inter 10s
----- docker app / network
docker_app: ...
networks:
public_network:
ipv4_address: 172.20.0.50
public_network:
name: public_network
driver: bridge
ipam:
driver: default
config:
- subnet: 172.20.0.0/24
gateway: 172.20.0.254
I have 2 Docker containers running in the same network and I want 1 of them to call another via spring Webclient.
I'm sure they all are in the same network -> docker network inspect <network_ID> proves this.
AFAIK I can ping one container from another to check if they can talk to each other by docker exec -ti attachment-loader-prim ping attachment-loader-sec
If I run this - I see responses from attachment-loader-sec like 64 bytes from 172.21.0.5: seq=0 ttl=64 time=0.220 ms, which means they can communicate.
When I send Postman request to attachment-loader-prim by its exposed port localhost:8085, I expect that after some business logic it calls for attachment-loader-sec via Webclient, but on that step I get a 500 error with such a message:
"finishConnect(..) failed: Connection refused:
attachment-loader-sec/172.21.0.5:80; nested exception is
io.netty.channel.AbstractChannel$AnnotatedConnectException:
finishConnect(..) failed: Connection refused:
attachment-loader-sec/172.21.0.5:80"
Both attachment-loader-prim and attachment-loader-sec can be accessed separately via postman and both send a response, no problem.
This is my docker-compose:
version: '3'
services:
attachment-loader-prim:
container_name: attachment-loader-prim
build:
context: ""
restart: always
image: attachment-loader:latest
environment:
SERVER_PORT: 8085
networks:
- loader_network
expose:
- 8085
ports:
- 8005:8005
- 8085:8085
attachment-loader-sec:
container_name: attachment-loader-sec
build:
context: ""
restart: always
image: attachment-loader:latest
environment:
SERVER_PORT: 8086
networks:
- loader_network
expose:
- 8086
ports:
- 8006:8005
- 8086:8086
networks:
loader_network:
driver: bridge
And this is a Webclient which makes a call:
class RemoteServiceCaller(private val fetcherWebClientBuilder: WebClient.Builder) {
suspend fun getAttachmentsFromRemote(id: String, params: List<Param>, username: String): Result? {
val client = fetcherWebClientBuilder.build()
val awaitExchange = client.post()
.uri("/{id}/attachment", id)
.contentType(MediaType.APPLICATION_JSON)
.bodyValue(params)
.header(usernameHeader, username)
.accept(MediaType.APPLICATION_OCTET_STREAM)
.awaitExchange {
if (it.statusCode().is2xxSuccessful) {
handleSucessCode(it)
} else it.createExceptionAndAwait().run {
LOG.error(this.responseBodyAsString, this)
throw ProcessingException(this)
}
}
return awaitExchange
}
private suspend fun handleSucessCode(response: ClientResponse) {
// some not important logic
}
}
P.S. BasicUri for Webclient defined as Config Bean like http://attachment-loader-sec/list
All my investigations brought me to such problems as:
Calling container using localhost instead of container name
Containers are not in the same network.
All that seems not relevant for me.
Any ideas will be really appreciated.
The problem was in calling a service without its port. The url became now http://attachment-loader-sec:8086/list and it is correct now. In my case I get 404, which means that my url path is not quite correct, but that is outside of current question
I am having troubles with microservice health checks in my consul docker setup, which i believe is a symptom of failure in service discovery as i only have one server in my registry.
Below is consul list of members from inside the docker container.
/ # consul members
Node Address Status Type Build Protocol DC Segment
7b1edb14a647 172.19.0.6:8301 alive server 1.7.4 2 dc1 <all>
/ #
Consul container logs repeat the same error below for all the microservices:
consul | 2020-06-16T12:19:11.087Z [WARN] agent: Check socket connection failed: check=service:ffa44b66c4869601c04abdbea6dc5be5 error="dial tcp 172.19.0.6:50044: connect: connection refused"
I am using docker-compose v.3.2 to create a network for containers.
This is a consul service definition
consul:
container_name: consul
ports:
- '8400:8400'
- '8500:8500'
- '8600:53/udp'
image: consul
command: ['agent', '-server', '-bootstrap', '-ui', '-client', '0.0.0.0']
Microservice definition
service-notification:
build:
context: .
dockerfile: apps/service-notification/Dockerfile
args:
NODE_ENV: development
depends_on:
- consul
image: 'service-notification:latest'
restart: always
environment:
- CONSUL_HOST=consul
ports:
- '50044:50044'
I am using CONSUL_HOST env variable to pass in correct host url.
Consul config for the microservice
consul:
host: ${{CONSUL_HOST}}
port: 8500
service:
discoveryHost: ${{CONSUL_HOST}}
healthCheck:
timeout: 1s
interval: 10s
tcp: ${{ service.discoveryHost }}:${{ service.port }}
maxRetry: 5
retryInterval: 5000
tags: ["v1.0.0", "microservice"]
name: io.ultimatebackend.srv.notification
port: 50044
My conclusion so far is that consul server container fails to reach the agents somehow. But i don't know why and i feel like i am missing some obvious peace of consul structure. Please advise.
I was incorrectly configuring my service. The dicoveryHost should be an IP and port of a micro-service inside docker network.
I'm testing a sample spring cloud stream application (running on a Ubuntu linux machine) with one source and one sink services. All my services are docker-containerized and I would like to use kafka as message broker.
Below the relevant parts of the docker-compose.yml:
zookeeper:
image: confluent/zookeeper
container_name: zookeeper
ports:
- "2181:2181"
kafka:
image: wurstmeister/kafka:0.9.0.0-1
container_name: kafka
ports:
- "9092:9092"
links:
- zookeeper:zk
environment:
- KAFKA_ADVERTISED_HOST_NAME=192.168.33.101
- KAFKA_ADVERTISED_PORT=9092
- KAFKA_DELETE_TOPIC_ENABLE=true
- KAFKA_LOG_RETENTION_HOURS=1
- KAFKA_MESSAGE_MAX_BYTES=10000000
- KAFKA_REPLICA_FETCH_MAX_BYTES=10000000
- KAFKA_GROUP_MAX_SESSION_TIMEOUT_MS=60000
- KAFKA_NUM_PARTITIONS=2
- KAFKA_DELETE_RETENTION_MS=1000
.
.
.
# not shown: eureka service registry, spring cloud config service, etc.
myapp-service-test-source:
container_name: myapp-service-test-source
image: myapp-h2020/myapp-service-test-source:0.0.1
environment:
SERVICE_REGISTRY_HOST: 192.168.33.101
SERVICE_REGISTRY_PORT: 8761
ports:
- 8081:8080
.
.
.
Here the relevant part of application.yml for my service-test-source service:
spring:
cloud:
stream:
defaultBinder: kafka
bindings:
output:
destination: messages
content-type: application/json
kafka:
binder:
brokers: ${SERVICE_REGISTRY_HOST:192.168.33.101}
zkNodes: ${SERVICE_REGISTRY_HOST:192.168.33.101}
defaultZkPort: 2181
defaultBrokerPort: 9092
The problem is the following, if I launch the docker-compose above, in the test-source container log I notice that the service fails to connect to zookeeper, giving a repeated set of Connection refused error, and finishing with a ZkTimeoutException which makes the service terminate (see below).
The strange fact is that, if instead of running my source (and sink) test services as docker containers I run them as jar files via maven mvn spring-boot:run <etc...> the services work fine and are able to exchange messages via kafka. (note that kafka, zookeeper, etc. are still running as docker containers).
.
.
.
*** THE FOLLOWING REPEATED n TIMES ***
2017-02-14 14:40:09.164 INFO 1 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2017-02-14 14:40:09.166 WARN 1 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_111]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[na:1.8.0_111]
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) ~[zookeeper-3.4.6.jar!/:3.4.6-1569965]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) ~[zookeeper-3.4.6.jar!/:3.4.6-1569965]
.
.
.
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:53)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.springframework.context.ApplicationContextException: Failed to start bean 'outputBindingLifecycle'; nested exception is org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
Any idea what the problem might be?
edit:
I discovered that in the "jar" execution logs the test-source service tries to connect to zookeeper through the IP 127.0.0.1, as can be seen from the log snipped below:
2017-02-15 14:24:04.159 INFO 10348 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2017-02-15 14:24:04.159 INFO 10348 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2017-02-15 14:24:04.178 INFO 10348 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Socket connection established to localhost/127.0.0.1:2181, initiating session
2017-02-15 14:24:04.201 INFO 10348 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x15a421fd9ec000a, negotiated timeout = 10000
2017-02-15 14:24:05.870 INFO 10348 --- [ main] org.apache.zookeeper.ZooKeeper : Initiating client connection, connectString=localhost:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient#72ba68e3
2017-02-15 14:24:05.882 INFO 10348 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
2017-02-15 14:24:05.883 INFO 10348 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Socket connection established to localhost/0:0:0:0:0:0:0:1:2181, initiating session
This explains why everything works on the jar execution but not the docker one (the zookeeper container exports its 2181 port to the host machine, so it's visible as localhost for the service process when running directly on the host machine), but doesn't solve the problem: Apparently the spring cloud stream kafka configuration is ignoring the property spring.cloud.stream.kafka.binder.zkNodes as set in the application.yml (note that if I log the value of such environment variable from the service, I see the correct value of 192.168.33.101 that I hardcoded there for debugging purposes).
You have set the defaultBinder to be rabbit while trying to use the Kafka binder configuration. Do you have both rabbit and kafka binders in the classpath of your application? In that case, you can enable here
zookeeper:
image: wurstmeister/zookeeper
container_name: 'zookeeper'
ports:
- 2181:2181
--------------------- kafka --------------------------------
kafka:
image: wurstmeister/kafka
container_name: 'kafka'
environment:
- KAFKA_ADVERTISED_HOST_NAME=kafka
- KAFKA_ADVERTISED_PORT=9092
- KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
- KAFKA_CREATE_TOPICS=kafka_docker_topic:1:1
ports:
- 9092:9092
depends_on:
- zookeeper
spring:
profiles: dev
cloud:
stream:
defaultBinder: kafka
kafka:
binder:
brokers: kafka # i added brokers and zkNodes property
zkNodes: zookeeper #
bindings:
input:
destination: message
content-type: application/json