How docker HTTP client container send HTTP request on start - docker

I have a docker-compose file which defines an HTTP server and client as follows
version: '3'
services:
serverA:
container_name: serverA
image: xxx/apiserver
restart: unless-stopped
ports:
- '9090:9090'
networks:
- apinet
clientA:
container_name: clientA
image: xxx/apiclient
restart: unless-stopped
ports:
- '20005:20005'
depends_on:
- serverA
networks:
- apinet
networks:
apinet:
driver: bridge
The server image contain a simple golang HTTP server code ready to handle request
func main() {
http.HandleFunc("/itemhandler", headers)
http.ListenAndServe(":9090", nil)
}
func itemhandler(w http.ResponseWriter, req *http.Request) {
//handle http request
}
while the client image contain a function SendhttpPOSTRequest which contain an HTTP client module that sends HTTP request to the server and retry until it is able to send the request to the server. The client image when run as container contain a configuration file config.toml file which contain the IP address and PORT of the server.
//
func main() {
SendhttpPOSTRequest() // sends http post request with retry
}
The problem is when start the docker-compose file, both container get created and running however the client keeps sending the HTTP request but could not reach the server even though in the config.toml I have changed the server information as shown below and restart the client container. I have used the server IP address obtained from the apinet network as well and restart the client container but still could not send the HTTP request.
config.toml:
serverIP = "serverA" or serverIP = "172.x.0.x"
serverPort = "9090"
A ping between serverA and ClientA using their name works, eg ping serverA from clientA and vise versa works. HTTP request from the host machine to the server works but the HTTP request from the client does not get to the server. An HTTP request to the server inside the client container works but that is not what I want. What I want is that the client sends HTTP request (executes the SendhttpPOSTRequest() ) to the server when its image container starts and retry until the server is up and running.
I have search stackoverflow but could not get similar problem. Can anyone help me. Am new to docker.

Related

How to fix 502 Bad Gateway error in nginx?

I have a server running docker-compose. Docker-compose has 2 services: nginx (as a reverse proxy) and back (as an api that handles 2 requests). In addition, there is a database that is not located on the server, but separately (database as a service).
Requests processed by back:
get('/api') - the back service simply replies "API is running" to it
get('/db') - the back service sends a simple query to an external database ('SELECT random() as random, current_database() as db')
request 1 - works fine, request 2 - the back service crashes, nginx continues to work and a 502 Bad Gateway error appears in the console.
An error occurs in the nginx service Logs: upstream prematurely
closed connection while reading response header from upstream.
The back service Logs: connection terminated due to connection timeout.
These are both rather vague errors. And I don’t know how else to get close to them, given that the code is not in a container, without Nginx and with the same database, it works as it should.
What I have tried:
increase the number of cores and RAM (now 2 cores and 4 GB of Ram);
add/remove/change proxy_read_timeout, proxy_send_timeout and proxy_connect_timeout parameters;
test the www.test.com/db request via postman and curl (fails with the same error);
run the code on your local machine without a container and compose and connect to the same database using the same pool and the same ip (everything is ok, both requests work and send what you need);
change the parameter worker_processes (tested with a value of 1 and auto);
add/remove attribute proxy_set_header Host $http_host, replace $http_host with "www.test.com".
Question:
What else can I try to fix the error and make the db request work?
My nginx.conf:
worker_processes 1;
events {
worker_connections 1024;
}
http{
upstream back-stream {
server back:8080;
}
server {
listen 80;
listen [::]:80;
server_name test.com www.test.com;
location / {
root /usr/share/nginx/html;
resolver 121.0.0.11;
proxy_pass http://back-stream;
}
}
}
My docker-compose.yml:
version: '3.9'
services:
nginx-proxy:
image: nginx:stable-alpine
container_name: nginx-proxy
ports:
- 80:80
- 443:443
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf
networks:
- network
back:
image: "mycustomimage"
container_name: back
restart: unless-stopped
ports:
- '81:8080'
networks:
- network
networks:
network:
driver: bridge
I can upload other files if needed. Just taking into account the fact that the code does not work correctly in the container, the problem is rather in setting up the container.
I will be grateful for any help.
Code of the back: here
The reason for the error is this: I forgot to add my server's ip to the list of allowed addresses in the database cluster.

Docker Registry behind HAProxy not working properly

I have configured Docker Registry using htpasswd authentication as docker service. I use a port mapping 443:5000 and it works perfectly. "docker login :443" works, "docker pull <mydomain:433/myimage" works.
The container service looks like this:
myhub:
image: myhub
ports:
- "443:5000"
environment:
- REGISTRY_HTTP_ADDR=0.0.0.0:5000
- REGISTRY_HTTP_TLS_KEY=/certs/efimeridopolis.privkey.pem
- REGISTRY_HTTP_TLS_CERTIFICATE=/certs/efimeridopolis.fullchain.pem
- REGISTRY_AUTH=htpasswd
- REGISTRY_AUTH_HTPASSWD_REALM='Registry Realm'
- REGISTRY_AUTH_HTPASSWD_PATH=/auth/registry.htpasswd
volumes:
- /mnt/volume1/hub:/var/lib/registry
networks:
- myoverlay
Now, I try to put it behind HAProxy. SSL is terminated at HAProxy, so the service looks like:
myhub:
image: myhub
environment:
- REGISTRY_HTTP_ADDR=0.0.0.0:5000
- REGISTRY_AUTH=htpasswd
- REGISTRY_AUTH_HTPASSWD_REALM='Registry Realm'
- REGISTRY_AUTH_HTPASSWD_PATH=/auth/registry.htpasswd
volumes:
- /mnt/volume1/hub:/var/lib/registry
networks:
- myoverlay
and the relevant part of HAProxy configuration is:
frontend fe_443
mode http
bind *:443 ssl crt /etc/ssl/private/mydomain.pem
http-request add-header X-Forwarded-Proto https if { ssl_fc }
acl host_registry hdr(host) -i hub.mydomain
use_backend be_registry_443 if host_registry
backend be_registry_443
mode http
option forwardfor
server hub1 myhub:5000 check
It seems something is going wrong here. While accessing
https://hub.mydomain/v2/_catalog
through browser works, which means I am asked for username/password and then I get the list or repositories, when I try to use the console to:
$ docker pull hub.mydomain:443/v2/myhaproxy
it gives me:
Using default tag: latest
Error response from daemon: received unexpected HTTP status: 503 Service Unavailable
The same when I try:
$ docker login mydomain:443
I am asked username and password but then I get the same 503 message.
Since browser can list the repositories, I know the registry is online and accessible.
What is wrong ?
https (443 port) should be tcp mode and using req.ssl_sni instead hdr(host).

Docker container can't reach another container using container name

I have 2 Docker containers running in the same network and I want 1 of them to call another via spring Webclient.
I'm sure they all are in the same network -> docker network inspect <network_ID> proves this.
AFAIK I can ping one container from another to check if they can talk to each other by docker exec -ti attachment-loader-prim ping attachment-loader-sec
If I run this - I see responses from attachment-loader-sec like 64 bytes from 172.21.0.5: seq=0 ttl=64 time=0.220 ms, which means they can communicate.
When I send Postman request to attachment-loader-prim by its exposed port localhost:8085, I expect that after some business logic it calls for attachment-loader-sec via Webclient, but on that step I get a 500 error with such a message:
"finishConnect(..) failed: Connection refused:
attachment-loader-sec/172.21.0.5:80; nested exception is
io.netty.channel.AbstractChannel$AnnotatedConnectException:
finishConnect(..) failed: Connection refused:
attachment-loader-sec/172.21.0.5:80"
Both attachment-loader-prim and attachment-loader-sec can be accessed separately via postman and both send a response, no problem.
This is my docker-compose:
version: '3'
services:
attachment-loader-prim:
container_name: attachment-loader-prim
build:
context: ""
restart: always
image: attachment-loader:latest
environment:
SERVER_PORT: 8085
networks:
- loader_network
expose:
- 8085
ports:
- 8005:8005
- 8085:8085
attachment-loader-sec:
container_name: attachment-loader-sec
build:
context: ""
restart: always
image: attachment-loader:latest
environment:
SERVER_PORT: 8086
networks:
- loader_network
expose:
- 8086
ports:
- 8006:8005
- 8086:8086
networks:
loader_network:
driver: bridge
And this is a Webclient which makes a call:
class RemoteServiceCaller(private val fetcherWebClientBuilder: WebClient.Builder) {
suspend fun getAttachmentsFromRemote(id: String, params: List<Param>, username: String): Result? {
val client = fetcherWebClientBuilder.build()
val awaitExchange = client.post()
.uri("/{id}/attachment", id)
.contentType(MediaType.APPLICATION_JSON)
.bodyValue(params)
.header(usernameHeader, username)
.accept(MediaType.APPLICATION_OCTET_STREAM)
.awaitExchange {
if (it.statusCode().is2xxSuccessful) {
handleSucessCode(it)
} else it.createExceptionAndAwait().run {
LOG.error(this.responseBodyAsString, this)
throw ProcessingException(this)
}
}
return awaitExchange
}
private suspend fun handleSucessCode(response: ClientResponse) {
// some not important logic
}
}
P.S. BasicUri for Webclient defined as Config Bean like http://attachment-loader-sec/list
All my investigations brought me to such problems as:
Calling container using localhost instead of container name
Containers are not in the same network.
All that seems not relevant for me.
Any ideas will be really appreciated.
The problem was in calling a service without its port. The url became now http://attachment-loader-sec:8086/list and it is correct now. In my case I get 404, which means that my url path is not quite correct, but that is outside of current question

Traefik v2 reverse proxy to a local server outside Docker

I have a simple server written in Python that listens on port 8000 inside a private network (HTTP communication). There is now a requirement to switch to HTTPS communications and every client that sends a request to the server should get authenticated with his own cert/key pair.
I have decided to use Traefik v2 for this job. Please see the block diagram.
Traefik runs as a Docker image on a host that has IP 192.168.56.101. First I wanted to simply forward a HTTP request from a client to Traefik and then to the Python server running outside Docker on port 8000. I would add the TLS functionality when the forwarding is running properly.
However, I can not figure out how to configure Traefik to reverse proxy from i.e. 192.168.56.101/notify?wrn=1 to the Python server 127.0.0.1:8000/notify?wrn=1.
When I try to send the above mentioned request to the server (curl "192.168.56.101/notify?wrn=1") I get "Bad Gateway" as an answer. What am I missing here? This is the first time that I am in contact with Docker and reverse proxy/Traefik. I believe it has something to do with ports but I can not figure it out.
Here is my Traefik configuration:
docker-compose.yml
version: "3.3"
services:
traefik:
image: "traefik:v2.1"
container_name: "traefik"
hostname: "traefik"
ports:
- "80:80"
- "8080:8080"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock:ro"
- "./traefik.yml:/traefik.yml:ro"
traefik.yml
## STATIC CONFIGURATION
log:
level: INFO
api:
insecure: true
dashboard: true
entryPoints:
web:
address: ":80"
providers:
docker:
watch: true
endpoint: "unix:///var/run/docker.sock"
file:
filename: "traefik.yml"
## DYNAMIC CONFIGURATION
http:
routers:
to-local-ip:
rule: "Host(`192.168.56.101`)"
service: to-local-ip
entryPoints:
- web
services:
to-local-ip:
loadBalancer:
servers:
- url: "http://127.0.0.1:8000"
First, 127.0.0.1 will resolve to the traefik container and not to the docker host. You need to provide a private IP of the node and it needs to be accessible form the traefik container.
There is some workaround to make proxy to localhost:
change 127.0.0.1 to IP of docker0 interface
It should be 172.17.0.1
and then try to listen your python server on all interfaces (0.0.0.0)
if you use simple python http server nothing change... on default it listen on all interfaces

Setting up more than one MQTT broker with Docker

Using Docker, I was able to use eclipse-mosquitto to set up an MQTT broker with my app, which subscribes to messages. I'm learning Docker right now, so wanted to try adding two brokers to Docker-compose with different ports mapped like this:
version: '3'
services:
myapp:
...
links:
- mqtt
- mqtt2
depends_on:
- mqtt
- mqtt2
mqtt:
image: eclipse-mosquitto:latest
container_name: mqtt-iot
ports:
- 1883:1883
mqtt2:
image: eclipse-mosquitto:latest
container_name: mqtt2-iot
ports:
- 1884:1883
From outside of the myapp container (i.e. from my OS X terminal), both mqtt and mqtt2 are working; I can publish and subscribe to messages as expected.
const mqtt = require('mqtt')
mqtt.connect('mqtt://mqtt', {port: 1883}) // Success
mqtt.connect('mqtt://mqtt2', {port: 1884}) // Success
However, when I'm inside the container of myapp, I can only connect to mqtt. mqtt2 connection fires the offline event right away, and no connection fails. What do I need to do to for myapp to be using both of those brokers properly?
Two issues here
links:
- mqtt
- mqtt2
Links is deprecated now and is not even required in your compose. Next when you use below
const mqtt = require('mqtt')
mqtt.connect('mqtt://mqtt', {port: 1883}) // Success
mqtt.connect('mqtt://mqtt2', {port: 1884}) // Success
From outside. This is based on the ports on the host. When you do it from app container you should do it like below
const mqtt = require('mqtt')
mqtt.connect('mqtt://mqtt', {port: 1883}) // Success
mqtt.connect('mqtt://mqtt2', {port: 1883}) // Success
The container cannot see mapped port on host. It will see what is inside the network. And in local network both are listen on 1883

Resources