Container always reach same backend on replicated services - docker-swarm

I'm deploying a 3 tier application using docker swarm, similar to:
--> BACK01-01 -- --> BACK02-01
| | |
FRONTEND-01 ----------------> BACK01-02 --------> BACK02-02
| | |
--> BACK01-03 -- --> BACK02-03
Frontend Back Service 01 Back Service 02
This is a 3 node swarm, where each *-01 service task is running on the manager-node, each *-02 service task is running on worker-node-01 and each *-03 service task is running on worker-node-02
All communication between services are using GRPC and creating a new connection per request.
All I want with this is to distribute the load over every replica.
Sequentially I made a request to frontend which make a request to back01, which make a request to back02. But after 50 requests, all inner requests where made to back01-03 and back02-03 and the other were never reached.
I using default service configuration and the stack was deployed using portainer GUI
Is there anything that I'm not understanding?
P.S: I had tested service load balance with a simple HTTP and GRPC server returning the container id, with 4 replicas in one node, and it was returning each one sequentially.

Related

docker hangs on internal connection loopback

Kind of new to docker but have a setup with multiple microservices in docker using FastAPI each with their own Postgres DB. Have an issue where I am sending an API request to microservice A, which calls an internal microservice B which works fine, the issue arises when I make an API request to microservice A, which calls microservice B, which sends another API request to back to microservice A to completely unrelated endpoint to fetch some other data type that only exists there, only the connection just hangs on "Starting new HTTP connection" in the docker logs. Not sure where to start to look on how to debug this, any help appreciated.
Tried making multiple simultaneous connections to microservice A, no issues there.

Cannot connect to local MQTT server (running on Windows 10) from docker instance

RabbitMQ server is running locally on Windows 10 and docker is running on it also.
I'm running a device simulator on docker and it has to talk to local RabbitMQ server through MQTT.
It had been working but one day it stopped.
Here is device logging -
mqtt-client.cpp:322 | Failed to connect to broker at 'xxx#xxx.xxxxxx.com/:1883': code=15, message='Lookup error.'
Keep in mind that from docker(latest version) calls have been made to local web server which has exact domain name -
https-commissioning-channel.cpp:81 | [HttpsCommissioningChannel] using token to contact bootstrap service at 'https://xxx.xxxxxx.com/apibst/alo/v1/bootstrap/device-info'
So you can see domain name has been resolved. For firewall configuration port is open on 1883 (consider it had been working). RabbitMQ is running.
What might be the issue and what should I do to make the call go through?

As per the comments xxx#xxx.xxxxxx.com/:1883 should not contain a slash (xxx#xxx.xxxxxx.com:1883) - see the URI Scheme.

Spring Boot Admin - Running in Docker Swarm weirdly

I am running multiple Spring-Boot servers all connected to a Spring Boot Admin instance. Everything is running in the same Docker Swarm.
Spring Boot Admin keeps reporting on these "fake" instances that pop up and die. They are up for 1 second and then become unresponsive. When I clear them, they come back. The details for that instance show this error:
Fetching live health status failed. This is the last known information.
Request failed with status code 502
Here's a screenshot:
This is the same for all my APIs. This is causing us to get an inaccurate health reading of our services. How can I get Admin to stop reporting on these non-existant containers ?
I've looked in all my nodes and can't find any containers (running or stopped) that match the unresponsive containers that Admin is reporting.

Connecting to scality/s3 server between docker containers

We are using a python based solution which shall load and store files from S3. For developing and local testing we are using a vagrant environment with docker and docker-compose. We have two docker-compose defintions - one for the assisting backend services (mongo, restheart, redis and s3) and the other one containing the python based REST API exposing solution using the backend services.
When our "front-end" docker-compose group interacts with restheart this works fine (using the name of the restheart container as server host in http calls). When we are doing the same with scality/s3 server this does not work.
The interesting part is, that we have created a test suite for using the scality/s3 server from a python test suite running on the host (windows10) over the forwarded ports through vagrant to the docker container of scality/s3 server within the docker-compose group. We used the endpoint_url localhost and it works perfect.
In the error case (when frontend web service wants to write to S3) the "frontend" service always responds with:
botocore.exceptions.ClientError: An error occurred (InvalidURI) when calling the CreateBucket operation: Could not parse the specified URI. Check your restEndpoints configuration.
And the s3server always responds with http 400 and the message:
s3server | {"name":"S3","clientIP":"::ffff:172.20.0.7","clientPort":49404,"httpMethod":"PUT","httpURL":"/raw-data","time":1521306054614,"req_id":"e385aae3c04d99fc824d","level":"info","message":"received request","hostname":"cdc8a2f93d2f","pid":83}
s3server | {"name":"S3","bytesSent":233,"clientIP":"::ffff:172.20.0.7","clientPort":49404,"httpMethod":"PUT","httpURL":"/raw-data","httpCode":400,"time":1521306054639,"req_id":"e385aae3c04d99fc824d","elapsed_ms":25.907569,"level":"info","message":"responded with error XML","hostname":"cdc8a2f93d2f","pid":83}
We are calling the scality with this boto3 code:
s3 = boto3.resource('s3',
aws_access_key_id='accessKey1',
aws_secret_access_key='verySecretKey1',
endpoint_url='http://s3server:8000')
s3_client = boto3.client('s3',
aws_access_key_id='accessKey1',
aws_secret_access_key='verySecretKey1',
endpoint_url='http://s3server:8000')
s3.create_bucket(Bucket='raw-data') # here the exception comes
bucket = self.s3.Bucket('raw-data')

This issue is quite common. In your config.json file, which you mount in your Docker container, I assume, there is a restEndpoints section, where you must associate a domain name with a default region. What that means is your frontend domain name should be specified in there, matching a default region.
Do note that that default region does not prevent you from using other regions: it's just where your buckets will be created if you don't specify otherwise.
In the future, I'd recommend you open an issue directly on the Zenko Forum, as this is where most of the community and core developpers are.
Cheers,
Laure

Does Jhipster support sticky sessions with docker stack or HAproxy?

Running a sample jhipster app (found at : https://github.com/ehcache/ehcache3-samples/tree/master/fullstack) , when I deployed it to a docker swarm (swarm mode) with docker stack, it worked fine and I could log-in
But when I started "scaling" the web app, I found out the session was lost whenever my request would hit another container than the first one.
Actually, I even saw in the logs :
worker2 | org.springframework.security.web.authentication.rememberme.CookieTheftException: Invalid remember-me token (Series/token) mismatch. Implies previous cookie theft attack.
worker2 | at org.terracotta.demo.security.CustomPersistentRememberMeServices.getPersistentToken(CustomPersistentRememberMeServices.java:173)
worker2 | at org.terracotta.demo.security.CustomPersistentRememberMeServices.processAutoLoginCookie(CustomPersistentRememberMeServices.java:83)
worker2 | at org.springframework.security.web.authentication.rememberme.AbstractRememberMeServices.autoLogin(AbstractRememberMeServices.java:130)
while I was trying to logging in again...
Is there something I need to setup to have the load balancer treat the session as unique ?

Resources