SignalR + Redis in cluster not working - docker

Question

Background
When running the website with single application instance (container) - SignalR is working perfectly.
When scaling out to more instances (>1), it throws errors and is just not working. I looked for an explanation on the internet and found that I need to configure my Signalr to work in cluster. I choose Redis as my backplane.
I worked through many tutorials on how to do it right and it's just not working for me.
Environment
I working asp.net core v2.1 hosted in Google cloud. The application is deployed as Docker container, using Kestrel + nginx. The docker container runs in a Kubernetes cluster behind a load balancer.
My Configuration
Startup.cs:
public class Startup
{
public Startup(IConfiguration configuration,
IHostingEnvironment hostingEnvironment)
{
Configuration = configuration;
HostingEnvironment = hostingEnvironment;
}
// This method gets called by the runtime. Use this method to add services to the container.
public void ConfigureServices(IServiceCollection services)
{
// ...
services.AddSignalR().AddRedis(options =>
{
options.Configuration = new ConfigurationOptions
{
EndPoints =
{
{ ConfigurationManager.Configuration["settings:signalR:redis:host"], int.Parse(ConfigurationManager.Configuration["settings:signalR:redis:port"])}
},
KeepAlive = 180,
DefaultVersion = new Version(2, 8, 8),
Password = ConfigurationManager.Configuration["settings:signalR:redis:password"]
};
});
// ...
}
// This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
public void Configure(IApplicationBuilder app, IHostingEnvironment env, IServiceProvider serviceProvider)
{
if (env.IsDevelopment())
{
app.UseDeveloperExceptionPage();
}
else
{
app.UseExceptionHandler("/errors/general");
app.UseHsts();
}
// nginx forward
app.UseForwardedHeaders(new ForwardedHeadersOptions
{
ForwardedHeaders = ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto
});
app.UseSignalR(routes =>
{
routes.MapHub<StatisticsHub>("/hubs/myhub");
});
}
}
In order to verify that the connection to Redis server succeeded, I checking the output window of the Kastrel:
Same behaviour (connected) found also on servers (2 replicas, not on development enviroment).
In order to verify that the Signalr is "really" using the Redis (not just connecting), I used redis-cli to connect the Redis server and found that:
From this I can understand that there are some "talks" on Redis.
I removed the website LoadBalancer (GCP) and deployed it agian. Now with Sticky-Session: ClientIP. This loadbalancer is routing requests to the different containers.
The only place that I didnt changed is the nginx configuration. I'm wrong?
The result
SignalR not working on the browser. These errors are from the browser console:
scripts.min.js:1 WebSocket connection to 'wss://site.com/hubs/myhub?id=VNxKLazEKr9FKM4GPZRDhA' failed: Error during WebSocket handshake: Unexpected response code: 404
scripts.min.js:1 Error: Failed to start the transport 'WebSockets': undefined
Error: Connection disconnected with error 'Error: Server returned handshake error: Handshake was canceled.'.
scripts.min.js:1 Error: Connection disconnected with error 'Error: Server returned handshake error: Handshake was canceled.'.
files.min.js:1 Uncaught (in promise) Error: Error connecting to signalR. Error= Error
...
Question
What I missed? What to check?