When an HTTP request sent to a cloud run service is cancelled by the initiating service, the cancelled/closed connection is not propagated to the request that is being serviced inside the cloud run container.
Example code:
package main
import (
"log"
"net/http"
"os"
"time"
)
func main() {
port := os.Getenv("PORT")
if port == "" {
port = "8080"
}
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
select {
case <-ctx.Done():
log.Printf("context cancelled")
case <-time.After(time.Minute):
w.Write([]byte("OK"))
log.Printf("request completed. path=%s", r.URL.Path)
}
})
log.Fatal(http.ListenAndServe(":"+port, nil))
}
Running this code locally, a curl request sent to http://localhost:8080 and then cancelled with ctrl-c will show up in the logs as "context cancelled". The same request with the service deployed in cloud run and cancelled will show up as a successful request after 1 minute in the logs.
I don't think Cloud Run offers such a guarantee today.
I am guessing this is because of an implementation detail. The external traffic to a Run service does not directly go to the container, but rather passes through one or more load balancers and proxies. These LBs/proxies may not be properly propagating the client disconnects (or simply buffering requests/responses). This would result in the behavior you’re seeing.
I will go ahead and file an internal feature request about it. Thanks for bringing it up.
Related
Kind of new to docker but have a setup with multiple microservices in docker using FastAPI each with their own Postgres DB. Have an issue where I am sending an API request to microservice A, which calls an internal microservice B which works fine, the issue arises when I make an API request to microservice A, which calls microservice B, which sends another API request to back to microservice A to completely unrelated endpoint to fetch some other data type that only exists there, only the connection just hangs on "Starting new HTTP connection" in the docker logs. Not sure where to start to look on how to debug this, any help appreciated.
Tried making multiple simultaneous connections to microservice A, no issues there.
I have created an azure durable function and put it in a docker container, everything is working in docker desktop, the azure durable function is using and MQ rabbit trigger.
this the code:
public async Task<List<string>> RunOrchestrator(
[OrchestrationTrigger] IDurableOrchestrationContext context
)
{
var job = context.GetInput<Job>();
var outputs = new List<string>();
try
{
job.JobCreationIdResult = await context.CallActivityAsync<string>("JobExecutor_CreateJobSimple", job);
}
catch (Exception ex)
{
_logger.ForContext<JobExecutorSimple>().Error(ex.Message, ex);
return outputs;
}
return outputs;
}
[FunctionName("JobExecutor_CreateJobSimple")]
public async Task<string> CreateJob([ActivityTrigger] Job job)
{
_logger.ForContext<JobExecutorSimple>().Information($"result JobId: {job.MktJobId}");
return "done";
}
[FunctionName("JobExecutor_RabbitMQStartSimple")]
public async Task RabbitMQStart(
[RabbitMQTrigger("mkt-executor-q-local", ConnectionStringSetting = "mkt-Executor-RabbitMqConnection")] Job job,
[DurableClient] IDurableOrchestrationClient starter)
{
string instanceId = await starter.StartNewAsync("JobExecutorSimple", job);
_logger.ForContext<JobExecutorSimple>().Information($"Started orchestration with ID = '{instanceId}'.");
}
nothing fancy, with the same image deploy to aks I'm getting the following log:
[40m[32minfo[39m[22m[49m: Host.Startup[0]
Job host started
[40m[32minfo[39m[22m[49m: Microsoft.Azure.WebJobs.Host.DrainModeManager[0]
DrainMode mode enabled
[40m[32minfo[39m[22m[49m: Microsoft.Azure.WebJobs.Host.DrainModeManager[0]
Calling StopAsync on the registered listeners
[40m[32minfo[39m[22m[49m: Host.Startup[0]
Stopping the listener 'Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableTaskListener' for function 'JobExecutorSimple'
[40m[32minfo[39m[22m[49m: Host.Startup[0]
Stopped the listener 'Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableTaskListener' for function 'JobExecutorSimple'
[40m[32minfo[39m[22m[49m: Host.Startup[0]
Stopping the listener 'Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableTaskListener' for function 'JobExecutor_CreateJobSimple'
[40m[32minfo[39m[22m[49m: Host.Triggers.DurableTask[0]
Stopping task hub worker. IsGracefulStop: False. InstanceId: . Function: . HubName: SimpleFunction. AppName: . SlotName: . ExtensionVersion: 2.6.0. SequenceNumber: 2.
[40m[32minfo[39m[22m[49m: Host.Startup[0]
Stopping the listener 'Microsoft.Azure.WebJobs.Extensions.RabbitMQ.RabbitMQListener' for function 'JobExecutor_RabbitMQStartSimple'
[40m[32minfo[39m[22m[49m: Host.Startup[0]
Stopped the listener 'Microsoft.Azure.WebJobs.Extensions.RabbitMQ.RabbitMQListener' for function 'JobExecutor_RabbitMQStartSimple'
[40m[32minfo[39m[22m[49m: Host.Triggers.DurableTask[0]
Task hub worker stopped. IsGracefulStop: False. Latency: 00:00:01.2084240. InstanceId: . Function: . HubName: SimpleFunction. AppName: . SlotName: . ExtensionVersion: 2.6.0. SequenceNumber: 3.
[40m[32minfo[39m[22m[49m: Host.Startup[0]
Stopped the listener 'Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableTaskListener' for function 'JobExecutor_CreateJobSimple'
[40m[32minfo[39m[22m[49m: Microsoft.Azure.WebJobs.Host.DrainModeManager[0]
Call to StopAsync complete, registered listeners are now stopped
[40m[32minfo[39m[22m[49m: Host.General[337]
Host lock lease acquired by instance ID '000000000000000000000000B7700D04'.
and basically the function doesn't do anything.
the first error that I was getting was relating to not being able to release the lease so I have to turn it off in the host file:
"extensions": {
"durableTask": {
"hubName": "SimpleFunction",
"useAppLease": false
}
}
there is a way a azure durable function to work inside aks without KEDA or it is mandatory?, really we don't expect to scale.
Since you are not expecting it to scale you can deploy it directly to deploy to AKS .
KEDA will only be required only if you are preempting a surge in demand for a small period of time and not much after that . Thus you don't need event driven auto-scaling .
To create a docker image for durable function please refer the below article :
Run a Durable Azure Function in a Container - Code it Yourself... (mendible.com)
To deploy your docker image to AKS refer the following documentation :
Kubernetes on Azure tutorial - Deploy a cluster - Azure Kubernetes Service | Microsoft Docs
You can also refer this
Running Azure Functions on AKS – Robert te Kaat (wordpress.com)
I’ve recently spent a few weeks dealing with the same problem where DrainMode would kick in and stop the listeners from running for seemingly no reason, and only on AKS, we were not ever able to recreate it locally.
In our case the reason turned out to be that when the application was starting up, the base image was setting an environment variable ASPNETCORE_URLS to bind to port 80. However in AKS we were setting a security context on our deployment to not allow the container to run as root, and only root users can bind to ports < 1024. This led to the same log trace you have shared, where drain mode kicks in and the listeners stop, but with no logged reason for it to happen.
Our fix was to expose port 8080 in the dockerfile and set ASPNETCORE_URLS to http://+:8080
I am trying to setup my container to send some of its outgoing traffic to Fiddler (for debugging my application).
I setup it up like this:
services.AddHttpClient("OAuthClient").ConfigurePrimaryHttpMessageHandler(x =>
{
var handler = new HttpClientHandler
{
Proxy = new WebProxy("http://host.docker.internal:8888"),
UseProxy = true
};
return handler;
});
And this works fine when I run normal HTTP traffic from the container. It also works fine when I run HTTPS traffic from my dev machine. When when I try to run HTTPS traffic from a container, I get the following error:
The SSL connection could not be established, see inner exception.
And the inner exception is:
System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
I looked that up and from what I can see it is a "Self Signed Certificate" issue. But I am not sure where it thinks it is is finding the self signed certificate. Is it the container, the host or the url I am trying to call?
I feel that a container using a fiddler proxy is common enough that this should be a solved problem.
How can I pass my container's HTTPS traffic to fiddler?
I'm spinning up a container with a service that returns a simple ping response to a GET request.
after 3 requests I get this error for every request afterwards.
only happens inside a docker container base on this
FROM mcr.microsoft.com/dotnet/core/aspnet:2.2
when I run same app on my windows machine all works fine ...
warn: Microsoft.AspNetCore.Server.Kestrel[0]
Connection processing ended abnormally.
System.Net.Sockets.SocketException (125): Operation canceled
at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketAwaitableEventArgs.<GetResult>g__ThrowSocketException|7_0(SocketError e)
at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketAwaitableEventArgs.GetResult()
at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketConnection.ProcessSends()
at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketConnection.DoSend()
at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
at System.IO.Pipelines.Pipe.GetReadAsyncResult()
at System.IO.Pipelines.Pipe.DefaultPipeReader.GetResult(Int16 token)
I am working on the progressive web application sample.
Following the instruction, I am on the step of register service worker.
This is the code for register service worker:
if ('serviceWorker' in navigator) {
navigator.serviceWorker.register('./service-worker.js').then(function () {
console.log('Service Worker Registered');
});
}
But it is always failed and get the error log as:
Could you please teach me why I cannot register the service worker?
Your service worker is not linked properly. Mentioned path is not correct and so it’s getting 404 response. Remove the “.” In the path and place service worker and the file containing this script in same place.