This question already has answers here:
Http response at 400 or 500 level
(2 answers)
Closed 10 months ago.
State of Servies:
Client (nuxt) is up on http://localhost:3000 and the client sends
requests to http://localhost:8080.
Server (django) is running on 0.0.0.0:50051.
Also docker is up
78496fef541f 5f9773709483 "/docker-entrypoint.…" 29 minutes
ago Up 29 minutes 0.0.0.0:8080->8080/tcp, :::8080-8080/tcp,
10000/tcp envoy
envoy.yaml Configurations:
I configured the envoy.yaml file as follows:
static_resources:
listeners:
- name: listener_0
address:
socket_address: { address: 0.0.0.0, port_value: 8080 }
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"#type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
codec_type: auto
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains: ["*"]
routes:
- match: { prefix: "/" }
route:
cluster: greeter_service
max_stream_duration:
grpc_timeout_header_max: 0s
cors:
allow_origin_string_match:
- prefix: "*"
allow_methods: GET, PUT, DELETE, POST, OPTIONS
allow_headers: keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,custom-header-1,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc-web,grpc-timeout
max_age: "1728000"
expose_headers: custom-header-1,grpc-status,grpc-message
http_filters:
- name: envoy.filters.http.grpc_web
- name: envoy.filters.http.cors
- name: envoy.filters.http.router
clusters:
- name: greeter_service
connect_timeout: 0.25s
type: logical_dns
http2_protocol_options: {}
lb_policy: round_robin
load_assignment:
cluster_name: cluster_0
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 0.0.0.0
port_value: 50051
Error:
But the following error occurs and, as it seems, the requests do not reach the Django 0.0.0.0:50051 server.
503 Service Unavailable
grpc-message: upstream connect error or disconnect/reset before
headers. reset reason: connection failure, transport failure reason:
delayed connect error: 111
I have encountered the same error. Here are my condition:
Envoy: running on docker with listener at port 8080 and redirect to port 9090
Next JS Web client: request to envoy proxy at port 8080
Node Grpc Server: listen on port 9090
Im starting all on local environment.
Based on this example about configuring the envoy proxy that refer to this issue, I change the address on envoy proxy to host.docker.internal on envoy.yaml.
Just refer to this section, if you want to try:
clusters:
- name: backend_service
connect_timeout: 0.25s
type: logical_dns
http2_protocol_options: {}
lb_policy: round_robin
load_assignment:
cluster_name: cluster_0
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: host.docker.internal // i change it, before was 0.0.0.0
port_value: 9090
Related
I have a web app using gRPC Web to interact with my gRPC service through a dockerized Envoy Proxy. When I try calling the endpoint exposed in Envoy, I receive the following error:
gRPC Error (code: 14, codeName: UNAVAILABLE, message: upstream connect error or disconnect/reset before headers. reset reason: protocol error, details: [], rawResponse: null, trailers: {content-length: 0})
Here is my client side code:
class FiltersService {
static ResponseFuture<Filters> getFilters() {
GrpcWebClientChannel channel =
GrpcWebClientChannel.xhr(Uri.parse('http://localhost:9000'));
FiltersServiceClient clientStub = FiltersServiceClient(
channel,
);
return clientStub.getFilters(Void());
}
}
Here is my Envoy.yaml:
admin:
access_log_path: /tmp/admin_access.log
address:
socket_address:
address: 0.0.0.0
port_value: 9901
static_resources:
listeners:
- name: listener_0
address:
socket_address: { address: 0.0.0.0, port_value: 9000 }
filter_chains:
- filters:
- name: envoy.http_connection_manager
typed_config:
"#type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
codec_type: auto
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains: ["*"]
routes:
- match: { prefix: "/" }
route:
cluster: greeter_service
max_grpc_timeout: 0s
cors:
allow_origin_string_match:
- prefix: "*"
allow_methods: GET, PUT, DELETE, POST, OPTIONS
allow_headers: keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,custom-header-1,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc-web,grpc-timeout
max_age: "1728000"
expose_headers: custom-header-1,grpc-status,grpc-message
http_filters:
- name: envoy.filters.http.grpc_http1_bridge
- name: envoy.filters.http.grpc_web
- name: envoy.filters.http.cors
- name: envoy.filters.http.router
clusters:
- name: greeter_service
connect_timeout: 0.25s
type: logical_dns
upstream_connection_options:
tcp_keepalive:
keepalive_time: 300
lb_policy: round_robin
load_assignment:
cluster_name: cluster_0
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: host.docker.internal
port_value: 9001
Where port 9001 is where my gRPC service is running. I was able to verify the service is working properly by directly calling it from Kreya, and I received the correct response.
I am also able to successfully access the admin server from localhost:9901.
However, when I try calling the endpoint exposed by Envoy (9000) through either Kreya or client side, I get
gRPC Error (code: 14, codeName: UNAVAILABLE, message: upstream connect error or disconnect/reset before headers. reset reason: protocol error, details: [], rawResponse: null, trailers: {content-length: 0})
I am running the following commands to run the Dockerized Envoy:
docker build -t my-envoy:1.0 .
docker run -p 9000:9000 -p 9901:9901 my-envoy:1.0
After a lot of frustration and playing around, I finally figured it out. Looking at the documentation, it seems like the envoy.filters.httl.grpc_web filter can translate a request to both HTTP/2 and HTTP/3. I am assuming (though if someone more knowledgeable in this field can correct me, please do) that without specifying, Envoy does not know what protocol to translate to. As such, simply adding http2_protocol_options: {} to my cluster resolved the issue. I'm including the full cluster block below for anyone that may come across the same issue in the future.
clusters:
- name: greeter_service
connect_timeout: 0.25s
type: logical_dns
lb_policy: round_robin
http2_protocol_options: {}
load_assignment:
cluster_name: cluster_0
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: host.docker.internal
port_value: 9001
For some reason, whenever I tried using the --networks=host flag in my docker run command, I was unable to access the admin portal, and so the solutions that I came across, and the tutorials for using Envoy to proxy for gRPC Web weren't very beneficial in resolving this issue. Hopefully this is helpful to someone facing the same issue.
I have a web application with the following stack:
UI: Flutter Web/Dart
Server: Go
Communication Protocol: gRPC/gRPC-Web
I have defined a few protobufs and compiled them into both Go and Dart successfully. When I run the Go server code, I am able to successfully make gRPC calls with Kreya, however when I try making the same call from Flutter using grpc/grpc_web.dart, though I keep running into the following error:
gRPC Error (code: 2, codeName: UNKNOWN, message: HTTP request completed without a status
(potential CORS issue), details: null, rawResponse: , trailers: {})
Here is my UI Code:
class FiltersService {
static ResponseFuture<Filters> getFilters() {
GrpcWebClientChannel channel =
GrpcWebClientChannel.xhr(Uri.parse('http://localhost:9000'));
FiltersServiceClient clientStub = FiltersServiceClient(
channel,
);
return clientStub.getFilters(Void());
}
}
Backend Code:
func StartServer() {
log.Println("Starting server")
listener, err := net.Listen("tcp", fmt.Sprintf(":%v", port))
if err != nil {
log.Fatalf("Unable to listen to port %v\n%v\n", port, err)
}
repositories.ConnectToMongoDB()
grpcServer = grpc.NewServer()
registerServices()
if err = grpcServer.Serve(listener); err != nil {
log.Fatalf("Failed to serve gRPC\n%v\n", err)
}
}
// Register services defined in protobufs to call from UI
func registerServices() {
cardsService := &services.CardsService{}
protos.RegisterCardsServiceServer(grpcServer, cardsService)
filtersService := &services.FiltersService{}
protos.RegisterFiltersServiceServer(grpcServer, filtersService)
}
As mentioned, the API call is successful when Kreya is used to make the call, however the Dart code keeps failing.
I have tried wrapping the gRPC server in the gRPC web proxy, however that also failed from both Dart and Kreya. Here is the code I tried:
func StartProxy() {
log.Println("Starting server")
listener, err := net.Listen("tcp", fmt.Sprintf(":%v", port))
if err != nil {
log.Fatalf("Unable to listen to port %v\n%v\n", port, err)
}
repositories.ConnectToMongoDB()
grpcServer = grpc.NewServer()
registerServices()
grpcWebServer := grpcweb.WrapServer(grpcServer)
httpServer := &http.Server{
Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.ProtoMajor == 2 {
grpcWebServer.ServeHTTP(w, r)
} else {
w.Header().Set("Access-Control-Allow-Origin", "*")
w.Header().Set("Access-Control-Allow-Methods", "POST, GET, OPTIONS, PUT, DELETE")
w.Header().Set("Access-Control-Allow-Headers", "Accept, Content-Type, Content-Length, Accept-Encoding, X-CSRF-Token, Authorization, X-User-Agent, X-Grpc-Web")
w.Header().Set("grpc-status", "")
w.Header().Set("grpc-message", "")
if grpcWebServer.IsGrpcWebRequest(r) {
grpcWebServer.ServeHTTP(w, r)
}
}
}),
}
httpServer.Serve(listener)
}
func StartServer() {
StartProxy()
}
I am also aware of Envoy Proxy which can be used in place of this gRPC web proxy, however if I do that, I would be exposing the endpoints on Envoy as REST APIs, which would then forward the request as a gRPC call. From what I understand, this would require maintaining 2 versions of the data models - one for communication between the UI and Envoy (in JSON), and the other for communication between Envoy and the server (as protobuf). Is this the correct understanding? How can I move past this?
*** EDIT: ***
As per the suggestion in the comments, I have tried using Envoy in place of the go proxy. However, even now, I'm having trouble getting it to work. I'm now getting upstream connect error or disconnect/reset before headers. reset reason: overflow when trying to access the port exposed by Envoy 9001, though I can successfully call the backend service directly from kreya on port 9000.
Here is my envoy.yaml:
static_resources:
listeners:
- name: listener_0
address:
socket_address: { address: host.docker.internal, port_value: 9001 }
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"#type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
codec_type: auto
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains: ["*"]
routes:
- match: { prefix: "/" }
route:
cluster: greeter_service
max_stream_duration:
grpc_timeout_header_max: 0s
cors:
allow_origin_string_match:
- prefix: "*"
allow_methods: GET, PUT, DELETE, POST, OPTIONS
allow_headers: keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc-web,grpc-timeout
max_age: "1728000"
expose_headers: id,token,grpc-status,grpc-message
http_filters:
- name: envoy.filters.http.grpc_web
- name: envoy.filters.http.cors
- name: envoy.filters.http.router
clusters:
- name: greeter_service
connect_timeout: 0.25s
type: logical_dns
http2_protocol_options: {}
lb_policy: round_robin
# win/mac hosts: Use address: host.docker.internal instead of address: localhost in the line below
load_assignment:
cluster_name: cluster_0
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: host.docker.internal
port_value: 9000
I was able to resolve the issue by taking the suggestion in the comments and using Envoy to proxy instead of the go-proxy, though the solution didn't work purely out of the box according to the linked post.
Here is the working envoy.yaml
admin:
access_log_path: /tmp/admin_access.log
address:
socket_address:
address: 0.0.0.0
port_value: 9901
static_resources:
listeners:
- name: listener_0
address:
socket_address: { address: 0.0.0.0, port_value: 9000 }
filter_chains:
- filters:
- name: envoy.http_connection_manager
typed_config:
"#type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
codec_type: auto
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains: ["*"]
routes:
- match: { prefix: "/" }
route:
cluster: greeter_service
max_grpc_timeout: 0s
cors:
allow_origin_string_match:
- prefix: "*"
allow_methods: GET, PUT, DELETE, POST, OPTIONS
allow_headers: keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,custom-header-1,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc-web,grpc-timeout
max_age: "1728000"
expose_headers: custom-header-1,grpc-status,grpc-message
http_filters:
- name: envoy.filters.http.grpc_web
- name: envoy.filters.http.cors
- name: envoy.filters.http.router
clusters:
- name: greeter_service
connect_timeout: 0.25s
type: logical_dns
lb_policy: round_robin
http2_protocol_options: {}
load_assignment:
cluster_name: cluster_0
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: host.docker.internal
port_value: 9001
The working Dockerfile
FROM envoyproxy/envoy:v1.20-latest
COPY ./envoy.yaml /etc/envoy/envoy.yaml
CMD /usr/local/bin/envoy -c /etc/envoy/envoy.yaml -l debug
And the commands used to run Envoy
docker build -t envoy .
The previous command has to be run in the same directory as the Dockerfile
docker run -p 9000:9000 -p 9901:9901 envoy
Where 9000 and 9901 are the ports I want to expose and be able to access externally (listed in the envoy.yaml)
**NOTE: **
Make sure to include http2_protocol_options: {}. Following some possible solutions online, I had removed it, which led to a connection reset due to protocol error. I had been stuck on this issue for hours until I saw that Envoy can forward a request as either HTTP/2 or HTTP/3, and decided to add it back, which finally allowed me to make a gRPC call using the gRPC Web client.
Hope this helps anyone else that may be coming across this issue.
I have gRPC server in scala Play Framework which exposes gRPC hello world example service on port 9000. I'm trying to connect it with React web client. It seems that I'm having connection issues with Envoy proxy which is deployed to docker container on Mac.
I'm always getting the same error which I believe means that Envoy is not able to connect with backend:
code: 2
message: "Http response at 400 or 500 level"
metadata: Object { }
My docker file to build Envoy is:
FROM envoyproxy/envoy:v1.12.2
COPY ./envoy.yaml /etc/envoy/envoy.yaml
CMD /usr/local/bin/envoy -c /etc/envoy/envoy.yaml -l trace --log-path /tmp/envoy_info.log
And I'm building it using this script:
echo --- Building my-envoy docker image ---
docker build -t my-envoy:1.0 .
echo --- Running my-envoy docker image ---
docker run -d -p 8080:8080 -p 9901:9901 --network=host my-envoy:1.0
Envoy configuration is defined in this yaml file:
admin:
access_log_path: /tmp/admin_access.log
address:
socket_address: { address: 0.0.0.0, port_value: 9901 }
static_resources:
listeners:
- name: listener_0
address:
socket_address: { address: 0.0.0.0, port_value: 8080 }
filter_chains:
- filters:
- name: envoy.http_connection_manager
config:
codec_type: auto
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains: ["*"]
routes:
- match: { prefix: "/" }
route:
cluster: greeter_service
max_grpc_timeout: 0s
cors:
allow_origin:
- "*"
allow_methods: GET, PUT, DELETE, POST, OPTIONS
allow_headers: keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,custom-header-1,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc-web,grpc-timeout
max_age: "1728000"
expose_headers: custom-header-1,grpc-status,grpc-message
http_filters:
- name: envoy.grpc_web
- name: envoy.cors
- name: envoy.router
clusters:
- name: greeter_service
connect_timeout: 0.25s
type: logical_dns
http2_protocol_options: {}
lb_policy: round_robin
hosts: [{ socket_address: { address: host.docker.internal, port_value: 9000 }}]
I could not find anything relevant in envoy log file except these logs:
[2020-12-15 01:28:18.747][8][debug][upstream] [source/common/upstream/logical_dns_cluster.cc:72] starting async DNS resolution for host.docker.internal
[2020-12-15 01:28:18.747][8][trace][upstream] [source/common/network/dns_impl.cc:160] Setting DNS resolution timer for 5000 milliseconds
[2020-12-15 01:28:18.748][8][trace][upstream] [source/common/network/dns_impl.cc:160] Setting DNS resolution timer for 5000 milliseconds
[2020-12-15 01:28:18.749][8][debug][upstream] [source/common/upstream/logical_dns_cluster.cc:79] async DNS resolution complete for host.docker.internal
[2020-12-15 01:28:21.847][8][debug][main] [source/server/server.cc:175] flushing stats
[2020-12-15 01:28:23.751][8][debug][upstream] [source/common/upstream/logical_dns_cluster.cc:72] starting async DNS resolution for host.docker.internal
[2020-12-15 01:28:23.751][8][trace][upstream] [source/common/network/dns_impl.cc:160] Setting DNS resolution timer for 5000 milliseconds
[2020-12-15 01:28:23.753][8][trace][upstream] [source/common/network/dns_impl.cc:160] Setting DNS resolution timer for 5000 milliseconds
[2020-12-15 01:28:23.753][8][debug][upstream] [source/common/upstream/logical_dns_cluster.cc:79] async DNS resolution complete for host.docker.internal
So I have a dapr service hosted at: 192.168.1.34:50459
I'm trying to have it communicate with my web application using grpc-web. To do so I have an envoy proxy (according to https://github.com/grpc/grpc-web/issues/347).
My envoy.yaml file is as follows:
static_resources:
listeners:
- name: listener_0
address:
socket_address: { address: 0.0.0.0, port_value: 4949 }
filter_chains:
- filters:
- name: envoy.http_connection_manager
config:
codec_type: auto
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains: ["*"]
routes:
- match: { prefix: "/" }
route:
cluster: greeter_service
max_grpc_timeout: 0s
cors:
allow_origin_string_match:
- prefix: "*"
allow_methods: GET, PUT, DELETE, POST, OPTIONS
allow_headers: keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,custom-header-1,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc-web,grpc-timeout
max_age: "1728000"
expose_headers: custom-header-1,grpc-status,grpc-message
http_filters:
- name: envoy.grpc_web
- name: envoy.cors
- name: envoy.router
clusters:
- name: greeter_service
connect_timeout: 0.25s
type: logical_dns
http2_protocol_options: {}
lb_policy: round_robin
hosts: [{ socket_address: { address: 192.168.1.34, port_value: 50459 }}]
It should listen at 0.0.0.0:4949, and forward it to 192.168.1.34:50459
But when I start this proxy with
docker run -d -v envoy.yaml:/etc/envoy/envoy.yaml:ro -p 4949:4949 -p 50459:50459 envoyproxy/envoy:v1.15.0
It routes it to 0.0.0.0:50459
enter image description here
Does anyone know how to resolve this?
I don't know much about envoy but can you configure it to log to mounted path? See if the route is actually configured? Also, I assume you're setting the --dapr-grpc-port explicitly in Dapr run command? Have you tried setting it to a diff port in case of collisions?
I am trying to configure my envoy proxy to allow for save requests from my angular application to my application server using grpc. I have a letsencrypt certificate loaded, but the requests fail and chrome prints a: ERR_CERT_COMMON_NAME_INVALID when trying to connect. I have an apache2 running serving my web application. The envoy proxy on docker and the web application are running on the same machine.
my envoy.yaml:
admin:
access_log_path: /tmp/admin_access.log
address:
socket_address: { address: 0.0.0.0, port_value: 9901 }
static_resources:
listeners:
- name: listener_0
address:
socket_address: { address: 0.0.0.0, port_value: 17887 }
filter_chains:
- filters:
- name: envoy.http_connection_manager
config:
codec_type: auto
stat_prefix: ingress_http
stream_idle_timeout: 0s
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains: ["myactualdomain.com"]
routes:
- match: { prefix: "/" }
route:
cluster: greeter_service
max_grpc_timeout: 0s
cors:
allow_origin_string_match:
- prefix: "*"
allow_methods: GET, PUT, DELETE, POST, OPTIONS
allow_headers: keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,custom-header-1,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc-web,grpc-timeout
max_age: "1728000"
expose_headers: grpc-status,grpc-message
http_filters:
- name: envoy.grpc_web
- name: envoy.cors
- name: envoy.router
tls_context:
common_tls_context:
alpn_protocols: "h2"
tls_certificates:
- certificate_chain: { filename: "/etc/fullchain.pem" }
private_key: { filename: "/etc/privkey.pem" }
clusters:
- name: greeter_service
connect_timeout: 1.00s
type: logical_dns
http2_protocol_options: {}
lb_policy: round_robin
hosts: [{ socket_address: { address: localhost, port_value: 17888 }}]
I was thinking i might be because I am not using the traditional https port.
Any help appreciated.
I actually got it working. First of all I added a new subdomain for the envoy proxy and created a new pair of certificates. Also don't do this: domains: ["myactualdomain.com"] but rather ["*"] as this leads to a CORS violation. If you only connect with grpc-web and the envoy don't use ssl as they run on the same machine anyway. If you wan't to do that though, you might want to take a look at that: https://medium.com/#farcaller/how-to-configure-https-backends-in-envoy-b446727b2eb3, I didn't try it though.