I run my application that grabs data from an external API in a Docker container (alpine). I use Docker Desktop 4.1.1 on macOS Monterey 12.5.
Every now and then my app needs to refresh its auth token. Everything works well.
But sometimes I get timeouts on request to refresh the token (lets say it's auth.example.com).
I think auth.example.com might be rate limiting those calls but:
It works no problem when I request same thing from my host (outside Docker) at the same time when it timing out in a container
After I restart Docker it works right away from inside a container
Issue disappears after some (random?) time. Sometimes it's 30 minutes, sometimes it's hours
I tested it from different containers made from different, clean (Debian, alpine, Ubuntu) images - calls to auth.example.com are timing out from all of them
I tried telneting telnet auth.example.com 443 and it timeouts inside Docker and works well from my host
At the same time telnet google.com 443 works well from inside my containers
I tried running hundreds of those requests from my host in a loop to see if it gets blocked but it doesn't (and my app inside a container requests that only once an hour maybe)
Seems like Docker is adding something in the request that allows auth.example.com to filter those requests maybe?
But I tried sending requests from inside my container and from my host to RequestBin and all headers look the same.
I tried using mitmproxy and Proxyman to watch the requests but auth.example.com uses SSL pinning and I was not able to configure it properly.
I don't know how to debug that further. Any ideas?
(I am using Spotify's API, with Spotipy library, and calls that time out are made to accounts.spotify.com).
I have a series of microservices that I have been testing. Originally it was using Service Fabric however I have switched to using Consul, Fabio, Nomad which I like better.
In development on my machine things work well however I am running into some issues actually getting Fabio to work in a cluster format.
I have a cluster of 5 nodes each running Consul, Fabio, Nomad.
Each service gets a dynamic port at runtime and successfully registers itself.
On the node which the service is running Fabio correctly forwards traffic.
However if the same fabio url is used on a different node then traffic is forwarded to the correct node/port however that is closed so the connection doesn't work.
For instance if ServiceA running on MachineA on port 1234 then http://MachineA:9999/ServiceA correctly works.
However http://MachineB/ServiceA fails after MachineA tries to initiate a connection to MachineB on port 1234.
A solution would be to add firewall rules, I would imagine, however this requires all the Services to run as Admin which I don't want.
Is there a way to support this through Fabio?
I've tried doing simple single node swarm just like in Docker tutorial part 3 and I've found out that if I use curl then I'm jumping between two replicas, but if I use Chrome then once I open the page then any following requests will be handled by the same replica. I'm sure I'm actually hitting it only once, because counter increases only by 1.
What is happening? Is it some kind of feature in Docker Swarm load balancing? If so, how would it work? No specific request headers are send to the server, so how would the load balancer recognize me? It can't be IP, because if I use incognito mode I'll be handled by different replica and I'll be stick to it as long as I'm in incognito.
It's not a Swarm thing, it's a chrome thing. Curl acts like you'd expect, each command is a new TCP request that shows as a new connection going through the Swarm VIP load balancer.
Chrome (and other browsers) have lots of methods to keep TCP connections open for future requests (HTTP keep-alives, etc). This is why it will stay connected to the same container because the connection is persistent through the LB to the replica. The LB will only shift to the "next in the round-robin pool" for a new connection.
I have together 6 containers running in docker swarm. Kafka+Zookeeper, MongoDB, A, B, C and Interface. Interface is the main access point from public - only this container publish the port - 5683. The interface container connects to A, B and C during startup. I am using docker-compose file + docker stack deploy, each service has a name which is used as host for interface. Everything starts successfully and works fine. After some time (20 mins,1h,..) I am not able to make request to interface. Interface receives my requests but application lost connection with service A,B,C or all of them. If I restart interface, it's able to reconnect to services A,B,C.
I firstly thought it's problem of application so I expose 2 new ports on each service (interface, A,B,C) and connect with profiler and debugger to them. Application is running properly, no leaks, no blocked threads, normally working and waiting for connections. Debugger shows me that when I make a request to interface and interface tries to request service A, Connection reset by peer exception was thrown.
During this debugging I found out interesting stuff. I attached debugger to interface when the services started and also debugger was disconnected after some time. + I was not able to reconnect it, until I made request to the container -> application. PRoblem - handshake failed.
Another interesting stuff that I found out was that I was not able to request neither interface. So I used wireshark to see what's going on and: SYN - ACK was fine. Then application post some data and interface respond with FIN,ACK. I assume that this also happen when interface tries to request service A and it FIN the connection. Codebase of Interface, A,B and C is the same regarding netty server.
Finally, I don't think it's a application issue. Why? I tried to deploy containers not as services. I run each container separately, published the ports of each and endpoint of services were set to localhost. (not overlay network). And it is working. Containers run without problem. + I didn't say at the beginning, that the the java applications (interface, A,B,C) runs without problem when they are running as standalone application - not in docker.
Could you please help me what could be the issue? Why the docker in case of overlay network is closing sockets?
I am using newest docker. I used also older.
Finally, I was able to solve the problem.
What was happening, one more time. Interface opens permanent TCP connection to A,B,C. When you try to run these services A,B,C as a standalone java applications, everything is working. When we dockerize them and run in swarm, it was working only few minutes. Strange was that the connection between Interface and another service was interrupted in the moment when you made a request from client to interface.
After many many unsuccessful tests and debugging each container I tried to run each docker container separately, with mapped ports and as endpoint I specified localhost. (each container exposed ports and interface was connecting to localhost) Funny thing happen, it was working. When you run containers like this, different network driver for container is used. Bridge one. If you run it in swarm, overlay network driver is used.
So it had to be something with the docker network, not with application itself. Next step was tcpdump from each container after couple of minutes, when it should stop working. It was very interesting.
Client -> Interface (OK, request accepted)
Interface ->(forward request because it belongs to A) A
Interface -> A [POST]
A -> Interface [RESET]
A was reseting opened TCP communication after couple of minutes without communication. Why?
Docker uses IP Virtual Server and IPVS maintains its own connection table. The default timeout for CLOSE_WAIT connections in IPVS table is 60 seconds. Hence when the server sends something after 60 seconds, the IPVS connection is no longer available and the packet looks invalid for a new TCP session and gets RST. On the client side, the connection remains forever in FIN_WAIT2 state because the app still has the socket open; kernel's fin_wait timer kicks in only for orphaned TCP sockets.
This is what I read about it and how understand it. I am not sure if my explanation of problem is correct, but based on these assumptions I implemented ping-pong between Interface and A,B,C services in case there is no communication for <60seconds. And, it’s working.
Got the same issue.
Specified
endpoint_mode: dnsrr
to properties of the service which plays "server" role and it works just fine.
https://forums.docker.com/t/tcp-timeout-that-occurs-only-in-docker-swarm-not-simple-docker-run/58179
(This is a follow up to rails app fast on server, but slow when accessed from another machine.)
I have a Rails web app that's incredibly slow when I access via its hostname, but runs at normal speeds when I access via its IP address (or via localhost, if I access it on the same server machine it's running on). This makes me think the problem is with the DNS. (Also, all these machines are running on the same corporate intranet.)
However, when I ping the hostname from a terminal, the ping seems to run fine. Does the fact that pinging works suggest that the problem is not with the DNS? (I don't really know much about DNS or servers and networking, so I'm kind of floundering around a bit here.)
Update to add: I also ran a simple "Hello world" Sinatra app, and this also runs super slowly when accessed via hostname (but not when accessed via IP address).
Fast ping from your terminal suggests that DNS between you and DNS server was fine and that network between you and server is fine.
This still does not help with the DNS on your server. Do you have any network operations that your server performs? If so, you need to make sure the network is reachable.
I suggest you get a simple "hello world" Rails application deployed there and see if it is Rails issue related (server wide) or your application related (very easy to do).
The other suggestion is to profile your Rails app and see which operation is taking the time to complete.
Your ping command is probably using cached DNS instead of hitting the server every time. Google for "flushdns" to find the right syntax to purge your cache for your particular operating system, then try it. You'll need to do this every time if you want to use ping to see about DNS response.