How to restart a Rails server *gracefully*? - ruby-on-rails

TL;DR: Rails-server restarts aren't handled gracefully in Action Cable, resulting in corrupted state. How to fix that?
I run a Rails server which uses Action Cable, among other things.
When I change a file, the server restarts as expected. But it doesn't exactly restart gracefully.
The Action Cable connection logs {"type":"disconnect","reason":"server_restart","reconnect":true} in the established socket. Then the socket appears to be cut and my channel's unsubscribed method does not run. Upon reconnect, however, the subscribed method does run—as it should, but now we have an asymmetry. Something as simple as keeping track of the number of connected clients has become unreliable because upon a server restart that number will now double although the true number remained the same. That's because it gets to increment the count when subscribed is called, but decrementing the counter happens in unsubscribed, which is not called when the server restarts.
If this only happened in development, it wouldn't be a huge issue, but it happens in production as well during server restarts caused by deployments.
How do I tell Rails to restart the server gracefully, by which I mean it should also run the unsubscribed method in my channel instead of just cutting the connection?
EDIT: I've also filed an issue with Rails: https://github.com/rails/rails/issues/41005

Related

Guacamole Disconnect Session on User Idle

I have google'ed a lot on this question but haven't found an answer. I have a Guacamole server that connects to a local VNC session and I would like for it to disconnect the user session if it detects no activity for an hour.
What I've tried and attempted
used xprintidle to show idle time of the user, this works so will be put in a script that might be used later to terminate an idle session.
I have looked at Guacamole's api-session-timeout and set it for a minute, I had hoped it would not reconnect to the same session once the VNC server stops abruptly. It seems this did not work.
I have tried to use guacamole's user-mappings.xml to specify the parameter "autoretry" and set that to "1". Restarted and this did not work, I'm thinking its not used this way.
I have went to the guacamole postgres database and manually inserted the entry for autoretry into the parameter table. Restarted, but this did not work.
I have went to the VNC side (I use TurboVNC) and looked at the flag -idletimeout and configuration option max-idle-timeout. This terminates the TurboVNC service when there are no active connections. It is not what I am after, I'm trying to only disconnect the session when the user is idle.
I figured that the VNC side would not work because even if the VNC session is terminated Guacamole would keep on retying forever to reconnect to that session.
From some posts on the Guacamole mailing list it seems that disabling auto reconnect is not possible without a recompile from source.
Is there a way to disconnect an active session after an idle timeout period? or maybe a way to stop Guacamole from reconnecting?

How to change ping time after we gets disconnected from server in Action Cable?

Currently, I can see when I close the server(or the connection gets lost) it retries after 13sec(or so). Is there a way, I can change this time in Rails 5?

NodeMCU resest with mqtt.client:close()), or mqtt.client:connect()

I have an application that uses mqtt for communication between modules and with a mobile terminal.
In some situations when the messages do not arrive, the node performs a self test of MQTT (sending a msg to itself), and when the selftest fails, tries to reconnect to the broker (mqtt offline not always arrives). And then two problems may arise:
If I perform a mqtt.client:close() to assure that the client is closed (to avoid the second problem) and the client is already closed, the node resets.
If I perform a mqtt.client:connect() and the client is still connected, an exception and a restet occurs.
is there a way to know if the mqtt client is connected or not?
Thanks for your comment. I am going to describe what I am doing, to see if you can help me:
I have two independent system, a master and a slave. The master publish a test message every 10 minutes. If there is no answer from the slave. it publish a test message to itself. If this self test does not arrive, a disconnection from the broker is assumed, and a reconnection is initiated.
And here is where the problem comes, sometimes the client is disconnected and everything go well, but sometimes it is still connected but unresponsive, and the node resets with an exception "already connected".
Performing a mqtt:close() previously to the reconnection, should be safe, but if I send it and the client is truly disconnected, the node resets without any reason (known to me).
All this is happening without receiving any offline message.
Instead of waiting for messages manually sent by the master client (which could fail to send for various reasons, leading a listening device to the wrong conclusion about the state of its connection to the broker), I recommend using MQTT's built-in connection management.
First, you can make sure that each client's initial connection has succeeded by including an error handler in :connect(). If the client really opens, nothing in the NodeMCU documentation indicates that it will close itself; it may go offline.
Once connected, the client only knows that something is wrong when it sends a message and does not receive a response. It sounds like you are not calling :publish() much (which would otherwise let you know by returning false), so pinging may be best. If you expect to receive a message from the broker every n seconds, set a keepalive time slightly higher than that on the client.
Then, failure to get a response to those messages should trigger an event that you can respond to. That might be something like the following (not tested, may work better called outside the callback):
m:on("offline", function(client) m:close() end)

Wink API Subscriptions Stop Sending Overnight

I'm using the Wink API to control lights and thermostats etc. In the API you can subscribe to device topics to receive events such as when a light is turned on or off. This is handled through PubNub. I have this all working except that the events stop being sent if I leave the system overnight.
If I try to "manually" toggle a light by tapping the physical switch then normally an event is fired and received by my application via PubNub. Restarting my application and thus reconnecting to PubNub does NOT resolve the issue. The only way I've found to resolve the issue is to open the Wink app itself. Nothing else needs to be done but to open the Wink app. Then if I manually toggle the switch the event will show up and be received by my application. This solution works even if I don't restart my application.
There appears to be some sort of wake or keep-alive going on. Although I don't see anything in the Wink API docs that state such. There is also nothing sent from PubNub during this downtime. I have logging being sent out for the status(), message(), and presence() callbacks and nothing is logged from these overnight. Therefore taking all the above into account I believe the issue resides on the Wink side of things.
Unfortunately, Wink does not provide developer support. The Wink app is obviously doing something but because the transactions are encrypted it is unknown what it is doing. Also my test cycle is 8 hours long currently so it is difficult to debug this by trial and error. I'm working to see if this "timeout" is shorter than the overnight 8 hours I've experienced so far. i've also tried power cycling my Wink Hub yesterday but that did not resolve the issue.
Anyone else run into this issue? How do you keep subscriptions alive so that they are always sent?

Kill off connections that are not consuming Rabbitmq

I am having a bit of an issue where if the client crashes or if it loses connectivity it does not properly close the connection. This results in several idle connections that keep adding up. The client I am using rabbitmq-c does not support heartbeats yet therefore I cannot use that as a possible solution.
Is there anyway to kill off any connections that are not at the same time consuming a queue?
I found out the client was not setting a timeout time to the server therefore the connections were always being kept open. I also updated the client to support Heartbeats so there is now two ways the connections can be killed instead of being kept open.

Resources