We have a dataflow process subscribed to "Subscribtion A" on a "Topic A" processing messages in streaming mode.
According to PubSub documentation, if the messages are not pulled they will be retained for at least 7 days per subscription. Am I right to assume that if I stop my Dataflow process for an hour and then restart, it should process all the messages accumulated in the subscription while subscriber process was down before proceeding with the streaming messages?
In our case Dataflow process does not seem to pick up queued messages after restart. Is there any explicit configuration option that we are missing?
We tested it again and it does seem to work fine. It appears to be related to some other issues.
Related
Our Google cloud dataflow pipeline worked well for more than a year. Recently we switched the data-flow region to US-central and ever since our pipeline is not able to get messages from Pubsub and process them with same amount of throughput it used to generate. After getting few messages from the topic, backlog starts growing forever and unacked message count on subscription continuously increases as long as messages are published, and when publishing stops, backlog gets stuck and won't reduce.
From Monitoring Metrics Explorer I captured following stats.
Total messages published: 20K
Number of workers: 5
Ack message count: 25/sec for a duration of 5 minutes.
Acknowledge Requests: 8/sec
Ack latency: 750 Millis for a duration of 5 minutes.
Pull requests: 8/sec
After 5 mins, the backlog messages never got into pipeline's user code and as stated earlier unacked message count on subscription got stuck at 14K.
The metric Unacked message count is showing 14K. Does this mean subscription received 14K messages but never delivered them to pipeline user code? What could be the reason for subscription not delivering pulled messages to the pipeline user code?
Any help would be greatly appreciated.
I have a requirement where I want to get some kind of notification while the message retention period is over and the message is about to be discarded from the MQTT topic.
So the actual requirement is, we have Bluetooth bands, which are send their presence through a centralized agent and an MQTT broker. Now we got a requirement where we need to upgrade the band firmware. For doing so, we will send a message to the topic with a message and a specific retention period. Infra will receive the message notification and look for the band. If the band is found then it's ok otherwise it will wait for new bands to be available. Once the retention period is over, in some cases we have to retry, so to implement the retry mechanism, I wanted to receive the notification from the MQTT broker if a message retention period is over.
Please help me if this is even possible into MQTT?
The broker won't tell you when it drops messages, but since you know when you sent the message and what expiry time you set there is nothing to stop you implementing this yourself.
I have a cloud run service deployed, with an eventarc trigger which creates a pubsub topic behind the scenes, to which the trigger is subscribed.
Recently to my horror I noticed the trigger was not being invoked anymore. Looking further I noticed there were 0 subscriptions to the pubsub topic. I went to the cloud run service console and the trigger still existed, so why no subscription to the topic?
I have redeployed the service multiple times before, and the subscription has never disappeared like this. Deleting and re-creating the trigger brought back the subscription, however now I lost all my messages. :-(
Can anyone explain how this could happen, and if it does happen again, how I can re-enable the subscription without re-creating the trigger and thus losing all my pubsub messages?
Thank you very much.
Pub/Sub subscriptions created by Eventarc triggers should definitely not expire. If you see this again, please open a bug with details: https://cloud.google.com/support/docs/issue-trackers
I received this message from the engineering team:
"As of May 3rd, pubsub subscriptions created by creating an Eventarc trigger are set to never expire. Previously, they had a default expiration of 30 days of inactivity.
It sounds like what the customer is experiencing is consistent with that. What happened was there were 30 days of inactivity and the subscription expired.
The workaround at the moment would be to create a new trigger."
I do use the PAHO C client library in my application. I do subscribe topics with MQTTAsync_subscribe() and QoS set to 1. From what I understand is that 1 means, that a message is send to the client at least one time.
I disconnect the client that subscribed the topic and the client that published the topic is still sending messages to the Mosquitto broker. If I start the subscriber lets say hours later, I get all the buffered messages beginning from the last one when the subscriber was shut down. So far so good! But the problem is, that the messages arrive in the same interval as the publisher sends new messages. By doing so, you will never get the latest message queued up by the publisher.
What I expect is that the Mosquitto broker tries to send all pending messages one after the other to the client instead of sending one old message when a new one is published.
Maybe someone can help me to understand why this happens or maybe how to overcome this situation?
Queued messages should all be batch delivered when the client reconnects.
They are not queued up and delivered with the same time offset, mosquitto wouldn't even keep time of arrival to be able to do time offset delayed delivery (Time of arrival is kept for MQTT v5 messages because they can now have TTL value, but this is only used to remove expired messages from the queue, not to delay late delivery).
Is there a best practice for publishing scheduled/delayed messages with MQTT, for example, using Mosquitto or HiveMQ brokers?
The use case is: Tell a subscriber to perform some maintenance in 15 minutes.
Optimally, the use case would be then solved by publishing the message "perform maintenance now please", and mark the message with "deliver no sooner than 15 minutes from now".
While I wouldn't recommend this to do in any scenario with high throughput, at least with HiveMQ you can do the following:
Implement a OnPublishReceivedCallback.
Schedule a Runnable that uses the PublishService to some kind of shared ScheduledExecutorService. The Runnable re-publishes the publish via the PublishService
The OnPublishReceivedCallback needs to throw away the original publish by throwing an OnPublishReceivedException (use false as constructor parameter so you don't disconnect the publishing client)
No, messages are delivered immediately for all connected clients subscribed to a topic and at reconnection for disconnected clients with persistent subscriptions.
If you want to do delayed messages you will have to implement your own store and forward mechanism before they are published to the broker.