Our Google cloud dataflow pipeline worked well for more than a year. Recently we switched the data-flow region to US-central and ever since our pipeline is not able to get messages from Pubsub and process them with same amount of throughput it used to generate. After getting few messages from the topic, backlog starts growing forever and unacked message count on subscription continuously increases as long as messages are published, and when publishing stops, backlog gets stuck and won't reduce.
From Monitoring Metrics Explorer I captured following stats.
Total messages published: 20K
Number of workers: 5
Ack message count: 25/sec for a duration of 5 minutes.
Acknowledge Requests: 8/sec
Ack latency: 750 Millis for a duration of 5 minutes.
Pull requests: 8/sec
After 5 mins, the backlog messages never got into pipeline's user code and as stated earlier unacked message count on subscription got stuck at 14K.
The metric Unacked message count is showing 14K. Does this mean subscription received 14K messages but never delivered them to pipeline user code? What could be the reason for subscription not delivering pulled messages to the pipeline user code?
Any help would be greatly appreciated.
Related
We implemented the change notifications subscription for channel messages as per the Microsoft Teams guidelines. We received the messages within 1 minute until 12-July-2021. Now we see a latency of about 2 - 15 minutes for sometimes notably on PST day time.
According to this link (https://learn.microsoft.com/en-us/graph/webhooks#latency) channel messages will be notified within 1 minute.
Could you please let us know what is causing this delay serving the request ?
We would also like to know if there is policy or priority on serving the notifications based on tenant type.
Thanks in advance.
We have a dataflow process subscribed to "Subscribtion A" on a "Topic A" processing messages in streaming mode.
According to PubSub documentation, if the messages are not pulled they will be retained for at least 7 days per subscription. Am I right to assume that if I stop my Dataflow process for an hour and then restart, it should process all the messages accumulated in the subscription while subscriber process was down before proceeding with the streaming messages?
In our case Dataflow process does not seem to pick up queued messages after restart. Is there any explicit configuration option that we are missing?
We tested it again and it does seem to work fine. It appears to be related to some other issues.
The documentation for Microsoft Graph webhooks is not clear on the durability of events. If a client creates a subscription and then subsequently goes down (say, for maintenance), what happens to events that occur while the client is unavailable? Are they queued and retried by the Microsoft Graph until the client comes back online, or are they lost? If they are queued, for how long?
Graph Webhook will try to send the events a number of times within 4 hours window. If the client does not come back online after 4 hours, then these events will be deleted.
When in QOS 1 & 2 it replays all past messages. Is there a way in standard implementations to receive the entire past queue (as array) when becoming live again? (Of course only subscribed ones)
When a client has subscribed to a topic at QOS 1 or 2 and then disconnects. If when that client reconnects with the same client id and with the clean session flag set to false, the broker should replay any missed messages.
The broker should not replay any messages that had already been sent during the first connected period (with the possible exception of any QOS 1 messages that may have been in flight at the time of the disconnect)
It appears that we aren't getting all AnyOfferChanged notifications on an Amazon SQS. Many are arriving in the queue, but a manual analysis is showing that many are also just going missing.
Is there any way to query MWS to see a list of notifications or even a simple count for the day?
Any common causes for losing MWS subscription notifications sent to SQS?
I don't believe there is a way to query the SQS system. They are just messages in a queue and you read them, process them, and then delete them. We have been using the AnyOfferChanged notifications for about a year and just have to trust that they work. If you're sure that the criteria is met (a product you sell, and a price changes in the top 20 offers, new or used) and there is not message in the queue for it, the I would open a ticket with Seller Central. We have seen that it is near-real time that an SQS notification arrives for one of our product price changes.
To get a count for the day, you'd just have to read messages from your queue starting at a certain point, add +1 to a count variable for each message, delete them from the queue, and then look at the counter after 24 hours. Best I can think of.