I have a cloud run service deployed, with an eventarc trigger which creates a pubsub topic behind the scenes, to which the trigger is subscribed.
Recently to my horror I noticed the trigger was not being invoked anymore. Looking further I noticed there were 0 subscriptions to the pubsub topic. I went to the cloud run service console and the trigger still existed, so why no subscription to the topic?
I have redeployed the service multiple times before, and the subscription has never disappeared like this. Deleting and re-creating the trigger brought back the subscription, however now I lost all my messages. :-(
Can anyone explain how this could happen, and if it does happen again, how I can re-enable the subscription without re-creating the trigger and thus losing all my pubsub messages?
Thank you very much.
Pub/Sub subscriptions created by Eventarc triggers should definitely not expire. If you see this again, please open a bug with details: https://cloud.google.com/support/docs/issue-trackers
I received this message from the engineering team:
"As of May 3rd, pubsub subscriptions created by creating an Eventarc trigger are set to never expire. Previously, they had a default expiration of 30 days of inactivity.
It sounds like what the customer is experiencing is consistent with that. What happened was there were 30 days of inactivity and the subscription expired.
The workaround at the moment would be to create a new trigger."
Related
Been working with subscriptions for the past few months and haven't had issues. Today our service has been down because changes made for Contacts and Events are not issuing a notification to our API. I have resubscribed several times in different environments to no avail. Code base has not changed for the past couple days.
The URL for the POST is: https://graph.microsoft.com/beta/subscriptions
This is the request payload (2018-12-13T14:40:48.321209 PST):
{
:changeType=>"created,updated", :notificationUrl=>"https://da532483.ngrok.io/office365/subscription/notification", :resource=>"/me/calendars/AQMkADAwATY3ZmYAZS1lZmRkAC1lOTZhLTAwAi0wMAoARgAAA6PhSGdMGt9Nkwdvvtt9NrIHAD-_YGH8PpRIvfFpb45sC10AAAIBBgAAAD-_YGH8PpRIvfFpb45sC10AAtZPfToAAAA=/events",
:expirationDateTime=>"2018-12-16T21:10:48+00:00",
:clientState=>"clientState"
}
And the response payload (2018-12-13T14:40:49.339139 PST):
{
"#odata.context":"https://graph.microsoft.com/beta/$metadata#subscriptions/$entity",
"id":"d07de8ab-2c6c-447c-90d1-2b583318c597",
"resource":"/me/calendars/AQMkADAwATY3ZmYAZS1lZmRkAC1lOTZhLTAwAi0wMAoARgAAA6PhSGdMGt9Nkwdvvtt9NrIHAD-_YGH8PpRIvfFpb45sC10AAAIBBgAAAD-_YGH8PpRIvfFpb45sC10AAtZPfToAAAA=/events",
"applicationId":"0e8eebe4-55dc-4bd6-91ac-c13f91bb145d",
"changeType":"created,updated",
"clientState":"clientState",
"notificationUrl":"https://da532483.ngrok.io/office365/subscription/notification",
"expirationDateTime":"2018-12-16T21:10:48Z",
"creatorId":"00067FFEEFDDE96A"
}
Any help would be greatly appreciated.
Sorry about the issues you experienced with Microsoft Graph webhook notifications.
Here is what happened:
We experienced a capacity issue that resulted in a delayed deliveries for a subset of notifications. Some subscriptions were affected more than others, with some subscribers seeing a significant drop in notifications delivered. The time period for this issue was between 2AM on 12-12-2018 and 6PM 12-13-2018 (PST).
What we are going to do:
1.We are investigating the root cause and will take steps to prevent this from occurring in the future.
2.We recognize that our outage communications for Graph notifications are lacking. We are working on improving our comms with the goal of pro-actively publishing service degradation info to our customers as soon as we realize they occur.
We appreciate your usage of Graph and our webhook notification framework. Sorry about the trouble!
Based on my understanding of how subcriptions work, it seems like we should build some sort of cron job to check if almost about to expire users were renewed since notifications won't be sent in some cases.
Based on reading these:
https://developer.apple.com/library/archive/technotes/tn2413/_index.html#//apple_ref/doc/uid/DTS40016228-CH1-SUBSCRIPTIONS-MY_SERVER_PROCESS_RARELY_RECEIVES_RENEWAL_NOTICES_WHEN_THE_AUTO_RENEWING_SUBSCRIPTION_RENEWS_
https://medium.com/revenuecat-blog/ios-subscriptions-are-hard-d9b29c74e96f
my question is:
is my conclusion true? are notifications not sent for renewals?
if I have to build a cron job and call verifyReceipt myself I can imagine that I would call it quite a lot per day, is there a limit to how many times I can call this endpoint? When will I be throttled?
Notifications are not sent for regular renewals. You will get a notification (INTERACTIVE_RENEWAL) if the user cancels and resubscribes. Best practice would be to check the /verifyReciept endpoint to get subscription status from Apple. There aren't any published throttles on this endpoint and would imagine it's extremely scalable.
Also, highly recommend RevenueCat to completely manage subscription status for you.
I am using quickfixj to subscribe to market data. Noticed that after midnight reset, I will not receiving incremental anymore. How do I know if I am still subscribed to the the session?
Do I need to resubscribe each time when I logged on again?
It of course depends on the target system that you're connecting to. Some systems will maintain subscriptions across sessions, but the vast majority will not.
You probably need to resubscribe.
This is a bit late, but I figured out that I need to resubscribe. Since the initiator is not down but the acceptor is down at that period of time. When Accepter started, the initiator was still trying to connect, once connected, the subscription is not triggered.
So the solution is to add subscription in the onLogon callback, so each time it got logged on, it will try to subscribe again, so the stream is not lost.
I'm using the Wink API to control lights and thermostats etc. In the API you can subscribe to device topics to receive events such as when a light is turned on or off. This is handled through PubNub. I have this all working except that the events stop being sent if I leave the system overnight.
If I try to "manually" toggle a light by tapping the physical switch then normally an event is fired and received by my application via PubNub. Restarting my application and thus reconnecting to PubNub does NOT resolve the issue. The only way I've found to resolve the issue is to open the Wink app itself. Nothing else needs to be done but to open the Wink app. Then if I manually toggle the switch the event will show up and be received by my application. This solution works even if I don't restart my application.
There appears to be some sort of wake or keep-alive going on. Although I don't see anything in the Wink API docs that state such. There is also nothing sent from PubNub during this downtime. I have logging being sent out for the status(), message(), and presence() callbacks and nothing is logged from these overnight. Therefore taking all the above into account I believe the issue resides on the Wink side of things.
Unfortunately, Wink does not provide developer support. The Wink app is obviously doing something but because the transactions are encrypted it is unknown what it is doing. Also my test cycle is 8 hours long currently so it is difficult to debug this by trial and error. I'm working to see if this "timeout" is shorter than the overnight 8 hours I've experienced so far. i've also tried power cycling my Wink Hub yesterday but that did not resolve the issue.
Anyone else run into this issue? How do you keep subscriptions alive so that they are always sent?
We have a dataflow process subscribed to "Subscribtion A" on a "Topic A" processing messages in streaming mode.
According to PubSub documentation, if the messages are not pulled they will be retained for at least 7 days per subscription. Am I right to assume that if I stop my Dataflow process for an hour and then restart, it should process all the messages accumulated in the subscription while subscriber process was down before proceeding with the streaming messages?
In our case Dataflow process does not seem to pick up queued messages after restart. Is there any explicit configuration option that we are missing?
We tested it again and it does seem to work fine. It appears to be related to some other issues.