I'd like to run a Python script in the cloud. It would use Tweepy Streaming to continuously listen for Tweets containing certain keywords. So it needs to run uninterrupted, 24/7.
Would Google Cloud Run be suitable for this use case?
The Quotas and Limits page mentions that requests timeout after 60 minutes max, but I don't know exactly what this means.
Thank you.
No, it would not be a good choice. Serverless infrastructure provided by products like Cloud Run and Cloud Functions is generally assumed to expand and contract server instances on demand, and server instances are never guaranteed a long uptime. If you absolutely require 24/7 uninterrupted operation of some background task not tied to an event or HTTP request, you should use a different cloud product, such as App Engine or Compute Engine.
"some background task not tied to an event or HTTP request"
Isn't what the OP wants? Merely to listen for tweets 24/7? Detecting a tweet is an event and an HTTP request. Cloud Run and Cloud Functions can be triggered 24/7 via its URL endpoints.
In the Cloud Functions Page, if you scroll down there is a section called "Integration with third-party services and APIs".
I quote this section:
"... capabilities such as sending a confirmation email after a successful Stripe payment or responding to Twilio text message events"
listening for tweets counts too. So it seems Google Cloud Functions/ Cloud Run can be used for the OP's use case.
Related
I have a large JAVA application which connects to hundreds of cloud based systems using their REST API's and fetch the data from those systems.
To connects those different cloud systems we have different modules and each one have different approach to call REST API's like some modules using apache rest client some module using googles rest client.
So there is no centralise place where the REST api is getting called.
I have to track performance of the application e.g. to fetch accounts info from test system takes 1 hour. and this process need
4 api calls for https://test/api/v2/accounts -- (this will return all account id's)
8000 api calls for https://test/api/v2/accounts/{accountId}. --- (this will return deaths of each account)
I need to track what is the time taken by each api to responds and based on that calculate time taken by application to process that data.
Important part here is deatiled api analysis and make graphical data if possible e.g.
4 api calls for https://test/api/v2/accounts --- taken 3 minutes
8000 api calls for https://test/api/v2/accounts/{accountId} -- taken
48 minutes
I need any any pointer how can I achieve that something like intercept all rest api made to https://test/api/v2
As you've probably already discovered, without some extra tweaking, wireshark just shows you the connections at the FQDN level: you can't see which individual endpoint is called (because TLS, by design, hides the content of the connection). You have a few options though:
if you control the APIs that are being connected to, you can load the
TLS keys into wireshark, and it'll let you decrypt the TLS
connection;
if you can force your app to use a proxy, you can use a Man-In-The-Middle (MITM) proxy (like Burp) to intercept the traffic; or
you can instrument your app to log destination and duration for all the API requests.
I'm trying to figure out how I can implement a retry policy in a Twilio Studio Flow. I see that they have an example, but it only has a delay of no more then 10-seconds.
I want something that can use to retry when my webhooks service is down. I did setup the sample from the Twilio docs but it only seems to work when you want a delay of no more then 10-seconds. But I need it to pause for an hour or two. So say the HTTP Post step fails because the webhooks service is offline, I want it to pause for an hour and try again. Then pause for 2, then 3, then 4, etc. and try again. The point being, I don't want to lose the user's response.
What I am trying to do is not lose any of the user responses from a survey if my webhooks application goes down. We saw this happen in production for a couple of hours and we lost survey response from 200 users.
If this is not possible, is there a way I can reach back out to Twilio logs and get access to the responses that failed while the webhooks service was down? I recall running into something where you can pull back the logs, which could then be used to identify the ones that failed.
This kind of logic isn't really built into Studio. Ten second waits are typically the most you will see due to both Twilio Functions & the http request widget timing out at this point.
If you wish to include this kind of wait then you will need some sort of workaround where you go into a send & wait for reply widget (which ignores responses from your customers with some additional logic) and has a timeout set to the amount of time you want to wait. You can then transition to the webhook request again and re-attempt.
Alternatively, you can create a utility which uses the Execution resource to find all the failed flows for a given time period so you can choose how best to move forward.
I'm looking for ideas to send Docker Logs for each runs to be sent to my application in realtime. I'm looking ways this can be done. Please let me know how this can be done.
Let me know if you have done this already or know how this can be achieved. I want to build feature similar to Netlify or vercel where they show you all build log on UI in realtime. I want something similar for my node application.
You can achieve this with Vercel and Log Drains.
Log Drains make it easy to collect logs from your deployments and forward them to archival, search, and alerting services by sending them via HTTPS, HTTP, TLS, and TCP once a new log line is created.
At the time of writing, we currently support 3 types of Log Drains:
JSON
NDJSON
Syslog
Along with Log Drains, we are introducing two new open-source integrations with logging services for you to start using them today: LogDNA and Datadog.
Install the integration: https://vercel.com/integrations?category=logging
See the announcement blog post: https://vercel.com/blog/log-drains
Note that Vercel does not allow Docker deployments, but does support Serverless Functions.
Using the Smooch API, I am trying to obtain all of the messages sent to my Facebook appid in the past few minutes or hours.
The Get Messages REST method does exactly what I need, except for that it only returns messages from a particular appUserId. This isn't useful unless you already know what users have sent you messages. I cannot use a webhook as the application resides behind a corporate firewall. Opening the firewall to connections that originate from the outside is not an option (even with white-listing).
Is there a way to invoke the Get Messages REST method such that it will ignore the appUserId filter? Perhaps some sort of wildcard character?
GET {{url}}/{{apiVersion}}/apps/{{appId}}/appusers/{{appUserId}}/messages
Unfortunately you do need to have the appUserId (or userId) on hand in order to query user messages.
Webhooks are a pretty essential part of building a Smooch integration. If you can't receive them through your firewall, then you might consider building an intermediary service outside of your corporate network for receiving Smooch webhooks. For each webhook event you receive it would either:
Forward it through a secure tunnel into your coprorate network
Store the appUserId (or the whole event) in its own database, and provide a secure endpoint that allows your corporate network service to query that data
I'm curious to know more about your use case, e.g which Smooch channels are you integrating? With more details I might be able to improve this answer.
#alavers We would like to leverage nearly every messaging integration you offer.
#alavers You may want to consider providing a Get Messages variant that is better suited for use within a corporate firewall environment. An excellent example is the http long poll implementation provided by APIs such Amazon's SQS API. Their receiveMessage method waits for up to the specified time period but returns as soon as a message is received. This provides nearly the same performance of a webhook but eliminates the need for a customer to open their corporate firewall to connections that originate from outside the corporation. Most IT departments will approve connections that originate from within the corporation, but permitting connections that originate from the outside becomes a very difficult sell.
Preamble: I'm trying to put together a proposal for what I assume to be a very common use-case, and I'd like to use Amazon's SWF and SQS to accomplish my goals. There may be other services that will better match what I'm trying to do, so if you have suggestions please feel free to throw them out.
Problem: The need at its most basic is for a client (mobile device, web server, etc.) to post a message that will be processed asynchronously without a response to the client - very basic.
The intended implementation is to for the client to post a message to a pre-determined SQS queue. At that point, the client is done. We would also have a defined SWF workflow responsible for picking up the message off the queue and (after some manipulation) placing it in a Dynamo DB - again, all fairly straightforward.
What I can't seem to figure out though, is how to trigger the workflow to start. From what I've been reading a workflow isn't meant to be an indefinite process. It has a start, a middle, and an end. According to the SWF documentation, a workflow can run for no longer than a year (Setting Timeout Values in SWF).
So, my question is: If I assume that a workflow represents one message-processing flow, how can I start the workflow whenever a message is posted to the SQS?
Caveat: I've looked into using SNS instead of SQS as well. This would allow me to run a server that could subscribe to SNS, and then start the workflow whenever a notification is posted. That is certainly one solution, but I'd like to avoid setting up a server for a single web service which I would then have to manage / scale according to the number of messages being processed. The reason I'm looking into using SQS/SWF in the first place is to have an auto-scaling system that I don't have to worry about.
Thank you in advance.
I would create a worker process that listens to the SQS queue. Upon receiving a message it calls into SWF API to start a workflow execution. The workflow execution id should be generated based on the message content to ensure that duplicated messages do not result in duplicated workflows.
You can use AWS Lambda for this purpose . A lambda function will be invoked by SQS event and therefore you don't have to write a queue poller explicitly . The lambda function could then make a post request to SWF to initiate the workflow