I am in the process of writing a zapier trigger in which I need to initiate our file download api, which in turn provides a url to poll, once I get a file ready status, I can construct a file download url.
Is it possible to have polling code in a trigger?
Are there any size limits on the files that zapier supports?
Once I have my download url, can simply use this code?
return _.map(records, function(record) {
// if you just do url, we'll include any standard authentication headers
record.file = z.dehydrateFile('//mysite.com/mypath/download' + record.id);
return record;
});
David here, from the Zapier Platform team.
You can do basically whatever you want in a trigger, but there's a couple of important things to keep in mind:
The goal is to fetch a lot of items, return them all, and zapier zapier decide which ones are new
you've got 30 seconds total for the trigger, so if you're polling for lots of results, you're going to timeout
there's a very important line in the docs:
Note: you should only be using z.stashFile() in a hydration method or a hook trigger's perform if you're sending over a short-lived URL to a file. Otherwise, it can be very expensive to stash dozens of files in a polling call
The worst case scenario here is that every time we poll (which happens every 5-15 minutes), you wait for a file for each result and stash it every time (without knowing if the user needs it or not).
You should follow the example in the docs, where during the Zapier poll, you dehydrate your polling operation. When we decide an item is new, we'll do that poll, stash the file once it's ready, and send it to the user appropriately.
Polling should be generally avoided as it causes performance issues. If you can, you may want to write your polling code in a background function (see serverless functions) or background code. It does the heavy-lifting for you. Create a Zapier Webhook. From the background code/function, do an HTTP POST of your file/data one-by-one (as if a new item created/updated) to the Webhook. This structure/architecture makes your trigger. With your use case above, you can send just the download url to Webhook. After that, you can add a zap to use this trigger and consume the incoming data and send to an action. This makes it cleaner and efficient.
Ref: Webhooks
Related
I'm trying to figure out how I can implement a retry policy in a Twilio Studio Flow. I see that they have an example, but it only has a delay of no more then 10-seconds.
I want something that can use to retry when my webhooks service is down. I did setup the sample from the Twilio docs but it only seems to work when you want a delay of no more then 10-seconds. But I need it to pause for an hour or two. So say the HTTP Post step fails because the webhooks service is offline, I want it to pause for an hour and try again. Then pause for 2, then 3, then 4, etc. and try again. The point being, I don't want to lose the user's response.
What I am trying to do is not lose any of the user responses from a survey if my webhooks application goes down. We saw this happen in production for a couple of hours and we lost survey response from 200 users.
If this is not possible, is there a way I can reach back out to Twilio logs and get access to the responses that failed while the webhooks service was down? I recall running into something where you can pull back the logs, which could then be used to identify the ones that failed.
This kind of logic isn't really built into Studio. Ten second waits are typically the most you will see due to both Twilio Functions & the http request widget timing out at this point.
If you wish to include this kind of wait then you will need some sort of workaround where you go into a send & wait for reply widget (which ignores responses from your customers with some additional logic) and has a timeout set to the amount of time you want to wait. You can then transition to the webhook request again and re-attempt.
Alternatively, you can create a utility which uses the Execution resource to find all the failed flows for a given time period so you can choose how best to move forward.
I am working on a plugin for TFS to hook certain operations. I'm able to successfully hook code pushes via using an ISubscriber on PushNotification, but am having trouble finding any type that matches up with the completion of a Pull Request.
A little more on what I'm trying to do. I currently have a PushNotification hook that has some branch specific checks that it does. Some reject a push, others provide notifications to users using some complex rules. I need a way to be able to provide the notifications at a minimum, and ideally prevent the pull request from going through. I can't provide notifications prior to the pull request going through as the notifications should only occur for code being placed in our main repository.
Long term, I want to switch it over to using the webhooks and some async approval, but I don't have the time to adapt the tools to work like that and setup the additional server needed to make that happen. If there's no good solution, I'll simply disable pull requests for now until I can write proper services for it, but if there is a way to reuse the adapted hooks that can run on the PushNotification ISubscriber, it would be extremely helpful.
I have a Rails API which can handle requests from the clients. Clients use that API to perform analysis of their data. Client POSTs the data to API, API checks if that data have been analysed before. If so API just respond with analysis result. If the data haven't been analyzed before API:
Tells client that analysis started.
Establishes the connection with analyzing microservice.
Performs asynchronous (or deferred or i don't know) request to the analyzing microservice and waiting for response. The analysis takes much time so neither the API nor the microservice should be blocked while doing it.
When the response from analyzing microservice is returned API hands it to the client.
The main issue for me is to set up things such way that client could receive somehow the message "Your data had been sent to analysis" right after he performed the request. And then when analysis will be done client could receive its result.
The question is what approach I have to use in that case? Async responses, deferred responses, something else? And what known solutions could help me with that? Any gems?
I'm new to that stuff so I'm really sorry if I ask dumb questions.
If using HTTP you can only have one response to every request. To send multiple responses, i.e. "work in progress", then later the "results", you would need to use a different protocol, e.g. web sockets.
Since HTTP is so very common I'd stick with that in combination with background jobs. There are a couple of options which spring to mind.
Polling: The API kicks off a background jobs (to call the microservice) and responds to the client with a URL which the client can ping periodically for the result. The URL would respond with some kind of "work in progress" status until the result is actually ready). The URL would need to include some kind of id so the API can lookup the background job.
The API would potentially have two URLS; /api/jobs/new and /api/jobs/<ID>. They would, in Rails, map to a controller new and show action.
Webhooks: Have the client include a URL of its own in the request. Once the result is available have the background job hit the given URL with the result.
Either way, if using HTTP, you will not be able to handle the whole thing within a request/response, you will have to use some kind of background processing (so request to the microservice happens in a different process). You could look at Sidekiq, for example.
Here is an example for polling:
URL: example.com/api/jobs/new
web app receives client request
generates a unique id for the request, SecureRandom.uuid.
starts a background job (Sidekiq) passing in the uuid and any other parameters needed
respond with URL such as example.com/api/jobs/
--
background job
sends request to microservice API and waits for response
saves result to database with uuid
--
URL: example.com/api/jobs/UUID
look in database for UUID, if not found respond that job is "in progress". If found return result found in database.
Depending on what kind of API you use. I assume your clients interact via HTTP.
If you want to build an asynchronous API over HTTP the first thing that you should do: accept the request, create a job, handle it in the background and immediately return.
For the client to get the response you have to 2 options:
Implement a status endpoint where clients can periodically poll the status of the job
Implement a callback via webhooks. So the client has to provide a URL which you then call after you're done.
A good start for background processing is the sidekiq gem or more general ActiveJob that ships with Rails.
Preamble: I'm trying to put together a proposal for what I assume to be a very common use-case, and I'd like to use Amazon's SWF and SQS to accomplish my goals. There may be other services that will better match what I'm trying to do, so if you have suggestions please feel free to throw them out.
Problem: The need at its most basic is for a client (mobile device, web server, etc.) to post a message that will be processed asynchronously without a response to the client - very basic.
The intended implementation is to for the client to post a message to a pre-determined SQS queue. At that point, the client is done. We would also have a defined SWF workflow responsible for picking up the message off the queue and (after some manipulation) placing it in a Dynamo DB - again, all fairly straightforward.
What I can't seem to figure out though, is how to trigger the workflow to start. From what I've been reading a workflow isn't meant to be an indefinite process. It has a start, a middle, and an end. According to the SWF documentation, a workflow can run for no longer than a year (Setting Timeout Values in SWF).
So, my question is: If I assume that a workflow represents one message-processing flow, how can I start the workflow whenever a message is posted to the SQS?
Caveat: I've looked into using SNS instead of SQS as well. This would allow me to run a server that could subscribe to SNS, and then start the workflow whenever a notification is posted. That is certainly one solution, but I'd like to avoid setting up a server for a single web service which I would then have to manage / scale according to the number of messages being processed. The reason I'm looking into using SQS/SWF in the first place is to have an auto-scaling system that I don't have to worry about.
Thank you in advance.
I would create a worker process that listens to the SQS queue. Upon receiving a message it calls into SWF API to start a workflow execution. The workflow execution id should be generated based on the message content to ensure that duplicated messages do not result in duplicated workflows.
You can use AWS Lambda for this purpose . A lambda function will be invoked by SQS event and therefore you don't have to write a queue poller explicitly . The lambda function could then make a post request to SWF to initiate the workflow
Following a specific action the user takes on my website, a number of messages must be sent to different emails. Is it possible to have a separate thread or worker take care of sending multiple emails so as to avoid having the response from the server take a while to return if there are a lot of emails to send?
I would like to avoid using system process or scheduled tasks, email queues.
You can definitely spawn off a background thread in your controller to handle the emails asynchronously.
I know you want to avoid queues, but another thing i have done in the past is written a windows service that pulls email from a DB queue and processes it at certain intervals. This way you can separate the 2 applications if there is a lot of email to be sent.
This can be done in many different ways, depending on how large your application is and what kind of reliability you want. Any of these ways should help you achieve what you want (in ascending order based on complexity):
If you're using IIS SMTP Server or another mail server that supports a pickup directory option, you can go with that. With this option, instead of sending the emails directly, they are saved first in the pickup directory. Your call will immediately return after the email is saved in the pickup directory, so the user won't have to wait until the email is sent. On the other hand, the server will try to send the email as soon as it's saved in the pickup directory so it's almost immediate (just without blocking the call).
You can use a background thread like described in other answers. You'll need to be careful with this option as the thread can end unexpectedly before it finishes its job. You'll need to add some code to make sure this works reliably (personally, I'd prefer not to use this option).
Using a messaging queue server like MSMQ. This is more work and you probably should only look into this if you have a large scale application or have good reasons not to use the first option with the pickup directory.
There are a few ways you could do this.
You could store enough details about the message in the database, and write a windows service to loop through them and send the email. When the user submits the form it just inserts the required data about the message and trusts the service will pick it up. Almost an email queue which you said you didn't want, but you're going to end up in a queue situation with almost any solution.
Another option would be to drop in NServiceBus. Use that for these kinds of tasks.
I typically compile the message body and store that in a table in the db along with the from and to addresses, a subject, and a timestamp indicating when the email was sent. Then I have a background task check the table periodically and pull any that haven't been sent. This task attempts to send each email and updates the timestamp accordingly. One advantage of storing the compiled message body up front is that the background task doesn't have to do any processing of context-specific data, and therefore can be pretty darn simple.
Whenever an operation like is hingent upon an event, there is always the possibility something will go wrong.
In ASP.NET you can spawn multiple threads and have those threads do the action. Make sure you tell the thread it's a background thread, otherwise ASP.NET might way for the thread to finish before rendering your page:
myThread.IsBackground = true;
I know you said you didn't want to use system process or scheduled tasks, but a windows service would be a viable approach to this as well. The approach would be to use MS Queue, or save the actions needing to be done in a DataBase table. Then have a windows service check every minute or so and do those actions.
This way, if something fails (Email server down) those emails / actions can still be done.
They will also be recorded for audit's (which is very nice to have).
This method allows you're web site to function as a website while offloading these tasks to another service. The last thing you need is for multiple ASP.NET processes to be used up waiting for emails to send. let something else handle that.