New to Twilio. Developing an IT alerting function with Twilio SMS/MMS API in Python. A postfix alias-executed program processes a message and sends essential data via Twilio MMS to designated recipients.
Media such as images are accessed through media_url property to Client.messages.create(), via a URL pointing to content that I must store and offer through my HTTP server.
I have verified that that is the case, so my question is:
How do I control access to those images so that only Twilio can access them, and only for the duration of the message sending process?
My current solution, which is a kludge, is for the postscript alias-executed program to write a list of media files associated with the message, and then write my own status_callback that erases the files in that list when I get a "delivered" status (or a certain time limit expires).
This is a problem because the media files are publicly accessible for however long it takes for the "delivered" status to arrive or for my timeout to occur.
I've tried various searches but no applicable security mechanism has presented itself.
I use Basic authentication and serve all my Twilio content from a dedicated directory which is password protected, Twilio seems quite happy to accept urls with inline username#password parameters.
I think Twilio publish a list of their IP address ranges somewhere too, so if you really want to lock your media directory down you could whitelist those and deny everything else access to that dir within your server config.
To delete them once they are processed I would probably write a basic script that is triggered by the Twilio status webhook and adds the filename of the image which can be deleted to a database table. I think you can pass some sort of verification tokens for Twilio to return with a callback for additional security.
Then run another script every few mins as a cron job (under a different user account with permission to delete files in your media dir) which reads the database, deletes any files listed from the directory and then clears the database ready for the next time.
Edit
Thinking about it you can probably delete the files as soon as Twilio has queued your message as I'm pretty sure they copy your media files to their server upon submission. These files are publicly accessible (but with names nobody is likely to guess). You can delete them with HTTP DELETE
Related
The problem at hand
I have a rails app.
Users will be uploading files. Anywhere between 1 file to 3000 files. Sometimes they are zip files, and sometimes they are not. I do not want hold up the server with these files uploads, so I am looking for a solution around this problem.
The zipped files will have to be unzipped.
I then want to check whether: the user has previously uploaded the same files? i.e. if the user has already uploaded the same file(2) one week ago, then this is a problem: (i) either we don’t allow that particular file to be uploaded, or we ask the user: are you sure you want to upload the same file again?
Then I want to store the keys/links to the files within the appropriate models/records on the back end.
Was wondering what the best workflow for handling the above could be: i.e a very general overview: in other words, could AWS Lambda / Google cloud computing etc. etc be best employed to handle the above problem? How would we use the Shrine gem, to best handle this situation? Would it make sense to use AWS Lambda rather than using background jobs?
My preferences are to use the Shrine gem for uploading.
My Ideas:
In the client side, the user drags and drops the files the user
wants to upload.
All the files are then uploaded (whether zipped or otherwise) to a temporary bucket location via the Shrine gem.
IF the zip files are uploaded then perhaps an AWS lambda function must be triggered to unzip the files. If that’s the case,then at the end of the day, the keys for these files must somehow be returned to the client, to handle validation issues – but then how would the AWS lambda function be able to return this request to the original client side where the request was originated? Or rather,should the AWS lambda function be generated from the client side,passing in the IDs of the unzipped blobs?
Then we need to run some validations: we want to handle the situation where there are duplicate files. We will need to check with our rails backed as to whether those files have already been uploaded.
After those validation issues are handled, then user submits the form, and all the keys are stored within the appropriate records.
These ideas are by no means prescriptive
Am seeking some very general advise on what the best way is of doing this all. I am by no means constrained to AWS: I could use Google or Azure just as easily. Any guidance on the above would be much appreciated.
Specific questions:
How would the AWS lambda function get triggered?
How would be be able to return the keys of the uploaded files back to the client?
What do I mean by general overview?
Here are some examples of general overviews:
(1) Uploading & Unzipping files to S3 through Rails hosted on Heroku?
(2) https://www.quora.com/How-do-I-extract-large-zip-files-in-AWS-Lambda
Any pointers in the right direction would be much appreciated.
Cheers!
This isn't a really difficult problem to solve if you are willing to change the process flow a little bit.
In the client side, the user drags and drops the files the user wants to upload.
When the user requests the upload operation to begin you can make HTTP GET requests to an API Gateway endpoint, backed with a Lambda. The Lambda can query for previous files uploaded by the client and send back a result set showing what files already exist. You then filter those out and send only what is considered new from the client to the server. This will save the user time in waiting for the upload to happen and save you time on the S3/Lambda side of not having to store duplicates or process them. This isn't a substitute for server-side validation though, you'll still want to do that. For legit clients, this will save you and them a lot of bandwidth and storage.
All the files are then uploaded (whether zipped or otherwise) to a temporary bucket location via the Shrine gem.
This works. As they enter the temp bucket, use a Lambda with an S3 event to process the files, unzip files, push any metadata needed into DynamoDb and delete the files from the temp bucket. In the temp bucket, I would place the files into a folder that is unique per request and user. I would take the user/client Id and a UUID of some kind and make that your folder name. Such as Johnathon+3b5339b8-c8db-4d5c-b678-406fcf073f4f, or encode this value into a Base64 string and make that your folder name. Store this in DynamoDb with each file uploaded into your permanent bucket with the Hash Key being the userid/clientid, a Sort Key being the full folder path + file name and an extra attribute of IsProcessed. The IsProcessed attribute will be updated by your Lambda that is processing the files and moving them to their permanent S3 bucket. If there are errors, you can put the error in this field. If it is successful then you put it in this field.
the keys for these files must somehow be returned to the client, to handle validation issues – but then how would the AWS lambda function be able to return this request to the original client side where the request was originated? Or rather,should the AWS lambda function be generated from the client side,passing in the IDs of the unzipped blobs?
The original API request to push the files to the temp S3 bucket would be able to return back to the client the folder name johnathon+3b5339b8-c8db-4d5c-b678-406fcf073f4f to the client. So let's say you made a HTTP POST to /jobs. You would return back 201 Created with a HTTP Header of Location /jobs/johnathon+3b5339b8-c8db-4d5c-b678-406fcf073f4f. Your client can then start polling /jobs/johnathon+3b5339b8-c8db-4d5c-b678-406fcf073f4f for the status of the process.
Your response back to /jobs/johnathon+3b5339b8-c8db-4d5c-b678-406fcf073f4f can return the DynamoDB records. This would include all DynamoDB records for the HashKey matching the folder name. Your client side can look at all of the objects in the result set and check the IsProcessed attribute to see if everything worked out ok, or if there were issues.
Then we need to run some validations: we want to handle the situation where there are duplicate files. We will need to check with our rails backed as to whether those files have already been uploaded.
Handle this with the Lambda that is executed by the temporary bucket. Grab the files from the temp bucket folder, handle your business logic and back-end queries then push them to their final permanent bucket.
After those validation issues are handled, then user submits the form, and all the keys are stored within the appropriate records.
All of this would happen asynchronously, starting when the user submits the form. The client side needs to be able to handle this by making HTTP GET requests to the endpoint mentioned above, checking for the status of the process. This gives you some more flexibility as you can also publish SNS messages on failures as well, such as sending an email to the clients if they upload 3,000 files and you need to spend 30 minutes processing them.
I created an interactive message that gets called via a slash command which gets distributed to the entire channel. Everything works fine. But I can't for the life of me figure out how to limit the message to the entire channel but to the person evoking it. Right now the message goes to everyone including the person that invoked it. If you've ever used /poll, the poll goes out to everyone but the person who created it.
If anyone knows how to do this, can you point me in the right direction?
The response message from a slash command can be only one of two things:
an ephemeral response, which can be seen by the user who issued the command only
an in_channel response, which can be seen by everyone in the channel
There is no feature or switch so that the response message would not be visible to the issuing user.
You can however build a workaround and manually send every user except the one issuing the command an ephemeral message. Here is an outline of how that would work:
Get the list of users of the channel via conversations.members
Send each user an ephemeral message via chat.postEphemeral
There are some significant caveat to this workaround:
You app needs a lot of additional scopes to be allowed to retrieve the member list and send messages (see documentation of those API methods for details)
There is rate limit of about 1 message per second, so it might take quite some time to send those message to all user depending on how big the group is
your slash command needs to respond with 3 seconds. So you will need to implement some fancy multi-threading to be able to send all those messages.
To make that work in private channels you have to work with an additional bot user
That would get you the result you are asking for, but there are some caveats:
Context of what I'm trying to accomplish:
User shares a file with the bot
Other users interact with the bot via a dialog
The bot shares the original file to the other users
For example, we want to share a file to the bot that contains this week's cafeteria menu. Each time users would interact with the bot in a certain way, it would share the cafeteria menu with them so that they can consult it.
I've tried calling files.share method but bots can't perform this action (get invalid token type error).
As far as I can tell, there is no way to do this currently. I've tried link unfurling in the message body but that only works if the file itself was already shared to the user. If not, the link simply won't unfurl and clicking it will fail.
The bot can perform a files.upload call and re-upload the contents of the file to each user individually. This seems incredibly wasteful but appears to be the only way to work currently.
Is there something I'm missing?
The reason your bot can not use file.share is that this is an undocumented API method and you need a legacy token to use it. No other token (user token, bot token) will work, because it requires the post scope, which only exists for legacy token.
Approach A: Legacy Token
So one approach would be to use a legacy token with your bot, which you can create here for your current workspace. That should work nicely if your Slack app is only used on your "own" Slack workspace where you can create and use a legacy token.
Approach B: File Mention
Another approach is to use the mention feature in messages to share a file. This works by sending the private link (url_private property) of an already shared file in a message to a new channel. This will automatically re-share the file in that channel. I believe this only works with files that how been previously shares in a public channel and can therefore be re-shared. Be aware though that the file mention feature is currently being reworked, so this behavior might change.
Example:
https://slack.com/api/chat.postMessage?token=TOKEN&channel=CHANNEL&as_user=true&text=URL_PRIVATE
For more details see the Slack tutorial Storing, retrieving, and modifying file uploads.
Approach C: External File / image file
If you host your file externally or create a public URL for a file uploaded to Slack you can share it in every channel by just adding the URL to a message. Slack will automatically unfurl it and therefore share it to the user in any channel. This is different to Approach B, because its not a file mention and requires a public URL. You get the public URL of an uploaded file by calling files.sharedPublicURL.
If i'm not wrong, you can do like this :
you share a file with your bot
you retrieve the file shared ID, so his url_private property (cf https://api.slack.com/types/file#authentication)
you then donwload the file
you can then re-share it several times later (without re-uploading to each user)...
I have a specific issue where I'm using WebRTC (voice and video).
I want to keep a connection/voice/video streams alive between webpages on a website. I thought I could use shared web workers to run in the background?
Any guidance would be great. I've looked at other posts but they're quite old and wondered if anyone had any, more, up-to-date information or ways I could tackle this issue?
UPDATE:
Shared Web Workers are the incorrect way of tackling this problem. Service Workers are the way forward for maintaining after the web page is terminated.
Keeping the webRTC connection alive between page loads seems like a rare use case. Normally, you start a call and remain on a single page. I guess it could make sense if you wanted to embed a customer support like webRTC widget on a site and have that widget follow a user through page navigations under a single domain.
I don't think saving/reusing the blob URL will allow you to reconnect on a page reload for security issues/hijacking potential.
There is the IceRestart constraint which might help. Apparently you can save the SDP info to local storage, reuse the negotiated SDP, then call an IceRestart to quickly reconnect.
As described in section 3, the nominated ICE candidate pair is
exchanged during an SDP offer/answer procedure, which is maintained
by the JavaScript. The JavaScript can save the SDP information on
the application server or in browser local storage. When a page
reload has happened, a new JavaScript will be reloaded, which will
create a new PeerConnection and retrieve the saved SDP information
including the previous nominated candidate pair. Then the JavaScript
can request the previous resource by sending a setLocalDescription(),
which includes the saved SDP information. Instead of restart an ICE
procedure without additional action hints, the new JavaScript SHALL
send an updateIce() which indicates that it has happended because of
a page reload. If the ICE agent then can allocate the previous
resource for the new JavaScript, it will use the previous nominated
candidate pair for the first connectivity check, and if it succeeds
the ICE agent will keep it marked as selected. The ICE agent can now
send media using this candidate pair, even if it is running in
Regular Nomination mode.
https://bugs.chromium.org/p/webrtc/issues/detail?id=979
https://datatracker.ietf.org/doc/html/draft-li-rtcweb-ice-page-reload-02
I'm building a Ruby on Rails app, and I'd like to integrate some Office365 features.
For instance : I would like to download a file from OneDrive and then attach it to an Email in order to send it via Outlook rest API.
I found this get Item content OneDrive REST API but I dont understand how to use it.
I understand that I have to send a GET request (formated as explained in msdn.microsoft.com) with Rails, which will then provide me a "a pre-authenticated download URL" to download the file.
Then I will have to send a second GET request with this a pre-authenticated download URL to start the download, but I don't understand how to deal with the Response in order to save the file into a variable.
How can I retrieve the file into a variable of my Ruby on Rails App, so that I can attach it to an Email with an Outlook REST API to send it from my own Rail controller ?
Also this workflow is really not optimized in term of Bandwidth and Processing (3 REST API request + 1 download + 1 upload), it will work.
However if it exist a single REST API that direclty attach a OneDrive file to an email to send it, that would ease a lot my life, save energy, save money from Microsoft datacenter, and spare the planet ecology.
Any tutorial, examples, or more explanatory doc would be much appreciated.
--- EDIT ---
Adding link to the email is not wished as the email may have to be send to someone outside of Office365 users, and public link are a security issue for confidential documents.
Any help is welcome.
There isn't a single REST API call you can make currently to do what you want, although being able to easily attach a file from OneDrive to a new email message is a great scenario for Microsoft Graph API, it just isn't supported right now.
If you want to attach the file, you need to do as you mentioned, download the contents of the file, and then upload it again as an attachment to the message.
However, I'd recommend sending a link to the file instead, even though you mentioned you don't want to do that. OneDrive for Business now supports "company shareable links" which are scoped to just the user's organization instead of being available totally anonymously.
Something else to consider: The security concerns of sending an anonymous link aren't that different than sending an attached file. In fact, the anonymous link can be more secure, because access to the file can be monitored and revoked in the future (unlike the attachment, which will always be out there).