From what I found both libraries aws-sdk and aws-s3 (Ruby) provide methods to download S3 object data only in string chunks. I'd like to be able to read it in binary chunks, so that it resembles file I/O and be more efficient. Does anyone know if there is a supported way to do it? One hack that I thought about is creating an access URL for the S3 object (S3 functionality) and download the file with some HTTP client library. What's a good library for that in Ruby?
P.S.: I need to stream data, so that I can decrypt files on-the-fly. AWS has client-side encryption library only in Java SDK.
So, it turns out in ruby (since 1.9?) a string of encoding ASCII-8BIT is used as a binary block, so you have to work with that. P.S.: net/http is a ruby http client library that can be used to read response body.
Related
I'm writing rails application. My task is to show data from old java application. One of the objects has images attached, I have to display these images and allow user to add new ones.
I want to know what is the best way to let user uploading images to old app.
Old app has http post method for adding images with params: multipart file, md5 hash of the image and object id.
I tried using paperclip or carrierwave but their documentation is about saving uploaded images and I just want to transfer them somewhere else and not save anything.
there are few ways to communicate with java from ruby.
REST API - expose Java functionalities through endpoints. then use an HTTP client.
Remote Procedure Call - Apache Thrift
JRuby - embedded java programme into Ruby
however, I think maybe you can just proxy the Java API without going through the rails application.
I have a rails-api project, which provide the api to access my data.
I use carrierwave to store my file, my model called User and file attribute called image.
So, the image attribute contained the file_name, url and some other info.
In order to translate the file through the api, I added the gem carrierwave-base64.
I understand the Upload process. The client app encode the file to base64 code, then sent to backend by a json message. For example:
{user: {email: "test#email.com", image: "data:image/jpg;base64,#{base64_image}"}
So when the backend receive the json request, the carrierwave will parse the base64 code to a file and store it to local or S3
What I do not understand, is the Download process:
When I request the user info, what I assume is that the image file would be transfered as a base64 code in a json message, and then the client app will encode the base64 code to a file(image), and then display.
But actually, what I can provide for the json data, is the file url, not the base64 code.
The reason I want to get the file(image) from the api-server is because I don't want to the client app directly access s3 by url. So every time when the client app want to get a file, it will request the api-server, and api-server will get the file and transfer to the client.
Does anyone can explain how to do the download?
Or if I was thought in a wrong strategy, that I need another api endpoint to response a file object, not just accompany with user model.
Cheers.
Restricting Access to Objects Stored on Amazon S3
https://github.com/thoughtbot/paperclip/wiki/Restricting-Access-to-Objects-Stored-on-Amazon-S3
you did a good thing with uploading ,But while downloading you need to send URL no base64 and its traditional
Also for securrity purpose you can put public read permission on s3 while uploading and use expiring_url(60, :thumb) for your clients
In this URL get expired in time that you have specified
I'm trying to make an app where I take pictures from users add them to a canvas, draw stuff in them, then convert them to a base64 string and upload them.
For this purpose I'm considering the possibility to use a cdn but can't find information on what I can upload to them and how the client side uploading works. I'd like to be able to send the image as base64 and the name to be given to the file, so that when it arrives to the origin cdn, the base64 image is decoded and saved under the specified name (which I will add to the database on the server).Is this possible?Can I have some kind of save.php file on the origin cdn where I write my logic to save the file and to which I'll send XHR requests? Or how this whole thing work?I know this question may sound trivial but I'm looking for it for hours and still didn't find anything which explains in detail how the client side uploading work for CDNs.
CDNs usually do not provide such uploading service for client side, so you can not do it in this way.
I'm writting api for webservice, and i need to provide ability for users to upload files using this api.
For my uploads i'm using carrierwave, but not sure how do i pass file from api request to carrierwave, and how the file should be sent from client machine to server
Basically it is based on the tool which is used by the API consumer. If API consumer is using ruby then can consume it by passing File object, or using httmultiparty gem we can upload the file.
For you reference https://github.com/jwagener/httmultiparty. Please let me know if you need more help.
Usually I implement file uploads over a REST API by allowing clients to send a PUT request with the base64 encoded binary data of the file inside the request body.
Then you can route the client request to your CarrierWave uploader, which can decode the binary data contained in the request body using something like FilelessIO.new(Base64.decode64(encoded_file))
Try RestClient. It encapsulates net/http with cool features like multipart form data:
require 'rest_client'
RestClient.post('http://localhost:3000/foo',
:name_of_file_param => File.new('/path/to/file'))
It also supports streaming.
gem install rest-client will get you started.
So here's the thing. I believe my case is pretty particular at this point, and some help from the experts it's highly advisable.
I have an API built on Rails (3.2.6) and what I want to able to do is receive a video file (mostly .mp4, .avi) and upload it to s3 through a Process Queue (Using Resque).
Now I'm kind of lost on how to do this. To my understanding, I would be receiving a byte[] (Array of bytes) through the request which is the video and send that as a param to my Resque job in order to upload it (Resque Job params can only be strings, not objects)?
Has anyone had any experience doing this sort of procedure. We're pretty much trying to mimic the http://docs.brightcove.com/en/media/ create_video method. Where a Video object can be created either by sending the direct file in the request or the link the file....
Any suggestions?
Try using the CarrierWave gem. You should allow someone to HTTP POST the file data to your API, and then save that file data on the backend and upload it to S3 using CarrierWave.