We are looking to support export of photos from our S3 location to users' Dropbox. Currently I am using the code like below:
#photo = Photo.find(id) #Photo.image has attachment
#photo.image.copy_to_local_file(nil, 'tmp/png/temp.png') #Get the file locally from S3
local_file = File.open('tmp/png/temp.png')
response = client.put_file('sample.png', local_file) # Then copy to Dropbox
The above method costs twice the bandwidth. Is there anyway I can transfer the images directly from S3 to Dropbox without copying them locally?
Thanks in advance!
How about trying something like mover and use their APIs?
https://mover.io/
http://support.mover.io/knowledgebase/articles/214572-how-to-transfer-or-backup-your-amazon-s3-buckets-t
Or you can also try SME Storage (Storage Made Easy)
http://storagemadeeasy.com/
It's kinda ironic that DropBox uses Amazon S3 to stores all its files.
Or you can also write your own streamer in Ruby and run it in an Amazon instance it will be much faster since all the data would be within Amazon.
How do I HTTP post stream data from memory in Ruby?
Related
I'd like to implement uploading a profile picture for users. I'm using a VueJs frontend with a Rails API. What I'm trying to do is upload the image only using the frontend. I'd like for the file to get uploaded without any calls API calls. I could then store the location of the file in the picture attribute in the backend and retrieve it. Is that possible? I'm also using Element library.
<el-upload :http-request="addAttachment">
<el-button size="small" type="primary">Click Upload</el-button>
</el-upload>```
What you are looking at is called,
direct uploads or browser based uploads.
There should be support from storage service you are using.
Example: using S3 and GCS it is possible.
Upload without any API calls? -
Not sure, I once had to make a small API call to get the signature key and use it with POST params to upload file to storage service(GCS)
Once the API response is returned, you then might want to write to db about the file path.
I am working on a project where the user joins a "stream". During stream setup, the person who is creating the stream (the stream creator) can choose to either:
Upload all photos added to the stream by members to our hosting solution (S3)
Upload all photos added to the stream by members to the stream creator's own Dropbox authenticated folder
In the future I would like to add more storage providers (such as Drive, Onesky etc)
There is a couple of different questions I have in regards to how to solve this.
What should the structure be in the database for photos? I currently only have photo_url, but that won't be easy to manage from a data perspective with pre-signed urls and when there are different ways a photo can be uploaded (s3, dropbox etc.)
How should the access tokens for each storage provider be stored? Remember that only the stream creator's access_token will be stored and everyone who is on the stream will share that token when uploading photos
I will add iOS and web clients in the future that will do a direct upload to the storage provider and bypass the server to avoid a heavy load on the server
As far as database storage, your application should dictate the structure based on the interface that you present both to the user and to the stream.
If you have users upload a photo and they don't get to choose the URI, and you don't have any hierarchy within a stream, then I'd recommend storing just an ID and a stream_id in your main photo table.
So at a minimum you might have something looking like
create table photos(id integer primary key, stream_id integer references streams(id) not null);
But you probably also want description and other information that is independent of storage.
The streams table would have all the generic information about a stream, but would have a polymorphic association to a class dependent on the type of stream. So you could use that association to get an instance of S3Stream or DropBoxStream based on what actual stream was used.
That instance (also an ActiveRecord resource) could store the access key, and for things like dropbox, the path to the folder etc. In addition, that instance could provide methods to construct a URI given your Photo object.
If a particular technology needs to cache signed URIs, then say the S3Stream object could reference a S3SignedUrl model where the URIs are signed.
If it turns out that the signed URL code is similar between DropBox and S3, then perhaps you have a single SignedUrl model.
When you design the ios and android clients, it is critical that they are not given access to the stream owner's access tokens. Instead, you'll need to do all the signing inside your server app. You wouldn't want a compromise of a device to lead to exposing the access token creating billing problems as well as privacy exposures.
Hope this helps.
we setup a lot of rails applications with different kind of file storages behind it.
Yes, just an url is not manageable in the future. To save a lot of time you could use gems like carrierwave or paperclip. They handle all the thumbnail generation and file validation. One approach is, that you could upload the file from the client directly to S3 or Dropbox to a tmp folder and just tell your Rails App "Hey, here is the url of a new upload file" and paperclip and carrierwave will take care of the thumbnail generation and storaging. (Example for paperclip)
Don't know exactly how your stream works, so I cannot give a good answer to this -.-
With the setup I mentioned in 1. you should upload form your different clients directly to S3 or Dropbox etc. and after uploading, the client tells the Rails Backend that it should import the file from that url. (And before paperclip or carrierwave finish their processing you could use the tmp url from the file to display something directly in your stream)
Is it possible to use carrier wave to upload directly to amazon's S3 without using my server?
What i mean is, I don't want the images first going to my ec2 instance, and then uploaded to s3. I believe there is a way to upload directly to S3 to save my server's resources from having to process/stream the file.
I am just looking into carierwave, does it support nice html5 uploads where the user can just drag and drop the file on the web page?
If you want to upload directly to S3 from the browser you must do it with Javascript.
Heroku provides a nice tutorial : https://devcenter.heroku.com/articles/direct-to-s3-image-uploads-in-rails
Once uploaded, you can pass the finale S3 public URL of the image in a hidden field and download it server-side with carrierwave for further manipulation (resizing, ...)
I am using Amazon S3 storage for my files and i need to provide download functionality in which user can select multiple files and download them at once (as dropbox)...
I tried to implement this functionality by downloading each file in memory stream and create a zip file and returned to user, but its too much time consuming, I need to know that is there any way that this process can be implemented asynchronously that user don't have to be wait longer and downloading starts immediately as dropbox do...
I am using MVC Web API...
Thanks in advance...
From a high level, I would probably do something like this.
Trigger a job with the file set the user wants to download. Have a worker that downloads the files from s3, compresses them into a zip file and then uploads it back to a temporary location in s3.
Once the job is completed, send the user a signed URL to the zip file itself.
Clean up the zip file after a certain amount of time. Maybe 24 hours?
I would like to know if Google App Engine can be used as a Content delivery network like aws S3. I'm running a RoR app on Heroku and I would like store my uploaded files on GAE instead of s3.
If it's possible what would be the best way to do it?
http://24ways.org/2008/using-google-app-engine-as-your-own-cdn
It won't be able to host files over 1MB though.
Make sure to read through the comments on that blog post as well, some have concerns about the terms of service.
GAE in itself isn't meant to be a CDN... that doesn't, however, stop you from writing a CDN application on top of it. The only limit you'll need to worry about is the 50 MB limit on the size of the blobstore. Such an app will have to provide a URL that you can hit to get the upload URL, which could then be used to upload the file. The download url can also be generated with the upload URL, and used to access the content.