I'm using Shrine with Rails to upload my images directly to S3, they are then served using Cloudfront.
My setup is working great, but I'm stuck trying to answer 2 issues:
1) User uploads a new Image via Uppy. They click "Create Image Page" which creates a new "page" object. The user is taken to a blank page, while the Image derivative is being created in a background process using Sidekiq. How can we have the image "pop-in" via JS once the derivative is successfully created and promoted to /store? Is there a callback we can pick up on, or is it possible to keep trying to find the derivative via JS until it exists?
2) Using the Derivatives Endpoint plugin, all of the image URLs are relative. Is there a way to serve this as absolute URLs from Cloudfront/Custom Domain? Here's an example using Rails:
Ruby code:
<%= image_tag(image.image_file.derivation_url(:banner, 800, 300)) %>
Resulting HTML:
<img src="/derivations/images/banner/800/300/...">
How can I instead serve this resulting URL:
<img src="http://abc123.cloudfront.net/derivations/images/banner/800/300/...">
Was this intentional to prevent DoS? Thanks.
Adding the host option to "derivation_endpoint" causes the following error returned as XML:
<Error>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
<RequestId>...</RequestId>
<HostId>...</HostId>
</Error>
Could this be my Cloudfront access settings? Maybe CORS.
Here is my derivation_endpoint setup in image_uploader.rb
plugin :derivation_endpoint, host: "http://abc123.cloudfront.net", prefix: "derivations", secret_key: ..., upload: true
routes.rb
mount Shrine.presign_endpoint(:cache) => "s3/params"
mount PhotoUploader.derivation_endpoint => "derivations/images"
config/initializers/Shrine.rb
Shrine.plugin :url_options, store: { host: "http://abc123.cloudfront.net" }
Since the background job will update the record with derivatives, the easiest would be to poll the "show" route of the record (i.e. GET /images/:id) on the client side. In this case the route could have an alternative JSON response via respond_to.
Alternatively, you could send a message from the background job to the client once processing is finished, via WebSockets or something like message_bus.
You can configure the :host option for the derivation_endpoint plugin:
Shrine.plugin :derivation_endpoint, host: "https://cloudfront.net"
Related
Please bare with me for a question for which it's nearly impossible to create a reproducible example.
I have an API setup with FastAPI using Docker, Serverless and deployed on AWS API Gateway. All routes discussed are protected with an api-key that is passed into the header (x-api-key).
I'm trying to accomplish a simple redirect from one route to another using fastapi.responses.RedirectResponse. The redirect works perfectly fine locally (though, this is without api-key), and both routes work perfectly fine when deployed on AWS and connected to directly, but something is blocking the redirect from route one (abc/item) to route two (xyz/item) when I deploy to AWS. I'm not sure what could be the issue, because the logs in CloudWatch aren't giving me much to work with.
To illustrate my issue let's say we have route abc/item that looks like this:
#router.get("/abc/item")
async def get_item(item_id: int, request: Request, db: Session = Depends(get_db)):
if False:
redirect_url = f"/xyz/item?item_id={item_id}"
logging.info(f"Redirecting to {redirect_url}")
return RedirectResponse(redirect_url, headers=request.headers)
else:
execution = db.execute(text(items_query))
return convert_to_json(execution)
So, we check if some value is True/False and if it's False we redirect from abc/item to xyz/item using RedirectResponse(). We pass the redirect_url, which is just the xyz/item route including query parameters and we pass request.headers (as suggested here and here), because I figured we need to pass along the x-api-key to the new route. In the second route we again try a query in a different table (other_items) and return some value.
I have also tried passing status_code=status.HTTP_303_SEE_OTHER and status_code=status.HTTP_307_TEMPORARY_REDIRECT to RedirectResponse() as suggested by some tangentially related questions I found on StackOverflow and the FastAPI discussions, but that didn't help either.
#router.get("/xyz/item")
async def get_item(item_id: int, db: Session = Depends(get_db)):
execution = db.execute(text(other_items_query))
return convert_to_json(execution)
Like I said, when deployed I can successfully connect directly to both abc/item and get a return value if True and I can also connect to xyz/item directly and get a correct value from that, but when I pass a value to abc/item that is False (and thus it should redirect) I get {"message": "Forbidden"}.
In case it can be of any help, I try debugging this using a "curl" tool, and the headers I get returned give the following info:
Content-Type: application/json
Content-Length: 23
Connection: keep-alive
Date: Wed, 27 Jul 2022 08:43:06 GMT
x-amzn-RequestId: XXXXXXXXXXXXXXXXXXXX
x-amzn-ErrorType: ForbiddenException
x-amz-apigw-id: XXXXXXXXXXXXXXXX
X-Cache: Error from cloudfront
Via: 1.1 XXXXXXXXXXXXXXXXXXXXXXXXX.cloudfront.net (CloudFront)
X-Amz-Cf-Pop: XXXXX
X-Amz-Cf-Id: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
So, this is hinting at a CloudFront error. Unfortunately I don't see anything slightly hinting at this API when I look into my CloudFront dashboard on AWS, there literally is nothing there (I do have permissions to view the contents though...)
The API logs in CloudWatch look like this:
2022-07-27T03:43:06.495-05:00 Redirecting to /xyz/item?item_id=1234...
2022-07-27T03:43:06.495-05:00 [INFO] 2022-07-27T08:43:06.495Z Redirecting to /xyz/item?item_id=1234...
2022-07-27T03:43:06.496-05:00 2022-07-27 08:43:06,496 INFO sqlalchemy.engine.Engine ROLLBACK
2022-07-27T03:43:06.496-05:00 [INFO] 2022-07-27T08:43:06.496Z ROLLBACK
2022-07-27T03:43:06.499-05:00 END RequestId: 6f449762-6a60189e4314
2022-07-27T03:43:06.499-05:00 REPORT RequestId: 6f449762-6a60189e4314 Duration: 85.62 ms Billed Duration: 86 ms Memory Size: 256 MB Max Memory Used: 204 MB
I have been wondering if my issue could be related to something I need to add to somewhere in my serverless.yml, perhaps in the functions: part. That currently looks like this for these two routes:
events:
- http:
path: abc/item
method: get
cors: true
private: true
request:
parameters:
querystrings:
item_id: true
- http:
path: xyz/item
method: get
cors: true
private: true
request:
parameters:
querystrings:
item_id: true
Finally, it's probably good to note that I have added custom middleware to FastAPI to handle the two different database connections I need for connecting to other_items and items tables, though I'm not sure how relevant this is, considering this functions fine when redirecting locally. For this I implemented the solution found here. This custom middleware is the reason for the redirect in the first place (we change connection URI based on route with that middleware), so I figured it's good to share this bit of info as well.
Thanks!
As noted here and here, it is mpossible to redirect to a page with custom headers set. A redirection in the HTTP protocol doesn't support adding any headers to the target location. It is basically just a header in itself and only allows for a URL (a redirect response though could also include body content, if needed—see this answer). When you add the authorization header to the RedirectResponse, you only send that header back to the client.
A suggested here, you could use the set-cookie HTTP response header:
The Set-Cookie HTTP response header is used to send a cookie from the
server to the user agent (client), so that the user agent can send it back to
the server later.
In FastAPI—documentation can be found here and here—this can be done as follows:
from fastapi import Request
from fastapi.responses import RedirectResponse
#app.get("/abc/item")
def get_item(request: Request):
redirect_url = request.url_for('your_endpoints_function_name') #e.g., 'get_item'
response = RedirectResponse(redirect_url)
response.set_cookie(key="fakesession", value="fake-cookie-session-value", httponly=True)
return response
Inside the other endpoint, where you are redirecting the user to, you can extract that cookie to authenticate the user. The cookie can be found in request.cookies—which should return, for example, {'fakesession': 'fake-cookie-session-value-MANUAL'}—and you retrieve it using request.cookies.get('fakesession').
On a different note, request.url_for() function accepts only path parameters, not query parameters (such as item_id in your /abc/item and /xyz/item endpoints). Thus, you can either create the URL in the way you already do, or use the CustomURLProcessor suggested here, here and here, which allows you to pass both path and query parameters.
If the redirection takes place from one domain to another (e.g., from abc.com to xyz.com), please have a look at this answer.
I am using an ajax request which works in local, not in remote, because of an url problem. It looks like :
$.ajax({
type: "POST",
url: "../classes/file_to_process.php",
data: "my data"
success: function(msg){...}
})
I keep on having an error message : "The requested URL /classes/file_to_process.php {without the double dots behind it} was not found on this server"
My working directory is in a folder /prod, in which there is the index.php. The /classes folder is at the same level as /prod. So to fetch it from an jquery request, I use ../classes/file_to_process
I tried an absolute path by using pwd to fetch the correct path on the remote server, but I have the same message
Anybody has an idea ?
'classes' folder is on the same level as 'public', then you can't access it directly from the client (AJAX, JavaScript, etc). You need to either put it in the 'public' or map it to /classes virtual path. Or you can have a trusted .php file in your 'public' folder that accesses the 'classes' on the server side.
TL;DR;
From the client side you cannot access a file that is not being served to the client.
Im writing an app to store files on Amazon S3. Im pretty close, Im able to save and retrieve files using a url. However since these links are public Im trying to use the following method in my assetscontroller to retrieve stored files from S3.
As links these files can be viewed/accessed in the browser, but if I use this code :
#This action will let the users download the files (after a simple authorization check)
def get
asset = current_user.assets.find_by_id(params[:id])
if asset
#Parse the URL for special characters first before downloading
data = open("#{URI.parse(URI.encode(asset.uploaded_file.url))}")
#then again, use the "send_data" method to send the above binary "data" as file.
send_data data, :filename => asset.uploaded_file_file_name
else
flash[:error]="Access Violation"
redirect_to assets_path
end
end
Im getting this error in my browser:
Errno::ENOENT in AssetsController#get
No such file or directory - http://s3.amazonaws.com/BUCKETNAME/assets/29/FILENAME.jpeg? 1339979591
When I click on the resource on the S3 site as Im logged into the S3 management console, the file is shown in my browser and its link is
https://s3.amazonaws.com/BUCKETNAME/assets/29/FILENAME.jpeg? AWSAccessKeyId=XXXXXXXXXXXExpires=1340003832&Signature=XXXXXXXXXXXXXXXXXX-amz-security- token=XXXXXXXXX//////////XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
So it does exist but cant be accessed through my app
Here is my Application Trace from my browser:
app/controllers/assets_controller.rb:99:in `initialize'
app/controllers/assets_controller.rb:99:in `open'
app/controllers/assets_controller.rb:99:in `get'
Any clues on whats going on?
Thanks
You can also just redirect the user to the file on S3.
Just try
redirect_to asset.uploaded_file.url
instead of send_file. send_file expects a path to a local file, which is then used by the webserver.
If you've set s3_permissions => :private, then you need to call
redirect_to asset.uploaded_file.expiring_url(10)
It's also interesting that in your error message it is http against https in s3 - you can also try to add the following option to your model's has_attached_file
:s3_protocol => 'https'
Hope this helps.
Because it is not possible to uploade files to a Heroku app.
I am using a Amazon S3 storage server for images.
The problem is just that the image url is to the Amazon S3 server.
I would like it to be mydomain.com/myimage.png instead of s3.amazon.com/bucket/myimage.png
How is it possible to show a image on a amazon s3 server when visiting example: mydomain.com/myimage.png ?
My solution, I only use png images:
In routes:
match "/images/vind/:style/:id/:basename.png" => "public#image_proxy"
In public controller:
def image_proxy
image_url = "http://s3-eu-west-1.amazonaws.com/konkurrencerher#{request.path}"
response.headers['Cache-Control'] = "public, max-age=#{84.hours.to_i}"
response.headers['Content-Type'] = 'image/png'
response.headers['Content-Disposition'] = 'inline'
render :text => open(image_url, "rb").read,
end
You can create a rack middleware that will look for a pattern in the url (/images/* or *.png) and when there's a match, it will act as a proxy, requesting the image from S3 and serving the content that it receives.
Make sure you set the caching headers right, so that Heroku's reverse proxy will cache it and serve it quickly.
I am working on a Rails App that Uses OmniAuth to gather Oauth/OAuth2 credentials for my users and then posts out to those services on their behalf.
Creating simple posts to update status feeds work great.. Now I am to the point of needing to upload files. Facebook says "To publish a photo, issue a POST request with the photo file attachment as multipart/form-data." http://developers.facebook.com/docs/reference/api/photo/
So that is what I am trying to do:
I have implemented the module here: Ruby: How to post a file via HTTP as multipart/form-data? to get the headers and data...
if appearance.post.post_attachment_content_type.to_s.include?('image')
fbpost = "https://graph.facebook.com/me/photos"
data, headers = Multipart::Post.prepare_query("title" => appearance.post.post_attachment_file_name , "document" => File.read(appearance.post.post_attachment.path))
paramsarray = {:source=>data, :message=> appearance.post.content}
response = access_token.request(:post, fbpost, paramsarray, headers)
appearance.result = response
appearance.save
end
I but I am getting a OAuth2::HTTPError - HTTP 400 Error
Any assistance would be Incredible... As I see this information will also be needed for uploading files to SoundCloud also.
Thanks,
Mark
Struggled with this myself. The oauth2 library is backed by Faraday for it's HTTP interaction. with a little configuration it supports uploaded files out of the box. First step is to add the appropriate Faraday middleware when building your connection. An example from my code:
OAuth2::Client.new client_id, secret, site: site do |stack|
stack.request :multipart
stack.request :url_encoded
stack.adapter Faraday.default_adapter
end
This adds the multipart encoding support to the Faraday connection. Next when making the request on your access token object you want to use a Faraday::UploadIO object. So:
upload = Faraday::UploadIO.new io, mime_type, filename
access_token.post('some/url', params: {url: 'params'}, body: {file: upload})
In the above code:
io - An IO object for the file you want to upload. Can be a File object or even a StringIO.
mime_type - The mime type of the file you are uploading. You can either try to detect this server-side or if a user uploaded the file to you, you should be able to extract the mime type from their request.
filename - What are are calling the file you are uploading. This can also be determined by your own choosing or you can just use whatever the user uploading the file calls it.
some/url - Replace this with the URL you want to post to
{url: 'params'} - Replace this with any URL params you want to provide
{file: upload} - Replace this with your multipart form data. Obviously one (or more) of the key/value pairs should have an instance of your file upload.
I'm actually using successfully this code to upload a photo on a fb page :
dir = Dir.pwd.concat("/public/system/posts/images")
fb_url = URI.parse("https://graph.facebook.com/#{#page_id}/photos")
img = File.open("myfile.jpg")
req = Net::HTTP::Post::Multipart.new(
"#{fb_url.path}?access_token=#{#token}",
"source" => UploadIO.new(img, "application/jpg", img.path),
"message" => "some messsage"
)
n = Net::HTTP.new(fb_url.host, fb_url.port)
n.use_ssl = true
n.verify_mode = OpenSSL::SSL::VERIFY_NONE
n.start do |http|
#result = http.request(req)
end