I have a service that generates S3 pre-signed PUT URLS for uploading.
While testing I noticed that Postman would have no issue uploading to the pre-signed URL as long as I clicked on the URL in "Pretty" mode. This would put the URL in its address bar.
But when I copied the raw output, it would 403 every time.
Does AWS S3 require & signs to be replaced with &?
EDIT: Updating to show how pre-signed URL is fetched with Node/Express/AWS API
router.get('/', function (req, res) {
var s3 = new AWS.S3();
var params = {Bucket: 'some-s3-bucket', Key: 'test/'};
s3.getSignedUrl('putObject', params, function (err, url) {
res.json({url: url});
});
});
No, you shouldn't html-encode & → & in the query string separators in an S3 signed URL.
It sounds like your service is generating the urls and either it, or something in the chain, is incorrectly html-encoding them, and by luck, postman or the browser is cleaning them back up again when you view them in "pretty" mode.
If you're deliberately doing this, then don't... but if you're unclear where the spurious encoding is coming from, switching to a text based tool for talking to your service -- like curl or wget -- is likely to get you closer to understanding what's going on than a browser-based tool.
Related
I have a python method that successfully creates a GET Signed Url that will download the video that is in the Google Cloud Bucket.
def _MakeUrlForApp(self, verb, path, content_type='', content_md5=''):
"""Forms and returns the full signed URL to access GCS."""
base_url = '%s%s' % (self.gcs_api_endpoint, path)
signature_string = self._MakeSignatureString(verb, path, content_md5,
content_type)
signature_signed = self._Base64Sign(signature_string)
"""replace # with %40 - and + with %2 and == with %3D"""
signature_signed = signature_signed.replace("+", "%2B")
signature_signed = signature_signed.replace("/", "%2F")
signature_signed = signature_signed.replace("=", "%3D")
self.client_id_email = self.client_id_email.replace("#", "%40")
signedURL = base_url + "?Expires=" + str(self.expiration) + "&GoogleAccessId=" + self.client_id_email + "&Signature=" + signature_signed
print 'this is the signed URL '
print signedURL
return signedURL
This is called in ios swift with a get post with http. It returns the signed url and it downloads the video to the ios app.
This method here, If i specify the bucketname, the objectname, text/plain as content type, and a couple words for the data, It creates and puts that file into the Google Cloud bucket for me.
def Put(self, path, content_type, data):
"""Performs a PUT request.
Args:
path: The relative API path to access, e.g. '/bucket/object'.
content_type: The content type to assign to the upload.
data: The file data to upload to the new file.
Returns:
An instance of requests.Response containing the HTTP response.
"""
md5_digest = base64.b64encode(md5.new(data).digest())
base_url, query_params = self._MakeUrl('PUT', path, content_type,
md5_digest)
headers = {}
headers['Content-Type'] = content_type
headers['Content-Length'] = str(len(data))
headers['Content-MD5'] = md5_digest
return self.session.put(base_url, params=query_params, headers=headers,
data=data)
What I want to know is one of these two things and nothing else. How do I upload data from a video to this data parameter in my python webapp2.requestHandler from ios? OR How do I get the correct put signed Url to upload video data?
Please do not comment with anything that will not solve this specific question and do not bash me for my methods. Please provide suggests that you feel will specifically help me and nothing else.
There are a few ways to upload images to GCS, and each way works with signed URLs. If the video files are small, your simplest option is to have the users perform a non-resumable upload, which has the same URL signature except that the verb is PUT instead of GET. You'll also need to add the "Content-Type" header to the signature.
Video files can be fairly large, though, so you may prefer to use resumable uploads. These are a bit more complicated but do work with signed URLs as well. You'll need to use the "x-goog-resumable: start" header (and include it in the signature) and set "Content-Length" to 0. You'll get back a response with a Location header containing a new URL. Your client will then use that URL to do the upload. Only the original URL needs to be signed. The client can use the followup URL directly.
I have tried to send files using multipart form data, but all I get is the following error:
{"description":"Error: Bad Request: chat_id is empty","error_code":400,"ok":false}
Here is my code snippet. Can anyone help me out where I am committing an error?
public HttpResponse<jsonnode> sendDocument(Integer chat_id,File f1) throws UnirestException {
return Unirest.post(endpoint + token + "/sendDocument")
header("accept", "application/json")
.field("chat_id", chat_id)
.field("document", f1)
.asJson();
}
well, it said chat_id is empty. that's an obvious error!
but for your question, there is only two ways to sendDocument to telegram.
that file is already is in their servers, so you should just pass the file_id in "document" field
you want to upload a file from your device and as they said
Must be posted using multipart/form-data in the usual way that files are uploaded via the browser
if you are doing the upload section correct, then just make sure chat_id is not empty.
In my Apigee API Proxy, I need to get the environment URL that is defined in my configuration, so that I can send it as part of the response.
For example: http://myorg-test.apigee.net/pricing
However, when I try to get it using proxy.url, I get an aliased path, like http://rrt18apigee.us-ea.4.apigee.com/pricing
Currently, I'm trying to get it like:
var response = {
proxyUrl : context.getVariable("proxy.url"),
};
Here is a work around. You can try to get the following variables and create the entire URL
Get the request scheme (http or https) from request.Headers.X-Forwarded-Proto (if you are using cloud version) or client.scheme if you are using on-prem
Get the host name from request.host
Get the entire request path from request.path
Entire list of URL query params and list from message.querystring
You can then construct the entire request URL.
( I know this should not be this painful. Please log a bug in case proxy.url is really broken. )
I'm trying out http requests to download a pdf file from google docs using google document list API and OAuth 1.0. I'm not using any external api for oauth or google docs.
Following the documentation, I obtained download URL for the pdf which works fine when placed in a browser.
According to documentation I should send a request that looks like this:
GET https://doc-04-20-docs.googleusercontent.com/docs/secure/m7an0emtau/WJm12345/YzI2Y2ExYWVm?h=16655626&e=download&gd=true
However, the download URL has something funny going on with the paremeters, it looks like this:
https://doc-00-00-docs.googleusercontent.com/docs/securesc/5ud8e...tMzQ?h=15287211447292764666&\;e=download&\;gd=true
(in the url '&\;' is actually without '\' but I put it here in the post to avoid escaping it as '&').
So what is the case here; do I have 3 parameters h,e,gd or do I have one parameter h with value 15287211447292764666&ae=download&gd=true, or maybe I have the following 3 param-value pairs: h = 15287211447292764666, amp;e = download, amp;gd = true (which I think is the case and it seems like a bug)?
In order to form a proper http request I need to know exectly what are the parameters names and values, however the download URL I have is confusing. Moreover, if the params names are h,amp;e and amp;gd, is the request containing those params valid for obtaining file content (if not it seems like a bug).
I didn't have problems downloading and uploading documents (msword docs) and my scope for downloading a file is correct.
I experimented with different requests a lot. When I treat the 3 parameters (h,e,gd) separetaly I get Unauthorized 401. If I assume that I have only one parameter - h with value 15287211447292764666&ae=download&gd=true I get 500 Internal Server Error (google api states: 'An unexpected error has occurred in the API.','If the problem persists, please post in the forum.').
If I don't put any paremeters at all or I put 3 parameters -h,amp;e,amp;gd, I get 302 Found. I tried following the redirections sending more requests but I still couldn't get the actual pdf content. I also experimented in OAuth Playground and it seems it's not working as it's supposed to neither. Sending get request in OAuth with the download URL responds with 302 Found instead of responding with the PDF content.
What is going on here? How can I obtain the pdf content in a response? Please help.
I have experimented same issue with oAuth2 (error 401).
Solved by inserting the oAuth2 token in request header and not in URL.
I have replaced &access_token=<token> in the URL by setRequestHeader("Authorization", "Bearer <token>" )
I have a DAV protocol that stores out-of-band data in the url anchor, e.g. the ghi in DELETE /abc.def#ghi. The server is a Flask application.
I can see the request come in on the wire via tcpdump, but when I look at the werkzeug Request object (such as url() or base_url()), all I get back is /abc.def. The #ghi has been stripped out.
Is there a method that returns this information, or do I have to subclass Request to handle this myself? If so, is there an example I can use as an inspiration?
I ran into the same problem. Facebook authentication API returns the access token behind a hash appended into the redirection url. In the same way, Flask's request.url drops everything in the URL behind the hash sign.
I'm also using Flask so I think you can use my brute-force workaround using Javascript's window.location.href to get the full URL. Then, I just extracted the piece that I needed (the access token), put it into a redirection URL where I can pass the access token as an argument to the receiving view function. Here's the code:
#app.route('/app_response/<response>', methods=['GET'])
def app_response_code(response):
return ''' <script type="text/javascript">
var token = window.location.href.split("access_token=")[1];
window.location = "/app_response_token/" + token;
</script> '''
#app.route('/app_response_token/<token>/', methods=['GET'])
def app_response_token(token):
return token
In case you manage(d) to do this within Werkzeug, I'm interested to know how.
From Wikipedia (Fragment Identifier) (don't have the time to find it in the RFC):
The fragment identifier functions differently than the rest of the URI: namely, its processing is exclusively client-side with no participation from the server
So Flask - or any other framework - doesn't have access to #ghi.
You can do this using flask.url_for with the _anchor keyword argument:
url_for('abc.def', _anchor='ghi')