Send multiple files using HTTParty - ruby-on-rails

Here is the code which is working using Net::HTTP::Post
request = Net::HTTP::Post.new(url)
...
form_data = [
['attachments[]', File.open('file1.txt')],
['attachments[]', File.open('file2.txt')]
]
request.set_form form_data, 'multipart/form-data'
http.request(request)
Now, I am trying to use httparty like below but it is not working.
body = { attachments: [ File.open('file1.txt'), File.open('file2.txt') ] }
HTTParty.post(url, body: body)
The response I am getting from web service call is below:
#<HTTParty::Response:0x557d7b549f90 parsed_response={"error"=>true, "error_code"=>"invalid_attachment", "error_message"=>"Attachmen
t(s) not found or invalid."}, #response=#<Net::HTTPBadRequest 400 Bad Request readbody=true>, #headers={"server"=>["nginx"], "date"=>[
"Mon, 20 May 2019 07:41:50 GMT"], "content-type"=>["application/json"], "content-length"=>["102"], "connection"=>["close"], "vary"=>["
Authorization"], "set-cookie"=>["c18664e1c22ce71c0c91742fbeaaa863=uv425hihrbdatsql1udrlbs9as; path=/"], "expires"=>["Thu, 19 Nov 1981
08:52:00 GMT", "-1"], "cache-control"=>["no-store, no-cache, must-revalidate", "private, must-revalidate"], "pragma"=>["no-cache", "no
-cache"], "x-ratelimit-limit"=>["60"], "x-ratelimit-remaining"=>["59"], "strict-transport-security"=>["max-age=63072000; includeSubdom
ains;"]}>
It looks like it is not able to read the contents of files. Does HTTParty support this or I need to use some other gem?

Something like this should work, I just tested it, worked for me no problem.
HTTParty.post(url,
body: { attachments: [
File.read('foo.txt'),
File.read('bar.txt')] })

With HTTParty you can pass IO/Files as parameters the same way (multipart is automatically set to true if there's a file in parameters).
But keep in mind that files should be closed after upload, otherwise you may run out of file descriptors before GC collects them:
files = ['file1.txt', 'file2.txt'].map{|fname| File.open(fname) }
begin
HTTParty.post(url, body: { attachments: files })
ensure
files.each(&:close)
end
That should work for you if net/http variant does (and is actually the same as your code).
Other thing to look at is content type detection by filename - because file upload consists of filename, content type and data itself.
Error 400 with "invalid_attachment" you're getting suggests that more probably it's related to content type or other validation on server side (so make sure you're testing with the same files and nothing else changes other than http lib), also check httparty to be a recent version

I've written a test program which sends the same multipart request using both Net::HTTP and HTTParty. Then it compares and prints the request strings so that we can compare them. The only substantive difference between the two requests is that HTTParty attempts to guess and set the Content-Type header (e.g. text/plain for a file named file1.txt), whereas Net::HTTP always uses application/octet-stream.
HTTParty definitely does read the files and send them in the request. So, I suggest you investigate if the server is returning an error due to the Content-Type (maybe the content type in your particular request is not supported).
For your reference, here is the test program and specific results.

Related

RestAssured - how to send a request without Content-Type?

I am using RestAssured to send a request:
Map<String, Object> headers = new HashMap<>();
headers.put("Accept", "*/*");
headers.put("Accept-Encoding", "gzip, deflate, br");
headers.put("Connection", "keep-alive");
Response response = RestAssured.given().baseUri(BASE_URL)
.headers(headers)
.log().all()
.post(URL_PREFIX + "/documents/request/" + username);
However, in the log I see that 1 more header was automatically added:
Content-Type=application/x-www-form-urlencoded; charset=ISO-8859-1
And I get 415 error.
Is it possible to send a request without Content-Type? I mean, without this header at all; if the request is sent with Content-Type equal to empty line, there is still a 400 error; the only way to make it work is to send the request without this header.
Seems like there is a bug in the RestAssured framework that is still open (I verified that in 4.3.3).
// https://mvnrepository.com/artifact/io.rest-assured/rest-assured
testImplementation group: 'io.rest-assured', name: 'rest-assured', version: '4.3.3'
Founded out, when creating negative tests for a API. Content type below is automatically generated when trying to send request.
Content-Type=application/x-www-form-urlencoded; charset=ISO-8859-1
Bug defined here:
https://github.com/rest-assured/rest-assured/issues/656
https://github.com/rest-assured/rest-assured/issues/986

Microsoft Graph (OneDrive) API - Resumable Upload Content-Type

I am trying to create the upload PUT request for the OneDrive API. It's the large file "resumable upload" version which requires the createUploadSession.
I have read the Microsoft docs here: As a warning the docs are VERY inaccurate and full of factual errors...
The docs simply say:
PUT
https://sn3302.up.1drv.com/up/fe6987415ace7X4e1eF866337Content-Length:
26Content-Range: bytes 0-25/128 <bytes 0-25 of the
file>
I am authenticated and have the upload session created, however when I pass the JSON body containing my binary file I receive this error:
{ "error": {
"code": "BadRequest",
"message": "Property file in payload has a value that does not match schema.", .....
Can anyone point me at the schema definition? Or explain how the JSON should be constructed?
As a side question, am I right in using "application/json" for this at all? What format should the request use?
Just to confirm, I am able to see the temp file created ready and waiting on OneDrive for the upload, so I know I'm close.
Thanks for any help!
If you're uploading the entire file in a single request then why do you use upload session when you can use the simple PUT request?
url = https://graph.microsoft.com/v1.0/{user_id}/items/{parent_folder_ref_id}:/{filename}:/content
and "Content-Type": "text/plain" header and in body simply put the file bytes.
If for some reason I don't understand you have to use single-chunk upload session then:
Create upload session (you didn't specified any problems here so i'm not elaborating)
Get uploadUrl from createUploadSession response and send PUT request with the following headers:
2.1 "Content-Length": str(file_size_in_bytes)
2.2 "Content-Range": "bytes 0-{file_size_in_bytes - 1}/{file_size_in_bytes}"
2.3 "Content-Type": "text/plain"
Pass the file bytes in body.
Note that in the PUT request the body is not json but simply bytes (as specified by the content-type header.
Also note that max chuck size is 4MB so if your file is larger than that, you will have to split into more than one chunks.
Goodlcuk

403 Forbidden for amazon s3 put api to upload file in Rails App

I have the following aws s3 api end point from third party application
https://xxx.s3.amazonaws.com/uploads/74d75512c28b49358846f959bd798536?Signature=lxBIZJD7DN4QK3LPmsHxR7D2eTA%3D&Expires=1506108780&AWSAccessKeyId=AKIAJG6Z6A5TUL7ULPXA
I am trying to make an PUT request with this url to upload an File, but always get the following error
#<Net::HTTPForbidden 403 Forbidden readbody=true>
Update And here is the response body
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>
The request signature we calculated does not match the
signature you provided. Check your key and signing method.
</Message>
<AWSAccessKeyId>AKIAJG6Z6A5TUL7ULPXA</AWSAccessKeyId>
<StringToSign>
PUT
application/xml
1506144576
/xxx/uploads/93a64f8081804604be917b4185c5ed58
</StringToSign>
<SignatureProvided>8pwtC2vtN4LuOwGc887AlT2ZnUc=</SignatureProvided>
<StringToSignBytes>50 55 </StringToSignBytes>
<RequestId>B2B18369BA23A92F</RequestId>
<HostId>YCtjOJsfskITKxjW96ouZq1BV=</HostId>
</Error>
If I inspect the response it shows following data, but did not understood what's wrong with this? no other clue, anyone can help will be appreciated.
{
"x-amz-request-id" => [
[0] "385D5CADFE6175FC"
],
"x-amz-id-2" => [
[0] "4uD51Gb/rJy0QooQVmF25Qbp0E568bB9v1P4Grg9CTM2dJ/Iiccad/IyuuEnWDphlGZrr8ZUnQw="
],
"content-type" => [
[0] "application/xml"
],
"transfer-encoding" => [
[0] "chunked"
],
"date" => [
[0] "Fri, 22 Sep 2017 19:28:01 GMT"
],
"server" => [
[0] "AmazonS3"
]
}
I am running the following code snippet
uri = URI.parse(aws_api_end_point)
request = Net::HTTP::Put.new(uri)
request.body = File.read(file_path)
request.content_type = 'application/xml'
http = Net::HTTP.start(uri.host, uri.port, :use_ssl => true)
response = http.request(request)
Note:: I am not using any aws SDK.
Request signing is deterministic -- for a given request at a given moment in time, there is exactly one valid signature. For all practical purposes, the opposite is also true -- for any signature, there's only one valid request you can make. Anything else, and the signature does not match.
The algorithm is designed not to be reverse-engineered, so we can't say what request they expected you to make when they gave you the signature.
But, we do have this. I believe some whitespace was lost, so I have added it back in:
<StringToSign>
PUT
application/x-www-form-urlencoded
1506144576
/xxx/uploads/93a64f8081804604be917b4185c5ed58
</StringToSign>
This is the request you made (not the request you were expected to make), converted to its canonical form using this pseudocode.
StringToSign =
HTTP-VERB + "\n" +
Content-MD5 + "\n" +
Content-Type + "\n" +
Expires + "\n" +
CanonicalizedAmzHeaders +
CanonicalizedResource;
The most obvious candidate for a problem is Content-Type. It is almost a certainty that they did not expect you to use application/x-www-form-urlencoded because the S3 PUT Object operation does not use HTML Form uploads. S3 PUT expects the raw octets of the object in the request body, and a Content-Type header to match. S3 itself does not actually validate whether they match (header vs. body), but without at least the expected Content-Type header, the upload will be blocked.
If you are providing a Content-Type to the 3rd party API, then your upload needs to use that same type on the S3 upload, because the signature expects it.
Unfortunately, for troubleshooting purposes, there is an infinite number of things that can be done wrong to invalidate a request intended to be used with a pre-signed URL. The documentation from the third party should clarify the structure of the subsequent request they expect you to make.

How to remove charset=utf-8 in a Content-Type header, generated by grails

I'm trying to send json data as a response body in grails. I've tried setting the Content-Type header to application/json using the following methods:
render (status: httpServletResponse, text: responseToRender as JSON, contentType: "application/json")
Each time the resulting header is as follows:
Content-Type: application/json;charset=utf-8
How do I get rid of the charset=UTF-8 postfix?
You can not get rid of the charset postfix.
You can change it with the charset parameter defined here:
https://docs.grails.org/latest/ref/Controllers/render.html
You can also provide no information by just handing json to render, such as:
response.setContentType("application/json")
render JsonOutput.toJson(responseToRender);
However, this will default to the standard encoding required by HTTP 1.1 which is ISO-8859-1. therefore your result would be application/json;charset=ISO-8859-1
https://www.w3.org/International/articles/http-charset/index.en
So, if you somehow need to use this parameter, you may use .split(";")[0] to access only the first part.

YouTube API: Delete video HTTP request from Ruby not working

I am trying to delete a video on YouTube from a Ruby on Rails application. I am following these instructions, from the YouTube API docs:
DELETE /feeds/api/users/default/uploads/VIDEO_ID HTTP/1.1
Host: gdata.youtube.com
Content-Type: application/atom+xml
Authorization: Bearer ACCESS_TOKEN
GData-Version: 2
X-GData-Key: key=DEVELOPER_KEY
I am not very familiar with Ruby's Net::HTTP class, but it seems that no matter what I try I cannot get the request to work properly. I have looked carefully at the many other StackOverflow questions regarding deleting videos from YouTube, but none that I could find address this particular problem. My code is below, where I've replaced the user name, video ID, access token, and developer key.
url = URI.parse("https://gdata.youtube.com/feeds/api/users/[USER_NAME]/uploads/[VIDEO_ID]")
post_args = { 'Host' => 'gdata.youtube.com', 'GData-Version' => '2', 'Content-Type' => 'application/atom+xml', 'Authorization' => "Bearer [ACCESS_TOKEN]", 'X-GData-Key' => 'key=[DEVELOPER_KEY]' }
req = Net::HTTP::Delete.new(url.path)
req.set_form_data(post_args)
httpreq = Net::HTTP.new(url.host, url.port)
httpreq.use_ssl = true
resp = httpreq.start {|http| http.request(req) }
Checking the response, I get an Error 400 (Bad Request) from YouTube. The response simply says "Your client has issued a malformed or illegal request. That's all we know".
Is there something wrong with the request I'm making? I've checked it against the template time and time again and I can't see anything wrong with it. I know that my access token and developer key are working because I can make other requests like video uploads just fine.
I printed the debug output from the HTTP request, and as far as I can tell it looks fine:
<- "DELETE /feeds/api/users/[USER_NAME]/uploads/[VIDEO_ID] HTTP/1.1\r\nAccept: */*\r\nUser-Agent: Ruby\r\nContent-Type: application/x-www-form-urlencoded\r\nHost: gdata.youtube.com\r\nContent-Length: 275\r\n\r\n"
<- "Host=gdata.youtube.com&GData-Version=2&Content-Type=application%2Fatom%2Bxml&Authorization=Bearer+[ACCESS_TOKEN]&X-GData-Key=key%3D[DEVELOPER_KEY]"
The only thing I could see as a possible problem was that in the first line of the request, the "Content-Type" is set to "application/x-www-form-urlencoded". Again, not being an expert on HTTP requests I'm not sure what the difference is between the Content-Type set in the first line and the Content-Type that I explicitly set as "application/atom+xml" which appears on the second line of the request. After some digging, though, I found out that the set_form_data method automatically sets the content type as "application/x-www-form-urlencoded", so I tried adding the following line to my code:
req.content_type = 'application/atom+xml'
right after the line
req.set_form_data(post_args)
When I do this, I do see a corresponding change in the request:
<- "DELETE /feeds/api/users/[USER_ID]/uploads/[VIDEO_ID] HTTP/1.1\r\nAccept: */*\r\nUser-Agent: Ruby\r\nContent-Type: application/atom+xml\r\nHost: gdata.youtube.com\r\nContent-Length: 275\r\n\r\n"
<- "Host=gdata.youtube.com&GData-Version=2&Content-Type=application%2Fatom%2Bxml&Authorization=Bearer+[ACCESS_TOKEN]&X-GData-Key=key%3D[DEVELOPER_KEY]"
However, I still get the exact same response from YouTube. Error 400, bad request. What the heck is going on here??
Of course, 10 minutes after asking my question, I find out the answer. I did not understand the distinction between the HTTP header fields and form arguments, which I don't feel so bad about since it's not explained anywhere either in the Ruby documentation on Net::HTTP or in the YouTube API. The reason I was confused was because for uploading a video, you can provide all the values like Authorization and Content-Type as form data, so the above approach from my question works fine. For deleting a video, you have to provide those values as part of the header, not form data. At least, that is now my understanding.
Anyway, in case anyone ever runs into this problem, this solved it for me:
req = Net::HTTP::Delete.new(url.path)
req['GData-Version'] = '2' # this syntax sets header fields & values
req['Authorization'] = "..."
req['X-GData-Key'] = "..."
req.content_type = 'application/atom+xml'
httpreq = Net::HTTP.new(url.host, url.port)
httpreq.use_ssl = true
resp = httpreq.start {|http| http.request(req) }
Another case where one explanatory sentence from the authors of the documentation would have saved two hours of wasted time. If I had a nickel...

Resources