we use Rails 3 and Spree Commerce for our online shop and we have a payment provider, that returns errors in a redirect URL if some occur. When an error occurs, we present that string with flash messages to the user.
Yesterday, something didn't work, and the payment provider returned this string in the redirect URL, which should be presented to the user inside a flash message:
errormsg=Bitte+versuchen+Sie+es+sp%E4ter+nochmals.
I debugged a little bit, and the string looks like this when decoded (e.g. is written to flash[:error]):
Bitte versuchen Sie es sp\xE4ter nochmals.
And after that, an error is raised, when rails tries to render the flash message:
invalid byte sequence in UTF-8
Can someone tell me, how to fix this? The error should contain a german ä and not \xE4. I tried setting # encoding: utf-8 to the beginning of the controller and the view, but this doesn't help.
'Obviously, your payment provider uses ISO-8859-1 or similar to send german umlauts.
As your rails app uses UTF-8, you can convert the message of your provicder.
Assume you stored the message in variable msg, use
utf_msg = params[:errormsg].force_encoding('ISO-8859-1').encode('UTF-8')
you can also check, if the resulting encoding is valid:
uft_msg.valid_encoding?
and outpunt a different message to avoid errors.
Related
Currently we are using SendGrid Inbound Parse to receive emails.
We handle the Inbound Parse webhook request by Azure HttpTrigger function implmented in C# (.NET 6).
When the received email is in UTF-8 encoding, everything's okay.
However, when we tried to receive email in shift_jis encoding, headers are okay,
but japanese characters in text and html are garbled.
From Inbound Parse request, we got the charsets as below:
subject: UTF-8
to: UTF-8
from: UTF-8
cc: UTF-8
html: shift_jis
text: shift_jis
And the string we got directly from request.form["text"] (or "html") was already garbled like "�e�L�X�gshiftJis-007"
(should be "テキストshiftJis-007"), so we cannot use string in request directly.
Then we tried to convert (System.Text.Encoding.Convert method) it from charset encoding (shift_jis) to utf-8,
and the result was different from original string but still unreadable "?e?L?X?gshiftJis-007".
Our questions are:
When using C# HttpTrigger Azure function to handle Inbound Parse webhook request
(request data is passed through AspNetCore.)
What encoding is in html/text string in Inbound Parse webhook request
when the email is send in encoding other than UTF-8?
How to read text and html in shift_jis encoding (or other encodings excluding UTF-8)
correctlyfrom an Inbound Parse webhook request?
Twilio Developer Evangelist here. I would recommend reaching out to the support team because it requires to investigate the payload to figure out what is going on.
I also tried to replicate the issue on my end with using send_raw option. Here's the payload, and it does contain shift_jis characters. You may be able to process the payload manually.
(stripped X-Mailer info)
'Content-Type: text/plain; charset="shift_jis"\n' +
'X-Mailer: \n' +
'Content-Transfer-Encoding: quoted-printable\n' +
'\n' +
'\n' +
'=83e=83L=83X=83gshiftJis-007\n'
I am trying to follow the guide to generate Oauth authentication tokens for YAHOO DSP API.
Base64 encoding is a way of encoding binary data into text so that it can be easily transmitted across a network without error.
In this step, you will take the client ID and client secret that the YDN console generated for you and encode them using the base64 protocol. You can use an online encoding service like base64encode.org.
No matter which service you use, ensure that no spaces are appended to the CLIENT_ID and CLIENT_SECRET keys and separate the CLIENT_ID and CLIENT_SECRET with a colon, i.e. CLIENT_ID:CLIENT_SECRET.
The generated value will now be referenced as ENCODED(CLIENT_ID:CLIENT_SECRET) in this guide.
An example is given:
CLIENT_ID = dj0yJmk9N2pIazlsZk1iTzIxJmQ9WVdrOWVEUmpVMFpWTXpRbWNHbzlNQS0tJnM9Y29uc3VtZXJzZWNyZXQmeD00NA–
CLIENT_SECRET= a7e13ea3740b933496d88755ff341bfb824805a6
AUTHORIZATION = ZGoweUptazlOMnBJYXpsc1prMWlUekl4Sm1ROVdWZHJPV1ZFVW1wVk1GcFdUWHBSYldOSGJ6bE5RUzB0Sm5NOVkyOXVjM1Z0WlhKelpXTnlaWFFtZUQwME5BLS06YTdlMTNlYTM3NDBiOTMzNDk2ZDg4NzU1ZmYzNDFiZmI4MjQ4MDVhNg==
Using the recommended website I get the wrong AUTHORIZATION.
I have tried both encoding the whole thing at once ie. encode(CLIENT_ID:CLIENT_SECRET), and each element individually encode(CLIENT_ID):encode(CLIENT_SECRET).
Attempt encoding whole thing:
ZGoweUptazlOMnBJYXpsc1prMWlUekl4Sm1ROVdWZHJPV1ZFVW1wVk1GcFdUWHBSYldOSGJ6bE5RUzB0Sm5NOVkyOXVjM1Z0WlhKelpXTnlaWFFtZUQwME5B4oCTOiBhN2UxM2VhMzc0MGI5MzM0OTZkODg3NTVmZjM0MWJmYjgyNDgwNWE2
Attempt encoding each element:
ZGoweUptazlOMnBJYXpsc1prMWlUekl4Sm1ROVdWZHJPV1ZFVW1wVk1GcFdUWHBSYldOSGJ6bE5RUzB0Sm5NOVkyOXVjM1Z0WlhKelpXTnlaWFFtZUQwME5B4oCT:YTdlMTNlYTM3NDBiOTMzNDk2ZDg4NzU1ZmYzNDFiZmI4MjQ4MDVhNg==
Expected result:
ZGoweUptazlOMnBJYXpsc1prMWlUekl4Sm1ROVdWZHJPV1ZFVW1wVk1GcFdUWHBSYldOSGJ6bE5RUzB0Sm5NOVkyOXVjM1Z0WlhKelpXTnlaWFFtZUQwME5BLS06YTdlMTNlYTM3NDBiOTMzNDk2ZDg4NzU1ZmYzNDFiZmI4MjQ4MDVhNg==
The difference between 'each element' and the expected result is only a few characters corresponding to the end of client_ID and the colon.
B4oCT: should be BLS06.
Links to full documentation:
https://developer.yahoo.com/dsp/api/docs/authentication/tokens.html
https://developer.yahoo.com/dsp/api/docs/traffic/info/sandbox.html
Update:
The final character of Client_ID is '–' . This is some sort of non-standard character that is interpreted as two dashes i.e.'--' in utf-8 and windows 1258.
One different, TO NOTE is, that when you decrypt the expected output you will get your client id as
dj0yJmk9N2pIazlsZk1iTzIxJmQ9WVdrOWVEUmpVMFpWTXpRbWNHbzlNQS0tJnM9Y29uc3VtZXJzZWNyZXQmeD00NA--
instead of
dj0yJmk9N2pIazlsZk1iTzIxJmQ9WVdrOWVEUmpVMFpWTXpRbWNHbzlNQS0tJnM9Y29uc3VtZXJzZWNyZXQmeD00NA–
NOTE, there are two "-" at the end.
OAuth client auth token is always generated using Base64 encoding with following format
Base64_Encoding(CLIENT_ID:CLIENT_SECRET)
Most of the usage perform this Base64 encoding with encoding type as "UTF-8".
It looks like, Yahoo requires this token with different encoding. On "https://www.base64encode.org/" if you try to encode your "CLIENT_ID:CLIENT_SECRET" with "Windows-1254" as destination charset, you will receive the expected result. So, it looks like both encoding and decoding here is done keeping "Windows-1254" charset in place.
please note that i had asked this question a while back but i was unable to attend to it because of some personal issue, so it got deleted, meta user suggested i reask the question and so here it is
Recently i encountered a website which doesn't send any dynamic information when you try to login, not even Username and Password !, what i mean is that after i use Fiddler and Http Analyzer i can see that 4 event happens :
All of these happens in the web browser :
1- A simple Get for the login main page
2- A Post which sends some data (These data do not include my Username and Password and they seem static! and even though there is a __USERCONTROLPATH which can be extracted from the step 1 response, it doesn't change, meaning even if i try to login multiple times using different browser at different times everything including the __USERCONTROLPATH is the same)
3- A Post which is of JSON type and the request is empty and the response only contains a single line as you can see at the image below:
4- A Get happens which is the login main page but if you look at HTML you can see that the user is logged in!
I tried doing these steps one by one, but i am unable to successfully log in to the website and i am unable to understand how my Username and Password is sent!
Here you can see some of the code i have written:
//for the sake of cookies and ... i do a Get on the main page
idHttp.get('MainLogPage');
//i send the first post which contains some information which never changes, they seems static and do not include the username and password
idhttp.post('someURL', requestList);
jsonRawRequest := TJSONObject.Create;
jsonRequest := TStringStream.Create(jsonRawRequest.ToString, TEncoding.UTF8);
/here i try to do the post that you saw in the image, no value is sent so i send an empty json request.
idhttp.post('someOtherURL', jsonRequest)
In the end my question is how can i login to such a site ?, what am i missing ?, currently i get error on Step 3, when i post the JSON, here is a picture of the error:
Update 01 :
I meant to update the whole post and explain every step in more detail but as i was messing with fiddler and checking the posts again i noticed something, when the Browser sends the POST "https://www.somewebsite.com//Index.aspx/LoginUser" the type is application/json and in the "HTTP analyzer" i can see that it send and receive a JSON but in the Fiddler the Webforms is empty! so i check the TextView thats where i saw it !, i again went and check HTTP analyzer's Raw stream, and in there i also saw a user and password being sent, i don't know how the post contains this string that you can see below, but in fiddler you can see it in "TextView" and in http analyzer you can see it in raw stream, now that i know this, can you tell me how can i imitate such a post ?, by the way the user and password are encrypted which i presume is a matter of its own, and i probably have to contact the site's admin for the algorithm ?, anyway, first things first, how can i send such a POST ?
In Fiddler :
HTTP Analyzer :
Update 02:
I tried what Remy Lebeau said,
But the JSON in the site was not standard as you can see in:
The input is not a valid Base-64 string Error, My JSON uses double quotation marks instead of single quotations
After that i tried a hard coded solution and i read the JSON from a file as you can see below:
loader := TStringList.Create;
loader.LoadFromFile('jsonWithCustomFormat.txt');
jsonRequest := TStringStream.Create(loader[0], TEncoding.UTF8);
After that i set the refer to the login page:
idHttp.Request.Referer := 'myLoginPageURL';
idHttp.Request.ContentType := 'application/json'
And Finally i send the POST:
idhttp.Post(URL, jsonRequest, ms);
The result of what i did was an alert error as you can in the image below:
I am very confused as to what the problem is, i have to solve this so any hint or help is much appreciated.
How to prevent a server from returning an error 400 code error when the URL contains % symbol using NGINX server?
Nginx configuration for my website:
....
rewrite ^/download/(.+)$ /download.php?id=$1 last;
....
When I tried to get access to this URL:
http://mywebsite.net/download/some-string-100%-for-example
I got this error:
400 Bad Request
With this url :
http://mywebsite.net/download/some-string-%25-for-example
it's work fine !
It's because it needs to be URL encoded first.
This will explain:
http://www.w3schools.com/tags/ref_urlencode.asp
URLs can only be sent over the Internet using the ASCII character-set.
Since URLs often contain characters outside the ASCII set, the URL has to be converted into a valid ASCII format.
URL encoding replaces unsafe ASCII characters with a "%" followed by two hexadecimal digits.
URLs cannot contain spaces. URL encoding normally replaces a space with a plus (+) sign or with %20.
The URL interpreter is confused to see a % without hexadecimals after it.
Why would you think of solving by changing Nginx configuration???
It's impossible to solve from the server side. It's a problem from the client side.
https://headteacherofgreenfield.wordpress.com/2016/03/23/100-celebrations/
In that URL, the title is 100% Celebrations! but the permalink is autogenerated to 100-celebrations. It's because they know putting 100% will cause a URL encode problem.
If even Wordpress doesn't do it your way, then why should you do it?
I am working on a flex application which communicates with a Rails backened.
When i request for some data, It sends back xml feed.
In some cases, if given parameters are not valid, then rails return an error feed with status code = 422 as following
email is wrong
But I dont get this feed in FaultEvent of Flex, How could i read error feed?
Thanks
Are you getting the result in ResultEvent in such cases? I am not sure for what all HTTP error codes FaultEvent will get invoke(I know only it goes for 404 and 500). May be its still going to ResultEvent as a valid result!
You can use HTTPService instead of URLLoader.
Flex HTTP results will not include the actual underlying HTTP response codes. It just doesn't work. (TM)