I have an application that builds an HTML email. Included in the content is an encoded URL parameter that might, for example, contain a promotional code or product reference. The email is generated by a Windows service (essentially a console application) and the link, when clicked is handled by an MVC web site. Here is the code for creating the email link:
string CreateLink(string domain, string code) {
// code == "xyz123"
string encrypted = DES3Crypto.Encrypt(code); // H3uKbdyzrUo=
string urlParam = encrypted.EncodeBase64(); // SDN1S2JkeXpyVW890
return domain + "/" + urlParam;
}
The action method on the MVC controller is constructed as follows:
public ActionResult Index(string id) {
string decoded = id.DecodeBase64();
string decrypted = DES3Crypto.Decrypt(decoded);
...
}
In all our testing, this mechanism has worked as expected, however, now we have gone live we are seeing around a 4% error rate where the conversion from base-64 fails with the following exception:
The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or a non-white space character among the padding characters.
The id parameter from the url 'looks' OK. The problem appears to be with the EncodeBase64/DecodeBase64 methods that are failing as DecodeBase64 method returns a 'garbled' string such as "�nl����□��7y�b�8�sJ���=" on the failed links.
Furthermore, most of the errors are from IE6 user agents leading me to think this is a character encoding problem but I don't see why.
For reference, here is the code for my base-64 URL encoding:
public static string EncodeBase64(this string source)
{
byte[] bytes = Encoding.UTF8.GetBytes(source);
string encodedString = HttpServerUtility.UrlTokenEncode(bytes);
return encodedString;
}
public static string DecodeBase64(this string encodedString)
{
byte[] bytes = HttpServerUtility.UrlTokenDecode(encodedString);
string decodedString = Encoding.UTF8.GetString(bytes);
return decodedString;
}
Any advice would be much appreciated.
To recap, I was creating a URL that used a base-64 encoded parameter which was itself a Triple DES encrypted string. So the URL looked like http://[Domain_Name]/SDN1S2JkeXpyVW890 The link referenced a controller action on an MVC web site.
The URL was then inserted into an HTML formatted email. Looking at the error log, we saw that around 5% of the public users that responded to the link were throwing an "invalid base-64 string error". Most, but not all, of these errors were related to the IE6 user agent.
After trying many possible solutions based around character and URL encoding, it was discovered that somewhere in the client's process the url was being converted to lower-case - this, of course, broke the base-64 encoding (as it is uses both upper and lower case encoding characters).
Whether the case corruption was caused by the client's browser, email client or perhaps local anti-virus software, I have not been able to determine.
The Solution
Do not use any of the standard base-64 encoding methods, instead use a base-32 or zBase-32 encoding instead - both of which are case-insensitive.
See the following links for more details
Base-32 - Wikipedia
MyTenPennies Base-32 .NET Implementation
The moral of the story is, Base-64 URL encoding can be unreliable in some public environments. Base-32, whilst slightly more verbose, is a better choice.
Hope this helps.
It looks like you were really close. You had an extra zero coming back from your encyrpted.EncodeBase64() function.
Try this:
string data = "H3uKbdyzrUo=";
string b64str = Convert.ToBase64String(UTF8Encoding.UTF8.GetBytes(data));
string clearText = UTF8Encoding.UTF8.GetString(Convert.FromBase64String(b64str));
This is an interesting issue. My guess is that IE 6 is eating some of the characters.
For example, the length of the string that you included "ywhar0xznxpjdnfnddc0yxzbk2jnqt090" is not a multiple of four (which is a requirement for FromBase64 to work http://msdn.microsoft.com/en-us/library/system.convert.frombase64string.aspx)
But if you were to pad that string until it's length is a multiple of four ("ywhar0xznxpjdnfnddc0yxzbk2jnqt090" + "a12") then that works.
The MSDN documentation says that one ("=") or two ("==") equal characters are used for padding to/fromBase64 methods and I suspect IE 6 is truncating that from the string that you send.
This is total speculation but I hope it helps.
Related
I am facing the following issue:
In my app, the user can enter special characters (like emojis) in a textfield also. So, while sending this entered text to server in request body, I am converting it using the following code:
func emojiToUTF8()->String
{
let data = self.data(using: .nonLossyASCII, allowLossyConversion: true)
let emoji = String.init(data: data!, encoding: .utf8)
return emoji ?? self
}
For instance, if I enter the text "☺️", it gets converted into "\u263a\ufe0f" using the above method. Things are fine till here.
The problem occurs when I add this to a dictionary for sending it as a request parameter to the server. Code i'm using:
var parameters = [String:String]()
parameters["feedback"] = feedBackTxt
print("Parameters:",parameters) /// output: ["feedback": "\\u263a\\ufe0f"]
So, the problem here is that an extra slash is getting appended before each slash due to char escaping. I checked the created dictionary value as well. It shows double slash there also. How do I avoid this? Why is this happening when I am simply creating a dictionary with a string? This is causing issue at server end.
I have tried a couple of things, but none of them seem to work.
Your problem is that you're double-encoding.
You're taking a string, converting it to ASCII, then re-parsing it as UTF8 and then encoding that (probably) as JSON, which is UTF8. In the process, the backslashes are being escaped by your second encoder.
The best solution to this is to rework your server to accept UTF8. However, if you can't do that, you need to ensure you encode this string just one time, in ASCII.
In short, you should get rid of emojiToUTF8 and ensure that your parameters processor encodes the way your server requires (which apparently is ASCII and not UTF8).
I'm returning a Character vector from a function in R to C# using R.NET. The only problem is that unicode characters, such as Greek Letters are being lost. The following line gives an example of the code I'm using:
CharacterVector cvAll = results[5].AsList().AsCharacter();
Where results is a list of results returned by the R function. The characters are also written by R to a text file and they display fine in notepad and other editors. Can I get R.Net to return the characters correctly?
Looks like you ran into an open issue with RDotNet : https://github.com/jmp75/rdotnet/issues/25
Unicode characters don't seem to be supported yet. I ran into the same issue while calling the engine.CreateDataFrame() method. It did return a DataFrame with all my accentuated strings wrong.
There seems to be a workaround though : when calling RDotNet functions, if I give strings encoded in my computer default encoding (Windows ANSI) and converted from UTF-8 (important), R takes them and gives back correctly interpreted accentuated strings to C#. I don't exactly know why it is working though... It might have something to do with the default encoding used with .Net for string being UTF-16. (cf. here : http://csharpindepth.com/Articles/General/Strings.aspx), hence the conversion from UTF-8 to default ANSI that seems to be working.
Here is an ugly example : when I'm building a RDotNet DataFrame, I convert all strings in a CharacterVector to ANSI (from UTF-8) encoded ones :
try
{
string[] colAsStrings = null;
colAsStrings = Array.ConvertAll<object, string>(uneColonne, s => StringEncodingHelper.EncodeToDefaultFromUTF8((string)s));
correctedDataArray[i] = colAsStrings;
columnConverted = true;
}
Here is the static method used for conversion :
public static string EncodeToDefaultFromUTF8(string stringToEncode)
{
byte[] utf8EncodedBytes = Encoding.UTF8.GetBytes(stringToEncode);
return Encoding.Default.GetString(utf8EncodedBytes);
}
I'm using MongoDB for my Go and Rails projects (using same database) and i have a bson.Binary data in my document (contain Base64 encoded publicKey)
type Device struct {
Id string `json:"id" form:"id" bson:"_id"`
PublicKey bson.Binary `json:"pub_key" form:"pub_key" bson:"public_key"`
Token string `json:"token" form:"token" bson:"token"`
CreatedAt time.Time `json:"created_at" bson:"created_at"`
UpdatedAt time.Time `json:"updated_at" bson:"updated_at"`
}
but when i retrieve it in my Go projects and the data is somehow corrupted (4 characters are missing)
device := models.Device{}
err = db.C(models.CollectionDevice).Find(bson.M{"_id": deviceId}).One(&device)
publicKey := device.PublicKey
publicKeyBase64 := base64.StdEncoding.EncodeToString(publicKey.Data)
fmt.Println("publicKey BinaryData: ", publicKey.Data)
t
"pub_key": {
"Kind": 0,
"Data": "Axj5A9BgkKJohGh/0BAQEFAAMBjYAAMYmCAYGACqW0smeNiaIgRgXJv7dLZ0n5gTeXxiI9q6h8EdbiPDLiFsv7Jgatwxm+OpucrUBp4sZp/othrSQWnJQR5vpnCZYZnNJUdjTAYw+PyjQ2qq9YEeLHMnMzhYgaq2ata+CSsCjalkOyzLt/rDEBn5WHaCfYm0Vm+QbbEWVLltUqcjvScOyAwEAAQn"
}
(220 characters length in Data)
the original Base64 encoded string is
Axj5A9BgkKJohGh/0BAQEFAAMBjYAAMYmCAYGACqW0smeNiaIgRgXJv7dLZ0\n5gTeXxiI9q6h8EdbiPDLiFsv7Jgatwxm+OpucrUBp4sZp/othrSQWnJQR5vp\nCZYZnNJUdjTAYw+PyjQ2qq9YEeLHMnMzhYgaq2ata+CSsCjalkOyzLt/rDEB\n5WHaCfYm0Vm+QbbEWVLltUqcjvScOyAwEAAQ\n
with 224 characters
i use this code in my Go projects:
fmt.Println("publicKey: ", publicKey.Data)
//publicKey is bson.Binary data type
i have tried to retrieve the data from my rails projects (same database) and it retrieve correctly
public_key.as_json["$binary"]
the result is exactly right (224 characters) :
"Axj5A9BgkKJohGh/0BAQEFAAMBjYAAMYmCAYGACqW0smeNiaIgRgXJv7dLZ0\nn5gTeXxiI9q6h8EdbiPDLiFsv7Jgatwxm+OpucrUBp4sZp/othrSQWnJQR5v\npnCZYZnNJUdjTAYw+PyjQ2qq9YEeLHMnMzhYgaq2ata+CSsCjalkOyzLt/rD\nEBn5WHaCfYm0Vm+QbbEWVLltUqcjvScOyAwEAAQn\n"
As you can see, there is still a \n at the last of string. Any one know why 4 characters are missing in Go ?
===== Additional information =====
When i store it in my mongoDB from my Rails API, i receive public_key params as Base64 format, but i decode it to binary and then i stored it with this code
def create
params = device_params
public_key = Base64.decode64 device_params[:public_key]
params[:public_key] = BSON::Binary.new(public_key, :generic)
device = Device.find_or_create_by(id: device_params[:id])
render_success device.update_attributes(params), device
end
Your original data contains exactly 4 newline characters \n:
Axj5A9BgkKJohGh/0BAQEFAAMBjYAAMYmCAYGACqW0smeNiaIgRgXJv7dLZ0\n
5gTeXxiI9q6h8EdbiPDLiFsv7Jgatwxm+OpucrUBp4sZp/othrSQWnJQR5vp\n
CZYZnNJUdjTAYw+PyjQ2qq9YEeLHMnMzhYgaq2ata+CSsCjalkOyzLt/rDEB\n
5WHaCfYm0Vm+QbbEWVLltUqcjvScOyAwEAAQ\n
These newline characters are not part of the Base64 encoded data, they are just to format the input.
Problem arises likely when you try to store this as a raw string literal, because this is an interpreted string literal, the \n sequences are to be discarded / removed.
Since the backslash is an invalid symbol for "normal" Base64, they are removed, though the subsequent n is not (because n is valid in Base64). As a result, you get corrupted Base64 data.
You should either completely remove the line endings and then you get a valid Base64 which you may store either as a string or binary; or store it as a general string which will preserve the line endings (which have to be dealt with during base64 decoding).
I'm developing a CakePHP application which uses the standard CakePHP URL schema.
If I use the following function in a Controller:
class IndexController extends AppController {
public $uses = false;
public function test($a) {
var_dump($a);
}
}
And call it with this URL:
http://server/index/test/Hello+%2BTest
I get this result:
string(11) "Hello++Test"
I would expect the first "+" to be unescaped to " ". Why doesn't this happen?
I think you mean "unencode" instead of "unescape".
This is happening probably because the + sign is a valid character for a URL. Only the urlencoded characters (i.e. %xx) will be converted back to "readable" characters. (though pretty sure this is done by Apache, not cake)
If you want to force conversion, you can just run it through PHP's urldecode() function.
From RFC 1738 :
Thus, only alphanumerics, the special characters "$-_.+!*'(),", and reserved characters used for their reserved purposes may be used unencoded within a URL.
I have two URLs with parameters
http://localhost:8041/Reforge.aspx?name=CyanГ
http://localhost:8041/Reforge.aspx?name=Cyanì
In first URL Firefox encodes last charecter (Г) as %D0%93 (correctly in UTF-8).
In second URL Firefox encodes last character (ì) as %EC (correctly in ISO-8859-1)
ASP.NET MVC can be configured using element in web.config to either assume UTF-8 or ISO-8859-1. But Firefox flips between encodings depending on the context.
Note that UTF-8 can be unambiguously distinguished from Latin-1 encoding.
Is there a way to teach ASP.NET MVC to decode parameter values using either one of the formats?
EDIT: Is there a class that I could use to decode raw query string that would handle encoding correctly? Note - Firefox uses either UTF-8 or Latin-1 encoding - but not both at the same time. So my plan is to try decode manually using UTF-8 and then look for "invalid" character (FFFD), if one is found - try Latin-1 decode.
Example:
Firefox encodes as following:
- v v
http://localhost:8041/Reforge.aspx?name=ArcânisГ
Firefox turns into
http://localhost:8041/Reforge.aspx?name=Arc%C3%A2nis%D0%93`
Notice that UTF8 encoding is used for both non-ASCII characters.
- v
http://localhost:8041/Reforge.aspx?name=Arcâ
Firefox turns into
http://localhost:8041/Reforge.aspx?name=Arc%E2
Notice that ISO-8859-1 (Latin-1) encoding is used for the non-ASCII character.
Here is my working solution, any way to improve on it? Specifically I would rather extend framework instead of handling it inside an action itself.
private string DecodeNameParameterFromQuery(string query) {
string nameUtf8 = HttpUtility.ParseQueryString(query, Encoding.UTF8)["name"];
const char invalidUtf8Character = (char) 0xFFFD;
if (nameUtf8.Contains(invalidUtf8Character)) {
const int latin1 = 0x6FAF;
var nameLatin1 = HttpUtility.ParseQueryString(query, Encoding.GetEncoding(latin1))["name"];
return nameLatin1;
}
return nameUtf8;
}