I have some NSString like :
test = #"this is %25test%25 string";
I am trying to replace test with some arabic text , but it is not replacing exactly as it is :
[test stringByReplacingOccurrencesOfString:#"test" withString:#"اختبار"];
and the result is :
this is %25 اختبار %25 string
Some where I read there could be some problem with encoding or text alignment.Is there extra adjustment needed to be done for arabic string operations .
EDIT : I have used NSMutable string insert property but still the same result .
EDIT 2:
One other thing that occurs to me that is causing most of your trouble in this specific example. You have a partially percent-encoded string above. You have spaces, but you also have %25. You should avoid doing that. Either percent-encode a string or don't. Convert it all at once when required (using stringByAddingPercentEscapesUsingEncoding:). Don't try to "hard-code" percent-encoding. If you just used "this is a %اختبار% string" (and then percent-encoded the entire thing at the end), all your directional problems would go away (see how that renders just fine?). The rest of these answers address the more general question when you really need to deal with directionality.
EDIT:
The original answer after the line relates to human-readable strings, and is correct for human-readable strings, but your actual question (based on your followups) is about URLs. URLs are not human-readable strings, even if they occasionally look like them. They are a sequence of bytes that are independent of how they are rendered to humans. "اختبار" cannot be in the path or fragment parts of an URL. These characters are not part of the legal set of characters for those sections (اختبار is allowed to be part of the host, but you have to follow the IDN rules for that).
The correct URL encoding for this is a %25<arabic>%25 string is:
this%20is%20a%20%2525%D8%A7%D8%AE%D8%AA%D8%A8%D8%A7%D8%B1%2525%20string
If you decode and render this string to the screen, it will appear like this:
this is a %25اختبار%25 string
But it is in fact exactly the string you mean (and it is the string you should pass to the browser). Follow the bytes (like the computer will):
this - this (ALPHA)
%20 - <space> (encoded)
is - is (ALPHA)
%20 - <space> (encoded)
a - a (ALPHA)
%20 - <space> (encoded)
%25 - % (encoded)
25 - 25 (DIGIT)
%D8%A7 - ا (encoded)
%D8%AE - خ (encoded)
%D8%AA - ت (encoded)
%D8%A8 - ب (encoded)
%D8%A7 - ا (encoded)
%D8%B1 - ر (encoded)
%25 - % (encoded)
25 - 25 (DIGIT)
%20 - <space> (encoded)
string - string (ALPHA)
The Unicode BIDI display algorithm is doing what it means to do; it just isn't what you expect. But those are the bytes and they're in the correct order. If you add any additional bytes (such as LRO) to this string, then you are modifying the URL and it means something different.
So the question you need to answer is, are you making an URL, or are you making a human-readable string? If you're making an URL, it should be URL-encoded, in which case you will not have this display problem (unless this is part of the host, which is a different set of rules, but I don't believe that's your problem). If this is a human-readable string, see below about how to provide hints and overrides to the BIDI algorithm.
It's possible that you really need both (a human-friendly string, and a correct URL that can be pasted). That's fine, you just need to handle the clipboard yourself. Show the string, but when the user goes to copy it, replace it with the fully encoded URL using UIPasteboard or by overriding copy:. See Copy, Cut, and Paste Operations. This is fairly common (note how in Safari, it displays just "stackoverflow.com" in the address bar but if you copy and paste it, it pastes "https://stackoverflow.com/" Same thing.
Original answer related to human-readable strings.
Believe it or not, stringByReplacingOccuranceOfString: is doing the right thing. It's just not displaying the way you expect. If you walk through characterAtIndex:, you'll find that it's:
% 2 5 ا ...
The problem is that the layout engine gets very confused around all the "neutral direction" characters. The engine doesn't understand whether you meant "%25" to be attached to the left to right part or right to left part. You have to help it out here by giving it some explicit directional characters to work with.
There are a few ways to go about this. First, you can do it the Unicode 6.3 tr9-29 way with Explicit Directional Isolates. This is exactly the kind of problem that Isolates are meant to solve. You have some piece of text whose direction you want to be considered completely independently of all other text. Unicode 6.3 isn't actually supported by iOS or OS X as best I can tell, but for many (though not all) uses, it "works."
You want to surround your Arabic with FSI (FIRST STRONG ISOLATE U+2068) and PDI (POP DIRECTIONAL ISOLATE U+2069). You could also use RLI (RIGHT-TO-LEFT ISOLATE) to be explicit. FSI means "treat this text as being in the direction of the first strong character you find."
So you could ideally do this:
NSString *test = #"this is a %25\u2068test\u2069%25 string";
NSString *arabic = #"اختبار";
NSString *result = [test stringByReplacingOccurrencesOfString:#"test" withString:arabic];
That works if you know what you're going to substitute before hand (so you know where to put the FSI and PDI). If you don't, you can do it the other way and make it part of the substitution:
NSString * const FSI = #"\u2068";
NSString * const PDI = #"\u2069";
NSString *test = #"this is %25test%25 string";
NSString *arabic = #"اختبار";
NSString *replaceString = [#[FSI, arabic, PDI] componentsJoinedByString:#""];
NSString *result = [test stringByReplacingOccurrencesOfString:#"test" withString:replaceString];
I said this "mostly" works. It's fine for UILabel, and it probably is fine for anything using Core Text. But in NSLog output, you'll get these extra "placeholder" characters:
You might get this other places, too. I haven't checked UIWebView for instance.
So there are some other options. You can use directional marks. It's a little awkward, though. LRM and RLM are zero-width strongly directional characters. So you can bracket the arabic with LRM (left to right mark) so that the arabic doesn't disturb the surrounding text. This is a little ugly since it means the substitution has to be aware of what it's substituting into (which is why isolates were invented).
NSString * const LRM = #"\u200e";
NSString *test = #"this is a %25test%25 string";
NSString *replaceString = [#[LRM, arabic, LRM] componentsJoinedByString:#""];
NSString *result = [test stringByReplacingOccurrencesOfString:#"test" withString:replaceString];
BTW, Directional Marks are usually the right answer. They should always be the first thing you try. This particular problem is just a little too tricky.
One more way is to use Explicit Directional Overrides. These are the giant "do what I tell you to do" hammer of the Unicode world. You should avoid them whenever possible. There are some security concerns with them that make them forbidden in certain places (<RLO>elgoog<PDF>.com would display as google.com for instance). But they will work here.
You bracket the whole string with LRO/PDF to force it to be left-to-right. You then bracket the substitution with RLO/PDF to force it to the right-to-left. Again, this is a last resort, but it lets you take complete control over the layout:
NSString * const LRO = #"\u202d";
NSString * const RLO = #"\u202e";
NSString * const PDF = #"\u202c";
NSString *test = [#[LRO, #"this is a %25test%25 string", PDF] componentsJoinedByString:#""];
NSString *arabic = #"اختبار";
NSString *replaceString = [#[RLO, arabic, PDF] componentsJoinedByString:#""];
NSString *result = [test stringByReplacingOccurrencesOfString:#"test" withString:replaceString];
I would think you could solve this problem with the Explicit Directional Embedding characters, but I haven't really found a way to do it without at least one override (for instance, you could use RLE instead of RLO above, but you still need the LRO).
Those should give you the tools you need to figure all of this out. See the Unicode TR9 for the gory details. And if you want a deeper introduction to the problem and solutions, see Cal Henderson's excellent Understanding Bidirectional (BIDI) Text in Unicode.
You should try like this:
NSString *test = #"this is %25test%25 string";
NSString *test2 = [[[test stringByReplacingPercentEscapesUsingEncoding:NSStringEncodingConversionAllowLossy] componentsSeparatedByString:#"test"] componentsJoinedByString:#"اختبار"];
Related
I found some weirdest thing in Firebase Database/Storage. The thing is that I don't know if Firebase or Swift is not detecting umlauts e.g(ä, ö, ü).
I did some easy things with Firebase like upload images to Firebase Storage and then download them into tableview. Some of my .png files had umlauts in the title for example(Röda.png).
So the problem occurs now if I download them. The only time my download url is nil is if the file name contains the umlauts I was talking about.
So I tried some alternatives like in HTML ö - ö. But this is not working. Can you guys suggest me something? I can't use ö - o, ü - u etc.
This is the code when url is nil when trying to set some values into Firebase:
FIRStorage.storage().reference()
.child("\(productImageref!).png")
.downloadURLWithCompletion({(url, error)in
FIRDatabase.database().reference()
.child("Snuses").child(productImageref!).child("productUrl")
.setValue(url!.absoluteString)
let resource = Resource(downloadURL: url!, cacheKey: productImageref)
After spending a fair bit of time research your problem, the difference boils down to how the character ö is encoded and I traced it down to Unicode normalization forms.
The letter ö can be written in two ways, and String / NSString considers them equal:
let str1 = "o\u{308}" // decomposed : latin small letter o + combining diaeresis
let str2 = "\u{f6}" // precomposed: latin small letter o with diaeresis
print(str1, str2, str1 == str2) // ö ö true
But when you percent-encode them, they produce different results:
print(str1.stringByAddingPercentEncodingWithAllowedCharacters(.URLPathAllowedCharacterSet())!)
print(str2.stringByAddingPercentEncodingWithAllowedCharacters(.URLPathAllowedCharacterSet())!)
// o%CC%88
// %C3%B6
My guess is that Google / Firebase chooses the decomposed form while Apple prefers the other in its text input system. You can convert the file name to its decomposed form to match Firebase:
let str3 = str2.decomposedStringWithCanonicalMapping
print(str3.stringByAddingPercentEncodingWithAllowedCharacters(.URLPathAllowedCharacterSet()))
// o%CC%88
This is irrelevant for ASCII-ranged characters. Unicode can be very confusing.
References:
The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (highly recommended)
Strings in Swift 2
NSString and Unicode
Horray for Unicode!
The short answer is that no, we're actually not doing anything special here. Basically all we do under the hood is:
// This is the list at https://cloud.google.com/storage/docs/json_api/ without the & because query parameters
NSString *const kGCSObjectAllowedCharacterSet =
#"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-._~!$'()*+,;=:#";
- (nullable NSString *)GCSEscapedString:(NSString *)string {
NSCharacterSet *allowedCharacters =
[NSCharacterSet characterSetWithCharactersInString:kGCSObjectAllowedCharacterSet];
return [string stringByAddingPercentEncodingWithAllowedCharacters:allowedCharacters];
}
What blows my mind is that:
let str1 = "o\u{308}" // decomposed : latin small letter o + combining diaeresis
let str2 = "\u{f6}" // precomposed: latin small letter o with diaeresis
print(str1, str2, str1 == str2) // ö ö true
returns true. In Objective-C (which the Firebase Storage client is built in), it totally shouldn't, as they're two totally different characters (in actuality, the length of str1 is 2 while the length of str2 is 1 in Obj-C, while in Swift I assume the answer is 1 for both).
Apple must be normalizing strings before comparison in Swift (probably a reasonable thing to do, since otherwise it leads to bugs like this where strings are "the same" but compare differently). Turns out, this is exactly what they do (see the "Extended Grapheme Clusters" section of their docs).
So, when you provide two different characters in Swift, they're being propagated to Obj-C as different characters and thus are encoded differently. Not a bug, just one of the many differences between Swift's String type and Obj-C's NSString type. When in doubt, choose a canonical representation you expect and stick with it, but as a library developer, it's very hard for us to choose that representation for you.
Thus, when naming files that contain Unicode characters, make sure to pick a standard representation (C,D,KC, or KD) and always use it when creating references.
let imageName = "smorgasbörd.jpg"
let path = "images/\(imageName)"
let decomposedPath = path.decomposedStringWithCanonicalMapping // Unicode Form D
let ref = FIRStorage.storage().reference().child(decomposedPath)
// use this ref and you'll always get the same objects
I've just come across this line in some legacy code I'm editing:
[UIImage imageNamed:#"data/visuals/interface/" #"backgroundViewController"];
^^^^
"Oops, what have I done here?"
I thought I must have accidentally just pasted something in the wrong place, but an undo didn't change that line. Out of curiosity, I built the program and it was successful!
Whaddyaknow? Obj-c has a more succinct way of concatenating string literals.
I added some more tests:
A simple log
NSLog(#"data/visuals/interface/" #"backgroundViewController");
data/visuals/interface/backgroundViewController
In parameters
NSURL *url = [NSURL URLWithString:#"http://" #"test.com" #"/path"];
NSLog(#"URL:%#", url);
URL:http://test.com/path
Using Variables
NSString *s = #"string1";
NSString *s2 = #"string2";
NSLog(#"%#", s s2);
Doesn't compile (not surprised by this one)
Other literals
NSNumber *number = #1 #2;
Doesn't compile
Some questions
Is this string concatenation documented anywhere?
How long has it been supported?
What is the underlying implementation? I expect it will be [s1 stringByAppendingString:s2]
Is it considered good practice by any authoritative body?
This method of concatenating static NSStrings is a compile-time compiler capability that has been available for over ten years. It is usually used to allow long constant strings to be split over several lines. Similar capabilities have been available in "C" for decades.
In the C Programming Language book, 1988 second edition, page 38 describes string concatenation so it has been around for a long time.
Excerpt from the book:
String constants can be concatenated at compile time:
"hello," " world" is equivalent to "hello, world"
This is useful for spitting long strings across several source lines.
Objective-C is a strict superset of "C" so it has always supported "C" string concatenation and my guess is that because of that static NSString concatenation has always been available.
It is considered good practice when used to split a static string across several lines for readability.
I have been having a lot of trouble with NSString's stringWithFormat: method as of late. I have written an object that allows you to align N lines of text (separated by new lines) either centered, right, or left. At the core of my logic is NSString's stringWithFormat. I use this function to pad my strings with spaces on the left or right of individual lines to produce the alignment I want. Here is an example:
NSString *str = #"$3.00" --> 3 dollars
[NSString stringWithFormat:#"%8s", [str cStringUsingEncoding:NSUnicodeStringEncoding]] --> returns --> " $3.00"
As you can see the above example works great, I padded 3 spaces on the left and the resulting text is right aligned/justified. Problems begin to arise when I start to pass in foreign currency symbols, the formatting just straight up does not work. It either adds extra symbols or just returns garbage.
NSString *str = #"Kč1.00" --> 3 Czech Koruna (Czech republic's currency)
[NSString stringWithFormat:#"%8s", [str cStringUsingEncoding:NSUnicodeStringEncoding]] --> returns --> " Kč1.00"
The above is just flat out wrong... Now I am not a string encoding expert but I do know NSString uses the international standardized unicode encoding for special characters well outside basic ASCII domain.
How can I fix my problem? What encoding should I use? I have tried so many different encoding enums I have lost count, everything from NSMACOSRomanEncoding to NSUTF32UnicodeBigEndian.. My last resort will be to just completely ditch using stringWithFormat all together, maybe it was only meant for simple UTF8Strings and basic symbols.
If you want to represent currency, is a lot better if you use a NSNumberFormatter with currency style (NSNumberFormatterCurrencyStyle). It reads the currentLocale and shows the currency based on it. You just need to ask its string representation and append to a string.
It will be a lot easier than managing unicode formats, check a tutorial here
This will give you the required result
NSString *str = #"Kč1.00";
str=[NSString stringWithFormat:#"%#%8#",[#" " stringByPaddingToLength:3 withString:#" " startingAtIndex:0],str];
Out Put : #" Kč1.00";
Just one more trick to achieve this -
If you like use it :)
[NSString stringWithFormat:#"%8s%#",[#"" cStringUsingEncoding:NSUTF8StringEncoding],str];
This will work too.
I have an issue in an application I'm writing where I need to compare one NSURL that points to a file and an NSString, which is an incoming string representation of the same file path.
I can't get them to compare – the output I'm given when NSLogging is confusing, perhaps it is a encoding issue?
I can make them look the same with this code: [urlString stringByRemovingPercentEncoding];
The raw output for the NSURL is:
file:///var/mobile/Applications/F14AFBD8-FF60-4094-8BBD-7AC2477E0B20/Documents/1.%20AKTIV%20SA%CC%88LJFOLDER/Sa%CC%88ljfolder2014-SP1.pdf
And for the NSString:
/var/mobile/Applications/F14AFBD8-FF60-4094-8BBD-7AC2477E0B20/Documents/1. AKTIV SÄLJFOLDER/Säljfolder2014-SP1.pdf
If I run stringByRemovingPercentEncoding on the NSURL it looks the same, but they don't compare.
If I run stringByAddingPercentEscapesUsingEncoding:NSUTF8StringEncoding to the NSString I get file:///var/mobile/Applications/F14AFBD8-FF60-4094-8BBD-7AC2477E0B20/Documents/nestle/1.%20AKTIV%20S%C3%84LJFOLDER/S%C3%A4ljfolder2014-SP1.pdf
Note that the percentages is not the same on the urls. I have tried so many things, changing encodings etc. but can't find a way to solve this.
Edit
So, I tried the precomposedStringWithCanonicalMapping as follows:
NSLog(#"EQUAL? :%hhd", [[strippedUrlString precomposedStringWithCanonicalMapping] isEqualToString:[filePath precomposedStringWithCanonicalMapping]]); – returns 0
I logged the strings and got
/Users/xxxxxx/Library/Application Support/iPhone Simulator/7.0/Applications/C05E0885-7B58-4B2F-A6B4-D9388E60462C/Documents/1. AKTIV SÄLJFOLDER/Säljfolder2014-SP1.pdf
with NSLog(#"Precompose url 1: %#", [strippedUrlString precomposedStringWithCanonicalMapping]);
for the first string and
/Users/xxxxxx/Library/Application%20Support/iPhone%20Simulator/7.0/Applications/C05E0885-7B58-4B2F-A6B4-D9388E60462C/Documents/1.%20AKTIV%20SA%CC%88LJFOLDER/Sa%CC%88ljfolder2014-SP1.pdf
with NSLog(#"Precompose file 1: %#", [filePath precomposedStringWithCanonicalMapping]);
for the second.
Tried same code, but with precomposedStringWithCompatibilityMapping and got exactly the same result :(
Probably you ran in a problem that in Unicode equivalent strings are not always binary equal.
http://en.wikipedia.org/wiki/Unicode_equivalence
You have
…SA%CC%88…:
This is the problem.
It means: We have an "A" and a combining diaeresis -> Ä. The diaeresis is the 0xCC88, which is UTF-8 for Unicode 0x0308 (COMBINING DIAERESIS). So the Ä is encoded as an A with an combining diaeresis.
…S%C3%84…:
This is easy. 0xC384 is UTF-8 for 0x00C4 that means A-Umlaut -> Ä
First of all: What is the source of the first string?
Addition: You can use precomposedStringWith…Mapping (NSString).
BTW: You can compare strings without diacritic marks using -compare:withOptions: et al. with the option NSDiacriticInsensitiveSearch. In this case, I assume, string 1 equals string 2. Butt it would equal an "A", too, what is probably not what you want.
So as I work my way through understanding string methods, I came across this useful class
NSCharacterSet
which is defined in this post quite well as being similar to a string excpet it is used for holding the char in an unordered set
What is differnce between NSString and NSCharacterset?
So then I came across the useful method invertedSet, and it bacame a little less clear what was happening exactly. Also I a read page a fter page on it, they all sort of glossed over the basics of what was happening and jumped into advanced explainations. So if you wanted to know what this is and why we use It SIMPLY put, it was not so easy instead you get statements like this from the apple documentation: "A character set containing only characters that don’t exist in the receiver." - and how do I use this exactly???
So here is what i understand to be the use. PLEASE provide in simple terms if I have explained this incorrectly.
Example Use:
Create a list of Characters in a NSCharacterSetyou want to limit a string to contain.
NSString *validNumberChars = #"0123456789"; //Only these are valid.
//Now assign to a NSCharacter object to use for searching and comparing later
validCharSet = [NSCharacterSet characterSetWithCharactersInString:validNumberChars ];
//Now create an inverteds set OF the validCharSet.
NSCharacterSet *invertedValidCharSet = [validCharSet invertedSet];
//Now scrub your input string of bad character, those characters not in the validCharSet
NSString *scrubbedString = [inputString stringByTrimmingCharactersInSet:invertedValidCharSet];
//By passing in the inverted invertedValidCharSet as the characters to trim out, then you are left with only characters that are in the original set. captured here in scrubbedString.
So is this how to use this feature properly, or did I miss anything?
Thanks
Steve
A character set is a just that - a set of characters. When you invert a character set you get a new set that has every character except those from the original set.
In your example you start with a character set containing the 10 standard digits. When you invert the set you get a set that has every character except the 10 digits.
validCharSet = [NSCharacterSet characterSetWithCharactersInString:validNumberChars];
This creates a character set containing the 10 characters 0, 1, ..., 9.
invertedValidCharSet = [validCharSet invertedSet];
This creates the inverted character set, i.e. the set of all Unicode characters without
the 10 characters from above.
scrubbedString = [inputString stringByTrimmingCharactersInSet:invertedValidCharSet];
This removes from the start and end of inputString all characters that are in
the invertedValidCharSet. For example, if
inputString = #"abc123d€f567ghj😄"
then
scrubbedString = #"123d€f567"
Is does not, as you perhaps expect, remove all characters from the given set.
One way to achieve that is (copied from NSString - replacing characters from NSCharacterSet):
scrubbedString = [[inputString componentsSeparatedByCharactersInSet:invertedValidCharSet] componentsJoinedByString:#""]
This is probably not the most effective method, but as your question was about understanding
NSCharacterSet I hope that it helps.