Ruby 1.9 doesn't support Unicode normalization yet - ruby-on-rails

Question

I'm trying to port over some of my old rails apps to Ruby 1.9 and I keep getting warnings about how "Ruby 1.9 doesn't support Unicode normalization yet." I've tracked it down to this function, but I'm getting about 20 warning messages per request:
rails-2.3.5/activesupport/lib/active_support/inflector.rb
def transliterate(string)
warn "Ruby 1.9 doesn't support Unicode normalization yet"
string.dup
end
Any ideas how I should start tracking these down and resolving it?

Answer 1

If you are aware of the consequences, i.e. accented characters will not be transliterated in Ruby 1.9.1 + Rails 2.3.x, place this in config/initializers to silence the warning:
# http://stackoverflow.com/questions/2135247/ruby-1-9-doesnt-support-unicode-normalization-yet
module ActiveSupport
module Inflector
# Calling String#parameterize prints a warning under Ruby 1.9,
# even if the data in the string doesn't need transliterating.
if Rails.version =~ /^2\.3/
undef_method :transliterate
def transliterate(string)
string.dup
end
end
end
end
Rails 3 does indeed solve this issue, so a more future-proof solution would be to migrate towards that.

Answer 2

The StringEx Gem seems to work pretty well. It has no dependency on Iconv either.
It adds some methods to the String class, like "to_ascii" which does beautiful transliteration out of the box:
require 'stringex'
"äöüÄÖÜßë".to_ascii #=> "aouAOUsse"
Also, the Babosa Gem does a great job transliterating UTF-8 strings, even with language support:
"Jürgen Müller".to_slug.transliterate.to_s #=> "Jurgen Muller"
"Jürgen Müller".to_slug.transliterate(:german).to_s #=> "Juergen Mueller"
Enjoy.

Answer 3

That method definition is wrapped in an if-statement for Ruby 1.9. Right above it, you will find the regular definition, which shows a bit more of what this is doing. It's a method used to convert accented characters into their regular variants. E.g.: á => a, or ë => e
But this method is only used in parameterize, which is in turn defined right above transliterate. This is all still in ActiveSupport. I can't find anything that is directly calling parameterize.
So perhaps you're using parameterize or transliterate yourself, somewhere in your Rails application?
Common usage (according to the parameterize documentation) is for creating friendly permalinks from arbitrary strings, much like SO does, for example:
http://stackoverflow.com/questions/2135247/ruby-1-9-doesnt-support-unicode-normalization-yet
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Answer 4

Replace the body of the method with
raise "transliterate called"
and observe a backtrace which will show you where the stuff is coming from at the first call. Your app will of course collapse as well but that will likely give you the culprit from the first try.

Answer 5

I appreciate that this is a dirty way to solve the problem, but having read the error message I'm aware of the issue. So I want to get rid of the warnings. I dropped this code in environment.rb:
module ActiveSupport
module Inflector
# Calling String#parameterize prints a warning under Ruby 1.9,
# even if the data in the string doesn't need transliterating.
# Maybe Rails 3 will have fixed it...?
if RAILS_GEM_VERSION =~ /^2\.3/
undef_method :transliterate
def transliterate(string)
string.dup
end
end
end
end

Answer 6

If you'd rather not monkey patch the Inflector module, you can also do this...
Both of the following worked for me to silence this annoying "Ruby 1.9 doesn't support Unicode normalization yet" warning:
silence_stream(STDERR) {
whatever_code_caused_transliterate_to_be_called
}
or
silence_warnings {
whatever_code_caused_transliterate_to_be_called
}
This does have the disadvantage that it requires cluttering up your calling code, but it is a technique you can use generally whenever you don't want to see warnings or other output.
activesupport provides silence_stream and silence_warnings in activesupport-2.3.11/lib/active_support/core_ext/kernel/reporting.rb

Answer 7

String#unicode_normalize, String#unicode_normalize!, String#unicode_normalized? will be introduced in Ruby 2.2. Sample code and implementation can be seen in test case, lib/unicode_normalize.rb and lib/unicode_normalize/normalize.rb.
// U+00E1: LATIN SMALL LETTER A WITH ACUTE
// U+U+0301: COMBINING ACUTE ACCENT
puts "\u00E1" == "a\u0301".unicode_normalize(:nfc)
puts true == "a".unicode_normalized?(:nfc)