Rails encoding in ASCII-8BIT - ruby-on-rails

I know this have been asked several times, but to me is happening something strange:
I have an index view where rendering certain characters (letters with accent) causes Rails to raise the exception
incompatible character encodings: ASCII-8BIT and UTF-8
so i checked my strings encoding and this is actually ASCII-8BIT everywhere, even though i set the proper encoding to UTF-8 in my application.rb
config.encoding = "utf-8"
and in my enviroment.rb
Encoding.default_external = Encoding::UTF_8
Encoding.default_internal = Encoding::UTF_8
and in my database it appear:
character_set_database = utf-8
as suggestend in some guides.
Strings are inserted with a textarea field and are not concatenated to any other already inserted string.
The strange things are:
this happens only in the index view, whereas this is not happening in the show (same resource)
this happens only for this model (which is an email, with subject and body, but this shouldn't affect anything)
In my development environment everything goes well setting str.force_encoding('utf-8'), whereas in my production environment this is not working. (dev i'm with Ruby 2.0.0, in production Ruby 2.1.0, both Rails4, and both MySql)
setting the file view with # encoding utf-8 also doesn't work
trying str.force_encoding('ascii-8bit').encode('utf-8') says Encoding::UndefinedConversionError "\xC3" from ASCII-8BIT to UTF-8 which is an à, while using body.force_encoding('ascii-8bit').encode('UTF-8', :invalid => :replace, :undef => :replace, :replace => '?'), replaces all accented charaters with a ?, while str.force_encoding('iso-8859-1').encode('utf-8') obviously generates the wrong character (a ?).
So my questions are 2:
- why is rails setting the string encodint to ascii-8bit?
- how to solve this issue?
I've already checked these questions (the newest ones with rails4):
Rails View Encoding Issues
"\xC2" to UTF-8 in conversion from ASCII-8BIT to UTF-8
How to convert a string to UTF8 in Ruby
Encoding::UndefinedConversionError: "\xE4" from ASCII-8BIT to UTF-8
and other resources also, but nothing worked.

You probably have a string literal in your source code somewhere that you then concatenate another string too. For instance:
some_string = "this is a string"
or even
some_string = "" #empty string
Those strings, stored in some_string, will be marked ASCII_8BIT, and if you then later do something like:
some_string = some_string + unicode_string
Then you'll get the error. That is, those strings will be marked ASCII-8BIT unless you add, to the top of the file where the string literals are created:
#encoding: utf-8
That declaration determines the default encoding that string literals in source code will have.
I am just guessing, because this pattern is a common source of this problem. To know more for sure, it would take more information than is in your question -- it would take debugging the actual source code, to figure out exactly what string is tagged as ASCII-8BIT when you expect it to be tagged UTF-8 instead, and exactly where that String came from.

Related

Rails admin encoding error when i try to use 'windows-1250'

I got error :_ incompatible character encodings: UTF-8 and Windows-1250_
when i try to show something with chars from Poland ie. 'ąęźć'
in my application.rb i got:
config.encoding = "windows-1250"
In database.yml:
encoding: windows-1250
How can i show params in windows-1250 in rails admin panel?
I would suggest you go with utf-8 encoding (which is ruby's default these days).
Your input 'ąęźć' is a valid utf-8 string, so you would face no problem in decoding it as a utf-8 string.
If you still want to hack around, you can use:
'ąęźć'.mb_chars.tidy_bytes.to_s
which should also give you the desired output.

How do I fix "Incompatible character encodings: UTF-8 and ASCII-8BIT"?

Incompatible character encodings: UTF-8 and ASCII-8BIT
How can I solve this error on Rails 3.2.3 and Ruby 1.9.3?
I tried to put these two lines in the environment.rb:
Encoding.default_external = Encoding::UTF_8
Encoding.default_internal = Encoding::UTF_8
I am sure that application.rb also contains this line "config.encoding = "utf-8".
Does anyone know how to solve this?
Consider using so called magic comments on the first line of your .rb file with encoding:
# encoding: UTF-8
class Xyz
...
end
It is very important in files where you place text with accents and other non-ASCII characters. They are the primary cause of the error you mention.
Sometimes it may happen that you mistype a character and, instead of a letter, you insert a hidden symbol. Also check your file for these. Look at the line ends and in spaces.
If you have data to store on the hdd, you can try data.force_encoding('UTF-8').

Unicode symbol in Rails' view

I'm trying to include a degree symbol into my Rails view. If I put the degree symbol (°) straight into .html.erb file, it is displayed by browser in a normal way.
But this symbol should be transferred to view via string. And here the problem begins.
If I put this:
<%= 176.chr %>
into view, or put
.... + 176.chr
into ruby source, I get
incompatible character encodings: UTF-8 and ASCII-8BIT
How to make Rails recognize all views as UTF-8 by default?
You can use special replacement for this symbol in HTML: °.
http://www.w3schools.com/charsets/ref_html_entities_4.asp
You have to put it in HTML, outside the <%= %>. Or use raw helper. Or mark it as html_safe. And by the way, did you try to supply encoding in your chr? Like 176.chr(__ENCODING__) (__ENCODING__ here isn't placeholder, it's Ruby thing) or 176.chr(Encoding::UTF_8). All these approaches should work.
This should already be specified inside your application.rb inside /config/.
The relevant section should look like this:
module App
class Application < Rails::Application
# Configure the default encoding used in templates for Ruby 1.9.
config.encoding = "utf-8"
end
end
I think the issue here is that you are generating a ASCII-8BIT character that should be inserted into the UTF-8 body.
If you want to use a UTF-8 String in your Ruby code you have to put this magic string into the first line of your ruby file:
# encoding: UTF-8
Details on Encoding in Ruby 1.9 can be found here

RoR character class regex

I have the following line of code in my Ruby on Rails app, which checks whether the given string contains Korean characters or not:
isKorean = !/\p{Hangul}/.match(word).nil?
It works perfectly in the console, but raises a syntax error for the actual app:
invalid character property name {Hangul}: /\p{Hangul}/
What am I missing and how can I get it to work?
This is a character encoding issue, you need to add:
# encoding: utf-8
to the top of the Ruby file you're using that regex in. You can probably use any encoding that the character class you're using exists in instead of UTF-8 if you wish. Note that in Ruby 2.0, UTF-8 is now the default, so this is not needed in Ruby 2.0+.
This is known as a "magic comment". You can and should read more about encoding in Ruby 1.9. Note that encoding in Rails views is handled automatically by config.encoding (set to UTF-8 by default in config/application.rb.
It was likely working in the console because your terminal is set to use UTF-8 already.

Rails: encoding woes with serialized hashes despite UTF8

I've just updated from ruby 1.9.2 to ruby 1.9.3p0 (2011-10-30 revision 33570). My rails application uses postgresql as its database backend. The system locale is UTF8, as is the database encoding. The default encoding of the rails application is also UTF8. I have Chinese users who input Chinese characters as well as English characters. The strings are stored as UTF8 encoded strings.
Rails version: 3.0.9
Since the update some of the existing Chinese strings in the database are no longer displayed correctly. This does not affect all strings, but only those that are part of a serialized hash. All other strings that are stored as plain strings still appear to be correct.
Example:
This is a serialized hash that is stored as a UTF8 string in the database:
broken = "--- !map:ActiveSupport::HashWithIndifferentAccess \ncheckbox: \"1\"\nchoice: \"Round Paper Clips \\xEF\\xBC\\x88\\xE5\\x9B\\x9E\\xE5\\xBD\\xA2\\xE9\\x92\\x88\\xEF\\xBC\\x89\\r\\n\"\ninfo: \"10\\xE7\\x9B\\x92\"\n"
In order to convert this string to a ruby hash, I deserialize it with YAML.load:
broken_hash = YAML.load(broken)
This returns a hash with garbled contents:
{"checkbox"=>"1", "choice"=>"Round Paper Clips ï¼\u0088å\u009B\u009Eå½¢é\u0092\u0088ï¼\u0089\r\n", "info"=>"10ç\u009B\u0092"}
The garbled stuff is supposed to be UTF8-encoded Chinese. broken_hash['info'].encoding tells me that ruby thinks this is #<Encoding:UTF-8>. I disagree.
Interestingly, all other strings that were not serialized before look fine, however. In the same record a different field contains Chinese characters that look just right---in the rails console, the psql console, and the browser. Every string---no matter if serialized hash or plain string---saved to the database since the update looks fine, too.
I tried to convert the garbled text from a possible wrong encoding (like GB2312 or ANSI) to UTF-8 despite ruby's claim that this was already UTF-8 and of course I failed. This is the code I used:
require 'iconv'
Iconv.conv('UTF-8', 'GB2312', broken_hash['info'])
This fails because ruby doesn't know what to do with illegal sequences in the string.
I really just want to run a script to fix all the old, presumably broken serialized hash strings and be done with it. Is there a way to convert these broken strings to something resembling Chinese again?
I just played with the encoded UTF-8 string in the raw string (called "broken" in the above example). This is the Chinese string that is encoded in the serialized string:
chinese = "\\xEF\\xBC\\x88\\xE5\\x9B\\x9E\\xE5\\xBD\\xA2\\xE9\\x92\\x88\\xEF\\xBC\\x89\\r\\n\"
I noticed that it is easy to convert this to a real UTF-8 encoded string by unescaping it (removing the escape backslashes).
chinese_ok = "\xEF\xBC\x88\xE5\x9B\x9E\xE5\xBD\xA2\xE9\x92\x88\xEF\xBC\x89\r\n"
This returns a proper UTF-8-encoded Chinese string: "(回形针)\r\n"
The thing falls apart only when I use YAML.load(...) to convert the string to a ruby hash. Maybe I should process the raw string before it is fed to YAML.load. Just makes me wonder why this is so...
Interesting! This is likely due to the YAML engine "psych" that's used by default now in 1.9.3. I switched to the "syck" engine with YAML::ENGINE.yamler = 'syck' and the broken strings are correctly parsed.
This seems to have been caused by a difference in the behaviour of the two available YAML engines "syck" and "psych".
To set the YAML engine to syck:
YAML::ENGINE.yamler = 'syck'
To set the YAML engine back to psych:
YAML::ENGINE.yamler = 'psych'
The "syck" engine processes the strings as expected and converts them to hashes with proper Chinese strings. When the "psych" engine is used (default in ruby 1.9.3), the conversion results in garbled strings.
Adding the above line (the first of the two) to config/application.rb fixes this problem. The "syck" engine is no longer maintained, so I should probably only use this workaround to buy me some time to make the strings acceptable for "psych".
From the 1.9.3 NEWS file:
* yaml
* The default YAML engine is now Psych. You may downgrade to syck by setting
YAML::ENGINE.yamler = 'syck'.
Apparently the Syck and Psych YAML engines treat non-ASCII strings in different and incompatible ways.
Given a Hash like you have:
h = {
"checkbox" => "1",
"choice" => "Round Paper Clips (回形针)\r\n",
"info" => "10盒"
}
Using the old Syck engine:
>> YAML::ENGINE.yamler = 'syck'
>> h.to_yaml
=> "--- \ncheckbox: "1"\nchoice: "Round Paper Clips \\xEF\\xBC\\x88\\xE5\\x9B\\x9E\\xE5\\xBD\\xA2\\xE9\\x92\\x88\\xEF\\xBC\\x89\\r\\n"\ninfo: "10\\xE7\\x9B\\x92"\n"
we get the ugly double-backslash format the you currently have in your database. Switching to Psych:
>> YAML::ENGINE.yamler = 'psych'
=> "psych"
>> h.to_yaml
=> "---\ncheckbox: '1'\nchoice: ! "Round Paper Clips (回形针)\\r\\n"\ninfo: 10盒\n"
The strings stay in normal UTF-8 format. If we manually screw up the encoding to be Latin-1:
>> Iconv.conv('UTF-8', 'ISO-8859-1', "\xEF\xBC\x88\xE5\x9B\x9E\xE5\xBD\xA2\xE9\x92\x88\xEF\xBC\x89")
=> "ï¼\u0088å\u009B\u009Eå½¢é\u0092\u0088ï¼\u0089"
then we get the sort of nonsense that you're seeing.
The YAML documentation is rather thin so I don't know if you can force Psych to understand the old Syck format. I think you have three options:
Use the old unsupported and deprecated Syck engine, you'd need to YAML::ENGINE.yamler = 'syck' before you YAML anything.
Load and decode all your YAML using Syck and then re-encode and save it using Psych.
Stop using serialize in favor of manually serializing/deserializing using JSON (or some other stable, predictable, and portable text format) or use an association table so that you're not storing serialized data at all.

Resources