iconv deprecation warning with ruby 1.9.3 - ruby-on-rails

I'm getting this warning when I run rspec:
/gems/activesupport-3.1.0/lib/active_support/dependencies.rb:240:in `block in require': iconv will be deprecated in the future, use String#encode instead.
I get the same warning with rails 3.1.0, 3.1.1, 3.1.2.rc2 versions. Seems it's related to sqlite3 gem, but I'm not sure. There are no warnings with ruby 1.9.2
Any suggestions how to deal with it?

You are getting this deprecation notice cause a library somewhere is requiring iconv.
iconv is a gem created by Matz that can be used to convert strings from one format to another.
For example this is often used:
Iconv.iconv('UTF-8//IGNORE', 'UTF-8', content) this little bit of magic takes a UTF-8 string that may have invalid chars and converts it to a proper UTF-8 string.
It has been decided that in Ruby 1.9.3 we should not be using iconv any more and instead use the built-in String#encode. encode is more powerful and allows you more flexibility.
The theory is that the above example could be replaced with:
string.encode("UTF-8", :invalid => :replace, :undef => :replace, :replace => "?")
In practice it seems this is imperfect.
This also leads to a less than easy story for gem creators who wish to support 1.8:
content = RUBY_VERSION.to_f < 1.9 ?
Iconv.iconv('UTF-8//IGNORE', 'UTF-8', "content") :
"#{content}".encode(Encoding::UTF_8, :invalid => :replace, :undef => :replace, :replace => '')
So, you have a gem somewhere that is requiring iconv, to find it:
Assuming your error message is: /gems/activesupport-3.1.0/lib/active_support/dependencies.rb:240
Open up /gems/activesupport-3.1.0/lib/active_support/dependencies.rb on line 240:
Add the line:
p caller if file =~ /iconv/
(just after: load_dependency(file) { result = super })
You will get a big fat stack trace:
rake --tasks
/home/sam/.rvm/gems/ruby-1.9.3-p125/gems/activesupport-3.2.6/lib/active_support/dependencies.rb:251:in `block in require': iconv will be deprecated in the future, use String#encode instead.
["/home/sam/.rvm/gems/ruby-1.9.3-p125/gems/calais-0.0.13/lib/calais.rb:5:in `'",
.. more omitted ..
This tells me it is the calais gem. Looking through pull requests, I am not the first. The pull has not been yanked in.
Depending on the gem, there may be an upgraded version that does not have this error, so I would recommend you upgrade your gems first. If you are unlucky you may be stuck with the unfortunate task of forking a gem to get rid of this (if for example your pull request to fix it languishes)

If you're seeing this, it's very probably not Rails. If you look at the method surrounding the line being referred to in the error you posted, you'll see the following:
def require(file, *)
result = false
load_dependency(file) { result = super }
result
end
I'm not saying it's your code, necessarily, but I'm certain that it's not actually the line in question where iconv is being called. In my case, I found that my project's code actually contained a reference to iconv.
If you want to check your code for such a reference, try grep -ir iconv ./ in your project directory.
When iconv is actually in a library it can be harder to find. By temporarily changing the above method to:
def require(file, *)
result = false
puts
puts caller.reverse
load_dependency(file) { result = super }
result
end
You can then easily run your code and grep out the relevant lines of the backtrace to find the root cause of the warning.
ruby your/code.rb 2>&1 | grep -B 5 iconv

Add this to the start of your program:
oldverb = $VERBOSE; $VERBOSE = nil
require 'iconv'
$VERBOSE = oldverb
and curse the people who think this is a professional way to handle deprecation.

You can pin down the exact location of the warning by generating exceptions for ActiveSupport::Deprecation, instead of just printing to the log. At the top of application.rb:
ActiveSupport::Deprecation.behavior = Proc.new do |message, backtrace|
raise message
end
Once you've figured out where the warning is coming from (by inspecting the full backtrace), remove this again.

To remove this warning...
go to your .rvm directory and find iconv.c (mine was at ~/.rvm/src/ruby-1.9.3-p125/ext/iconv/iconv.c)
edit that file are remove or comment out the call to warn_deprecated() (should be near the bottom)
from that file's directory, run ruby extconf.rb
then make
then make install
Should do the trick

Related

Activesupport / Multi json: "Did not recognize your adapter specification"

I have a Ruby 1.9.3 / Rails 3.1 project with the following in the gemfile:
gem 'rails', '3.1.12'
gem 'json'
gem 'multi_json', '1.7.7'
That version of rails sets activesupport to 3.1.12 as well. I'm not sure what the exact cause of the problem is, but when running bundle exec rake test, I got the error:
/home/user/.gem/ruby/1.9.3/gems/multi_json-1.7.7/lib/multi_json.rb:121:in 'rescue in load_adapter': Did not recognize your adapter specification. (ArgumentError)
...
(more stack trace, including activesupport methods)
Fortunately I found a solution! See below.
Edit: My original answer is outdated and incorrect; read it if you please, but please read the updated information at the bottom.
After viewing a ton of other questions such as these ones:
OmniAuth Login With Twitter - "Did not recognize your adapter specification." Error
Capistrano deploy: "Did not recognize your adapter specification" during assets:precompile
https://github.com/intridea/multi_json/issues/132
I hadn't found a solution, so I dove into the library and determined that load_adapter was receiving the parameter "JSONGem". The alias was failing, and the method attempted to load
/home/user/.gem/ruby/1.9.3/gems/multi_json-1.7.7/lib/multi_json/adapters/JSONGem.rb
This file doesn't exist, but .../json_gem.rb does exist! So I modified load adapter as follows:
def load_adapter(new_adapter)
# puts "new_adapter: #{new_adapter}" # Debugging
# puts "new_adapater.class: #{new_adapter.class}" # Debugging
case new_adapter
when String, Symbol
new_adapter = ALIASES.fetch(new_adapter.to_s, new_adapter)
new_adapter = "json_gem" if new_adapter =~ /^jsongem$/i # I added this line
# puts "final adapter: #{new_adapter}" # debugging
require "multi_json/adapters/#{new_adapter}"
klass_name = new_adapter.to_s.split('_').map(&:capitalize) * ''
MultiJson::Adapters.const_get(klass_name)
when NilClass, FalseClass
load_adapter default_adapter
when Class, Module
new_adapter
else
raise NameError
end
rescue NameError, ::LoadError
raise ArgumentError, 'Did not recognize your adapter specification.'
end
This fixed the problem for me. It's probably not an optimal solution (ideally I would understand WHY the ALIASES.fetch failed, if that is indeed what happened, and fix that), but if your problem is similar then hopefully this quick fix can help.
Update
It's not viable for deployability reasons to modify someone else's gem. Fortunately I found the root cause of the problem. In project_root/config/initializers/security_patches.rb, we had the line
ActiveSupport::JSON.backend = "JSONGem"
This was the recommended fix to a security bug in older versions of rails. Now that we are on a newer version of rails (i.e, > 3.0), we can simply replace "JSONGem" with "json_gem" (which is what my original modification was doing, in a roundabout way) and not worry about the security issue.

How to run Ruby on Rails 3 with ruby 2.0

Is there an easy fix, how I could continue an old rails 3-0.20 installation under ruby 2.0?
The first error, caused by this line:
<%= stylesheet_link_tag :all %>
is
ActionView::Template::Error (no implicit conversion of nil into String):
An upgrade of the rails version would be the best, but unfortunately it is not possible in my case.
Hotfix the problem with the following line at the end in application.rb
ActionController::Base.config.relative_url_root = ''
I ran into the same issue. After drilling down into the stylesheet_link_tag method, I found that the issue comes from here
# actionpack-3.0.20/lib/action_view/helpers/asset_tag_helper.rb:749
if has_request && include_host && !source.start_with?(controller.config.relative_url_root)
The problem is String#starts_with?. In 1.9.3, that method will handle a nil as an input. 2.0.0 does not allow that.
ruby-1.9.3> 'whatever'.start_with? nil
=> false
ruby-2.0.0> 'whatever'.start_with? nil
TypeError: no implicit conversion of nil into String
It's probably also true that later versions of Rails set the value to '' if it's not set to prevent this issue. The hotfix mentioned above does fix the issue, but the root cause is differences between 1.9.3 and 2.0.0.

JSON encoding/decoding with unicode in rails

I upgraded downgraded to rails 2.3.17 due to the security bugs, but now I can't decode json strings that I have saved down to a DB if they have unicode in them :(. Is there a way to process the string such that it decodes properly?
e = ActiveSupport::JSON.encode({'a' => "Hello Unicode \u2019"})
ActiveSupport::JSON.decode(e)
gives me
RangeError: 8217 out of char range
from /app/vendor/bundle/ruby/1.9.1/gems/activesupport-2.3.17/lib/active_support/json/backends/okjson.rb:314:in `unquote'
from /app/vendor/bundle/ruby/1.9.1/gems/activesupport-2.3.17/lib/active_support/json/backends/okjson.rb:251:in `strtok'
from /app/vendor/bundle/ruby/1.9.1/gems/activesupport-2.3.17/lib/active_support/json/backends/okjson.rb:215:in `tok'
from /app/vendor/bundle/ruby/1.9.1/gems/activesupport-2.3.17/lib/active_support/json/backends/okjson.rb:178:in `lex'
from /app/vendor/bundle/ruby/1.9.1/gems/activesupport-2.3.17/lib/active_support/json/backends/okjson.rb:46:in `decode'
from /app/vendor/bundle/ruby/1.9.1/gems/activesupport-2.3.17/lib/active_support/json/backends/okjson.rb:612:in `decode'
from /app/vendor/bundle/ruby/1.9.1/gems/activesupport-2.3.17/lib/active_support/json/decoding.rb:14:in `decode'
from (irb):30
from /usr/local/bin/irb:12:in `<main>'
I can't change the first line since it's coming from the DB like that.
This used to work.
You can change the backend JSON provider in ActiveSupport.
Add ActiveSupport::JSON.backend = "JSONGem" into an application initialiser (I added it to application.rb). This fixed the unicode parsing issues I had after I upgraded activesupport to 3.0.20.
See the vulnerability notice which caused this update - It mentions that this workaround should apply to 2.3.16 as well.
From rails console:
> ActiveSupport::VERSION::STRING
=> "3.0.20"
> ActiveSupport::JSON.decode('{"test":"string\u2019"}')
RangeError: 8217 out of char range
> ActiveSupport::JSON.backend = "JSONGem"
> ActiveSupport::JSON.decode('{"test":"string\u2019"}')
 => {"test"=>"string’"}
The JSON gem will handle this correctly.
As a note, the gem is much more strict than the other JSON parsers out there. For example:
{ 'test' : 'value' }
This is not valid JSON even though it looks okay.
For whatever reason the non-UTF-8 savvy JSON parser shipped as part of the 2.3.16 patch which is really sloppy on the part of the maintainer.
Switch to 2.3.15 which should be fine because that's when the fixes landed.
Curse the developer who started this project in rails
Begin work on porting to python post haste

XML parsing in Ruby

I am using a REXML Ruby parser to parse an XML file. But on a 64 bit AIX box with 64 bit Ruby, I am getting the following error:
REXML::ParseException: #<REXML::ParseException: #<RegexpError: Stack overflow in
regexp matcher:
/^<((?>(?:[\w:][\-\w\d.]*:)?[\w:][\-\w\d.]*))\s*((?>\s+(?:[\w:][\-\w\d.]*:)?[\w:][\-\w\d.]*\s*=\s*(["']).*?\3)*)\s*(\/)?>/mu>
The call for the same is something like this:
REXML::Document.new(File.open(actual_file_name, "r"))
Does anyone have an idea regarding how to solve this issue?
I've had several issues for REXML, it doesn't seem to be the most mature library. Usually I use Nokogiri for Ruby XML parsing stuff, it should be faster and more stable than REXML. After installing it with sudo gem install nokogiri, you can use something like this to get a DOM instance:
doc = Nokogiri.XML(File.open(actual_file_name, 'rb'))
# => #<Nokogiri::XML::Document:0xf1de34 name="document" [...] >
The documentation on the official webpage is also much better than that of REXML, IMHO.
I almost immediately found the answer.
The first thing I did was to search in the ruby source code for the error being thrown.
I found that regex.h was responsible for this.
In regex.h, the code flow is something like this:
/* Maximum number of duplicates an interval can allow. */
#ifndef RE_DUP_MAX
#define RE_DUP_MAX ((1 << 15) - 1)
#endif
Now the problem here is RE_DUP_MAX. On AIX box, the same constant has been defined somewhere in /usr/include.
I searched for it and found in
/usr/include/NLregexp.h
/usr/include/sys/limits.h
/usr/include/unistd.h
I am not sure which of the three is being used(most probably NLregexp.h).
In these headers, the value of RE_DUP_MAX has been set to 255! So there is a cap placed on the number of repetitions of a regex!
In short, the reason is the compilation taking the system defined value than that we define in regex.h!
This also answers my question which i had asked recently:
Regex limit in ruby 64 bit aix compilation
I was not able to answer it immediately as i need to have min of 100 reputation :D :D
Cheers!

Rails + Ruby 1.9 "invalid byte squence in US-ASCII"

After upgrading to ruby 1.9 we began to notice pages failing to render from the rails template renderer when a user used a non-ASCII character. Specifically "é". I was able to resolve this issue on one of our staging servers, but I have not been able to reproduce the fix on our production server.
The fix that seemed to work the first time:
Converted the database from latin1 to utf8 using the convert_charset tool available here: http://www.mysqlperformanceblog.com/2009/03/17/converting-character-sets/. (including setting default_character_set=utf8 in my.cnf and running SET GLOBAL character_set_server=utf8
Switched to the sam-mysql-ruby adapter (instead of the standard mysql adapter: http://gemcutter.org/gems/sam-mysql-ruby)
Restarted rails
The error is:
"invalid byte sequence in US-ASCII"
Oddly, after following the steps above the error has not changed on our production server. Setting encoding: utf8 in database.yml does not change the error either.
The error raised on the following line of code:
<%= link_to h(question.title), question_path(question) %>
This blog seems to suggest a fix, but it mentions that this should not be a problem in 1.9: http://www.igvita.com/2007/04/11/secure-utf-8-input-in-rails/ (and it's over 2 years old).
I imagine this problem might soon affect a lot of people as more rails developers people switch to 1.9.
I found the solution:
The problem is:
Fetching data from any database (Mysql, Postgresql, Sqlite2 & 3), all configured to have UTF-8 as it's character set, returns the data with ASCII-8BIT in ruby 1.9.1 and rails 2.3.2.1.
(Taken from: https://rails.lighthouseapp.com/projects/8994/tickets/2476)
My attempt to use the patched mysql adapter likely failed because my database was not configured to natively use utf8, so the patched adapter failed to work properly.
The fix ended up being to use the patch file available here: http://gnuu.org/2009/11/06/ruby19-rails-mysql-utf8/
require 'mysql'
class Mysql::Result
def encode(value, encoding = "utf-8")
String === value ? value.force_encoding(encoding) : value
end
def each_utf8(&block)
each_orig do |row|
yield row.map {|col| encode(col) }
end
end
alias each_orig each
alias each each_utf8
def each_hash_utf8(&block)
each_hash_orig do |row|
row.each {|k, v| row[k] = encode(v) }
yield(row)
end
end
alias each_hash_orig each_hash
alias each_hash each_hash_utf8
end
(Placed in lib/mysql_utf8fix.rb and required in enviornment.rb using require 'lib/mysql_utf8fix.rb')
it is only require 'mysql_utf8fix.rb' (rails 2.3.11)
Please user mysql2(gem) adapter instead of mysql adapter in database.yml
and remove the mysql patches(If exists) and add the following lines in environment.rb.
Encoding.default_external = Encoding::UTF_8
Encoding.default_internal = Encoding::UTF_8
Then run in apache and passenger it ll work fine
Thanks,
Ramanavel Selvaraju.

Resources