I am upgrading my app from rails 2 to 3 and when i 'require' this file that has an email address validator i get an 'invalid multibyte escape' error with:
dtext = '[^\\\\x80]'
pattern = /\A#{dtext}\z/
Any thoughts?
Try using:
pattern = /\A#{dtext}\z/, nil, 'n'
Check out details on encodings and regexp for more.
And I use and recommend this awesome article on encodings in Ruby.
Modify the rfc822.rb file and change the addr_spec line to the following:
addr_spec = Regexp.new("#{local_part}\\x40#{domain}", nil, 'n')
That should resolve the issue. I got the solution from another gem, see https://github.com/saepia/rfc822/blob/master/lib/rfc822.rb
Related
I'm using the following code to show description in template:
json.description resource.description if resource.description.present?
It gives me invalid byte sequence in UTF-8 error. I dig this a little bit, and find out the issue is my description has single quote as ’ instead of '. Wondering what's the best way to fix this encoding issue? Description is input by user and I have no control over it. Another weird issue is, I have multiple test environments, they all have the same Ruby and Rails version and they are running the same code, but only one of the environment has this error.
def to_utf8(str)
str = str.force_encoding("UTF-8")
return str if str.valid_encoding?
str = str.force_encoding("BINARY")
str.encode("UTF-8", invalid: :replace, undef: :replace)
end
ref: https://stackoverflow.com/a/17028706/455770
I upgraded downgraded to rails 2.3.17 due to the security bugs, but now I can't decode json strings that I have saved down to a DB if they have unicode in them :(. Is there a way to process the string such that it decodes properly?
e = ActiveSupport::JSON.encode({'a' => "Hello Unicode \u2019"})
ActiveSupport::JSON.decode(e)
gives me
RangeError: 8217 out of char range
from /app/vendor/bundle/ruby/1.9.1/gems/activesupport-2.3.17/lib/active_support/json/backends/okjson.rb:314:in `unquote'
from /app/vendor/bundle/ruby/1.9.1/gems/activesupport-2.3.17/lib/active_support/json/backends/okjson.rb:251:in `strtok'
from /app/vendor/bundle/ruby/1.9.1/gems/activesupport-2.3.17/lib/active_support/json/backends/okjson.rb:215:in `tok'
from /app/vendor/bundle/ruby/1.9.1/gems/activesupport-2.3.17/lib/active_support/json/backends/okjson.rb:178:in `lex'
from /app/vendor/bundle/ruby/1.9.1/gems/activesupport-2.3.17/lib/active_support/json/backends/okjson.rb:46:in `decode'
from /app/vendor/bundle/ruby/1.9.1/gems/activesupport-2.3.17/lib/active_support/json/backends/okjson.rb:612:in `decode'
from /app/vendor/bundle/ruby/1.9.1/gems/activesupport-2.3.17/lib/active_support/json/decoding.rb:14:in `decode'
from (irb):30
from /usr/local/bin/irb:12:in `<main>'
I can't change the first line since it's coming from the DB like that.
This used to work.
You can change the backend JSON provider in ActiveSupport.
Add ActiveSupport::JSON.backend = "JSONGem" into an application initialiser (I added it to application.rb). This fixed the unicode parsing issues I had after I upgraded activesupport to 3.0.20.
See the vulnerability notice which caused this update - It mentions that this workaround should apply to 2.3.16 as well.
From rails console:
> ActiveSupport::VERSION::STRING
=> "3.0.20"
> ActiveSupport::JSON.decode('{"test":"string\u2019"}')
RangeError: 8217 out of char range
> ActiveSupport::JSON.backend = "JSONGem"
> ActiveSupport::JSON.decode('{"test":"string\u2019"}')
=> {"test"=>"string’"}
The JSON gem will handle this correctly.
As a note, the gem is much more strict than the other JSON parsers out there. For example:
{ 'test' : 'value' }
This is not valid JSON even though it looks okay.
For whatever reason the non-UTF-8 savvy JSON parser shipped as part of the 2.3.16 patch which is really sloppy on the part of the maintainer.
Switch to 2.3.15 which should be fine because that's when the fixes landed.
Curse the developer who started this project in rails
Begin work on porting to python post haste
I'm getting this warning when I run rspec:
/gems/activesupport-3.1.0/lib/active_support/dependencies.rb:240:in `block in require': iconv will be deprecated in the future, use String#encode instead.
I get the same warning with rails 3.1.0, 3.1.1, 3.1.2.rc2 versions. Seems it's related to sqlite3 gem, but I'm not sure. There are no warnings with ruby 1.9.2
Any suggestions how to deal with it?
You are getting this deprecation notice cause a library somewhere is requiring iconv.
iconv is a gem created by Matz that can be used to convert strings from one format to another.
For example this is often used:
Iconv.iconv('UTF-8//IGNORE', 'UTF-8', content) this little bit of magic takes a UTF-8 string that may have invalid chars and converts it to a proper UTF-8 string.
It has been decided that in Ruby 1.9.3 we should not be using iconv any more and instead use the built-in String#encode. encode is more powerful and allows you more flexibility.
The theory is that the above example could be replaced with:
string.encode("UTF-8", :invalid => :replace, :undef => :replace, :replace => "?")
In practice it seems this is imperfect.
This also leads to a less than easy story for gem creators who wish to support 1.8:
content = RUBY_VERSION.to_f < 1.9 ?
Iconv.iconv('UTF-8//IGNORE', 'UTF-8', "content") :
"#{content}".encode(Encoding::UTF_8, :invalid => :replace, :undef => :replace, :replace => '')
So, you have a gem somewhere that is requiring iconv, to find it:
Assuming your error message is: /gems/activesupport-3.1.0/lib/active_support/dependencies.rb:240
Open up /gems/activesupport-3.1.0/lib/active_support/dependencies.rb on line 240:
Add the line:
p caller if file =~ /iconv/
(just after: load_dependency(file) { result = super })
You will get a big fat stack trace:
rake --tasks
/home/sam/.rvm/gems/ruby-1.9.3-p125/gems/activesupport-3.2.6/lib/active_support/dependencies.rb:251:in `block in require': iconv will be deprecated in the future, use String#encode instead.
["/home/sam/.rvm/gems/ruby-1.9.3-p125/gems/calais-0.0.13/lib/calais.rb:5:in `'",
.. more omitted ..
This tells me it is the calais gem. Looking through pull requests, I am not the first. The pull has not been yanked in.
Depending on the gem, there may be an upgraded version that does not have this error, so I would recommend you upgrade your gems first. If you are unlucky you may be stuck with the unfortunate task of forking a gem to get rid of this (if for example your pull request to fix it languishes)
If you're seeing this, it's very probably not Rails. If you look at the method surrounding the line being referred to in the error you posted, you'll see the following:
def require(file, *)
result = false
load_dependency(file) { result = super }
result
end
I'm not saying it's your code, necessarily, but I'm certain that it's not actually the line in question where iconv is being called. In my case, I found that my project's code actually contained a reference to iconv.
If you want to check your code for such a reference, try grep -ir iconv ./ in your project directory.
When iconv is actually in a library it can be harder to find. By temporarily changing the above method to:
def require(file, *)
result = false
puts
puts caller.reverse
load_dependency(file) { result = super }
result
end
You can then easily run your code and grep out the relevant lines of the backtrace to find the root cause of the warning.
ruby your/code.rb 2>&1 | grep -B 5 iconv
Add this to the start of your program:
oldverb = $VERBOSE; $VERBOSE = nil
require 'iconv'
$VERBOSE = oldverb
and curse the people who think this is a professional way to handle deprecation.
You can pin down the exact location of the warning by generating exceptions for ActiveSupport::Deprecation, instead of just printing to the log. At the top of application.rb:
ActiveSupport::Deprecation.behavior = Proc.new do |message, backtrace|
raise message
end
Once you've figured out where the warning is coming from (by inspecting the full backtrace), remove this again.
To remove this warning...
go to your .rvm directory and find iconv.c (mine was at ~/.rvm/src/ruby-1.9.3-p125/ext/iconv/iconv.c)
edit that file are remove or comment out the call to warn_deprecated() (should be near the bottom)
from that file's directory, run ruby extconf.rb
then make
then make install
Should do the trick
i found a ruby encryption library which is using this function.
def process(text)
0.upto(text.length-1) {|i| text[i] = text[i] ^ round}
text
end
in ruby 1.9.x it throws an error - undefined method ^' for "\x1A":String__
is there any work around in ruby 1.9.x?
after googling i came to know that "In Ruby 1.9 string[index] returns character not code of the character (like it was in 1.8)." (https://rails.lighthouseapp.com/projects/8994/tickets/3144-undefined-method-for-string-ror-234)
Thanks in advance
Try text[i].ord ^ round. See Getting an ASCII character code in Ruby using `?` (question mark) fails for more info.
After upgrading to ruby 1.9 we began to notice pages failing to render from the rails template renderer when a user used a non-ASCII character. Specifically "é". I was able to resolve this issue on one of our staging servers, but I have not been able to reproduce the fix on our production server.
The fix that seemed to work the first time:
Converted the database from latin1 to utf8 using the convert_charset tool available here: http://www.mysqlperformanceblog.com/2009/03/17/converting-character-sets/. (including setting default_character_set=utf8 in my.cnf and running SET GLOBAL character_set_server=utf8
Switched to the sam-mysql-ruby adapter (instead of the standard mysql adapter: http://gemcutter.org/gems/sam-mysql-ruby)
Restarted rails
The error is:
"invalid byte sequence in US-ASCII"
Oddly, after following the steps above the error has not changed on our production server. Setting encoding: utf8 in database.yml does not change the error either.
The error raised on the following line of code:
<%= link_to h(question.title), question_path(question) %>
This blog seems to suggest a fix, but it mentions that this should not be a problem in 1.9: http://www.igvita.com/2007/04/11/secure-utf-8-input-in-rails/ (and it's over 2 years old).
I imagine this problem might soon affect a lot of people as more rails developers people switch to 1.9.
I found the solution:
The problem is:
Fetching data from any database (Mysql, Postgresql, Sqlite2 & 3), all configured to have UTF-8 as it's character set, returns the data with ASCII-8BIT in ruby 1.9.1 and rails 2.3.2.1.
(Taken from: https://rails.lighthouseapp.com/projects/8994/tickets/2476)
My attempt to use the patched mysql adapter likely failed because my database was not configured to natively use utf8, so the patched adapter failed to work properly.
The fix ended up being to use the patch file available here: http://gnuu.org/2009/11/06/ruby19-rails-mysql-utf8/
require 'mysql'
class Mysql::Result
def encode(value, encoding = "utf-8")
String === value ? value.force_encoding(encoding) : value
end
def each_utf8(&block)
each_orig do |row|
yield row.map {|col| encode(col) }
end
end
alias each_orig each
alias each each_utf8
def each_hash_utf8(&block)
each_hash_orig do |row|
row.each {|k, v| row[k] = encode(v) }
yield(row)
end
end
alias each_hash_orig each_hash
alias each_hash each_hash_utf8
end
(Placed in lib/mysql_utf8fix.rb and required in enviornment.rb using require 'lib/mysql_utf8fix.rb')
it is only require 'mysql_utf8fix.rb' (rails 2.3.11)
Please user mysql2(gem) adapter instead of mysql adapter in database.yml
and remove the mysql patches(If exists) and add the following lines in environment.rb.
Encoding.default_external = Encoding::UTF_8
Encoding.default_internal = Encoding::UTF_8
Then run in apache and passenger it ll work fine
Thanks,
Ramanavel Selvaraju.