ActiveSupport::JSON.decode does not properly handle literal line breaks - ruby-on-rails

Is this the expected behavior? Note how the line break character gets lost.
ruby-1.9.2-p136 :001 > ActiveSupport::JSON.decode("{\"content\": \"active\n\nsupport\"}")
=> {"content"=>"active\nsupport"}
The same happens with unicode-escaped line breaks:
ruby-1.9.2-p136 :002 > ActiveSupport::JSON.decode("{\"content\": \"active\u000a\u000asupport\"}")
=> {"content"=>"active\nsupport"}
I'm using rails 3.0.3.

I eventually came across this ticket: https://rails.lighthouseapp.com/projects/8994/tickets/3479-activesupportjson-fails-to-decode-unicode-escaped-newline-and-literal-newlines
It seems this is a bug in ActiveSupport that will be fixed in Rails 3.0.5. For now I have patched activesupport and things are working as expected.
ruby-1.9.2-p136 :001 > ActiveSupport::JSON.decode("{\"content\": \"active\n\nsupport\"}")
=> {"content"=>"active\n\nsupport"}

To represent a newline in JSON data using double quotes, you must escape the newline:
ActiveSupport::JSON.decode("{\"content\": \"active\\n\\nsupport\"}")
Otherwise, you are inserting a newline into the JSON source and not the JSON data. Note that this would also work:
ActiveSupport::JSON.decode('{"content": "active\n\nsupport"}')
By using single quotes, you are no longer inserting a literal newline into the source of the JSON.
It is interesting to note the way ActiveSupport handles this by default (the default JSON backend is ActiveSupport::JSON::Backends::Yaml). By installing the json gem and changing the JSON backend to it (ActiveSupport::JSON.backend = 'JSONGem') and attempting to decode the same text (ActiveSupport::JSON.decode("{\"content\": \"active\\n\\nsupport\"}")) you get the following:
JSON::ParserError: 737: unexpected token at '{"content": "active
support"}'

Related

Rails loading ActiveSupport regex with ISO-8859-1 encoding instead of UTF-8

When I call '返回'.titleize on a Chinese-language string in the Rails 6 app on my server, I get an error:
Encoding::CompatibilityError: incompatible encoding regexp match (ISO-8859-1 regexp with UTF-8 string)
The source for the titleize function leads to the following in ActiveSupport::Inflector:
def titleize(word, keep_id_suffix: false)
humanize(underscore(word), keep_id_suffix: keep_id_suffix).gsub(/\b(?<!\w['’`()])[a-z]/) do |match|
match.capitalize
end
end
Although calling ActiveSupport::Inflector.titleize('返回') gives the error above, if I just copy the function body and run it as follows, there's no error --- I just get the correct titleize behaviour:
ActiveSupport::Inflector.humanize(ActiveSupport::Inflector.underscore('返回'), keep_id_suffix: false).gsub(/\b(?<!\w['’`()])[a-z]/) do |match|
match.capitalize
end
I guess that the regular expression /\b(?<!\w['’()])[a-z]/ is getting compiled with a ISO-8859-1 encoding when ActiveSupport::Inflector loads but is compiling with a different encoding when I run it in my Rails console, but I don't see anything I can do with this information.
What can I do to get the Rails helper titleize to work on my server as intended?
I'm on Rails 6, Ruby 2.6.6, and ENV["LANG"] is en_US.utf8.

Rails 4.2 syntax error, unexpected ':', expecting =>

I have two computers that I mainly use to develop my Rails application. While working on Computer 1, I added some bootstrap elements to some inputs. For example:
= f.select :transport_from_state, options_for_select(state_populator, #invoice_ambulance.transport_from_state), { include_blank: true}, { class: 'chosen-select', 'data-placeholder': 'State' }
I added the 'data-placeholder': 'State' and used the 'newer' syntax instead of the old :data-placeholder' => 'State' which works fine. The page works with no errors on Computer 1.
I pulled down on computer 2, and now I am getting an error for every instance of 'data-placeholder'. Here is my error:
syntax error, unexpected ':', expecting =>
...en-select', 'data-placeholder': 'State' }
I can replace it with the old syntax and it works fine. However, I shouldn't have to switch 100 instances of this to a deprecated syntax. I have since bundle installed, bundle updated, and rebuilt the db with no luck.
Computer 1 (works)
ruby 2.2.0p0
Rails 4.2.0
Computer 2 (doesnt work)
ruby 2.2.0preview1
Rails 4.2.0
You need to upgrade Computer 2 to the real Ruby 2.2.0 rather than this beta-ish "preview" version you have. Using quoted symbols with the JavaScript-style trailing colon syntax:
{ 'some string': value }
wasn't valid before Ruby 2.2, the 2.2.0preview1 version you have on Computer 2 apparently doesn't support it.
BTW, there is no old and new syntax, there is an alternate JavaScript-style notation that can be use when the keys in a Hash-literal are some symbols. Whoever told you that the hashrocket is deprecated is, at best, confused.
The "newer" syntax is only for symbols.
{hello: 'world'} is equivalent to {:hello => 'world'} but if your key is a string then you still have to use the "hash rocket" syntax: {'hello' => 'world'}
http://ruby-doc.org/core-2.2.0/Hash.html

Ruby / Sinatra JSON.parse is failing

the String that I'm trying to parse is:
{\"user_name\":\"test#test.com\",\"pass\":\"bla\"}
[3] pry(#<Flockers::WebApp>)> JSON.parse(request.body.read)
JSON::ParserError: A JSON text must at least contain two octets!
from c:/WebTools/Ruby193/lib/ruby/gems/1.9.1/gems/json_pure-1.8.1/lib/json/commo
n.rb:155:in `initialize'
When I execute JSON.parse in javascript, this works, but this is not parsing properly in ruby console.
This error can be caused by passing JSON.parse an empty string. Try running request.body.rewind before request.body.read.

JSON encoding/decoding with unicode in rails

I upgraded downgraded to rails 2.3.17 due to the security bugs, but now I can't decode json strings that I have saved down to a DB if they have unicode in them :(. Is there a way to process the string such that it decodes properly?
e = ActiveSupport::JSON.encode({'a' => "Hello Unicode \u2019"})
ActiveSupport::JSON.decode(e)
gives me
RangeError: 8217 out of char range
from /app/vendor/bundle/ruby/1.9.1/gems/activesupport-2.3.17/lib/active_support/json/backends/okjson.rb:314:in `unquote'
from /app/vendor/bundle/ruby/1.9.1/gems/activesupport-2.3.17/lib/active_support/json/backends/okjson.rb:251:in `strtok'
from /app/vendor/bundle/ruby/1.9.1/gems/activesupport-2.3.17/lib/active_support/json/backends/okjson.rb:215:in `tok'
from /app/vendor/bundle/ruby/1.9.1/gems/activesupport-2.3.17/lib/active_support/json/backends/okjson.rb:178:in `lex'
from /app/vendor/bundle/ruby/1.9.1/gems/activesupport-2.3.17/lib/active_support/json/backends/okjson.rb:46:in `decode'
from /app/vendor/bundle/ruby/1.9.1/gems/activesupport-2.3.17/lib/active_support/json/backends/okjson.rb:612:in `decode'
from /app/vendor/bundle/ruby/1.9.1/gems/activesupport-2.3.17/lib/active_support/json/decoding.rb:14:in `decode'
from (irb):30
from /usr/local/bin/irb:12:in `<main>'
I can't change the first line since it's coming from the DB like that.
This used to work.
You can change the backend JSON provider in ActiveSupport.
Add ActiveSupport::JSON.backend = "JSONGem" into an application initialiser (I added it to application.rb). This fixed the unicode parsing issues I had after I upgraded activesupport to 3.0.20.
See the vulnerability notice which caused this update - It mentions that this workaround should apply to 2.3.16 as well.
From rails console:
> ActiveSupport::VERSION::STRING
=> "3.0.20"
> ActiveSupport::JSON.decode('{"test":"string\u2019"}')
RangeError: 8217 out of char range
> ActiveSupport::JSON.backend = "JSONGem"
> ActiveSupport::JSON.decode('{"test":"string\u2019"}')
 => {"test"=>"string’"}
The JSON gem will handle this correctly.
As a note, the gem is much more strict than the other JSON parsers out there. For example:
{ 'test' : 'value' }
This is not valid JSON even though it looks okay.
For whatever reason the non-UTF-8 savvy JSON parser shipped as part of the 2.3.16 patch which is really sloppy on the part of the maintainer.
Switch to 2.3.15 which should be fine because that's when the fixes landed.
Curse the developer who started this project in rails
Begin work on porting to python post haste

iconv deprecation warning with ruby 1.9.3

I'm getting this warning when I run rspec:
/gems/activesupport-3.1.0/lib/active_support/dependencies.rb:240:in `block in require': iconv will be deprecated in the future, use String#encode instead.
I get the same warning with rails 3.1.0, 3.1.1, 3.1.2.rc2 versions. Seems it's related to sqlite3 gem, but I'm not sure. There are no warnings with ruby 1.9.2
Any suggestions how to deal with it?
You are getting this deprecation notice cause a library somewhere is requiring iconv.
iconv is a gem created by Matz that can be used to convert strings from one format to another.
For example this is often used:
Iconv.iconv('UTF-8//IGNORE', 'UTF-8', content) this little bit of magic takes a UTF-8 string that may have invalid chars and converts it to a proper UTF-8 string.
It has been decided that in Ruby 1.9.3 we should not be using iconv any more and instead use the built-in String#encode. encode is more powerful and allows you more flexibility.
The theory is that the above example could be replaced with:
string.encode("UTF-8", :invalid => :replace, :undef => :replace, :replace => "?")
In practice it seems this is imperfect.
This also leads to a less than easy story for gem creators who wish to support 1.8:
content = RUBY_VERSION.to_f < 1.9 ?
Iconv.iconv('UTF-8//IGNORE', 'UTF-8', "content") :
"#{content}".encode(Encoding::UTF_8, :invalid => :replace, :undef => :replace, :replace => '')
So, you have a gem somewhere that is requiring iconv, to find it:
Assuming your error message is: /gems/activesupport-3.1.0/lib/active_support/dependencies.rb:240
Open up /gems/activesupport-3.1.0/lib/active_support/dependencies.rb on line 240:
Add the line:
p caller if file =~ /iconv/
(just after: load_dependency(file) { result = super })
You will get a big fat stack trace:
rake --tasks
/home/sam/.rvm/gems/ruby-1.9.3-p125/gems/activesupport-3.2.6/lib/active_support/dependencies.rb:251:in `block in require': iconv will be deprecated in the future, use String#encode instead.
["/home/sam/.rvm/gems/ruby-1.9.3-p125/gems/calais-0.0.13/lib/calais.rb:5:in `'",
.. more omitted ..
This tells me it is the calais gem. Looking through pull requests, I am not the first. The pull has not been yanked in.
Depending on the gem, there may be an upgraded version that does not have this error, so I would recommend you upgrade your gems first. If you are unlucky you may be stuck with the unfortunate task of forking a gem to get rid of this (if for example your pull request to fix it languishes)
If you're seeing this, it's very probably not Rails. If you look at the method surrounding the line being referred to in the error you posted, you'll see the following:
def require(file, *)
result = false
load_dependency(file) { result = super }
result
end
I'm not saying it's your code, necessarily, but I'm certain that it's not actually the line in question where iconv is being called. In my case, I found that my project's code actually contained a reference to iconv.
If you want to check your code for such a reference, try grep -ir iconv ./ in your project directory.
When iconv is actually in a library it can be harder to find. By temporarily changing the above method to:
def require(file, *)
result = false
puts
puts caller.reverse
load_dependency(file) { result = super }
result
end
You can then easily run your code and grep out the relevant lines of the backtrace to find the root cause of the warning.
ruby your/code.rb 2>&1 | grep -B 5 iconv
Add this to the start of your program:
oldverb = $VERBOSE; $VERBOSE = nil
require 'iconv'
$VERBOSE = oldverb
and curse the people who think this is a professional way to handle deprecation.
You can pin down the exact location of the warning by generating exceptions for ActiveSupport::Deprecation, instead of just printing to the log. At the top of application.rb:
ActiveSupport::Deprecation.behavior = Proc.new do |message, backtrace|
raise message
end
Once you've figured out where the warning is coming from (by inspecting the full backtrace), remove this again.
To remove this warning...
go to your .rvm directory and find iconv.c (mine was at ~/.rvm/src/ruby-1.9.3-p125/ext/iconv/iconv.c)
edit that file are remove or comment out the call to warn_deprecated() (should be near the bottom)
from that file's directory, run ruby extconf.rb
then make
then make install
Should do the trick

Resources