Rails decode json with \u0022 character fail - ruby-on-rails

Question

I would like to decode a json string with a \u0022 character in it.
I am successful with:
>> ActiveSupport::JSON.decode("{\"json\":{\"difficulty\":1}}")
=> {"json"=>{"difficulty"=>1}}
But fail with:
>> ActiveSupport::JSON.decode("{\"json\":{\"difficulty\":\"test\\u0022test\"}}")
StandardError: Invalid JSON string
from /home/.../.rbenv/versions/1.8.7-p358/lib/ruby/gems/1.8/gems/activesupport-2.3.15/lib/active_support/json/backends/yaml.rb:14:in `decode'
from /home/.../.rbenv/versions/1.8.7-p358/lib/ruby/gems/1.8/gems/activesupport-2.3.15/lib/active_support/json/decoding.rb:14:in `__send__'
from /home/.../.rbenv/versions/1.8.7-p358/lib/ruby/gems/1.8/gems/activesupport-2.3.15/lib/active_support/json/decoding.rb:14:in `decode'
from (irb):11
I would love to replace the \u0022 character with another one, but I can't do it because it's inside the parameters parsing process of rails when the app receives a request; except if I override the json decode chore method, which I would prefer to avoid.
FYI : I'm on Ruby 1.8.7-p358 & Rails 2.3.15 & I can't change that.

Answer 1

I think this has to do with Syck, the old YAML library used in (very) old versions of Ruby. What does this have to do with YAML? In ActiveSupport 2.3.15, JSON decoding uses YAML.load to parse JSON, because JSON happens to be a subset of YAML.
A quick aside: In ActiveSupport 2.3.18, ActiveSupport::JSON::Backends::Yaml no longer uses the YAML backend decode YAML, and ActiveSupport::JSON::Backends::Yaml#decode looks like this:
def decode(json)
raise "The Yaml backend has been deprecated due to security risks, you should set ActiveSupport::JSON.backend = 'OkJson'"
end
There are very serious consequences to using outdated versions of Ruby and Rails. Upgrade to versions that are still supported by security releases, or you will be sorry. Seriously.
Anyway, let's take a closer look at your data. As I suspect you know, the UTF-8 character U+0022 is a double-quote. All of the escaping makes things confusing, so let's see what string we actually have:
str = "{\"json\":{\"difficulty\":\"test\\u0022test\"}}"
puts str
# => {"json":{"difficulty":"test\u0022test"}}
FWIW it's more readable to use one of Ruby's alternate string syntaxes here, e.g.:
str = %Q[{"json":{"difficulty":"test\\u0022test"}}]
puts str
# => {"json":{"difficulty":"test\u0022test"}}
Either way, that's a literal \ followed by u0022. So far so good. Now let's try to parse it with Syck (I'm using Ruby 2.2, but it doesn't make a difference in this case):
require "syck"
str = %Q[{"json":{"difficulty":"test\\u0022test"}}]
Syck.load(str)
# => .../syck.rb:145:in `load': syntax error on line 0, col 38: `# Transforms the subclass name found i}}' (ArgumentError)
I don't know why Syck's error messages are so weird, but anyway we can see that Syck falls over trying to parse this JSON. Now let's try it with a modern YAML parser:
require "psych"
str = %Q[{"json":{"difficulty":"test\\u0022test"}}]
Psych.load(str)
# => {"json":{"difficulty":"test\u0022test"}}
Works great!
So you have a few options.
You could upgrade to a version of Ruby that's still maintained, doesn't have security holes, and has a YAML parser that isn't broken.
You could upgrade to a version of Rails that's still maintained, doesn't have security holes, and doesn't use a broken YAML parser to parse JSON.
You could install Psych, which is available as a gem for older versions of Ruby.