Rails 3:
{"a" => "<br/>"}.to_json
=> "{\"a\":\"<br/>\"}"
Rails 4:
{"a" => "<br/>"}.to_json
=> "{\"a\":\"\\u003Cbr/\\u003E\"}"
WHY???
It appears to be causing the error
Encoding::UndefinedConversionError: "\xC3" from ASCII-8BIT to UTF-8
When my Rails 3 app tries to parse JSON generated by my rails 4 app.
WHY???
To defend against a common weakness in web applications. If you say in an HTML page eg:
<script type="text/javascript">
var something = <%= #something.to_json.html_safe %>;
</script>
then you might think you're fine because you've JSON-escaped the data you're injecting into JavaScript. But actually you're not safe: aside from JSON syntax you also have surrounding HTML syntax, and in an HTML script block </ is in-band signalling. Practically, if #something contains the string </script> you've got a cross-site scripting vulnerability as this comes out:
<script type="text/javascript">
var something = {"attack": "abc</script><script>alert('XSS');//"};
</script>
The first script block ends halfway through the string (leaving an unclosed string literal syntax error) and the second <script> is treated as a new script block and the potentially-user-submitted content within it executed.
Escaping the < character to \u003C is not required by JSON but it is a perfectly valid alternative and it automatically avoids this class of problems. If a JSON parser rejects it, that is a severe bug in the reader.
What is the code that is producing that error? I'm not convinced the error is anything to do with the <-escaping, as it is talking about byte 0xC3 rather than 0x3C. That could be indicative of a string with UTF-8 encoded content not having been marked as UTF-8... maybe you need a force_encoding("UTF-8") on the input?
You can retain the original string with JSON::dump:
JSON::dump "a" => "<br/>"
=> "{\"a\":\"<br/>\"}"
JSON::dump "a" => "x&y"
=> {\"a\":\"x&y\"}" # instead of x\u0026y
Use it with care for the reasons bobince mentions and particularly avoid it with any user-generated input (or at least make sure that's sanitized).
Here's an example I encountered where it's a legitimate use. Generating a JavaScript hash argument in a helper function:
# application_helper.rb
def widget_js(post)
options = {
color: ColorCalculator(post.color).to_rgb_hex,
...
}
"third_party_widget(#{JSON::dump options});"
end
I encountered this issue too and as others have mentioned, it's caused by using the ActiveSupport to_json method. To resolve, use the JSON gem directly with JSON.generate(data) where data is an Array or Hash. See https://github.com/flori/json for all JSON gem documentation.
Was having a similar problem with Rails 7 sending "<" in JSON output like:
..., "legend":[{"text":"<96.8%","color":"#FFAFFF"},{"text":"96.8% to 98.8%","color":"#E37DE3"},{"text":"98.8% to 100%","color":"#BA50BA"}], ...
from something like:
{entry: dataset.entry, legend: dataset.legend, ...
The "<" sign was showing up "legend":[{"text":"\u003c96.8%", ...
In my case `JSON.generate({entry: ...})` fixed the issue
Related
I need to output some JSON for a customer in a somewhat unusual format. My app is written with Rails 5.
Desired JSON:
{
"key": "\/Date(0000000000000)\/"
}
The timestamp value needs to have a \/ at both the start and end of the string. As far as I can tell, this seems to be a format commonly used in .NET services. I'm stuck trying to get the slashes to output correctly.
I reduced the problem to a vanilla Rails 5 application with a single controller action. All the permutations of escapes I can think of have failed so far.
def index
render json: {
a: '\/Date(0000000000000)\/',
b: "\/Date(0000000000000)\/",
c: '\\/Date(0000000000000)\\/',
d: "\\/Date(0000000000000)\\/"
}
end
Which outputs the following:
{
"a": "\\/Date(0000000000000)\\/",
"b": "/Date(0000000000000)/",
"c": "\\/Date(0000000000000)\\/",
"d": "\\/Date(0000000000000)\\/"
}
For the sake of discussion, assume that the format cannot be changed since it is controlled by a third party.
I have uploaded a test app to Github to demonstrate the problem. https://github.com/gregawoods/test_app_ignore_me
After some brainstorming with coworkers (thanks #TheZanke), we came upon a solution that works with the native Rails JSON output.
WARNING: This code overrides some core behavior in ActiveSupport. Use at your own risk, and apply judicious unit testing!
We tracked this down to the JSON encoding in ActiveSupport. All strings eventually are encoded via the ActiveSupport::JSON.encode. We needed to find a way to short circuit that logic and simply return the unencoded string.
First we extended the EscapedString#to_json method found here.
module EscapedStringExtension
def to_json(*)
if starts_with?('noencode:')
"\"#{self}\"".gsub('noencode:', '')
else
super
end
end
end
module ActiveSupport::JSON::Encoding
class JSONGemEncoder
class EscapedString
prepend EscapedStringExtension
end
end
end
Then in the controller we add a noencode: flag to the json hash. This tells our version of to_json not to do any additional encoding.
def index
render json: {
a: '\/Date(0000000000000)\/',
b: 'noencode:\/Date(0000000000000)\/',
}
end
The rendered output shows that b gives us what we want, while a preserves the standard behavior.
$ curl http://localhost:3000/sales/index.json
{"a":"\\/Date(0000000000000)\\/","b":"\/Date(0000000000000)\/"}
Meditate on this:
Ruby treats forward-slashes the same in double-quoted and single-quoted strings.
"/" # => "/"
'/' # => "/"
In a double-quoted string "\/" means \ is escaping the following character. Because / doesn't have an escaped equivalent it results in a single forward-slash:
"\/" # => "/"
In a single-quoted string in all cases but one it means there's a back-slash followed by the literal value of the character. That single case is when you want to represent a backslash itself:
'\/' # => "\\/"
"\\/" # => "\\/"
'\\/' # => "\\/"
Learning this is one of the most confusing parts about dealing with strings in languages, and this isn't restricted to Ruby, it's something from the early days of programming.
Knowing the above:
require 'json'
puts JSON[{ "key": "\/value\/" }]
puts JSON[{ "key": '/value/' }]
puts JSON[{ "key": '\/value\/' }]
# >> {"key":"/value/"}
# >> {"key":"/value/"}
# >> {"key":"\\/value\\/"}
you should be able to make more sense of what you're seeing in your results and in the JSON output above.
I think the rules for this were originally created for C, so "Escape sequences in C" might help.
Hi I think this is the simplest way
.gsub("/",'//').gsub('\/','')
for input {:key=>"\\/Date(0000000000000)\\/"} (printed)
first gsub will do{"key":"\\//Date(0000000000000)\\//"}
second will get you
{"key":"\/Date(0000000000000)\/"}
as you needed
I'm using Test::Unit in an old Rails app (3.2.22) and I'm trying to test a service class that hits an external api.
I'm using webmock and trying to get a json file fixture working, but I keep getting parsing errors from the json file.
My test stub looks like this:
response_data = fixture_file_upload('easypost/order_response.json')
stub_request(:post, 'https://api.easypost.com/v2/orders').
to_return(:status => 200, :body => File.read(response_data))
My order_response.json file looks like this:
{
'mode':'test',
'reference':'Order',
'is_return':false,
'options':{'currency':'USD','label_date':null}
}
When I run the tests, I get a parse error:
JSON::ParserError: 757: unexpected token at '{'mode':'test','reference':'Order','is_return':false,'options':{'currency':'USD','label_date':null}}'
What is going on?
UPDATE:
Got it working by using double quotes in the JSON file:
{
"mode":"test",
"reference":"Order",
"is_return":false,
"options":{"currency":"USD","label_date":null}
}
Could anyone explain why this is necessary?
From the fine specification:
A string is a sequence of zero or more Unicode characters, wrapped in double quotes, using backslash escapes.
and and object is a set of key/value pairs where the keys are strings:
That means that this:
{
'mode':'test',
'reference':'Order',
'is_return':false,
'options':{'currency':'USD','label_date':null}
}
is not JSON because JSON strings use double quotes and only double quotes, that's just something that sort of looks like JSON. When you switch to double quotes for your strings:
{
"mode":"test",
"reference":"Order",
"is_return":false,
"options":{"currency":"USD","label_date":null}
}
then you have JSON and everything should work.
This is to make it simpler and to avoid having to have another escape method for javascript reserved keywords
I think this post might be helpful in understanding the need for quotations (and also where I found the above quote): JSON Spec - does the key have to be surrounded with quotes?
the current hash is
{\"report_name\"=>\"Study/Control: ABIRATERONE ACETATE - 20151413355\", \"left_mue_start_date\"=>\"02-26-2015\", \"left_mue_end_date\"=>\"03-19-2015\", \"right_mue_start_date\"=>\"02-26-2015\", \"right_mue_end_date\"=>\"03-19-2015\", \"report_formulary_id\"=>\",7581\", \"mue\"=>\"true\", \"mue_type\"=>\"study/control\", \"chain_id\"=>\"1\", \"left_mue_formulary_ids\"=>[\"7581\"], \"action\"=>\"create_report\", \"controller\"=>\"informatics\", \"user_id\"=>339}
now I need to convert it in proper hash like
{"report_name" => "Study/Control: ABIRATERONE ACETATE - 20151413355"}
so I am trying to get it with JSON.parse but I am getting error like:
JSON::ParserError: 757: unexpected token at '{
So if someone know about that so please help me.
and I am using Rails 3.2
What you have is a hash printed as String. To convert it into a Hash use eval.
ch = "{\"report_name\"=>\"Study/Control: ABIRATERONE ACETATE - 20151413355\", \"left_mue_start_date\"=>\"02-26-2015\", \"left_mue_end_date\"=>\"03-19-2015\", \"right_mue_start_date\"=>\"02-26-2015\", \"right_mue_end_date\"=>\"03-19-2015\", \"report_formulary_id\"=>\",7581\", \"mue\"=>\"true\", \"mue_type\"=>\"study/control\", \"chain_id\"=>\"1\", \"left_mue_formulary_ids\"=>[\"7581\"], \"action\"=>\"create_report\", \"controller\"=>\"informatics\", \"user_id\"=>339}"
hash = eval(ch)
# => {"report_name"=>"Study/Control: ABIRATERONE ACETATE - 20151413355", "left_mue_start_date"=>"02-26-2015", "left_mue_end_date"=>"03-19-2015", "right_mue_start_date"=>"02-26-2015", "right_mue_end_date"=>"03-19-2015", "report_formulary_id"=>",7581", "mue"=>"true", "mue_type"=>"study/control", "chain_id"=>"1", "left_mue_formulary_ids"=>["7581"], "action"=>"create_report", "controller"=>"informatics", "user_id"=>339}
PS: A JSON string should look as follows, meaning what you have is not JSON and hence you got JSON::ParserError for using JSON.parse on a non-JSON string :
"{\"report_name\":\"Study/Control: ABIRATERONE ACETATE - 20151413355\",\"left_mue_start_date\":\"02-26-2015\",\"left_mue_end_date\":\"03-19-2015\",\"right_mue_start_date\":\"02-26-2015\",\"right_mue_end_date\":\"03-19-2015\",\"report_formulary_id\":\",7581\",\"mue\":\"true\",\"mue_type\":\"study/control\",\"chain_id\":\"1\",\"left_mue_formulary_ids\":[\"7581\"],\"action\":\"create_report\",\"controller\":\"informatics\",\"user_id\":339}"
To avoid using eval you could use JSON.parse ch.gsub('=>', ':') this way you will get a HASH from your HASH stored as STRING
Last time when I got this issue, since the json file that I got from a API that contains a BOM
UTF-8 BOM is a sequence of bytes (EF BB BF)
What's different between UTF-8 and UTF-8 without BOM?
at the beginning of that string, but you know that part wouldn't display or as readable when we got the string from response. I try to use Ruby JSON to parse it, but I failed, I got the same exception with yours. just a reminder for others, when you get a Json response. By the way, this would be no problem while you are handling that in javascript, but with problems in Python or Ruby languages.
I ran into a similar problem, though it was failing while parsing \"\". This is in regards to using Pact.IO. I mention it here since this is the highest ranked Google result while looking for the error I encountered. The solution in my case was to change the body of a POST in my C# application so that it wasn't using empty string, but a null string. Basically I added this before my HTTP call.
if (request.Method == HttpMethod.Post && request.Content!=null && request.Content.GetType()==typeof(StringContent) && request.Content.ReadAsStringAsync().Result == String.Empty)
request.Content = new StringContent(null);
So HAML 4 includes a coffeescript filter, which allows us coffee-loving rails people to do neat things like this:
- word = "Awesome."
:coffeescript
$ ->
alert "No semicolons! #{word}"
My question: For the end user, is this slower than using the equivalent :javascript filter? Does using the coffeescript filter mean the coffeescript will be compiled to javascript on every page load (which would obviously be a performance disaster), or does this only happen once when the application is started?
It depends.
When Haml compiles a filter it checks to see if the filter text contains any interpolation (#{...}). If there isn’t any then it will be the same text to transform on each request, so the conversion is done once at compile time and the result included in the template.
If there is interpolation in the filter text, then the actual text to transform will vary on each request, so the Coffeescript will need to be compiled each time.
Here’s an example. First with no interpolation:
:coffeescript
$ ->
alert "No semicolons! Awesome"
This generates the code (use haml -d to see the generated Ruby code):
_hamlout.buffer << "<script>\n (function() {\n $(function() {\n return alert(\"No semicolons! Awesome\");\n });\n \n }).call(this);\n</script>\n";
This code simply adds a string to the buffer, so no Coffeescript is being recompiled.
Now with interpolation:
- word = "Awesome."
:coffeescript
$ ->
alert "No semicolons! #{word}"
This generates:
word = "Awesome."
_hamlout.buffer << "#{
find_and_preserve(Haml::Filters::Coffee.render_with_options(
"$ ->
alert \"No semicolons! #{word}\"\n", _hamlout.options))
}\n";
Here, since Haml needs to wait to see what the value of the interpolation is, the Coffeescript is recompiled each time.
You can avoid compiling the Coffeescript on each request by not having any interpolation inside your :coffeescript filters.
The :javascript filter behaves similarly, checking to see if there is any interpolation, but since the :javascript filter only outputs some text to the buffer when it runs there is much less of a performance hit using it. You could possibly combine :javascript and :coffeescript filters, putting interpolated data in :javascript and keeping your :coffeescript static:
- word = "Awesome"
:javascript
var message = "No semicolons! #{word}";
:coffeescript
alert message
matt's answer is clear on what is going on. I made a helper to add locals to :coffeescript filters from a hash. This way you don't need to use global JavaScript variables. As a side note: on Linux, the slowdown is really negligible. On Windows however, the impact on performance is quite important (easily more than 100ms per block to compile).
module HamlHelper
def coffee_with_locals locals={}, &block
block_content = capture_haml do
block.call
end
return block_content if locals.blank?
javascript_locals = "\nvar "
javascript_locals << locals.map{ |key, value| j(key.to_s) + ' = ' + value.to_json.gsub('</', '<\/') }.join(",\n ")
javascript_locals << ";\n"
content_node = Nokogiri::HTML::DocumentFragment.parse(block_content)
content_node.search('script').each do |script_tag|
# This will match the '(function() {' at the start of coffeescript's compiled code
split_coffee = script_tag.content.partition(/\(\s*function\s*\(\s*\)\s*\{/)
script_tag.content = split_coffee[0] + split_coffee[1] + javascript_locals + split_coffee[2]
end
content_node.to_s.html_safe
end
end
It allows you to do the following:
= coffee_with_locals "test" => "hello ", :something => ["monde", "mundo", "world"], :signs => {:interogation => "?", :exclamation => "!"} do
:coffeescript
alert(test + something[2] + signs['exclamation'])
Since there is no interpollation, the code is actually compiled as normal.
I am working on very complex and dynamic form where I do lot of calls to various methods to render partials depending upon values chosen for drop downs using jQuery. Problem is that after filling the form if it fails validation the form loses all the filled in values upon re-load. I got around this by sending some specific values from params{} hash to methods for my partials on re-load. But it is very cumbersome and I have large number of elements in params hash. How can send the whole params{} to another method in the same controller using jQuery?
Ok I tried this in my form:
$.post("/collections/show_selected_media_fields",{media_type: $("#collections_controller_ev0_media_id option:selected").text(), parent_form_action: "<%=params[:action]%>",ev0_manufacturer_id:"<%=params[:collections_controller_ev0].inspect%>", }, function(data) {$("#show_selected_media_fields").html(data);});
It produces following string sent as parameter:
Parameters: {"ev0_manufacturer_id"=>"{"client_asset_id"=>"", "status_id"=>"6", "server_
name"=>"", "media_id"=>"11", "serial_number"=>"", "evidence_number"
;=>"qwe", "notes"=>"", "model"=>"", "manufacturer_id"=>"69"
;, "interface"=>"SATA", "obtained_from"=>"wr", "evidence_type_id"=>"1"}
", "media_type"=>"Server", "parent_form_action"=>"quick_save"}
how can I convert this raw string to a hash in controller?
{"client_asset_id"=>
needs to be converted to
{"client_asset_id"=>"",... etc}
===========
ok I tried Tom's method. That produces the params as a string in the following shape. I tried to convert it into hash by doing an eval on it. But it errors out.
{commit=>Save, ev1_current_location_id=>, collections_controller_ev1=>{file_system=>NTFS, obtained_from=>, evidence_number=>, interface=>SAT
A, size_unit=>GB, manufacturer_id=>, encryption_version=>, media_id=>3, size=>, evidence_type_id=>3, other_encryption=>, encryption_method=>
N/A, serial_number=>, model=>, encryption_key=>}, ev0_from_location_category=>, ev0_obtained_from_email_id=>, collections_controller_ev0=>{o
btained_from=>, evidence_number=>, media_id=>1, evidence_type_id=>1, status_id=>6}, custody_action=>Create, collection=>{acquired_by=>Amande
ep Singh, custodian_id=>12, matter_id=>58, location=>sa Nose, client_id=>11, software_version=>, collection_date_time=>Fri Nov 30 14:28:06 -
0800 2012, acquisition_method=>Direct Collection, notes=>, software_id=>1}, _method=>put, utf8=>Γ£ô, ev1_current_location_category=>, ev0_cu
rrent_location_category=>, add_working_copy=>No, ev1_obtained_from_email_id=>, authenticity_token=>3vyn6057DDIyfgTnbckeh5heRTIgcVBfxtY89Krfr
/c=, ev1_existing_artifact_type=>, ev0_from_location_id=>, action=>quick_save, ev1_from_location_id=>, ev1_from_location_category=>, ev0_cur
rent_location_id=>, controller=>collections}
using .to_json and HTMLEntities gem I have gotten to a point where I have the following string. I need to convert it back to params hash. How to?
{"utf8"=>"Γ£ô","collection"=>{"acquired_by"=>"Amandeep Singh","notes"=>"","matter_id"=>"58","software_id"=>"1","acquisition_method"=>"Direct
Collection","client_id"=>"11","custodian_id"=>"0","collection_date_time"=>"Fri Nov 30 15=>52=>12 -0800 2012","software_version"=>"","locati
on"=>"sa Nose"},"ev0_current_location_id"=>"","ev1_existing_artifact_type"=>"","ev1_obtained_from_email_id"=>"","custody_action"=>"Create","
action"=>"quick_save","ev0_current_location_category"=>"","ev1_from_location_category"=>"","_method"=>"put","ev0_obtained_from_email_id"=>""
,"ev0_from_location_category"=>"","ev1_current_location_category"=>"","commit"=>"Save","controller"=>"collections","authenticity_token"=>"3v
yn6057DDIyfgTnbckeh5heRTIgcVBfxtY89Krfr/c=","ev0_from_location_id"=>"","ev1_current_location_id"=>"","collections_controller_ev0"=>{"status_
id"=>"6","media_id"=>"9","obtained_from"=>"","evidence_number"=>"","evidence_type_id"=>"1"},"collections_controller_ev1"=>{"media_id"=>"3","
encryption_key"=>"","encryption_version"=>"","size"=>"","obtained_from"=>"","encryption_method"=>"N/A","model"=>"","evidence_number"=>"","ev
idence_type_id"=>"3","size_unit"=>"GB","other_encryption"=>"","interface"=>"SATA","file_system"=>"NTFS","serial_number"=>"","manufacturer_id
"=>""},"ev1_from_location_id"=>"","add_working_copy"=>"No"}
========== edit==========
I am using the following in my view.
JSON( params[:collections_controller_ev0])
It sends following quoted string to my controller.
{"evidence_number":"","notes":"","size":"","model":"","evidence_type_id":"1","media_id":"1","obtained_from":"","status_id":"6","manufacturer_id":"","size_unit":"GB"}
I convert it into valid JSON string as below using gsub('"','"'). The output is a VALID JSON string as validated by http://jsonlint.com/ shown below.
{"evidence_number":"","notes":"123","size":"123","model":"123","evidence_type_id":"1","media_id":"1","obtained_from":"","status_id":"6","manufacturer_id":"69","size_unit":"GB"}
JSON.parse works without error on this string but does not produce a properly formed hash. It produces following:
notes123evidence_numbersize123model123evidence_type_id1media_id1obtained_fromstatus_id6size_unitGBmanufacturer_id69
Can someone please tell me how to correct this?
Try <%=raw params[:action]%> and <%=raw params[:collections_controller_ev0]%> instead
I think it will work fine.
$.post("/collections/show_selected_media_fields",{media_type: $("#collections_controller_ev0_media_id option:selected").text(), parent_form_action: <%=raw params[:action]%>,ev0_manufacturer_id:<%=raw params[:collections_controller_ev0].inspect%>, }, function(data) {$("#show_selected_media_fields").html(data);});
params is basically a hash. So you can use .to_json on it. Then in your controller you can convert it from JSON to a hash with JSON.parse().
It's generally a much better idea to use JSON if you're going to be dealing with it in Javascript or HTML at all.
The method showed in my last edit works! it produces correct hash.