Can't correctly parse JSON fixture in tests - ruby-on-rails

I'm using Test::Unit in an old Rails app (3.2.22) and I'm trying to test a service class that hits an external api.
I'm using webmock and trying to get a json file fixture working, but I keep getting parsing errors from the json file.
My test stub looks like this:
response_data = fixture_file_upload('easypost/order_response.json')
stub_request(:post, 'https://api.easypost.com/v2/orders').
to_return(:status => 200, :body => File.read(response_data))
My order_response.json file looks like this:
{
'mode':'test',
'reference':'Order',
'is_return':false,
'options':{'currency':'USD','label_date':null}
}
When I run the tests, I get a parse error:
JSON::ParserError: 757: unexpected token at '{'mode':'test','reference':'Order','is_return':false,'options':{'currency':'USD','label_date':null}}'
What is going on?
UPDATE:
Got it working by using double quotes in the JSON file:
{
"mode":"test",
"reference":"Order",
"is_return":false,
"options":{"currency":"USD","label_date":null}
}
Could anyone explain why this is necessary?

From the fine specification:
A string is a sequence of zero or more Unicode characters, wrapped in double quotes, using backslash escapes.
and and object is a set of key/value pairs where the keys are strings:
That means that this:
{
'mode':'test',
'reference':'Order',
'is_return':false,
'options':{'currency':'USD','label_date':null}
}
is not JSON because JSON strings use double quotes and only double quotes, that's just something that sort of looks like JSON. When you switch to double quotes for your strings:
{
"mode":"test",
"reference":"Order",
"is_return":false,
"options":{"currency":"USD","label_date":null}
}
then you have JSON and everything should work.

This is to make it simpler and to avoid having to have another escape method for javascript reserved keywords
I think this post might be helpful in understanding the need for quotations (and also where I found the above quote): JSON Spec - does the key have to be surrounded with quotes?

Related

Rails remove '\' from json response

json response
{"skill"=>"{\"dept_id\"=>\"01\", \"user_id\"=>\"001\", \"level_cd\"=>\"04_swim\", \"first_name\"=>\"rohit\", \"last_name\"=>\"patel\", \"dept_full_name\"=>\"swiming\", \"rank\"=>\"04_swim\"}, {\"dept_id\"=>\"02\", \"user_id\"=>\"002\", \"level_cd\"=>\"04_swim\", \"first_name\"=>\"ranjit\", \"last_name\"=>\"shinde\", \"dept_full_name\"=>\"running\", \"rank\"=>\"03_run\"}, {\"dept_id\"=>\"04\", \"user_id\"=>\"004\", \"level_cd\"=>\"02_jump\", \"first_name\"=>\"kedar\", \"last_name\"=>\"patil\", \"dept_full_name\"=>\"jumping\", \"rank\"=>\"02_jump\"}, {\"dept_id\"=>\"05\", \"user_id\"=>\"005\", \"level_cd\"=>\"03_run\", \"first_name\"=>\"kapil\", \"last_name\"=>\"bote\", \"dept_full_name\"=>\"Hammer\", \"rank\"=>\"03_run\"}"
How to remove only \ from this response
expected output is
"skill"=>{"dept_id"=>"01", "user_id"=>"001", "level_cd"=>"04_swim", "first_name"=>"rohit", "last_name"=>"patel", "dept_full_name"=>"swiming", "rank"=>"04_swim"}, {"dept_id"=>"02", "user_id"=>"002", "level_cd"=>"04_swim", "first_name"=>"ranjit", "last_name"=>"shinde", "dept_full_name"=>"running", "rank"=>"03_run"}, {"dept_id"=>"04", "user_id"=>"004", "level_cd"=>"02_jump", "first_name"=>"kedar", "last_name"=>"patil", "dept_full_name"=>"jumping", "rank"=>"02_jump"}, {"dept_id"=>"05", "user_id"=>"005", "level_cd"=>"03_run", "first_name"=>"kapil", "last_name"=>"bote", "dept_full_name"=>"Hammer", "rank"=>"03_run"}
There are currently no backslashes in the string. The backslash is only there because the context is within a double quoted string context.
If you want to use a double quote in double quoted string context you need to escape it with a backslash, otherwise the compiler thinks you want to end the string.
"John Doe said: "Hello Word!""
The above is not valid. The " before Hello World! will end the string. Meaning that Hello World! will not be in string context and Ruby tries to parse Hello and World as constants.
To prevent this from happening you escape the " with a backslash \.
"John Doe said: \"Hello Word!\""
\" will be interpreted as one " character. There is no backslash present within the resulting string. See the Ruby literals documentation.
When using single quotes for string delimiters there is no need to escape the double quotes (but you do need to escape single quotes). The above could also be written as:
'John Doe said: "Hello Word!"'
Similarly your data can be written as:
{"skill"=>'{"dept_id"=>"01", "user_id"=>"001", "level_cd"=>"04_swim", "first_name"=>"rohit", "last_name"=>"patel", "dept_full_name"=>"swiming", "rank"=>"04_swim"}, {"dept_id"=>"02", "user_id"=>"002", "level_cd"=>"04_swim", "first_name"=>"ranjit", "last_name"=>"shinde", "dept_full_name"=>"running", "rank"=>"03_run"}, {"dept_id"=>"04", "user_id"=>"004", "level_cd"=>"02_jump", "first_name"=>"kedar", "last_name"=>"patil", "dept_full_name"=>"jumping", "rank"=>"02_jump"}, {"dept_id"=>"05", "user_id"=>"005", "level_cd"=>"03_run", "first_name"=>"kapil", "last_name"=>"bote", "dept_full_name"=>"Hammer", "rank"=>"03_run"}'
The above clearly demonstrates that there are no backslash characters present in the string.
However the string is not JSON. I suggest changing the server response if possible. You can eval the current response, but I would advise not to use eval ever (eval is evil). If the server would send malicious Ruby code, eval will execute it without any issues and might corrupt your machine.
Looks like the hash example needs to end with an } to be valid. So I added it in my example. Further more it looks to be a collection of records, but it also looks like it's missing a list. If it were inside a list it would be valid but as the example stands now, it is not a valid hash.
But let's say just for fun, I did want to take the string and put it inside an array. Maybe something like this:
data = {"skill"=>"{\"dept_id\"=>\"01\", \"user_id\"=>\"001\", \"level_cd\"=>\"04_swim\", \"first_name\"=>\"rohit\", \"last_name\"=>\"patel\", \"dept_full_name\"=>\"swiming\", \"rank\"=>\"04_swim\"}, {\"dept_id\"=>\"02\", \"user_id\"=>\"002\", \"level_cd\"=>\"04_swim\", \"first_name\"=>\"ranjit\", \"last_name\"=>\"shinde\", \"dept_full_name\"=>\"running\", \"rank\"=>\"03_run\"}, {\"dept_id\"=>\"04\", \"user_id\"=>\"004\", \"level_cd\"=>\"02_jump\", \"first_name\"=>\"kedar\", \"last_name\"=>\"patil\", \"dept_full_name\"=>\"jumping\", \"rank\"=>\"02_jump\"}, {\"dept_id\"=>\"05\", \"user_id\"=>\"005\", \"level_cd\"=>\"03_run\", \"first_name\"=>\"kapil\", \"last_name\"=>\"bote\", \"dept_full_name\"=>\"Hammer\", \"rank\"=>\"03_run\"}"}
parsed_data = data["skill"].split("}, ").map{|x| x.end_with?("\"") ? x + '}' : x}.map{|x| eval(x)}
puts parsed_data
{"dept_id"=>"01", "user_id"=>"001", "level_cd"=>"04_swim", "first_name"=>"rohit", "last_name"=>"patel", "dept_full_name"=>"swiming", "rank"=>"04_swim"}
{"dept_id"=>"02", "user_id"=>"002", "level_cd"=>"04_swim", "first_name"=>"ranjit", "last_name"=>"shinde", "dept_full_name"=>"running", "rank"=>"03_run"}
{"dept_id"=>"04", "user_id"=>"004", "level_cd"=>"02_jump", "first_name"=>"kedar", "last_name"=>"patil", "dept_full_name"=>"jumping", "rank"=>"02_jump"}
{"dept_id"=>"05", "user_id"=>"005", "level_cd"=>"03_run", "first_name"=>"kapil", "last_name"=>"bote", "dept_full_name"=>"Hammer", "rank"=>"03_run"}
Now with the data in an array you can convert it to json if you'd like
require 'json'
2.6.5 :007 > parsed_data.to_json
=> "[{\"dept_id\":\"01\",\"user_id\":\"001\",\"level_cd\":\"04_swim\",\"first_name\":\"rohit\",\"last_name\":\"patel\",\"dept_full_name\":\"swiming\",\"rank\":\"04_swim\"},{\"dept_id\":\"02\",\"user_id\":\"002\",\"level_cd\":\"04_swim\",\"first_name\":\"ranjit\",\"last_name\":\"shinde\",\"dept_full_name\":\"running\",\"rank\":\"03_run\"},{\"dept_id\":\"04\",\"user_id\":\"004\",\"level_cd\":\"02_jump\",\"first_name\":\"kedar\",\"last_name\":\"patil\",\"dept_full_name\":\"jumping\",\"rank\":\"02_jump\"},{\"dept_id\":\"05\",\"user_id\":\"005\",\"level_cd\":\"03_run\",\"first_name\":\"kapil\",\"last_name\":\"bote\",\"dept_full_name\":\"Hammer\",\"rank\":\"03_run\

Read multiple concatenated json objects in Ruby

I have a file that contains multiple JSON objects that are not separated by comma :
{
"field" : "value",
"another_field": "another_value"
} // no comma
{
"field" : "value"
}
Each of the objects standalone is a valid json object.
Is there a way that I can process this file easily?
I know this is NOT a valid json, but unfortunately this file is being generated by a 3rd party tool. I have no option of changing the way the output looks like.
I can't open a text editor and smart-insert commas / square brackets before the run, since this is an automated process (I also really don't want to write code that opens the file and manipulates it).
In .NET there's a library that has this exact feature :
https://stackoverflow.com/a/29480032/2970729
https://www.newtonsoft.com/json/help/html/P_Newtonsoft_Json_JsonReader_SupportMultipleContent.htm
Is there anything equivalent in Ruby?
As long as your file is that simple you might want to do something like this:
# content = File.read(filename)
content =<<-EOF
{
"field" : "value",
"another_field": "another_value"
} // no comma
{
"field" : "value"
}
EOF
require 'json'
JSON.parse("[#{content.gsub(/\}.*?\{/m, '},{')}]")
#=> [{"field"=>"value", "another_field"=>"another_value"}, {"field"=>"value"}]
The yajl-ruby gem enables processing concatenated JSON in Ruby. The parser can read from a String or an IO. Each complete object is yielded to a block.
require 'yajl'
File.open 'file.json' do |f|
Yajl.load f do |object|
# do something with object
end
end
See the documentation for other options (buffer size, symbolized keys, etc).

JSON::ParserError: 757: unexpected token at '{

the current hash is
{\"report_name\"=>\"Study/Control: ABIRATERONE ACETATE - 20151413355\", \"left_mue_start_date\"=>\"02-26-2015\", \"left_mue_end_date\"=>\"03-19-2015\", \"right_mue_start_date\"=>\"02-26-2015\", \"right_mue_end_date\"=>\"03-19-2015\", \"report_formulary_id\"=>\",7581\", \"mue\"=>\"true\", \"mue_type\"=>\"study/control\", \"chain_id\"=>\"1\", \"left_mue_formulary_ids\"=>[\"7581\"], \"action\"=>\"create_report\", \"controller\"=>\"informatics\", \"user_id\"=>339}
now I need to convert it in proper hash like
{"report_name" => "Study/Control: ABIRATERONE ACETATE - 20151413355"}
so I am trying to get it with JSON.parse but I am getting error like:
JSON::ParserError: 757: unexpected token at '{
So if someone know about that so please help me.
and I am using Rails 3.2
What you have is a hash printed as String. To convert it into a Hash use eval.
ch = "{\"report_name\"=>\"Study/Control: ABIRATERONE ACETATE - 20151413355\", \"left_mue_start_date\"=>\"02-26-2015\", \"left_mue_end_date\"=>\"03-19-2015\", \"right_mue_start_date\"=>\"02-26-2015\", \"right_mue_end_date\"=>\"03-19-2015\", \"report_formulary_id\"=>\",7581\", \"mue\"=>\"true\", \"mue_type\"=>\"study/control\", \"chain_id\"=>\"1\", \"left_mue_formulary_ids\"=>[\"7581\"], \"action\"=>\"create_report\", \"controller\"=>\"informatics\", \"user_id\"=>339}"
hash = eval(ch)
# => {"report_name"=>"Study/Control: ABIRATERONE ACETATE - 20151413355", "left_mue_start_date"=>"02-26-2015", "left_mue_end_date"=>"03-19-2015", "right_mue_start_date"=>"02-26-2015", "right_mue_end_date"=>"03-19-2015", "report_formulary_id"=>",7581", "mue"=>"true", "mue_type"=>"study/control", "chain_id"=>"1", "left_mue_formulary_ids"=>["7581"], "action"=>"create_report", "controller"=>"informatics", "user_id"=>339}
PS: A JSON string should look as follows, meaning what you have is not JSON and hence you got JSON::ParserError for using JSON.parse on a non-JSON string :
"{\"report_name\":\"Study/Control: ABIRATERONE ACETATE - 20151413355\",\"left_mue_start_date\":\"02-26-2015\",\"left_mue_end_date\":\"03-19-2015\",\"right_mue_start_date\":\"02-26-2015\",\"right_mue_end_date\":\"03-19-2015\",\"report_formulary_id\":\",7581\",\"mue\":\"true\",\"mue_type\":\"study/control\",\"chain_id\":\"1\",\"left_mue_formulary_ids\":[\"7581\"],\"action\":\"create_report\",\"controller\":\"informatics\",\"user_id\":339}"
To avoid using eval you could use JSON.parse ch.gsub('=>', ':') this way you will get a HASH from your HASH stored as STRING
Last time when I got this issue, since the json file that I got from a API that contains a BOM
UTF-8 BOM is a sequence of bytes (EF BB BF)
What's different between UTF-8 and UTF-8 without BOM?
at the beginning of that string, but you know that part wouldn't display or as readable when we got the string from response. I try to use Ruby JSON to parse it, but I failed, I got the same exception with yours. just a reminder for others, when you get a Json response. By the way, this would be no problem while you are handling that in javascript, but with problems in Python or Ruby languages.
I ran into a similar problem, though it was failing while parsing \"\". This is in regards to using Pact.IO. I mention it here since this is the highest ranked Google result while looking for the error I encountered. The solution in my case was to change the body of a POST in my C# application so that it wasn't using empty string, but a null string. Basically I added this before my HTTP call.
if (request.Method == HttpMethod.Post && request.Content!=null && request.Content.GetType()==typeof(StringContent) && request.Content.ReadAsStringAsync().Result == String.Empty)
request.Content = new StringContent(null);

rails decode json cyrrilic string

I have such string example, which i get from json (cp1251):
Ôèëüòð ìàñëÿíûé OPEL/GM/DAEWOO
which mean:
Фильтр масляный OPEL/GM/DAEWOO
this tool http://www.artlebedev.ru/tools/decoder/ say that i must use CP1252 → CP1251 decoder. I try it so:
my_string.force_encoding('cp1252').force_encoding('1251')
but it didn't solve my problem. What i do wrong?
how could i convert to normall view my json cyrrillic string in RoR?
i get json from url so:
jsonAE = JSON.load(open('http://******/portal.api?l=*****&p=Sih2*****&act=price_by_nr_firm&nr='+article_nr+'&oe=true'))
from json i get:
{"result":[{"nr":"OC90","brand":"Knecht","name":"Фильтр масляный OPEL/GM/DAEWOO","stock":"-","delivery":"не известно","minq":"1","upd":"16.03.15 23:40","price":"130.34","currency":"руб."},{"nr":"OC90","brand":"Knecht","name":‌​"Фильтр масляный OPEL/GM/DAEWOO","stock":"-","delivery":"не известно","minq":"1","upd":"17.03.15 00:05","price":"130.34","currency":"руб."}]}
but it turn's to something bad with JSON.load
▶ puts 'Ôèëüòð ìàñëÿíûé OPEL/GM/DAEWOO'
.encode(Encoding::CP1252)
.force_encoding(Encoding::CP1251)
.encode(Encoding::UTF_8)
#⇒ Фильтр масляный OPEL/GM/DAEWOO
The string in ruby is suspected to be utf8ed. So, the first action is to inform ruby that the string is actually in one-byte. Than we say “hey, don’t care, I know this one-byte is actually cyrillic.” And, finally, turn it back to utf-8.
Hope it helps.

Send params{} hash to another method in same controller

I am working on very complex and dynamic form where I do lot of calls to various methods to render partials depending upon values chosen for drop downs using jQuery. Problem is that after filling the form if it fails validation the form loses all the filled in values upon re-load. I got around this by sending some specific values from params{} hash to methods for my partials on re-load. But it is very cumbersome and I have large number of elements in params hash. How can send the whole params{} to another method in the same controller using jQuery?
Ok I tried this in my form:
$.post("/collections/show_selected_media_fields",{media_type: $("#collections_controller_ev0_media_id option:selected").text(), parent_form_action: "<%=params[:action]%>",ev0_manufacturer_id:"<%=params[:collections_controller_ev0].inspect%>", }, function(data) {$("#show_selected_media_fields").html(data);});
It produces following string sent as parameter:
Parameters: {"ev0_manufacturer_id"=>"{"client_asset_id"=>"", "status_id"=>"6", "server_
name"=>"", "media_id"=>"11", "serial_number"=>"", "evidence_number&quot
;=>"qwe", "notes"=>"", "model"=>"", "manufacturer_id"=>"69&quot
;, "interface"=>"SATA", "obtained_from"=>"wr", "evidence_type_id"=>"1"}
", "media_type"=>"Server", "parent_form_action"=>"quick_save"}
how can I convert this raw string to a hash in controller?
{"client_asset_id"=>
needs to be converted to
{"client_asset_id"=>"",... etc}
===========
ok I tried Tom's method. That produces the params as a string in the following shape. I tried to convert it into hash by doing an eval on it. But it errors out.
{commit=>Save, ev1_current_location_id=>, collections_controller_ev1=>{file_system=>NTFS, obtained_from=>, evidence_number=>, interface=>SAT
A, size_unit=>GB, manufacturer_id=>, encryption_version=>, media_id=>3, size=>, evidence_type_id=>3, other_encryption=>, encryption_method=>
N/A, serial_number=>, model=>, encryption_key=>}, ev0_from_location_category=>, ev0_obtained_from_email_id=>, collections_controller_ev0=>{o
btained_from=>, evidence_number=>, media_id=>1, evidence_type_id=>1, status_id=>6}, custody_action=>Create, collection=>{acquired_by=>Amande
ep Singh, custodian_id=>12, matter_id=>58, location=>sa Nose, client_id=>11, software_version=>, collection_date_time=>Fri Nov 30 14:28:06 -
0800 2012, acquisition_method=>Direct Collection, notes=>, software_id=>1}, _method=>put, utf8=>Γ£ô, ev1_current_location_category=>, ev0_cu
rrent_location_category=>, add_working_copy=>No, ev1_obtained_from_email_id=>, authenticity_token=>3vyn6057DDIyfgTnbckeh5heRTIgcVBfxtY89Krfr
/c=, ev1_existing_artifact_type=>, ev0_from_location_id=>, action=>quick_save, ev1_from_location_id=>, ev1_from_location_category=>, ev0_cur
rent_location_id=>, controller=>collections}
using .to_json and HTMLEntities gem I have gotten to a point where I have the following string. I need to convert it back to params hash. How to?
{"utf8"=>"Γ£ô","collection"=>{"acquired_by"=>"Amandeep Singh","notes"=>"","matter_id"=>"58","software_id"=>"1","acquisition_method"=>"Direct
Collection","client_id"=>"11","custodian_id"=>"0","collection_date_time"=>"Fri Nov 30 15=>52=>12 -0800 2012","software_version"=>"","locati
on"=>"sa Nose"},"ev0_current_location_id"=>"","ev1_existing_artifact_type"=>"","ev1_obtained_from_email_id"=>"","custody_action"=>"Create","
action"=>"quick_save","ev0_current_location_category"=>"","ev1_from_location_category"=>"","_method"=>"put","ev0_obtained_from_email_id"=>""
,"ev0_from_location_category"=>"","ev1_current_location_category"=>"","commit"=>"Save","controller"=>"collections","authenticity_token"=>"3v
yn6057DDIyfgTnbckeh5heRTIgcVBfxtY89Krfr/c=","ev0_from_location_id"=>"","ev1_current_location_id"=>"","collections_controller_ev0"=>{"status_
id"=>"6","media_id"=>"9","obtained_from"=>"","evidence_number"=>"","evidence_type_id"=>"1"},"collections_controller_ev1"=>{"media_id"=>"3","
encryption_key"=>"","encryption_version"=>"","size"=>"","obtained_from"=>"","encryption_method"=>"N/A","model"=>"","evidence_number"=>"","ev
idence_type_id"=>"3","size_unit"=>"GB","other_encryption"=>"","interface"=>"SATA","file_system"=>"NTFS","serial_number"=>"","manufacturer_id
"=>""},"ev1_from_location_id"=>"","add_working_copy"=>"No"}
========== edit==========
I am using the following in my view.
JSON( params[:collections_controller_ev0])
It sends following quoted string to my controller.
{"evidence_number":"","notes":"","size":"","model":"","evidence_type_id":"1","media_id":"1","obtained_from":"","status_id":"6","manufacturer_id":"","size_unit":"GB"}
I convert it into valid JSON string as below using gsub('"','"'). The output is a VALID JSON string as validated by http://jsonlint.com/ shown below.
{"evidence_number":"","notes":"123","size":"123","model":"123","evidence_type_id":"1","media_id":"1","obtained_from":"","status_id":"6","manufacturer_id":"69","size_unit":"GB"}
JSON.parse works without error on this string but does not produce a properly formed hash. It produces following:
notes123evidence_numbersize123model123evidence_type_id1media_id1obtained_fromstatus_id6size_unitGBmanufacturer_id69
Can someone please tell me how to correct this?
Try <%=raw params[:action]%> and <%=raw params[:collections_controller_ev0]%> instead
I think it will work fine.
$.post("/collections/show_selected_media_fields",{media_type: $("#collections_controller_ev0_media_id option:selected").text(), parent_form_action: <%=raw params[:action]%>,ev0_manufacturer_id:<%=raw params[:collections_controller_ev0].inspect%>, }, function(data) {$("#show_selected_media_fields").html(data);});
params is basically a hash. So you can use .to_json on it. Then in your controller you can convert it from JSON to a hash with JSON.parse().
It's generally a much better idea to use JSON if you're going to be dealing with it in Javascript or HTML at all.
The method showed in my last edit works! it produces correct hash.

Resources