With my current project, I'm receiving a large JSON file that I'm parsing and storing into my database. The problem is I feel like I'm structuring my database in a very inefficient way.
Example of JSON:
{
first_name: "John",
records: {
ids: [110, 725, 2250],
count: [1, 1, 6]
},
items: {
top: {
title: "My top",
values: { value: [51, 50, 70] }
},
middle: {
title: "Middle Stuff",
values: { value: [51] }
},
},
values: {
health: 100,
strength: 250,
mana: 50
}
}
As you can see the JSON is fairly complex, with nested Objects.
While building it, I started with the main Object ( user ), then slowly started adding more objects. Values was easy, so I added that as another table and just with a reference to the user_id.
Then I did records, which is a bit more complex, but works. However, I'm very worried about the most nested parts, that could be 5+ objects deep. I feel like I shouldn't have an entire column row for a simple value.
What would be the best way to improve on this? Should I somehow crunch the data and store it differently?
Thanks for your help.
// JSON response
{
"name": "John Smith",
...
"progression": {
"levels": [{
"name": "Level 1",
"bosses": [
{
"name": "Boss 1",
"difficultyCompleted": "Hard"
},
{
"name": "Boss 2",
"difficultyCompleted": "Hard"
}]
},{
"name": "Level 2",
"bosses": [
{
"name": "Boss 3",
"difficultyCompleted": "Normal"
},
{
"name": "Boss 4",
"difficultyCompleted": "Easy"
}]
}
}
}
In this example JSON, there is a few layers for each boss that the user has completed. What I would have thought to do initially was to create models and tables but that seemed like it would be wasteful not only in memeory, but also would take longer to fetch the current progression for Boss 4.
Example:
User has_one Progression.
Progression has_one Levels.
Levels has_many Dungeons.
Dungeons has_many bosses.
What I did instead was trying to compress the bosses into a single field, and just convert the JSON at runtime.
So, instead my structure would be like this.
User has_one Progression.
Progression has_many Dungeons.
Models:
// Progression.rb
class Progression < ApplicationRecord
has_many :dungeons
def self.initialize(params={})
params = params['levels']
prog = Progression.new()
params.each do |dungeon|
prog.dungeons.append( Dungeon.initialize( dungeon ) )
end
return prog
end
end
// Dungeon.rb
class Dungeon < ApplicationRecord
belongs_to :progression
def self.initialize(params={})
Dungeon.new(params.reject { |k| !Dungeon.attribute_method?(k) }) # Used to ignore any unused parameters that don't exist on the model.
end
# To convert `bosses` from JSON into a hash for easy use.
def get_bosses
JSON.parse bosses.gsub( '=>', ':' )
end
end
Migration:
// xxxxxx_create_progression.rb
class CreateProgression < ActiveRecord::Migration[5.0]
def change
create_table :progressions do |t|
t.integer :character_id
end
create_table :dungeons do |t|
t.integer :character_id
t.integer :progression_id
t.string :name
t.text :bosses
end
add_index :progressions, :character_id
add_index :dungeons, :progression_id
end
end
Now, when a User is updated to fetch their progression, I can set their progression.
// users_controller.rb
def update_progression
progression = ... fetched from the response
#user.progression = Progression.initialize(progression)
end
After that's all saved to the user, you can now fetch the progression back by going:
<% user.progression.dungeons.each do |dungeon| %>
<%= dungeon.name %>
<% end %>
This solution seems like a decent mix, but I'm a bit worried about the parsing of the JSON. It could become too much, but I'll have to keep watching it. Any other ideas or improvements would be greatly appreicated.
Related
In my Rails (api only) learning project, I have 2 models, Group and Artist, that have a many-to-many relationship with a joining model, Role, that has additional information about the relationship. I have been able to save m2m relationships before by saving the joining model by itself, but here I am trying to save the relationship as a nested relationship. I'm using the jsonapi-serializer gem, but not married to it nor am I tied to the JSON api spec. Getting this to work is more important than following best practice.
With this setup, I'm getting a 500 error when trying to save with the following errors:
Unpermitted parameters: :artists, :albums and ActiveModel::UnknownAttributeError (unknown attribute 'relationships' for Group.)
I'm suspecting that my problem lies in the strong param and/or the json payload.
Models
class Group < ApplicationRecord
has_many :roles
has_many :artists, through: :roles
accepts_nested_attributes_for :artists, :roles
end
class Artist < ApplicationRecord
has_many :groups, through: :roles
end
class Role < ApplicationRecord
belongs_to :artist
belongs_to :group
end
Controller#create
def create
group = Group.new(group_params)
if group.save
render json: GroupSerializer.new(group).serializable_hash
else
render json: { error: group.errors.messages }, status: 422
end
end
Controller#group_params
def group_params
params.require(:data)
.permit(attributes: [:name, :notes],
relationships: [:artists])
end
Serializers
class GroupSerializer
include JSONAPI::Serializer
attributes :name, :notes
has_many :artists
has_many :roles
end
class ArtistSerializer
include JSONAPI::Serializer
attributes :first_name, :last_name, :notes
end
class RoleSerializer
include JSONAPI::Serializer
attributes :artist_id, :group_id, :instruments
end
Example JSON payload
{
"data": {
"attributes": {
"name": "Pink Floyd",
"notes": "",
},
"relationships": {
"artists": [{ type: "artist", "id": 3445 }, { type: "artist", "id": 3447 }]
}
}
Additional Info
It might help to know that I was able to save another model with the following combination of json and strong params.
# Example JSON
"data": {
"attributes": {
"title": "Wish You Were Here",
"release_date": "1975-09-15",
"release_date_accuracy": 1
"notes": "",
"group_id": 3455
}
}
# in albums_controller.rb
def album_params
params.require(:data).require(:attributes)
.permit(:title, :group_id, :release_date, :release_date_accuracy, :notes)
end
From looking at https://api.rubyonrails.org/classes/ActiveRecord/NestedAttributes/ClassMethods.html I think the data format that Rails is normally going to expect will look something like:
{
"group": {
"name": "Pink Floyd",
"notes": "",
"roles_attributes": [
{ "artist_id": 3445 },
{ "artist_id": 3447 }
]
}
}
with a permit statement that looks something like (note the . before permit has moved):
params.require(:group).
permit(:name, :notes, roles_attributes: [:artist_id])
I think you have a few options here:
Change the data format coming into the action.
Craft a permit statement that works with your current data (not sure how tricky that is), you can test your current version in the console with:
params = ActionController::Parameters.new({
"data": {
"attributes": {
"name": "Pink Floyd",
"notes": "",
},
"relationships": {
"artists": [{ type: "artist", "id": 3445 }, { type: "artist", "id": 3447 }]
}
}
})
group_params = params.require(:data).
permit(attributes: [:name, :notes],
relationships: [:artists])
group_params.to_h.inspect
and then restructure the data to a form the model will accept; or
Restructure the data before you try to permit it e.g. something like:
def group_params
params_hash = params.to_unsafe_h
new_params_hash = {
"group": params_hash["data"]["attributes"].merge({
"roles_attributes": params_hash["data"]["relationships"]["artists"].
map { |a| { "artist_id": a["id"] } }
})
}
new_params = ActionController::Parameters.new(new_params_hash)
new_params.require(:group).
permit(:name, :notes, roles_attributes: [:artist_id])
end
But ... I'm sort of hopeful that I'm totally wrong and someone else will come along with a better solution to this stuff.
What should the strong parameters for my chapters_controller be if I have a Book entity and a Chapter entity?
Note: I am using JSON API.
In my chapters_controller, should my strong parameters be:
:title, :order, :content, :published, :book, :picture
Or should it be:
:title, :order, :content, :published, :book_id, :picture
If I use :book instead of :book_id, then in my Ember application, when I go to create a new chapter, I am able to create it and associate this chapter to the parent book, however, my test fails:
def setup
#book = books(:one)
#new_chapter = {
title: "Cooked Wolf Dinner",
order: 4,
published: false,
content: "The bad wolf was very mad. He was determined to eat the little pig so he climbed down the chimney.",
book: #book
}
end
def format_jsonapi(params)
params = {
data: {
type: "books",
attributes: params
}
}
return params
end
...
test "chapter create - should create new chapter assigned to an existing book" do
assert_difference "Chapter.count", +1 do
post chapters_path, params: format_jsonapi(#new_chapter), headers: user_authenticated_header(#jim)
assert_response :created
json = JSON.parse(response.body)
attributes = json['data']['attributes']
assert_equal "Cooked Wolf Dinner", attributes['title']
assert_equal 4, attributes['order']
assert_equal false, attributes['published']
assert_equal #book.title, attributes['book']['title']
end
end
I get error in my console saying Association type mismatch.
Perhaps my line:
book: #book
is causing it?
Either way, gut feeling is telling me I should be using :book in my chapters_controller strong parameters.
It's just my test isn't passing, and I am not sure how to write the parameter hash for my test to pass.
After a few more hours of struggle and looking at the JSON API docs:
http://jsonapi.org/format/#crud-creating
It has come to my attention, in order to set a belongsTo relationship to an entity with JSON API, we need do this:
POST /photos HTTP/1.1
Content-Type: application/vnd.api+json
Accept: application/vnd.api+json
{
"data": {
"type": "photos",
"attributes": {
"title": "Ember Hamster",
"src": "http://example.com/images/productivity.png"
},
"relationships": {
"photographer": {
"data": { "type": "people", "id": "9" }
}
}
}
}
This also led me to fixing another problem I had in the past which I couldn't fix. Books can be created with multiple genres.
The JSON API structure for assigning an array of Genre to a Book entity, we replace the data hash with a data array in the relationship part like this:
"data": [
{ "type": "comments", "id": "5" },
{ "type": "comments", "id": "12" }
]
Additonally, in my controllers, anything strong parameters like so:
:title, :content, genre_ids: []
Becomes
:title, :content, :genres
To comply with JSON API.
So for my new test sample datas I now have:
def setup
...
#new_chapter = {
title: "Cooked Wolf Dinner",
order: 4,
published: false,
content: "The bad wolf was very mad. He was determined to eat the little pig so he climbed down the chimney.",
}
...
end
def format_jsonapi(params, book_id = nil)
params = {
data: {
type: "chapters",
attributes: params
}
}
if book_id != nil
params[:data][:relationships] = {
book: {
data: {
type: "books",
id: book_id
}
}
}
end
return params
end
Special note on the relationship settings - only add relationships to params if there is a relationship, otherwise, setting it to nil is telling JSON API to remove that relationship, instead of ignoring it.
Then I can call my test like so:
test "chapter create - should create new chapter assigned to an existing book" do
assert_difference "Chapter.count", +1 do
post chapters_path, params: format_jsonapi(#new_chapter, #book.id), headers: user_authenticated_header(#jim)
assert_response :created
json = JSON.parse(response.body)
attributes = json['data']['attributes']
assert_equal "Cooked Wolf Dinner", attributes['title']
assert_equal 4, attributes['order']
assert_equal false, attributes['published']
assert_equal #book.id, json['data']['relationships']['book']['data']['id'].to_i
end
I am pretty new to rails and have the following problem:
I have an array of serialized hashes stored in a database. Now I want to extend the hash data with a new key-value pair (calculated from stored data), e.g. like this in Ruby:
Data stored database:
coordinates = Array.new
c1 = Hash.new('x' => x1, 'y' => y1)
c2 = Hash.new('x' => x2, 'y' => y2)
c3 = Hash.new('x' => x3, 'y' => y3)
coordinates << c1 << c2 << c3
Extending data:
coordinates.each_with_index do |c, i|
c['z'] = c['x'] + c['y']
Representation in the database looks like:
"coordinates_table": [
{
"id": 1,
"coordinates": [
{ "x": 1.0, "y": 0.5 },
{ "x": 0.5, "y": 0.4 }
],
"created_at": "2015-11-22T00:18:38.592Z",
"updated_at": "2015-11-22T00:18:38.592Z"
}
]
This is the original migration:
class CreateCoordinates < ActiveRecord::Migration
def change
create_table :coordinates do |t|
t.text :coordinates_param
t.timestamps null: false
end
add_index :coordinates, :id
end
end
and Coordinate class:
class Coordinate < ActiveRecord::Base
serialize :coordinates_param
end
Since I do not intend to change the input data, but only extend the data stored in the database, I figured the best way to realize this was a migration, but I did not find any way to access single Hash objects stored in my table in the generated migration model.
Is there any possibility to do this in rails, or do I have to provide a separate extension class where I could store the new data?
My RABL template seems to be very un-DRY and over complex. Because of this I think I may be using it wrong, or that there are better ways at generating my desired output.
As you can see from the show.rabl code, I have to turn the plugins_vulnerability.vulnerability association into a JSON hash, explicitly selecting which keys I need, then merge the plugins_vulnerability.fixed_in value into the hash, and finally adding the new hash, which now contains the fixed_in value, to the vulnerabilities_array array.
I'm doing this because I want the fixed_in value to be within the vulnerability node.
plugins_controller.rb
class Api::V1::PluginsController < Api::V1::BaseController
def show
#plugin = Plugin.friendly.includes(:plugins_vulnerability, :vulnerabilities).find(params[:id])
end
end
show.rabl:
object #plugin
cache #plugin if Rails.env == 'production'
attributes :name
# Add the 'vulnerabilities' node.
node :vulnerabilities do |vulnerabilities|
vulnerabilities_array = []
# turn the plugins_vulnerability association into an array
vulnerabilities.plugins_vulnerability.to_a.each do |plugins_vulnerability|
vulnerability = plugins_vulnerability.vulnerability.as_json # turn the plugins_vulnerability.vulnerability association into json
vulnerability = vulnerability.select {|k,v| %w(id title references osvdb cve secunia exploitdb created_at updated_at metasploit fixed_in).include?(k) } # only select needed keys
vulnerabilities_array << {
:vulnerability => vulnerability.merge(:fixed_in => plugins_vulnerability.fixed_in)
} # merge the fixed_in attribute into the vulnerability hash and add them to an array (fixed_in is from plugins_vulnerabilities)
end
vulnerabilities_array
end
output.json
{
"plugin": {
"name": "simple-share-buttons-adder",
"vulnerabilities": [
{
"vulnerability": {
"id": 88157,
"title": "Simple Share Buttons Adder 4.4 - options-general.php Multiple Admin Actions CSRF",
"references": "https:\/\/security.dxw.com\/advisories\/csrf-and-stored-xss-in-simple-share-buttons-adder\/,http:\/\/packetstormsecurity.com\/files\/127238\/",
"osvdb": "108444",
"cve": "2014-4717",
"secunia": "",
"exploitdb": "33896",
"created_at": "2014-07-15T17:16:51.227Z",
"updated_at": "2014-07-15T17:16:51.227Z",
"metasploit": "",
"fixed_in": "4.5"
}
},
{
"vulnerability": {
"id": 88158,
"title": "Simple Share Buttons Adder 4.4 - options-general.php ssba_share_text Parameter Stored XSS Weakness",
"references": "https:\/\/security.dxw.com\/advisories\/csrf-and-stored-xss-in-simple-share-buttons-adder\/,http:\/\/packetstormsecurity.com\/files\/127238\/",
"osvdb": "108445",
"cve": "",
"secunia": "",
"exploitdb": "33896",
"created_at": "2014-07-15T17:16:51.341Z",
"updated_at": "2014-07-15T17:16:51.341Z",
"metasploit": "",
"fixed_in": "4.5"
}
}
]
}
}
I guess you can do something like this:
object #plugin
cache #plugin if Rails.env == 'production'
attributes :name
child(#plugin.vulnerabilities => :vulnerabilities) {
attributes :id, :title, :references, :osvdb, :cve, :secunia, :exploitdb, :created_at, :updated_at, :metasploit
# Add the 'fixed_in' node.
node :fixed_in do |vulnerability|
#plugin.plugins_vulnerability.fixed_in
end
}
This should create the same output that you need. And it doesn't look awefully complex to me.
I tried to do this:
Things.order("name").group("category_name")
I was expecting the results to be something like this:
[
{
"category_name": "Cat1",
"things":
[
{ "name": "Cat1_Thing1" },
{ "name": "Cat1_Thing1" }
]
},
{
"category_name": "Cat2",
"things":
[
{ "name": "Cat2_Thing3" },
{ "name": "Cat2_Thing4" }
]
}
]
So I would have expected to get an array of "categories" each with an array of "items" which are within that category. Instead, it appears to give me a list of things, sorted by the field I grouped on.
Note: category_name is a column in the thing table.
Try something like
my_grouping = Category.includes(:things).
select("*").
group('categories.id, things.id').
order('name')
=> #<ActiveRecord::Relation [#<Category id: 1, name: "Oranges">, #<Category id: 2, name: "Apples">]>
Though, you'll still have to access the Thing objects via my_grouping.things, they'll already be at your hand, and you won't have to wait for the results. This is likely the sort of interaction you're looking for, vs. mapping them into an actual Array.
One option is to do the grouping in Rails (it returns a hash)
Things.order("name").group_by(&:category_name)
#=> {"cat1" => [thing1,thing2,..], "cat2" => [thing3,thing4,..],..}
ActiveRecord::Base#group performs a SQL GROUP BY. I think, but i'm not sure (depends on your db adapter) that as you don't specify any SELECT clause, you get the first record for each category.
To achieve what you want, there are different ways.
For instance, using #includes :
Category.includes(:things).map do |category|
{
category_name: category.name,
things: things.sort_by(&:name).map{|t| {name: t.name} }
}
end.to_json
Note that the standard (albeit often frowned upon) way to serialize models as json is to use (and override if need be) as_json and to_json. so you would have something along the lines of this :
class Category < ActiveRecord::Base
def as_json( options = {} )
defaults = { only: :name, root: false, include: {links: {only: :name}} }
super( defaults.merge(options) )
end
end
Use it like this :
Category.includes(:links).map(&:to_json)
EDIT
As category_name is only a column, you can do :
Thing.order( :category_name, :name ).sort_by( &:category_name ).map do |category, things|
{ category_name: category, things: things.map{|t| {name: t.name} } }
end.to_json
such thing could belong in the model :
def self.sorted_by_category
order( :category_name, :name ).sort_by( &:category_name ).map do |category, things|
{ category_name: category, things: things.map{|t| {name: t.name} } }
end
end
so you can do :
Thing.sorted_by_category.to_json
this way, you can even scope things further :
Thing.where( foo: :bar ).sorted_by_category.to_json