I'm using Toptal's Chewy gem to connect and query my Elasticsearch, just like an ODM.
I'm using Chewy along with Elasticsearch 6, Ruby on Rails 5.2 and Active Record.
I've defined my index just like this:
class OrdersIndex < Chewy::Index
define_type Order.includes(:customer) do
field :id, type: "keyword"
field :customer do
field :id, type: "keyword"
field :name, type: "text"
field :email, type: "keyword"
end
end
end
And my model:
class Order < ApplicationRecord
belongs_to :customer
end
The problem here is that when I perform any query using Chewy, the customer data gets deserialized as a hash instead of an Object, and I can't use the dot notation to access the nested data.
results = OrdersIndex.query(query_string: { query: "test" })
results.first.id
# => "594d8e8b2cc640bb78bd115ae644637a1cc84dd460be6f69"
results.first.customer.name
# => NoMethodError: undefined method `name' for #<Hash:0x000000000931d928>
results.first.customer["name"]
# => "Frederique Schaefer"
How can I access the nested association using the dot notation (result.customer.name)? Or to deserialize the nested data inside an Object such as a Struct, that allows me to use the dot notation?
try to use
results = OrdersIndex.query(query_string: { query: "test" }).objects
It converts query result into active record Objects. so dot notation should work. If you want to load any extra association with the above result you can use .load method on Index.
If you want to convert existing ES nested object to accessible with dot notation try to reference this answer. Open Struct is best way to get things done in ruby.
Unable to use dot syntax for ruby hash
also, this one can help too
see this link if you need openStruct to work for nested object
Converting the just-deserialized results to JSON string and deserializing it again with OpenStruct as an object_class can be a bad idea and has a great CPU cost.
I've solved it differently, using recursion and the Ruby's native Struct, preserving the laziness of the Chewy gem.
def convert_to_object(keys, values)
schema = Struct.new(*keys.map(&:to_sym))
object = schema.new(*values)
object.each_pair do |key, value|
if value.is_a?(Hash)
object.send("#{key}=", convert_to_object(value.keys, value.values))
end
end
object
end
OrdersIndex.query(query_string: { query: "test" }).lazy.map do |item|
convert_to_object(item.attributes.keys, item.attributes.values)
end
convert_to_object takes an array of keys and another one of values and creates a struct from it. Whenever the class of one of the array of values items is a Hash, then it converts to a struct, recursively, passing the hash keys and values.
To presence the laziness, that is the coolest part of Chewy, I've used Enumerator::Lazy and Enumerator#map. Mapping every value returned by the ES query into the convert_to_object function, makes every entry a complete struct.
The code is very generic and works to every index I've got.
Related
Perhaps my understanding of how this is supposed to work is wrong, but I seeing strings stored in my DB when I would expect them to be a jsonb array. Here is how I have things setup:
Migration
t.jsonb :variables, array: true
Model
attribute :variables, :variable, array: true
Custom ActiveRecord::Type
ActiveRecord::Type.register(:variable, Variable::Type)
Custom Variable Type
class Variable::Type < ActiveRecord::Type::Json
include ActiveModel::Type::Helpers::Mutable
# Type casts a value from user input (e.g. from a setter). This value may be a string from the form builder, or a ruby object passed to a setter. There is currently no way to differentiate between which source it came from.
# - value: The raw input, as provided to the attribute setter.
def cast(value)
unless value.nil?
value = Variable.new(value) if !value.kind_of?(Variable)
value
end
end
# Converts a value from database input to the appropriate ruby type. The return value of this method will be returned from ActiveRecord::AttributeMethods::Read#read_attribute. The default implementation just calls #cast.
# - value: The raw input, as provided from the database.
def deserialize(value)
unless value.nil?
value = super if value.kind_of?(String)
value = Variable.new(value) if value.kind_of?(Hash)
value
end
end
So this method does work from the application's perspective. I can set the value as variables = [Variable.new, Variable.new] and it correctly stores in the DB, and retrieves back as an array of [Variable, Variable].
What concerns me, and the root of this question, is that in the database, the variable is stored using double escaped strings rather than json objects:
{
"{\"token\": \"a\", \"value\": 1, \"default_value\": 1}",
"{\"token\": \"b\", \"value\": 2, \"default_value\": 2}"
}
I would expect them to be stored something more resembling a json object like this:
{
{"token": "a", "value": 1, "default_value": 1},
{"token": "b", "value": 2, "default_value": 2}
}
The reason for this is that, from my understanding, future querying on this column directly from the DB will be faster/easier if in a json format, rather than a string format. Querying through rails would remain unaffected.
How can I get my Postgres DB to store the array of jsonb properly through rails?
So it turns out that the Rails 5 attribute api is not perfect yet (and not well documented), and the Postgres array support was causing some problems, at least with the way I wanted to use it. I used the same approach to the problem for the solution, but rather than telling rails to use an array of my custom type, I am using a custom type array. Code speaks louder than words:
Migration
t.jsonb :variables, default: []
Model
attribute :variables, :variable_array, default: []
Custom ActiveRecord::Type
ActiveRecord::Type.register(:variable_array, VariableArrayType)
Custom Variable Type
class VariableArrayType < ActiveRecord::ConnectionAdapters::PostgreSQL::OID::Jsonb
def deserialize(value)
value = super # turns raw json string into array of hashes
if value.kind_of? Array
value.map {|h| Variable.new(h)} # turns array of hashes into array of Variables
else
value
end
end
end
And now, as expected, the db entry is no longer stored as a string, but rather as searchable/indexable jsonb. The whole reason for this song and dance is that I can set the variables attribute using plain old ruby objects...
template.variables = [Variable.new(token: "a", default_value: 1), Variable.new(token: "b", default_value: 2)]
...then have it serialized as its jsonb representation in the DB...
[
{"token": "a", "default_value": 1},
{"token": "b", "default_value": 2}
]
...but more importantly, automatically deserialized and rehydrated back into the plain old ruby object, ready for me to interact with it.
Template.find(123).variables = [#<Variable:0x87654321 token: "a", default_value: 1>, #<Variable:0x12345678 token: "b", default_value: 2>]
Using the old serialize api causes a write with every save (intentionally by Rails architectural design), regardless of whether or not the serialized attribute had changed. Doing this all manually by overriding setters/getters is an unnecessary complication due to the numerous ways attributes can be assigned, and is partly the reason for the newer attributes api.
If it helps anyone else, Rails wants you to provide the possible keys to permit in the controller as well if you're using strong params:
def controller_params
params.require(:parent_key)
.permit(
jsonb_field: [:allowed_key1, :allowed_key2, :allowed_key3]
)
end
One solution could be to just parse the variable via JSON.parse, push it inside an empty array, then assign it to the attribute.
variables = []
variable = "{\"token\": \"a\", \"value\": 1, \"default_value\": 1}"
variable.class #String
parsed_variable = JSON.parse(variable) #{"token"=>"a", "value"=>1, "default_value"=>1}
parsed_variable.class #Hash
variables.push parsed_variable
I save an array of strings to my rails database, but when I go to use it in the view, I believe it is printing the string definition of the array. Am I dealing with JSON here? (aka when it saves to the database is it just an array wrapped in a string?)
How do I have it so that in my view, it simply displays the items?
<%= record.items %>
displays inside my html tag:
["item1", "item2", "item3"]
I tried iterating through record.items.each do |item| but that did not work.
If you're saving an "exact" array as a String, then Array#each won't work, because isn't a method in the String class.
Maybe isn't the best option, but you could use JSON.parse and this way get your array and be able to iterate over each object inside:
require 'json'
str = '["item1", "item2", "item3"]'
JSON.parse(str).each { |item| p item }
# "item1"
# "item2"
# "item3"
In order this work your string must be an array, in your example the second item is missing its double quote.
You could consider working with serialization or array data types depending on you current database.
A better approach to your issue is to serialize your items column. I think by default it's Array but you can use Hash or JSON.
class Record < ActiveRecord::Base
serialize :items
end
Calling record.items returns the data exactly the way you need. If you go with this you'll have to update your old records to support it.
Given the following document (snippet):
{
udid: "0E321DD8-1983-4502-B214-97D6FB046746",
person: {
"firstname": "Jacob",
"lastname": "Prince"
}
}
I'n my console I can basically do:
mycollection.first.attributes.values_at("udid", "person")
This returns the person as a hash.
Now I want a single field. But these doesn't work (person.firstname):
mycollection.first.attributes.values_at("udid", "person.firstname")
mycollection.first.attributes.values_at("udid", "person[:firstname]")
mycollection.first.attributes.values_at("udid", "person['firstname']")
How how do you access the person child-document?
I'm in the need to have users select which fieds they want to export. I was thinking along the lines of doing something like this:
class Foo
include Mongoid::Document
# fields definitions
embeds_one :person # two fields: firstname, lastname
def to_csv *columns
attributes.values_at *columns
end
end
Whats a (the most) efficient way to select specific fields?
If you already know the fields and its nested keys, using Ruby v2.3+ you can utilise the dig() built-in method. For example:
document = collection.find({},{'projection' => {'uid' => 1, "person.firstname" => 1 }}).first
result = [document.dig("uid"), document.dig("person", "firstname")]
puts result.inspect
Alternatively, depending on your application use case you could also utilise MongoDB Aggregation Pipeline, especially $project operator
For example:
document = collection.aggregate([{"$project"=>{ :uid=>"$uid", :person_firstname=>"$person.firstname"}}]).first
puts document.values_at("uid", "person_firstname").inspect
Note that the projection above renames the nested person.firstname into a flatten field called person_firstname.
See also MongoDB Ruby Driver: Aggregation Tutorial
In my Rails 4 app, I actually send an active record relation in JSON with:
[...]
wine['varietals'] = record.varietals
#wines << wine
format.json { render :json => { :success => "OK", :items => #wines } }
[...]
wine['varietals'] is an array of AR relations. My problem is the varietal model contains a field named grape_id that is an integer. I need to send it in string for my WS. I don't want to make a custom conversion to JSON just for this field.
How to force this field to be string before the automatic JSON conversion ? If possible I don't want to make an array of hashes and keep the AR style with dot: model.field
wine['varietals'].each do |varietal|
varietal.grape_id.to_s
end
Of course this doesn't work.
All Rails models have an as_json method that gets called when rednering the model to JSON. You can override this method within your models to set up custom JSON formatting. In your case, you may want to add something like this to your Wine model:
def as_json(opts = {})
json = super(opts)
json["grape_id"] = self.grape_id.to_s
json
end
The method gives you the default model JSON when you call the super method and set it to the json variable, then stringifies grape_id and sets it in the JSON, and finally returns the updated JSON.
Now, any time a controller returns a JSON version of single Wine model, or an association of multiple Wine models, the JSON will be formatted through this updated method and the grape_id will be stringified every time.
I'm having problems with weird behaviour in RoR. I'm having a Hash that i'm converting to json using to_json() like so:
data = Hash.new
# ...
data = data.to_json()
This code appears inside a model class. Basically, I'm converting the hash to JSON when saving to database. The problem is, the string gets saved to database with its surrounding quotes. For example, saving an empty hash results in: "{}". This quoted string fails to parse when loading from the database.
How do I get rid of the quotes?
The code is:
def do_before_save
#_data = self.data
self.data = self.data.to_json()
end
EDIT:
Due to confusions, I'm showing my entire model class
require 'json'
class User::User < ActiveRecord::Base
after_find { |user|
user.data = JSON.parse(user.data)
}
after_initialize { |user|
self.data = Hash.new unless self.data
}
before_save :do_before_save
after_save :do_after_save
private
def do_before_save
#_data = self.data
self.data = self.data.to_json()
end
def do_after_save
self.data = #_data
end
end
The data field is TEXT in mysql.
I'm willing to bet money that this is the result of you calling .to_json on the same data twice (without parsing it in between). I've had a fair share of these problems before I devised a rule: "don't mutate data in a lossy way like this".
If your original data was {}, then first .to_json would produce "{}". But if you were to jsonify it again, you'd get "\"{}\"" because a string is a valid json data type.
I suggest that you put a breakpoint in your before_save filter and see who's calling it the second time and why.
Update
"call .to_json twice" is a general description and can also mean that there are two subsequent saves on the same object, and since self.data is reassigned, this leads to data corruption. (thanks, #mudasobwa)
It depends on your model's database field type.
If the field is string type (like VARCHAR or TEXT) it should be stored as string (no need to get rid of the quotes - they are fine). Make sure calling to_json once.
If the field is Postgres JSON type, then you can just assign a hash to the model's field, no need to call to_json at all.
If you are saving hash as a JSON string in a varchar column you can use serialize to handle marshalling/unmarshaling the data:
class Thing < ActiveRecord::Base
serialize :foo, JSON
end
Knowing exactly when to convert the data in the lifecycle of a record is actually quite a bit harder than your naive implementation. So don't reinvent the wheel.
However a huge drawback is that the data cannot be queried in the DB*. If you are using Postgres or MySQL you can instead use a JSON or JSONB (postgres only) column type which allows querying. This example is from the Rails guide docs:
# db/migrate/20131220144913_create_events.rb
create_table :events do |t|
t.json 'payload'
end
# app/models/event.rb
class Event < ApplicationRecord
end
# Usage
Event.create(payload: { kind: "user_renamed", change: ["jack", "john"]})
event = Event.first
event.payload # => {"kind"=>"user_renamed", "change"=>["jack", "john"]}
## Query based on JSON document
# The -> operator returns the original JSON type (which might be an object), whereas ->> returns text
Event.where("payload->>'kind' = ?", "user_renamed")
use {}.as_json instead of {}.to_json
ex:
a = {}
a.as_json # => {}
a.to_json # => "{}"
http://api.rubyonrails.org/classes/ActiveModel/Serializers/JSON.html#method-i-as_json