Rails audited search - ruby-on-rails

I'm trying to find every items that has been set to a specifique status last year.
I'm using Rail 5 with audited so I created a specific Audit Model and I try to write a scope to return my condition :
Audit = Audited.audit_class
class Audit
scope :followupRebus, -> { where(auditable_type: 'Followup')
.where(action: 'update')
.where("audited_changes LIKE '%step_id:[#{Step::REBUS}%'")
}
end
the content of the audited text field in postgres looks like this when I take it and show it with a .to_s
{"step_id"=>[9, 4], "active"=>[false, true]}
How can I get all audit with step_id = 9 ?
EDIT
Great thanks to DRSE, I finally found a working solution :
changing the default TEXT type of the Column with the migration sent by DRSE
Change the request like this :
class Audit
scope :followupRebus, -> { where(auditable_type: 'Followup')
.where(action: 'update')
.where("((audited_changes -> 'step_id')::json->>0)::int = :step_id", step_id: Step::REBUS)
}
end

You need to use Postgres JSON functions to query the JSON column audited_changes instead of the LIKE operator.
To find audits where the step_id was changed, you can use
.where("(audited_changes -> 'step_id')::jsonb ? :step_id", step_id: '9')
Note the use of the named bind variable :step_id, instead of using active record question mark (?) replacement because Postgres uses the question mark as a JSON query operator.
The clause above will find any audits where step_id = 9, whether that was the value set in the previous version or the updated version of your model.
To find audits where step_id = 9 in the the previous version:
# Check if the first value (indexed by 0) in 'step_id' array is '9'
.where("(audited_changes -> 'step_id')::jsonb->0 ? :step_id", step_id: '9')
To find audits with step_id = 9 set in the updated version:
# Check if the second value (indexed by 1) in 'step_id' array is 9
.where("(audited_changes -> 'step_id')::jsonb->1 ? :step_id", step_id: '9')
(Note: you should not directly string interpolate your conditions for your where clause because you are opening your app to a SQL injection vulnerability. Use rails style conditions with sanitized inputs instead.)
EDIT
Since you have indicated that your audited_changes column type is TEXT and not a JSON type, you will need to either run a migration to change the column type or else cast the column in the query.
To cast the column to JSON at query execution time, use audited_changes::json, so the example would be like this:
.where("(audited_changes::json -> 'step_id')::json ? :step_id", step_id: '9')
To change the column to JSON, start with rails g migration ChangeAuditedChangesColumnToJSONB. Then in your migration file (db/migrate/{timestamp}_change_audited_changes_column_to_jsonb.rb) write:
class ChangeAuditedChangesColumnToJSONB < ActiveRecord::Migration[5.1]
def up
change_column :audits, :audited_changes, :jsonb, using: 'audited_changes::JSONB'
end
def down
change_column :audits, :audited_changes, :text
end
end
Then run rails db:migrate and you should be good to go.
For a new project using Audited, you can add a param to the install step to specify the use of JSON or JSONB type for the audited_changes column.
rails generate audited:install --audited-changes-column-type jsonb # or json

Related

In Rails, how do you write a finder with "where" that compares two dates?

I’m using Rails 4.2. I have a model with these date time attributes
submitted_at
created_at
How do I write a finder method that returns all the models where the submitted_at field occurs chronologically before the created_at field? I tried this
MyModel.where(:submitted_at < :created_at)
But that is returning everything in my database, even items that don’t match.
where(:submitted_at < :created_at) is actually where(false). When you compare two symbols with the lt/gt operators you're actually just comparing them alphabetically:
:a < :b # true
:c > :b # true
where(false) or any other argument thats blank? just returns an "unscoped" relation for chaining.
The ActiveRecord query interface doesn't really have a straight forward way to compare columns like this.
You either use a SQL string:
Resource.where('resources.submitted_at < resources.created_at')
Or use Arel to create the query:
r = Resource.arel_table
Resource.where(r[:submitted_at].lt(r[:created_at]))
The results are exactly the same but the Arel solution is arguably more portable and avoids hardcoding the table name (or creating an ambigeous query).
You can use .to_sql to see what query is generated.
For yours it looks like this:
Resource.where(:submitted_at < :created_at).to_sql
# => SELECT `resources`.* FROM `resources`
If you update like below, you will get some results:
Resource.where('submitted_at < created_at')
# => SELECT * FROM `resources` WHERE (submitted_at < created_at)

Heroku error with PostgreSQL "could not identify an equality operator for type json"

I'm using JSON type in postgreSQL. I am able to solve the could not identify an equality operator for type json problem on my local machine by adding:
execute <<-SPROC
-- This creates a function named hashjson that transforms the
-- json to texts and generates a hash
CREATE OR REPLACE FUNCTION hashjson(
json
) RETURNS INTEGER LANGUAGE SQL STRICT IMMUTABLE AS $$
SELECT hashtext($1::text);
$$;
-- This creates a function named json_eq that checks equality (as text)
CREATE OR REPLACE FUNCTION json_eq(
json,
json
) RETURNS BOOLEAN LANGUAGE SQL STRICT IMMUTABLE AS $$
SELECT bttextcmp($1::text, $2::text) = 0;
$$;
-- This creates an operator from the equality function
CREATE OPERATOR = (
LEFTARG = json,
RIGHTARG = json,
PROCEDURE = json_eq
);
-- Finaly, this defines a new default JSON operator family with the
-- operators and functions we just defined.
CREATE OPERATOR CLASS json_ops
DEFAULT FOR TYPE json USING hash AS
OPERATOR 1 =,
FUNCTION 1 hashjson(json);
SPROC
However, I do not have the superuser privilege in Heroku. How should I solve the problem?
UPDATE
I am using JSON to store a nested hash, the record is like this:
Company has column properties. properties can be:
{websites:[{url:1,images:[1,2,3],description:1},{url:2,images:[1,2,3],description:2}], wikipedia:{url:1, description:2},facebook:{id:1}}
UPDATE 2
The error message:
PG::UndefinedFunction: ERROR: could not identify an equality operator for type json
LINE 1: SELECT DISTINCT "companies".* FROM "companies" INNER JOIN "reviews...
the error indicator arrow is on SELECT DISTINCT .
the sql is made by the this part in the view:
#review.companies
In model:
class Company < ActiveRecord::Base
store_accessor :properties, [:websites, :fb, :twitter, :linkedin,:country, :city, :street, :postcode]
In migration:
class AddPropertiesToCompany < ActiveRecord::Migration
def change
add_column :companies, :properties, :json
end
end
Thanks for help again.
One solution is to use hstore instead of json field type. The problem with this is casting your existing json to hstore.
Another solution is to use GROUP BY, instead of SELECT DISTINCT, in your queries.
But there are some cases where it would be easier to just change the field type to hstore, for example when using active admin.
I had a HABTM association in one of my models, and active admin would try to query:
db_dev=# SELECT DISTINCT "api_v1_test".* FROM "api_v1_test" INNER JOIN "api_v1_test_users" ON "api_v1_test"."id" = "api_v1_test_users"."test_id" WHERE "api_v1_test_users"."user_id" = 200;
ERROR: could not identify an equality operator for type json
LINE 1: SELECT DISTINCT "api_v1_test".* FROM "api_v1_test" INNER JOI...
Without wanting to spend much time fixing this and figuring out how to rewrite these queries I just converted the field to a hstore and everything is now working as it should.
Also, can you not enter a psql console from heroku to create your functions? or create a new migration for them?

Find records where an attribute is present

I have a User model with the attributes username, email and name.
username and email are required upon signup, but not name.
What would be the query to find all users that have filled out name (i.e. it is no nil)?
The query should be at least Rails 3.2 and 4.0 compatible.
I'm thinking something in the lines of:
User.where(name: present?)
[UPDATED 13/5/2022]
To get all records where an attribute is not present in SQL, we would write
WHERE attr IS NULL or attr = ''
an easy mistake to make is then to negate this and write
WHERE attr is not null and attr != ''
However in SQL this equates to writing
WHERE attr != ''
since the NULL value is always ignored when using the equality operator.
So this translates to rails as follows:
User.where.not(name: '')
[MY OLD ANSWER]
An empty value in a database gets a special value NULL. Testing whether is set uses the special comparator IS NULL or IS NOT NULL.
Then there still remains the possibility that an empty string was filled in, so a complete test would be
#users = User.where("name is NOT NULL and name != ''")
[UPDATED for rails 4+]
Since rails 4 we can write:
User.where.not(name: [nil, ""])
which will generate the same query. Awesome :)
present?
present? is essentially not nil and not empty?:
class Object
def present?
!blank?
end
def blank?
respond_to?(:empty?) ? !!empty? : !self
end
end
ActiveRecord condition
In Rails 4, not conditions can be done without raw sql code.
# Both lines lead to the same result:
User.where.not(name: [nil, ""])
User.where.not(name: nil).where.not(name: "")
Since there is no raw sql code, you don't have to worry about if they work with every database adapter. In fact, this works fine for both, mysql and postgres.
to_sql
You can see how they translate to sql queries if you append .to_sql, for example in the rails console.
# rails console
User.where.not(name: [nil, ""]).to_sql
# => "SELECT \"users\".* FROM \"users\" WHERE (NOT ((\"users\".\"name\" = '' OR \"users\".\"name\" IS NULL)))"
User.where.not(name: nil).where.not(name: "").to_sql
# => "SELECT \"users\".* FROM \"users\" WHERE (\"users\".\"name\" IS NOT NULL) AND (\"users\".\"name\" != '')"
Further Reading
[1] Rails 4 guide "ActiveRecord Query Interface"
[2] Definition of present? on github
NOT SQL queries can be built by where.not
#users = User.where.not(name: nil)
Try this:
User.where("name IS NOT NULL AND name != ?", "")
I edited my answer as per #nathavanda comments, which his answer in my opinion should be the accepted one.
You can simply do this:
User.where.not(name: '')
Because of the nature of not, it won't include records where name is nil nor empty string. See this article for more details about not and nil

How to set a counter cache value to a given one?

I am using Ruby on Rails 3.2.2 and I would like to set a counter cache value to a "custom" one. That is, at this time (in my migration file) I am trying to use the following code:
def up
add_column :articles, :comments_count, :integer, :default => 0
Article.reset_column_information
Article.find_each do |article|
# Note: The following code doesn't work (when I migrate the database it
# raises the error "comments_count is marked as readonly").
Article.update_column(:comments_count, article.custom_comments.count)
end
end
In other words, I would like to set the :comments_count value (a counter cache database table column) to a custom value (in my case that value is article.custom_comments.count - note: the custom_comments is not an ActiveRecord Association but a method stated in the Article model class; it returns an integer value as well) that is not related to a has_many associations.
Maybe, I could / should use something like
Article.reset_column_information
Article.find_each do |article|
Article.reset_counters(article.id, ...)
end
but it seems that the reset_counters method cannot work without has_many associations.
How can I set the :comments_count counter cache value to a given value that is related to a "custom association"?
The accept answer includes the iterating method, which is wrong for existing values of comment_count other than 0: update_counter sets the counter relative to it's current values. To set an absolute value, do:
Article.update_counters(article.id, comments_count: comments.count - article.comments_count)
If you have to fetch each row's correct count anyway, you can also more easily use Article.reset_counters(article.id, :comments)
To do it with far fewer queries, use this:
Author
.joins(:books)
.select("authors.id, authors.books_count, count(books.id) as count")
.group("authors.id")
.having("authors.books_count != count(books.id)")
.pluck(:id, :books_count, "count(books.id)")
.each_with_index do |(author_id, old_count, fixed_count), index|
puts "at index %7i: fixed author id %7i, new books_count %4i, previous count %4i" % [index, author_id, fixed_count, old_count] if index % 1000 == 0
Author.update_counters(author_id, books_count: fixed_count - old_count)
end
You describe comments_count as a counter cache, yet a counter cache is strictly defined as the number of associated records in a has_many relationship, which you say this isn't.
If the only way to get the value you want is via method on Article, then you're going to have to iterate over all your Article objects and update each one.
Article.find_each do |article|
article.update_attribute(:comments_count, article.custom_comments.count)
end
This is pretty inefficient, since it's loading and saving every object.
If the definition of custom_comments (which you don't actually explain) is something you can express in SQL, it would undoubtedly be faster to do this update in the database. Which might look something like this:
CREATE TEMP TABLE custom_comment_counts_temp AS
SELECT articles.id as id, count(comments.id) as custom_comments
FROM articles
LEFT JOIN comments ON articles.id = comments.article_id
WHERE <whatever condition indicates custom comments>
GROUP BY articles.id;
CREATE INDEX ON custom_comments_counts_temp(id);
UPDATE articles SET comments_count = (SELECT custom_comments FROM custom_comment_counts_temp WHERE custom_comment_counts_temp.id = articles.id);
DROP TABLE custom_comment_counts_temp;
(this assumes postgresql - if you're using mySQL or some other database, it may look different. If you're not using a relational database at all, it may not be possible)
Additionally, since it's not a counter cache according to Rails' fairly narrow definition, you'll need to write some callbacks that keep these values updated - probably an after_save callback on comment, something like this:
comment.rb:
after_save :set_article_custom_comments
def set_article_custom_comments
a = self.article
a.update_attribute(:comments_count, a.custom_comments.count)
end

Searching serialized data, using active record

I'm trying to do a simple query of a serialized column, how do you do this?
serialize :mycode, Array
1.9.3p125 :026 > MyModel.find(104).mycode
MyModel Load (0.6ms) SELECT `mymodels`.* FROM `mymodels` WHERE `mymodels`.`id` = 104 LIMIT 1
=> [43565, 43402]
1.9.3p125 :027 > MyModel.find_all_by_mycode("[43402]")
MyModel Load (0.7ms) SELECT `mymodels`.* FROM `mymodels` WHERE `mymodels`.`mycode` = '[43402]'
=> []
1.9.3p125 :028 > MyModel.find_all_by_mycode(43402)
MyModel Load (1.2ms) SELECT `mymodels`.* FROM `mymodels` WHERE `mymodels`.`mycode` = 43402
=> []
1.9.3p125 :029 > MyModel.find_all_by_mycode([43565, 43402])
MyModel Load (1.1ms) SELECT `mymodels`.* FROM `mymodels` WHERE `mymodels`.`mycode` IN (43565, 43402)
=> []
It's just a trick to not slow your application. You have to use .to_yaml.
exact result:
MyModel.where("mycode = ?", [43565, 43402].to_yaml)
#=> [#<MyModel id:...]
Tested only for MySQL.
Basically, you can't. The downside of #serialize is that you're bypassing your database's native abstractions. You're pretty much limited to loading and saving the data.
That said, one very good way to slow your application to a crawl could be:
MyModel.all.select { |m| m.mycode.include? 43402 }
Moral of the story: don't use #serialize for any data you need to query on.
Serialized array is stored in database in particular fashion eg:
[1, 2, 3, 4]
in
1\n 2\n 3\n etc
hence the query would be
MyModel.where("mycode like ?", "% 2\n%")
put space between % and 2.
Noodl's answer is right, but not entirely correct.
It really depends on the database/ORM adapter you are using: for instance PostgreSQL can now store and search hashes/json - check out hstore. I remember reading that ActiveRecord adapter for PostgreSQl now handles it properly. And if you are using mongoid or something like that - then you are using unstructured data (i.e. json) on a database level everywhere.
However if you are using a db that can't really handle hashes - like MySQL / ActiveRecord combination - then the only reason you would use serialized field is for somet data that you can create / write in some background process and display / output on demand - the only two uses that I found in my experience are some reports ( like a stat field on a Product model - where I need to store some averages and medians for a product), and user options ( like their preferred template color -I really don't need to query on that) - however user information - like their subscription for a mailing list - needs to be searchable for email blasts.
PostgreSQL hstore ActiveRecord Example:
MyModel.where("mycode #> 'KEY=>\"#{VALUE}\"'")
UPDATE
As of 2017 both MariaDB and MySQL support JSON field types.
You can query the serialized column with a sql LIKE statement.
MyModel.where("mycode LIKE '%?%'", 43402)
This is quicker than using include?, however, you cannot use an array as the parameter.
Good news! If you're using PostgreSQL with hstore (which is super easy with Rails 4), you can now totally search serialized data. This is a handy guide, and here's the syntax documentation from PG.
In my case I have a dictionary stored as a hash in an hstore column called amenities. I want to check for a couple queried amenities that have a value of 1 in the hash, I just do
House.where("amenities #> 'wifi => 1' AND amenities #> 'pool => 1'")
Hooray for improvements!
There's a blog post from 2009 from FriendFeed that describes how to use serialized data within MySQL.
What you can do is create tables that function as indexes for any data that you want to search.
Create a model that contains the searchable values/fields
In your example, the models would look something like this:
class MyModel < ApplicationRecord
# id, name, other fields...
serialize :mycode, Array
end
class Item < ApplicationRecord
# id, value...
belongs_to :my_model
end
Creating an "index" table for searchable fields
When you save MyModel, you can do something like this to create the index:
Item.where(my_model: self).destroy
self.mycode.each do |mycode_item|
Item.create(my_model: self, value: mycode_item)
end
Querying and Searching
Then when you want to query and search just do:
Item.where(value: [43565, 43402]).all.map(&:my_model)
Item.where(value: 43402).all.map(&:my_model)
You can add a method to MyModel to make that simpler:
def find_by_mycode(value_or_values)
Item.where(value: value_or_values).all.map(&my_model)
end
MyModel.find_by_mycode([43565, 43402])
MyModel.find_by_mycode(43402)
To speed things up, you will want to create a SQL index for that table.
Using the following comments in this post
https://stackoverflow.com/a/14555151/936494
https://stackoverflow.com/a/15287674/936494
I was successfully able to query a serialized Hash in my model
class Model < ApplicationRecord
serialize :column_name, Hash
end
When column_name holds a Hash like
{ my_data: [ { data_type: 'MyType', data_id: 113 } ] }
we can query it in following manner
Model.where("column_name = ?", hash.to_yaml)
That generates a SQL query like
Model Load (0.3ms) SELECT "models".* FROM "models" WHERE (column_name = '---
:my_data:
- :data_type: MyType
:data_id: 113
')
In case anybody is interested in executing the generated query in SQL terminal it should work, however care should be taken that value is in exact format stored in DB. However there is another easy way I found at PostgreSQL newline character to use a raw string containing newline characters
select * from table_name where column_name = E'---\n:my_data:\n- :data_type: MyType\n :data_id: 113\n'
The most important part in above query is E.
Note: The database on which I executed above is PostgreSQL.
To search serialized list you need to prefix and postfix the data with unique characters.
Example:
Rather than something like:
2345,12345,1234567 which would cause issues you tried to search for 2345 instead, you do something like <2345>,<12345>,<1234567> and search for <2345> (the search query get's transformed) instead. Of course choice of prefix/postfix characters depends on the valid data that will be stored. You might instead use something like ||| if you expect < to be used and potentially| to be used. Of course that increases the data the field uses and could cause performance issues.
Using a trigrams index or something would avoid potential performance issues.
You can serialize it like data.map { |d| "<#{d}>" }.join(',') and deserialize it via data.gsub('<').gsub('>','').split(','). A serializer class would do the job quite well to load/extract tha data.
The way you do this is by setting the database field to text and using rail's serialize model method with a custom lib class. The lib class needs to implement two methods:
def self.dump(obj) # (returns string to be saved to database)
def self.load(text) # (returns object)
Example with duration. Extracted from the article so link rot wouldn't get it, please visit the article for more information. The example uses a single value, but it's fairly straightforward to serialize a list of values and deserialize the list using the methods mentioned above.
class Duration
# Used for `serialize` method in ActiveRecord
class << self
def load(duration)
self.new(duration || 0)
end
def dump(obj)
unless obj.is_a?(self)
raise ::ActiveRecord::SerializationTypeMismatch,
"Attribute was supposed to be a #{self}, but was a #{obj.class}. -- #{obj.inspect}"
end
obj.length
end
end
attr_accessor :minutes, :seconds
def initialize(duration)
#minutes = duration / 60
#seconds = duration % 60
end
def length
(minutes.to_i * 60) + seconds.to_i
end
end
If you have serialized json column and you want to apply like query on that. do it like that
YourModel.where("hashcolumn like ?", "%#{search}%")

Resources