I'm experimenting with a few concepts (actually playing and learning by building a RoR version of the 1978 database WHATSIT?).
It basically is a has_many :through structure with Subject -> Tags <- Value. I've tried to replicate a little of the command line structure by using a query text field to enter the commands. Basically things like: What's steve's phone.
Anyhow, with that interface most of the searches use ILIKE. I though about enhancing it by allowing OR conditions using some form of an array. Something like What's steve's [son,daugher]. I got it working by creating the ILIKE clause directly, but not with string replacement.
def bracket_to_ilike(arrel,name,bracket)
bracket_array = bracket.match(/\[([^\]]+)\]/)[1].split(',')
like_clause = bracket_array.map {|i| "#{name} ILiKE '#{i}' "}.join(" OR ")
arrel.where(like_clause)
end
bracket_to_ilike(tags,'tags.name','[son,daughter]') produces the like clause tags.name ILiKE 'son' OR tags.name ILiKE 'daughter'
And it get the relations, but with all the talk about using the form ("tags.name ILiKE ? OR tags.name ? ",v1,v2,vN..)., I though I'd ask if anyone has any ideas on how to do that.
Creating variables on the fly is doable from what I've searched, but not in favor. I just wondered if anyone has tried creating a method that can add a where clause that has a variable number parameters.I tried sending the where clause to the relation, but it didn't like that.
Steve
Couple of things to watch out for in your code...
What will happen when one of the elements of bracket_array contains a single quote?
What will happen if I take it step farther and set an element to say "'; drop tables..."?
My first stab at refactoring your code would be to see if Arel can do it. Or Sequeel, or whatever they call the "metawhere" gem these days. My second stab would be something like this:
arrel.where( [ bracket_array.size.times.map{"#{name} ILIKE ?"}.join(' OR '), *bracket_array ])
I didn't test it, but the idea is to use the size of bracket_array to generate a string of OR'd conditions, then use the splat operator to pass in all the values.
Thanks to Phillip for pointing me in the right direction.
I didn't know you could pass an array to a where clause - that opened up some options
I had used the splat operator a few times, but it didn't hit me that it actually creates an object(variable)
The [son,daughter] stuff was just a console exercise to see what I could do, but not sure what I was going to do with it. I ended up taking the model association and creating the array out of the picture and implemented OR searches.
def array_to_ilike(col_name,keys)
ilike = [keys.map {|i| "#{col_name} ILiKE ? "}.join(" OR "), *keys ]
#ilike = [keys.size.times.map{"#{col_name} ILIKE ?"}.join(' OR '), *keys ]
#both work, guess its just what you are use to.
end
I then allowed a pipe(|) character in my subject,tag,values searches, so a WHATSIT style question
What's Steve's Phone Home|Work => displays home and work phone
steve phone home|work The 's stuff is just for show
steve son|daughter => displays children
phone james%|lori% => displays phone number for anyone who's name starts with james or lori
james%|lori% => dumps all information on anyone who's name starts with james or lori
The query then parses the command and if it encounters a | in any of the words, it will do things like:
t_ilike = array_to_ilike('tags.name',name.split("|"))
# or I actually stored it off on the inital parse
t_ilike = #tuple[:tag][:ilike] ||= ['tags.name ilike ?',tag]
Again this is just a learning exercise in creating a non-CRUD class to deal with the parsing and searching.
Steve
Related
In our application, the Recipe model has many ingredients (many-to-many relationship implemented using :through). There is a query to return all the recipes where at least one ingredient from the list is contained (using ILIKE or SIMILAR TO clause). I would like to pose two questions:
What is the cleanest way to write the query which will return this in Rails 6 with ActiveRecord. Here is what we ended up with
ingredients_clause = '%(' + params[:ingredients].map { |i| i.downcase }.join("|") + ')%'
recipes = recipes.where("LOWER(ingredients.name) SIMILAR TO ?", ingredients_clause)
Note that recipes is already created before this point.
However, this is a bit dirty solution.
I also tried to use ILIKE = any(array['ing1', 'ing2',..]) with the following:
ingredients_clause = params[:ingredients].map { |i| "'%#{i}%'" }.join(", ")
recipes = recipes.where("ingredients.name ILIKE ANY(ARRAY[?])", ingredients_clause)
This won't work since ? automatically adds single quotes so it would be
ILIKE ANY (ARRAY[''ing1', 'ing2', 'ing3'']) which is of course wrong.
Here, ? is used to sanitise parameters for SQL query, so avoid possible SQL injection attacks. That is why I don't want to write a plain query formed from params.
Is there any better way to do this?
What is the best approach to order results by the number of ingredients that are matched? For example, if I search for all recipes that contains ingredients ing1 and ing2 it should return those which contains both before those which contains only one ingredient.
Thanks in advance
For #1, a possible solution would be something like (assuming the ingredients table is already joined):
recipies = recipies.where(Ingredients.arel_table[:name].lower.matches_any(params[:ingredients]))
You can find more discussion on this kind of topic here: Case-insensitive search in Rails model
You can access a lot of great SQL query features via #arel_table.
#2 If we assume all the where clauses are applied to recipies already:
recipies = recipies
.group("recipies.id")
# Lets Rails know you meant to put a raw SQL expression here
.order(Arel.sql("count(*) DESC"))
Hi im fetching the user input and displaying the records that matches the condition, my query will look like
customers = customers.where('customers.contact_num ilike :search', {search: "%#{options[:search_contact]}%"})
here in db the contact number is stored in string with the format (091)-234-5678 like that
on while searching the user on basis of contact number if i search like this
091 it filters the number correctly, but while searching like 0912, it doesn't display record due to the braces, so how to modify the query to neglect the ) and - while searching..
As im new to the domain please help me out
thanks in advance
What about using REGEXP_REPLACE to remove all non-digit chars from the search - something like below?
customers = customers.where("REGEXP_REPLACE(customers.contact_num,'[^[:digit:]]','','g') ilike :search", {search: "%#{options[:search_contact]}%"})
Changing the query is hard. Let's not do that.
Instead right a quick script to transforms your numbers into
1112223333 form. No formatting at all. Something like:
require 'set';
phone = "(234)-333-2323"
numbers = Set.new(["1","2","3","4","5","6","7","8","9","0"])
output = phone.chars().select{|n| numbers.include?(n)}.join("")
puts output
=> "2343332323"
Then write a little function to transform them into display form for use in the views.
This will make your query work as is.
I have been building an app these days. The functionality is nothing fancy at all, I have to connect to a client's SOAP webservice, fetch some data, save it into my pg database and build a search functionality based on this data.
The search has to be performed on two tables, both combined are like 80K rows. It needs to look for every word in the input text in several fields from these two tables, which have a classical assocation one to many.
Previous to get my hands dirty I was looking at the choices I had to get the functionality done (ransack, searchkick, scoped_search etc), but I ended up trying first just vanilla Active Record and I was very surprised to find that I could achieve the functionality way easier than I thought and with an acceptable response time, about to 400ms active record time for the most expensive queries in local.
So the problem is, the performance of this app in Heroku is way worse than in local (I'm developing using a vagrant box btw). On average, queries take 2-3 times longer than in local, so the user experience goes from acceptable to poor. I was wondering If someone could help to improve my query. I'm also worried about how the background job that fetchs the data is also way les performant than in local and about some issues with the memory, but that's a different story though.
The relevant snippets are these:
part_master.rb where the search method is implemented:
class PartMaster < ApplicationRecord
has_many :part_variants, foreign_key: 'sap_cod', primary_key: 'sap_cod'
has_many :locations, foreign_key: 'sap_cod', primary_key: 'sap_cod'
scope :con_stock, -> { where("stock > 0") }
scope :planta, -> (planta) { where planta_cod: planta}
def self.search(params)
recordset = PartMaster.joins(:part_variants).all
recordset = recordset.con_stock if params[:stock].present?
recordset = recordset.planta(params[:planta]) if params[:planta].present?
recordset = search_keywords(params[:search], recordset)
recordset
end
private
def self.search_keywords(query, recordset)
keywords = query.to_s.strip.split
if query
keywords.each do |keyword|
recordset = recordset.where('part_masters.sap_cod ILIKE :q OR
unaccent(descripcion_maestro) ILIKE unaccent(:q)
OR fabricante ILIKE :q OR ref_fabricante ILIKE :q
OR fabricante_prov ILIKE :q OR ref_prov ILIKE :q',
q: "%#{keyword}%")
end
recordset.distinct.order(:sap_cod)
end
end
end
And this is the call to the method from the controller:
def index
parts = params[:search].present? ? PartMaster.search(params) :
PartMaster.none
#parts = parts.page(params[:page]).per(50)
end
I have an index in every searchable field.
EDIT: Finally I have tried a mix of the proposal in the answers. I have created one field in each table that is a concatenation of the relevant fields for the search, having so 2 OR statements instead of 5, and I also have put trigram GIN indexes in both new fields. I haven't seen any improvement though, the times corresponding to ActiveRecord are very similar, perhaps marginally better.
The thing is, the output for the query using EXPLAIN dones't show any info about the indexes being used.
Hash Join (cost=2243.29..6067.41 rows=2697 width=132)
Hash Cond: ((part_variants.sap_cod)::text = (part_masters.sap_cod)::text)
Join Filter: ((part_masters.combinada_maestro ~~* '%rodamiento%'::text) OR (part_variants.combinada_info ~~* '%rodamiento%'::text))
-> Seq Scan on part_variants (cost=0.00..1264.96 rows=54896 width=18)
-> Hash (cost=1128.13..1128.13 rows=34813 width=132)
-> Seq Scan on part_masters (cost=0.00..1128.13 rows=34813 width=132)
(6 rows)
Suggestions to improve the AR query speed use a direct Postgresql query in your model
Example for in you keywords loop
query = "SELECT * FROM part_masters WHERE......"
PartMaster.connection.execute(query, :skip_logging)
I agree with spikermann. Also the multiple ORs in a loop is not helping neither.
If you only want to work on a vanilla solution vs adding SOLR or any other engine, you could have one separate field to hold copies of the strings that you would like to search. (ex. name, description, ...). The search this field only. You will need some method to keep the field updated when the name, description or other values change.
So let's say i have a Customer model with array column phones.
It's pretty easy to find all customers with given phone
Customer.where('? = ANY(phones)', '+79851234567')
But i can't figure out how to use LIKE with wildcard when i want to find customers with phones similar to given one, something like:
Customer.where('ANY(phones) LIKE ?', '+7985%')
I'm using PostgreSQL 9.5 and Rais 4.2
Any ideas?
I think, first of all, its better to use second table phones with fields customer_id, phone_number. I think it's more rails way ). In this way you can use this query
Phone.where("phone_number LIKE ?", '%PART%').first.customer
If you serialize your array in some text field, by example JSON, you should use % on both sides of your pattern:
Customer.where('phones LIKE ?', '%+7985%')
If you have an array in your database, you should use unnest() function to expand an array to a set of rows.
Can you try this
Customer.where("array_to_string(phones, ', ') like ?", '+7985%')
I believe this will work.
I want my code to do two things that is currently not doing
#students = Student.where(["first_name = ? OR middle_name = ? OR last_name = ?", params[:query].split])
Work. (it says im supposed to pass 4 parameters but I want the user to be able to type words and find by those words on each of those fields and return whatever matches)
Actually use Like clause instead of rigid equal clause.
Please Help.
This looks like a problem that would be better suited to using search rather than SQL. Have you considered something like thinking sphinx or act as ferret (solr would probably be overkill).
...if you must do this in sql, you could build a query something like this:
cols = ['first_name', 'last_name', 'middle_name']
query = 'John Smith'
sql_query = cols.map{|c| query.split.map{|q| "#{c} like '?'"}}.join(' OR ')
sql_query_array = query.split * cols.count
Student.where(sql_query, sql_query_array)
I agree with the previous advice that if you need to do search you should look at something like Solr or Sphinx.
Anyhow, this should help you out.
def search
query = params[:query].split.map {|term| "%#{term}%" }
sql = "first_name LIKE ? OR middle_name LIKE ? OR last_name LIKE ?"
#students = Student.where([sql, *query])
end
The answer to step 1 is using Ruby's awesome little feature called the "splat operator" which allows you to take an array and evaluate it as a list of arguments.
The answer to step 2 is to just massage the query string you get back from the params and turn it into something you can use with the LIKE operator. I basically stole this from Railscasts Simple Search Form episode.
I'm not sure if you want all fields to search the same term if that is the case then you can do this:
where("first_name LIKE :term OR middle_name LIKE :term OR last_name LIKE :term", { term: "%#{params[:term]}%"})
No need for any crazy split or map or anything else, this is just straight ActiveRecord