I have FTS table and query that matches all rows where column contains both "all" and "in".
try db.executeQuery("SELECT * FROM table WHERE column MATCH '\"all\" AND \"in\"'", values: nil)
How can I make this work with parameter binding? So I can provide:
values: ["all", "in"]
SQLite uses plain strings as full-text patterns. You thus have to build yourself one full-text pattern string from your multiple words, and provide it as one FMDB parameter, as below.
let words = ["all", "in"]
let pattern = words
.map { "\"\($0)\"" } // wrap each word inside quotes
.joined(separator: " AND ")
try db.executeQuery("SELECT * FROM table WHERE column MATCH ?", values: [pattern])
Beware that not all patterns are valid. See the Full-Text Index Queries Grammar. This means that some inputs will trigger SQLite errors. For example, if the words come from a text field, and the user types a quote, as in "attack, then you may build the invalid pattern ""attack" that SQLite won't parse (SQLite error 1: malformed MATCH expression).
You can try to sanitize user input yourself, but this is difficult to do while preserving the maximum usable information. After all, application users are usually not aware of SQLite subtleties: there's no point punishing them with empty search results if they happen to type a funny search string.
If you need to perform pattern sanitization, I recommend you have a look at GRDB, an alternative to FMDB with great support for full-text search. It can build safe patterns from user input:
let userInput = textField.text
let pattern = FTS3Pattern(matchingAllTokensIn: userInput)
let rows = try Row.fetchAll(db, "SELECT * FROM table WHERE column MATCH ?", arguments: [pattern])
When user types "attack, the FTS3Pattern(matchingAllTokensIn:) would build the sane attack FTS pattern instead of failing. GRDB uses SQLite itself to perform the sanitization, which means that it's pretty solid.
For more information, see GRDB full-text search documentation.
Related
I am implementing an iOS app which contains a CoreData DB with one of the columns containing a text field, on which the user is supposed to perform several types of search (semantic, exact matches, among other options). Right now, I am stuck at trying to implement the following MySQL query (Python implementation):
'''select * from quotes where match section against ('\\"%s\\"' in boolean mode) ''' % (query)
The above query returns all the text rows containing all the words of string "query" without any particular order, as long as the text field contains all of the words. As an example, a query such as "tomorrow is Saturday" should find a match in a field containing the sentence "Saturday we are going somewhere. Tomorrow is Friday."
I have tried a few approaches, such as NSPredicate, and regex. I couldn't find a direct match in CoreData to the MySQL boolean mode, but the best I have been able to achieve so far was using regex with NSPredicate such as:
let queryArray = query.components(separatedBy: " ")
let regex = ".*\\b(" + queryArray.map {
NSRegularExpression.escapedPattern(for: $0)
}.joined(separator: "|") + ")\\b.*"
let predicate = NSPredicate(format: "textField MATCHES %#", regex)
However, this query is returning any textField row that contains one or more of the words in the initial query, which is not the desired result in my case.
How can I achieve the desired result in CoreData?
For those that may get stuck, NSCompoundPredicate did the trick for me, just create an array of predicates, where which NSPredicate instance is looking for one of the words in the string, ignoring case and apply the NSCompoundPredicate to the NSFetchRequest.
I have a field in the database which contains strings that look like: 58XBF2022L1001390 I need to be able to query results which match the last letter(in this case 'L'), and match or resemble the last four digits.
The regular expression I've been using to find records which match the structure is: \d{2}[A-Z]{3}\d{4}[A-Z]\d{7}, So far I've tried using a scope to refine the results, but I'm not getting any results. Here's my scope
def self.filter_by_shortcode(input)
q = input
starting = q.slice!(0)
ending = q
where("field ~* ?", "\d{2}[A-Z]{3}\d{4}/[#{starting}]/\d{3}[#{ending}]\g")
end
Here are some more example strings, and the substring that we would be looking for. Not every string stored in this database field matches this format, so we would need to be able to first match the string using the regex provided, then search by substring.
36GOD8837G6154231
G4231
13WLF8997V2119371
V9371
78FCY5027V4561374
V1374
06RNW7194P2075353
P5353
57RQN0368Y9090704
Y0704
edit: added some more examples as well as substrings that we would need to search by.
I do not know Rails, but the SQL for what you want is relative simple. Since your string if fixed format, once that format is validated, simple concatenation of sub-strings gives your desired result.
with base(target, goal) as
( values ('36GOD8837G6154231', 'G4231')
, ('13WLF8997V2119371', 'V9371')
, ('78FCY5027V4561374', 'V1374')
, ('06RNW7194P2075353', 'P5353')
, ('57RQN0368Y9090704', 'Y0704')
)
select substr(target,10,1) || substr(target,14,4) target, goal
from base
where target ~ '^\d{2}[A-Z]{3}\d{4}[A-Z]\d{7}$';
I am having trouble with a DB query in a Rails app. I want to store various search terms (say 100 of them) and then evaluate against a value dynamically. All the examples of SIMILAR TO or ~ (regex) in Postgres I can find use a fixed string within the query, while I want to look the query up from a row.
Example:
Table: Post
column term varchar(256)
(plus regular id, Rails stuff etc)
input = "Foo bar"
Post.where("term ~* ?", input)
So term is VARCHAR column name containing the data of at least one row with the value:
^foo*$
Unless I put an exact match (e.g. "Foo bar" in term) this never returns a result.
I would also like to ideally use expressions like
(^foo.*$|^second.*$)
i.e. multiple search terms as well, so it would match with 'Foo Bar' or 'Search Example'.
I think this is to do with Ruby or ActiveRecord stripping down something? Or I'm on the wrong track and can't use regex or SIMILAR TO with row data values like this?
Alternative suggestions on how to do this also appreciated.
The Postgres regular expression match operators have the regex on the right and the string on the left. See the examples: https://www.postgresql.org/docs/9.3/static/functions-matching.html#FUNCTIONS-POSIX-TABLE
But in your query you're treating term as the string and the 'Foo bar' as the regex (you've swapped them). That's why the only term that matches is the exact match. Try:
Post.where("? ~* term", input)
Inside my model at searchable block I have index time added_at.
At search block for searching I added with(:added_at, nil), made reindex and now inside search object I have:
<Sunspot::Search:{:fq=>["-added_at_d:[* TO *]"]...}>
What is the meaning of this [* TO *] ? Something went wrong?
By adding with(:added_at, nil) you narrow down the search results to documents having no values in the field added_at, so we can expect the corresponding query filter to be defined as :
fq=>["added_at_d:null"] # not valid
The problem is that Solr Standard Query Parser does not support searching a field for empty/null value. In this situation the filter needs to be negated (exluding documents having any value in the field) so that the query remains valid.
The operator - can be used to exclude the field, and the wildcard character * can be used to match any value, now we can expect the query filter to look like :
fq=>["-added_at_d:*"]
However, although the above is valid for the query parser, using a range query should be preferred to prevent inconsitent behaviors when using wildcard within negative subqueries.
Range Queries allow one to match documents whose field(s) values are
between the lower and upper bound specified by the Range Query. Range
Queries can be inclusive or exclusive of the upper and lower bounds.
A * may be used for either or both endpoints to specify an open-ended range query.
Eventually there is nothing wrong with this filter that ends up looking like :
fq=>["-added_at_d:[* TO *]"]
cf. Lucene Range Queries, Solr Standard Query Parser
I don't know the name for this kind of search, but I see that it's getting pretty common.
Let's say I have records with the following file names:
'order_spec.rb', 'order.sass', 'orders_controller_spec.rb'
If I search with the following string 'oc' I would like the result to return 'orders_controller_spec.rb' due to match the o in orders and the c in controller.
If the string is 'os' then I'd like all 3 to match, 'order_spec.rb', 'order.sass', 'orders_controller_spec.rb'.
If the string is 'oco' then I'd like 'orders_controller_spec.rb'
What is the name for this kind of search and how would I go about getting this done in Postgresql?
This is a called a subsequence search. One simple way to do it in Postgres is to use the LIKE operator (or several of the other options in those docs) and fill the spaces between your letters with a wildcard, which for LIKE is %. To match anything with an o followed by an s in the words column, that would look like this:
SELECT * FROM table WHERE words LIKE '%o%s%';
This is a relatively expensive search, but you can improve performance with a varchar_pattern_ops or text_pattern_ops index to support faster pattern matching.
CREATE INDEX pattern_index ON table (words varchar_pattern_ops);