Just an quick and easy one, I need to be able to search our database minus the case sensitivity, I know how to do it, just not with the Neo4jClient. Here's the code:
client.Cypher
.Match("(person:Person)")
.Where((Person person) => person.Email == search)
where 'search' is a parameter of type string that is passed to the method. I have read that using =~ '(?i)text' works, but that doesn't allow me to pass in the parameter, and I have tried this:
client.Cypher
.Match("(person:Person)")
.Where((Person person) => person.Email =~ "(?i){terms}")
.WithParam("terms",search)
But it doesn't like this.
I would like to be able to search without case, and if possible at the same time, using LIKE (or ILIKE as it seems to be for pattern matching).
Thanks
EDIT & ANSWER
The final code ended up as this:
return client.Cypher
.Match("(person:Person)")
.Where("person.Email =~ {terms}")
.OrWhere("person.Name =~ {terms}")
.WithParam("terms", "(?ui).*" + search + ".*")
.Return<Person>("person").Results.ToList();
Which does exactly what I want it to.
Also took the advice of a lowercase field with the value in, we already have one in the account so that logon names are not case sensitive, I am going to do this on the email and name fields, seems better than using toLower() (either in Cypher or in C#)
So thank to #Stefan Armbruster for his help.
You cannot have partial parameters. Instead add (?i) to the parameter value:
query: person.Email =~ term
parameter: term = "(?i)<myvalue"
Note 1: You need to use (?ui) for gracefully dealing with non-ascii case sensitivity (e.g. German umlauts).
Note 2: the =~ operator is not backed by an index, so the query above will touch every Person node and apply the regex to the property value. In Neo4j 2.3 there will be a index backed LIKE which supports string prefix matches.
If you want to use index based case insensitive search, the recommended approach is to store the property value converted to lower case (Cypher has a toLower function) and then do a exact match on the lower cased search value.
Related
When Ecto queries get more complex and require clauses like CASE...WHEN...ELSE...END, we tend to depend on Ecto's fragment to solve it.
e.g. query = from t in <Model>, select: fragment("SUM(CASE WHEN status = ? THEN 1 ELSE 0 END)", 2)
In fact the most popular Stack Overflow post about this topic suggests to create a macro like this:
defmacro case_when(condition, do: then_expr, else: else_expr) do
quote do
fragment(
"CASE WHEN ? THEN ? ELSE ? END",
unquote(condition),
unquote(then_expr),
unquote(else_expr)
)
end
end
so you can use it this way in your Ecto queries:
query = from t in <Model>,
select: case_when t.status == 2
do 1
else 0
end
at the same time, in another post, I found this:
(Ecto.Query.CompileError) to prevent SQL injection attacks, fragment(...) does not allow strings to be interpolated as the first argument via the `^` operator, got: `"exists (\n SELECT 1\n FROM #{other_table} o\n WHERE o.column_name = ?)"
Well, it seems Ecto's team figured out people are using fragment to solve complex queries, but they don't realize it can lead to SQL injection, so they don't allow string interpolation there as a way to protect developers.
Then comes another guy who says "don't worry, use macros."
I'm not an elixir expert, but that seems like a workaround to DO USE string interpolation, escaping the fragment protection.
Is there a way to use fragment and be sure the query was parameterized?
SQL injection, here, would result of string interpolation usage with an external data. Imagine where: fragment("column = '#{value}'") (instead of the correct where: fragment("column = ?", value)), if value comes from your params (usual name of the second argument of a Phoenix action which is the parameters extracted from the HTTP request), yes, this could result in a SQL injection.
But, the problem with prepared statement, is that you can't substitute a paremeter (the ? in fragment/1 string) by some dynamic SQL part (for example, a thing as simple as an operator) so, you don't really have the choice. Let's say you would like to write fragment("column #{operator} ?", value) because operator would be dynamic and depends on conditions, as long as operator didn't come from the user (harcoded somewhere in your code), it would be safe.
I don't know if you are familiar with PHP (PDO in the following examples), but this is exactly the same with $bdd->query("... WHERE column = '{$_POST['value']}'") (inject a value by string interpolation) in opposite to $stmt = $bdd->prepare('... WHERE column = ?') then $stmt->execute([$_POST['value']]); (a correct prepared statement). But, if we come back to my previous story of dynamic operator, as stated earlier, you can't dynamically bind some random SQL fragment, the DBMS would interpret "WHERE column ? ?" with > as operator and 'foo' as value like (for the idea) WHERE column '>' 'foo' which is not syntactically correct. So, the easiest way to turn this operator dynamic is to write "WHERE column {$operator} ?" (inject it, but only it, by string interpolation or concatenation). If this variable $operator is defined by your own code (eg: $operator = some_condition ? '>' : '=';), it's fine but, in the opposite, if it involves some superglobal variable which comes from the client like $_POST or $_GET, this creates a security hole (SQL injection).
TL;DR
Then comes another guy who says "don't worry, use macros."
The answer of Aleksei Matiushkin, in the mentionned post, is just a workaround to the disabled/forbidden string interpolation by fragment/1 to dynamically inject a known operator. If you reuse this trick (and can't really do otherwise), as long as you don't blindly "inject" any random value coming from the user, you'll be fine.
UPDATE:
It seems, after all, that fragment/1 (which I didn't inspect the source) doesn't imply a prepared statement (the ? are not placeholder of a true prepared statement). I tried some simple and stupid enough query like the following:
from(
Customer,
where: fragment("lastname ? ?", "LIKE", "%")
)
|> Repo.all()
At least with PostgreSQL/postgrex, the generated query in console appears to be in fact:
SELECT ... FROM "customers" AS c0 WHERE (lastname 'LIKE' '%') []
Note the [] (empty list) at the end for the parameters (and absence of $1 in the query) so it seems to act like the emulation of prepared statement in PHP/PDO meaning Ecto (or postgrex?) realizes proper escaping and injection of values directly in the query but, still, as said above LIKE became a string (see the ' surrounding it), not an operator so the query fails with a syntax error.
I have a field in the database which contains strings that look like: 58XBF2022L1001390 I need to be able to query results which match the last letter(in this case 'L'), and match or resemble the last four digits.
The regular expression I've been using to find records which match the structure is: \d{2}[A-Z]{3}\d{4}[A-Z]\d{7}, So far I've tried using a scope to refine the results, but I'm not getting any results. Here's my scope
def self.filter_by_shortcode(input)
q = input
starting = q.slice!(0)
ending = q
where("field ~* ?", "\d{2}[A-Z]{3}\d{4}/[#{starting}]/\d{3}[#{ending}]\g")
end
Here are some more example strings, and the substring that we would be looking for. Not every string stored in this database field matches this format, so we would need to be able to first match the string using the regex provided, then search by substring.
36GOD8837G6154231
G4231
13WLF8997V2119371
V9371
78FCY5027V4561374
V1374
06RNW7194P2075353
P5353
57RQN0368Y9090704
Y0704
edit: added some more examples as well as substrings that we would need to search by.
I do not know Rails, but the SQL for what you want is relative simple. Since your string if fixed format, once that format is validated, simple concatenation of sub-strings gives your desired result.
with base(target, goal) as
( values ('36GOD8837G6154231', 'G4231')
, ('13WLF8997V2119371', 'V9371')
, ('78FCY5027V4561374', 'V1374')
, ('06RNW7194P2075353', 'P5353')
, ('57RQN0368Y9090704', 'Y0704')
)
select substr(target,10,1) || substr(target,14,4) target, goal
from base
where target ~ '^\d{2}[A-Z]{3}\d{4}[A-Z]\d{7}$';
I am having trouble with a DB query in a Rails app. I want to store various search terms (say 100 of them) and then evaluate against a value dynamically. All the examples of SIMILAR TO or ~ (regex) in Postgres I can find use a fixed string within the query, while I want to look the query up from a row.
Example:
Table: Post
column term varchar(256)
(plus regular id, Rails stuff etc)
input = "Foo bar"
Post.where("term ~* ?", input)
So term is VARCHAR column name containing the data of at least one row with the value:
^foo*$
Unless I put an exact match (e.g. "Foo bar" in term) this never returns a result.
I would also like to ideally use expressions like
(^foo.*$|^second.*$)
i.e. multiple search terms as well, so it would match with 'Foo Bar' or 'Search Example'.
I think this is to do with Ruby or ActiveRecord stripping down something? Or I'm on the wrong track and can't use regex or SIMILAR TO with row data values like this?
Alternative suggestions on how to do this also appreciated.
The Postgres regular expression match operators have the regex on the right and the string on the left. See the examples: https://www.postgresql.org/docs/9.3/static/functions-matching.html#FUNCTIONS-POSIX-TABLE
But in your query you're treating term as the string and the 'Foo bar' as the regex (you've swapped them). That's why the only term that matches is the exact match. Try:
Post.where("? ~* term", input)
I am using:
c.customerName =~ '(?i).*$q.*'
in order to find insensitive case any kind of customername and this is working absolutely fine for all standard character. In German unfortunately there are special chars e.g. like Ä,Ö,Ü. In this cases the cypher statement is case sensitive, e.g. if we have two customer names like Ötest and ötest it will find only one of them depending if you type a lower or an upper Ö.
Anyone has a hint what I can do to expand the insensitive case search also on such special chars?
EDIT: The problem exists also when you have a name including e.g. a '&' - you'll find e.g. the company D&A Construction when you type 'D&' - the moment you add a thrid character 'D&A' the search fails and no result is shown. Any idea?
You need to add a 'u' in your regex to transform it in a case-insensitive unicode regex. Like this:
c.customerName =~ '(?ui).*$q.*'
Works here:
From this StackOverflow question.
Is there a way to set an attribute in mongoid for case insensitive searches?
Lets say that somebody has a username: IAmGreat and I want to find the users data using their unique username without butchering it and changing it to iamgreat.
Thanks
Actually you can search case insensitive. But you have to search with an regex!
Here is an example how I'm using it at http://zeit.io
User.where(email: /\A#{Regexp.escape(email)}\z/i).first
With the / you are starting and ending the regex. The i after the regex means case insensitive. \A Means the string has to start with the search string and \z means the string has to end with the search string. This is important if you are looking for an exact match.
You can even try something like:
User.where(username: /#{username}/i).first
if you are using rails or mongoid you can try the ff:
#user = User.where({:username => /.*#{name}.*/i })
Why not just down a User.login.downcase (or whatever your model/attribute combination is) when making the comparison? This will leave the capitalization in the DB as-is, but downcase the field just for comparison.
If your application doesn't need to store user-input as case-sensitive, just convert the input to uppercase or lowercase on the way in. Example,
username = params[:username].to_s.downcase
Otherwise, if performance is an issue for you (case-insensitive regex cannot take advantage for indexes) the right way to go about it is to store a backup field for username
field :username_downcase
And then do the query:
User.where(username_downcase: params[:username].to_s.downcase)