I'm trying to use Contains() in a search procedure. The full text indexes are created and working. The issue arises because you cannot used Contains() call on a NULL variable or parameter, it throws an error.
This takes 9 sec to run (passing in non-null param):
--Solution I saw on another post
IF #FirstName is null OR #FirstName = '' SET #FirstName = '""'
...
Select * from [MyTable] m
Where
(#FirstName = '""' OR CONTAINS(m.[fname], #FirstName))
This runs instantly (passing in non-null param)
IF #FirstName is null OR #FirstName = '' SET #FirstName = '""'
...
Select * from [MyTable] m
Where
CONTAINS(m.[fname], #FirstName)
Just by adding that extra 'OR' in front of the 'contains' completely changed the Query Plan. I have also tried using a 'case' statement instead of 'OR' to no avail, I still get the slow query.
Has anyone solved this the problem of null parameters in full text searching or experience my issue? Any thoughts would help, thanks.
I'm using SQL Server 2012
You are checking value of bind variable in SQL. Even worse, you do it in OR with access predicate. I am not an expert on SQL Server, but it is generally a bad practice, and such predicates lead to full table scans.
If you really need to select all values from table when #FirstName is null then check it outside of SQL query.
IF #FirstName is null
<query-without-CONTAINS>
ELSE
<query-with-CONTAINS>
I believe, in the majority of times #FirstName is not null. This way you will access table using your full text index most of the time. Getting all the rows from table is a lost cause anyway.
From a logical standpoint, the first query takes longer to execute because it has to evaluate 2 conditions:
#FirstName = '""'
and ,in case the first condition fails, which should be the majority of the time,
CONTAINS(m.[fname], #FirstName)
My guess is that in your table, you don't have any null or empty FirstName, that's why the results are the same. Otherwise, you would have a few "" in the result set as FirstName.
Maybe you should try reversing the order to see if it makes any difference:
WHERE (CONTAINS(m.[fname], #FirstName) OR #FirstName = '""')
Related
When Ecto queries get more complex and require clauses like CASE...WHEN...ELSE...END, we tend to depend on Ecto's fragment to solve it.
e.g. query = from t in <Model>, select: fragment("SUM(CASE WHEN status = ? THEN 1 ELSE 0 END)", 2)
In fact the most popular Stack Overflow post about this topic suggests to create a macro like this:
defmacro case_when(condition, do: then_expr, else: else_expr) do
quote do
fragment(
"CASE WHEN ? THEN ? ELSE ? END",
unquote(condition),
unquote(then_expr),
unquote(else_expr)
)
end
end
so you can use it this way in your Ecto queries:
query = from t in <Model>,
select: case_when t.status == 2
do 1
else 0
end
at the same time, in another post, I found this:
(Ecto.Query.CompileError) to prevent SQL injection attacks, fragment(...) does not allow strings to be interpolated as the first argument via the `^` operator, got: `"exists (\n SELECT 1\n FROM #{other_table} o\n WHERE o.column_name = ?)"
Well, it seems Ecto's team figured out people are using fragment to solve complex queries, but they don't realize it can lead to SQL injection, so they don't allow string interpolation there as a way to protect developers.
Then comes another guy who says "don't worry, use macros."
I'm not an elixir expert, but that seems like a workaround to DO USE string interpolation, escaping the fragment protection.
Is there a way to use fragment and be sure the query was parameterized?
SQL injection, here, would result of string interpolation usage with an external data. Imagine where: fragment("column = '#{value}'") (instead of the correct where: fragment("column = ?", value)), if value comes from your params (usual name of the second argument of a Phoenix action which is the parameters extracted from the HTTP request), yes, this could result in a SQL injection.
But, the problem with prepared statement, is that you can't substitute a paremeter (the ? in fragment/1 string) by some dynamic SQL part (for example, a thing as simple as an operator) so, you don't really have the choice. Let's say you would like to write fragment("column #{operator} ?", value) because operator would be dynamic and depends on conditions, as long as operator didn't come from the user (harcoded somewhere in your code), it would be safe.
I don't know if you are familiar with PHP (PDO in the following examples), but this is exactly the same with $bdd->query("... WHERE column = '{$_POST['value']}'") (inject a value by string interpolation) in opposite to $stmt = $bdd->prepare('... WHERE column = ?') then $stmt->execute([$_POST['value']]); (a correct prepared statement). But, if we come back to my previous story of dynamic operator, as stated earlier, you can't dynamically bind some random SQL fragment, the DBMS would interpret "WHERE column ? ?" with > as operator and 'foo' as value like (for the idea) WHERE column '>' 'foo' which is not syntactically correct. So, the easiest way to turn this operator dynamic is to write "WHERE column {$operator} ?" (inject it, but only it, by string interpolation or concatenation). If this variable $operator is defined by your own code (eg: $operator = some_condition ? '>' : '=';), it's fine but, in the opposite, if it involves some superglobal variable which comes from the client like $_POST or $_GET, this creates a security hole (SQL injection).
TL;DR
Then comes another guy who says "don't worry, use macros."
The answer of Aleksei Matiushkin, in the mentionned post, is just a workaround to the disabled/forbidden string interpolation by fragment/1 to dynamically inject a known operator. If you reuse this trick (and can't really do otherwise), as long as you don't blindly "inject" any random value coming from the user, you'll be fine.
UPDATE:
It seems, after all, that fragment/1 (which I didn't inspect the source) doesn't imply a prepared statement (the ? are not placeholder of a true prepared statement). I tried some simple and stupid enough query like the following:
from(
Customer,
where: fragment("lastname ? ?", "LIKE", "%")
)
|> Repo.all()
At least with PostgreSQL/postgrex, the generated query in console appears to be in fact:
SELECT ... FROM "customers" AS c0 WHERE (lastname 'LIKE' '%') []
Note the [] (empty list) at the end for the parameters (and absence of $1 in the query) so it seems to act like the emulation of prepared statement in PHP/PDO meaning Ecto (or postgrex?) realizes proper escaping and injection of values directly in the query but, still, as said above LIKE became a string (see the ' surrounding it), not an operator so the query fails with a syntax error.
I am trying to make a fairly simple query with an optional parameter in Interbase. I am using Firedac in Delphi 10 Seattle to call it.
SELECT STATUS_ID FROM TABLENAME
WHERE
STATUS_ID=:STATUSID OR :STATUSID IS NULL
fails with Dynamic SQL Error Code = -804 Unknown Datatype.
I can isolate just the :STATUSID IS NULL part and it fails.
Setting the parameter to null with just STATUS_ID=:STATUSID works just fine so it is the :STATUSID IS NULL part that is throwing the error.
Just remembered:
some_field = :some_param or :some_param is null
can be replaced by
some_field = coalesce(:some_param, some_field)
Yet another solution to try.
The answer I came up with is to use zero as my 'all' clause.
WHERE
STATUS_ID=:STATUSID OR :STATUSID=0;
that works. Null is better because it is correctly asking for No Entity, but it seems the implementation in Firedac does not support it.
I know it's very late answer. But I had a similar problem.
I guess your problem is because you test STATUS_ID=:STATUSID before :STATUSID IS NULL. Maybe it'll work with (:STATUSID IS NULL) OR (STATUS_ID=:STATUSID)
For numeric fields, I came up with a slightly different solution. For integer PKeys, often you never have a value of 0. Meaning you can either pass NULL or 0 and this will still return all rows. Lil more flexible.
SELECT * FROM WHATEVER
WHERE
((:ID IS NULL) OR (:ID <= 0) OR (:ID = ID))
For string fields, either NULL or '' (empty string) will return all rows.
SELECT * FROM WHATEVER
WHERE
((:STR IS NULL) OR (TRIM(:STR) = CAST('' AS VARCHAR(10))) OR (:STR = CLIENT_NI))
You can even chain them up like
SELECT * FROM WHATEVER
WHERE
((:ID IS NULL) OR (:ID <= 0) OR (:ID = ID)) AND
((:STR IS NULL) OR (TRIM(:STR) = CAST('' AS VARCHAR(10))) OR (:STR = CLIENT_NI))
Probably a little less efficient than having a separate query for all scenarios, but if you're using an OK RDB, it'll optimize the parameter WHERE clause as static, and you won't lose much performance. I love this solution.
You could use a cast
SELECT STATUS_ID
FROM TABLENAME
WHERE STATUS_ID=:STATUSID OR cast( :STATUSID as integer ) IS NULL
This works in firebird, so I guess that it should work in Interbase as well.
I'm searching for a way to remove every property, of any node in the DB, having a specific value using Cypher.
Context
I got a csv bulk file from a relational table with plenty of NULL values. LOAD CSV brings them as values. Removing them (replacing them with empty '' within the csv file) resulted in the same issue (properties without values). Tried many (many) Cypher operations to discard NULL values but nothing worked.
Can't find anything in the docs neither by Googling. Can this be done using only Cypher? It seems to me not (yet) supported.
Thanks.
How about this (when you know the property-names):
MATCH (n:Label)
WHERE n.property = ''
REMOVE n.property;
MATCH (u:User)
WHERE u.age = ''
SET u.age = null;
If you know which columns these are in your import you can do something like this
load csv with headers from "" as line
with line, case line.foo when '' then null else line.foo end as foo
create (:User {name:line.name, foo:foo})
It won't create the properties with null.
For numeric values it's easier as toInt() and toFloat() return null on unparseable values like ''.
No, there is no way to do this only with chypher. I suppose that you have already seen the way to do it via REST. That is the best solution for now.
I have a complex query that contains more than one place where the same primary key value must be substituted. It looks like this:
select Foo.Id,
Foo.BearBaitId,
Foo.LinkType,
Foo.BugId,
Foo.GooNum,
Foo.WorkOrderId,
(case when Goo.ZenID is null or Goo.ZenID=0 then
IsNull(dbo.EmptyToNull(Bar.FanName),dbo.EmptyToNull(Bar.BazName))+' '+Bar.Strength else
'#'+BarZen.Description end) as Description,
Foo.Init,
Foo.DateCreated,
Foo.DateChanged,
Bug.LastName,
Bug.FirstName,
Goo.BarID,
(case when Goo.ZenID is null or Goo.ZenID=0 then
IsNull(dbo.EmptyToNull(Bar.BazName),dbo.EmptyToNull(Bar.FanName))+' '+Bar.Strength else
'#'+BarZen.Description end) as BazName,
GooTracking.Status as GooTrackingStatus
from
Foo
inner join Bug on (Foo.BugId=Bug.Id)
inner join Goo on (Foo.GooNum=Goo.GooNum)
left join Bar on (Bar.Id=Goo.BarID)
left join BarZen on (Goo.ZenID=BarZen.ID)
inner join GooTracking on(Goo.GooNum=GooTracking.GooNum )
where (BearBaitId = :aBaitid)
UNION
select Foo.Id,
Foo.BearBaitId,
Foo.LinkType,
Foo.BugId,
Foo.GooNum,
Foo.WorkOrderId,
Foo.Description,
Foo.Init,
Foo.DateCreated,
Foo.DateChanged,
Bug.LastName,
Bug.FirstName,
0,
NULL,
0
from Foo
inner join Bug on (Foo.BugId=Bug.Id)
where (LinkType=0) and (BearBaitId= :aBaitid )
order by BearBaitId,LinkType desc, GooNum
When I try to use an integer parameter on this non-trivial query, it seems impossible to me. I get this error:
Error
Incorrect syntax near ':'.
The query works fine if I take out the :aBaitid and substitute a literal 1.
Is there something else I can do to this query above? When I test with simple tests like this:
select * from foo where id = :anid
These simple cases work fine. The component is TADOQuery, and it works fine until you add any :parameters to the SQL string.
Update: when I use the following code at runtime, the parameter substitutions are actually done (some glitch in the ADO components is worked around) and a different error surfaces:
adoFooContentQuery.Parameters.FindParam('aBaitId').Value := 1;
adoFooContentQuery.Active := true;
Now the error changes to:
Incorrect syntax near the keyword 'inner''.
Note again, that this error goes away if I simply stop using the parameter substitution feature.
Update2: The accepted answer suggests I have to find two different copies of the parameter with the same name, which bothered me so I reworked the query like this:
DECLARE #aVar int;
SET #aVar = :aBaitid;
SELECT ....(long query here)
Then I used #aVar throughout the script where needed, to avoid the repeated use of :aBaitId. (If the number of times the parameter value is used changes, I don't want to have to find all parameters matching a name, and replace them).
I suppose a helper-function like this would be fine too: SetAllParamsNamed(aQuery:TAdoQuery; aName:String;aValue:Variant)
FindParam only finds one parameter, while you have two with the same name. Delphi dataset adds each parameter as a separate one to its collection of parameters.
It should work if you loop through all parameters, check if the name matches, and set the value of each one that matches, although I normally choose to give each same parameter a follow-up number to distingish between them.
Is it possible to pass null values to parameter queries? For example
Sql = "insert into TableX values (?,?)".
Params = [{sql_integer, [Val1]}, {sql_float, [Val2]}].
% Val2 may be a float, or it may be the atom, undefined
odbc:param_query(OdbcRef, Sql, Params).
Now, of course odbc:param_query/3 is going to complain if Val2 is undefined when trying to match to a sql_float, but my question is... Is it possible to use a parameterized query, such as:
Sql = "insert into TableY values (?,?,?,?,?,?,?,?,?)".
with any null parameters? I have a use case where I am dumping a large number of real-time data into a database by either inserting or updating. Some of the tables I am updating have a dozen or so nullable fields, and I do not have a guarantee that all of the data will be there.
Concatenating a SQL together for each query, checking for null values seems complex, and the wrong way to do it.
Having a parameterized query for each permutation is simply not an option.
Any thoughts or ideas would be fantastic! Thank you!
You can use the atom null to denote a null value. For instance:
Sql = "insert into TableX values (?,?)".
Params = [{sql_integer, [Val1]}, {sql_float, [null]}].