What should I do to properly remove 'event' entries from Thingsboard?
As far as I know, the current API does not provide a way to remove events. It seems like the only way is to directly delete the records in DB.
By the way, I'm using PostgreSQL as DB.
After two hours of research in Thingsboard source code, I found the solution.
The date is contained in the uid_event field in V1 UUID format.
So first, you need to write a function uuid_timestampin order to convert the UUID to a timestamp. I found the solution, here: https://stackoverflow.com/a/24191574/5300212
CREATE FUNCTION uuid_timestamp(id uuid) RETURNS timestamptz AS $$
select TIMESTAMP WITH TIME ZONE 'epoch' +
(((('x' || lpad(split_part(id::text, '-', 1), 16, '0'))::bit(64)::bigint) +
(('x' || lpad(split_part(id::text, '-', 2), 16, '0'))::bit(64)::bigint << 32) +
((('x' || lpad(split_part(id::text, '-', 3), 16, '0'))::bit(64)::bigint&4095) << 48) - 122192928000000000) / 10000000 ) * INTERVAL '1 second';
$$ LANGUAGE SQL
IMMUTABLE
RETURNS NULL ON NULL INPUT;
After that, to delete all events older than 30 days ago, you can run a query like:
DELETE FROM public.event WHERE uuid_timestamp(event_uid::uuid) < now() - '30 days'::interval;
Your assumption is correct. You will need to execute a SQL script to cleanup the "events" table. I must note that for Cassandra DB we already have "cassandra.query. ts_key_value_ttl" and "cassandra.query.events_ttl" configuration parameter to automate this process.
Related
I have a select that return a single column, using as where clause 4 columns, and I created a specific index for this query. When I do this select using Dbeaver, the result return in 30~50 milliseconds.
SELECT
column1
FROM
myTable
WHERE
column2 = a
AND column3 = b
AND (column4 = c AND column5 = d )
FOR READ ONLY WITH UR;
Now I created a procedure with this same simple select. The proc declare a P1, a 'cursor with return', do the select, open the cursor and close the P1. When I use the same dbeaver connection the result is returning between 1.2 ~ 1.6 seconds.
CREATE PROCEDURE myProc (
IN a DECIMAL(10),
IN b DECIMAL(10),
IN c DECIMAL(10),
IN d DECIMAL(10)
)
LANGUAGE SQL
DYNAMIC RESULT SETS 1
SPECIFIC myProc
P1: BEGIN
DECLARE cursor1 CURSOR WITH RETURN for
SELECT
column1
FROM
myTable
WHERE
column2 = a
AND column3 = b
AND (column4 = c AND column5 = d )
FOR READ ONLY WITH UR;
OPEN cursor1;
END P1
Is this huge return difference correct? If wrong, Is there something wrong in my procedure that justifies this return time difference? Or could be something wrong in the DB configuration or the server that justifies this difference, something like few resources in the DB server or something like the proc not using the index( I don't know if procs in the DB2 use the index by default like the queries )?
I am new in DB2 and new in procedures creation. Thank you in advance.
Best regards,
Luis
I don't know if is the best way, but I solved the problem using a solution that I read for SQL Server. The solution is create local variables that will receive the parameters values, and use the variable in the queries, to guarantee always the best execution plan. I didn't knew if this was valid to DB2, but my proc now has almost the same response time compared to query. Worked!
Link of SQL Server post: SQL Server: Query fast, but slow from procedure
In this link above a user called #Jake give this explanation:
"The reason this happens is because the procedures query plan is being cached, along with the parameters that were passed to it. On subsequent calls, this query plan generated will be reused with new parameters. This can cause problems because if the data is unevenly distributed, one parameter can generate a sub-optimal plan vs. another. Using local variables essentially does the same as OPTIMIZE FOR UNKNOWN because local variables cannot be sniffed."
I think that is the same for DB2, because worked. After I change these old procedures to use local variables my execution plan begun to use the indexes recently created
I have a model, Lease, with attributes ends_on (date), and notice_days (integer) I would like to write a query that returns all records that have an ends_on date that is less than today's date plus notice_days for that record.
In straight SQL (for my Postgresql db) the query is:
SELECT * FROM leases WHERE ends_on < CURRENT_DATE + notice_days
I can easily write a query to compare against today's date + some fixed buffer
Lease.where(ends_on: ...Date.current + 30).all
But I don't want the fixed buffer, I want a buffer (notice_days) that is unique to each record.
I do realize that I could execute the sql shown above using exec_sql, but I would like to learn how to do this with a straightforward ActiveRecord query.
Edit
Stefanyuk's answer below does in fact answer my question, but I am looking for a way to write the ActiveRecord query with key value pairs as arguments to the where method in the hopes of making the query less database dependent. It seems that this should be possible but I have tried many ways of passing the attribute name without success.
This does not work, but I am looking for something similar that does:
Lease.where(ends_on: ...(Date.current + :notice_days))
Is there a way to replace the symbol :notice_days in that example with a reference to the attribute notice_days?
You can do this by using interval
SELECT * FROM leases WHERE ends_on < CURRENT_DATE + interval '1 day' * notice_days
Or with Rails
Lease.where("ends_on < CURRENT_DATE + interval '1 day' * notice_days")
Or with Arel
Lease.where(Lease.arel_table[:ends_on].lt(Arel.sql("CURRENT_DATE + notice_days")))
I have a timestamp with time zone field named statdate and the entry looks like this 2021-11-17 12:47:54-08. I want to create a field with just the time of day expressed locally, so it would look like 12:47:54. (This data was recorded on an iPhone and it's 12:28 PST). (Go to bottom of post for solution using views from #AdrianKalver)
select *,statdate :: timestamp :: time as stattime from table
works in PGAdmin and an example result is 12:47:54 as desired. How do I make this an alter table
ALTER TABLE tablename add COLUMN stattime timestamp generated always AS (select *,statdate :: timestamp :: time as stattime from tablename) stored;
is the wrong syntax.
ALTER TABLE tablename add COLUMN stattime timestamp generated always AS ( EXTRACT(HOUR FROM statdate) || ':' || EXTRACT(MINUTE FROM statdate) || ':' || EXTRACT(SECOND FROM statdate)) stored;
ERROR: generation expression is not immutable which I'm presuming is a type problem, although postgres can concatenate strings and numbers with this syntax.
Just tried something else
ALTER TABLE tablename add COLUMN stattime timestamp generated always AS ( Cast(EXTRACT(HOUR FROM statdate) as text) || ':' || cast(EXTRACT(MINUTE FROM statdate) as text) || ':' || cast(EXTRACT(SECOND FROM statdate) as text) ) stored; -- ERROR: generation expression is not immutable
I'm using the hours and minutes for a graph and I can't get in the middle of the Chartkick. Could do it in High Charts, but think it will be simpler to create the view chart and use that. The Rails/Chartkick looks like
<%= line_chart TableName.where(statdate: start..current_date).pluck(:statdate, :y_axis) %>
and can't break that apart. So will go with creating a View Table.
What's the right way to do this? I've looked here and at the postgresql docs and not having much luck.
Following comments, the solution
CREATE OR REPLACE VIEW public.view_bp_with_time AS
SELECT
id,
statdate,
statdate :: time AS stattime,
y-axis
FROM table_name
ORDER BY statdate
Now to bring into Rails. Not as straightforward as I thought. And I'm off the computer for the next week.
Per here:
https://www.postgresql.org/docs/current/sql-createtable.html
GENERATED ALWAYS AS ( generation_expr ) STORED
This clause creates the column as a generated column. The column cannot be written to, and when read the result of the specified expression will be returned.
The keyword STORED is required to signify that the column will be computed on write and will be stored on disk.
The generation expression can refer to other columns in the table, but not other generated columns. Any functions and operators used must be immutable. References to other tables are not allowed.
Basically the cast from timestamptz to timestamp is not immutable as there are time zones involved.
For more information see:
https://www.postgresql.org/docs/14/xfunc-volatility.html
Either:
Create a view that does the conversion.
Include it in your query as you show for the pgAdmin4 example.
Create a timestamp field on the table and either add the value to that field as part of INSERT\UPDATE or add a trigger that does that.
I have a view which is exporting some columns, one column is called
'created_at' type:"timestamp without timezone" format: "2014-03-20 12:46:36.590253"
In rails I have a method, which is getting data from the view and is trying to filter the data by a date. I tried rapping created_at into date() but is still not working.
Any ideas?
return ActiveRecord::Base.connection.select_all("
select * from db_functions where date(created_at) >= #{date_p} AND date(created_at) <= #{date_p}")
PG::UndefinedFunction: ERROR: operator does not exist: date >= integer
LINE 2: ...select * from db_functions where date(created_at) >= 2014-03...
^
HINT: No operator matches the given name and argument type(s). You might need to add explicit type casts.
The first noticeable problem is that your time is not quoted. This is causing the time to be treated as integer. To fix this, you could simply wrap date_p with quotes or use ActiveReocrd::ConnectionAdapters#quote as:
conn = ActiveRecord::Base.connection
return conn.select_all("select * from db_functions
where date(created_at) >= #{conn.quote(date_p)}
AND date(created_at) <= #{conn.quote(date_p)}")
Another option, instead of converting each created_at to date in where clause, you could modify date_p to be beginning of the day value and remove the "date" conversion altogether. Also, instead of using values directly in the query, it's better to use prepared statements (Linked article is a couple of years old, but explains prepared statements clearly with examples).
Then there is also the task of modifying the date time parameter to beginning of day. Given that date_p is a string and not a time, here is what you can do:
date_p = Time.zone.parse("2014-03-20 12:46:36.590253").beginning_of_day.utc
return ActiveRecord::Base.connection.select_all("select * from db_functions
where created_at >= ?
AND created_at <= ?", date_p, date_p)
After many hours, I finally realize that I am working correctly with the Cache object in my ASP.NET application but my stored procedures stops it from working correctly.
This stored procedure works correctly:
CREATE PROCEDURE [dbo].[ListLanguages]
#Page INT = 1,
#ItemsPerPage INT = 10,
#OrderBy NVARCHAR (100) = 'ID',
#OrderDirection NVARCHAR(4) = 'DESC'
AS
BEGIN
SELECT ID, [Name], Flag, IsDefault FROM dbo.Languages
END
But this (the one I wanted) doesn't:
CREATE PROCEDURE [dbo].[ListLanguages]
#Page INT = 1,
#ItemsPerPage INT = 10,
#OrderBy NVARCHAR (100) = 'ID',
#OrderDirection NVARCHAR(4) = 'DESC',
#TotalRecords INT OUTPUT
AS
BEGIN
SET #TotalRecords = 10
EXEC('SELECT ID, Name, Flag, IsDefault FROM (
SELECT ROW_NUMBER() OVER (ORDER BY ' + #OrderBy + ' ' + #OrderDirection + ') as Row, ID, Name, Flag, IsDefault
FROM dbo.Languages) results
WHERE Row BETWEEN ((' + #Page + '-1)*' + #ItemsPerPage + '+1) AND (' + #Page + '*' + #ItemsPerPage + ')')
END
I gave the #TotalRecords parameter the value 10 so you can be sure that the problem is not from the COUNT(*) function which I know is not supported well.
Also, when I run it from SQL Server Management Studio, it does exactly what it should do. In the ASP.NET application the results are retrieved correctly, only the cache is somehow unable to work!
Can you please help?
Maybe a hint
I believe that the reason why the dependency HasChanged property is related to the fact that the column Row generated from the ROW_NUMBER is only temporary and, therefore, the SQL SERVER is not able to to say whether the results are changed or not. That's why HasChanged is always set to true.
Does anyone know how to paginate results from SQL SERVER without using COUNT or ROW_NUMBER functions?
not enough cache size.
Sql cache dependency for .NET 3.5 only works for simple queries. Maybe .NET 4 will surprise me.
1 - Can you copy & paste the code you actually use to cache the results of that sproc ?
2 - Have you tried a sproc where you use straight query instead of EXEC-ing a string ?
Yes #2 means that you can't change the structure of the query on the fly :-) but unless you are calculating your own caching criteria in #1 that's the rule of caching you have to abide by in general. No caching mechanism is ever going to parse a string from EXEC from you.
EXEC-ing a string in a sproc makes that sproc a total toss of a coin on each and every run even for SQL Server itself. It also leaves you open to script injection attacks since your query is still being composed by strings at run time - it's not any different from composing the whole string in C# and passing it to sproc to "just EXEC whatever is inside"