Related
I want to query the database but only find out if there is at least one result or not. I am trying to minimize the cost for this transaction. What would the structure be in Rails to have the query be SELECT TOP or SELECT FIRST in SQL?
You could try exists?
Person.exists?(5) # by primary key
Person.exists?(name: 'David')
Person.exists? # is there at least one row in the table?
Person.where(name: 'Spartacus', rating: 4).exists?
Person.active.exists? # if you have an "active" scope
Note that this limits the result set to 1 in the SQL query and the select clause is something like SELECT 1 AS one
I have a timestamp with time zone field named statdate and the entry looks like this 2021-11-17 12:47:54-08. I want to create a field with just the time of day expressed locally, so it would look like 12:47:54. (This data was recorded on an iPhone and it's 12:28 PST). (Go to bottom of post for solution using views from #AdrianKalver)
select *,statdate :: timestamp :: time as stattime from table
works in PGAdmin and an example result is 12:47:54 as desired. How do I make this an alter table
ALTER TABLE tablename add COLUMN stattime timestamp generated always AS (select *,statdate :: timestamp :: time as stattime from tablename) stored;
is the wrong syntax.
ALTER TABLE tablename add COLUMN stattime timestamp generated always AS ( EXTRACT(HOUR FROM statdate) || ':' || EXTRACT(MINUTE FROM statdate) || ':' || EXTRACT(SECOND FROM statdate)) stored;
ERROR: generation expression is not immutable which I'm presuming is a type problem, although postgres can concatenate strings and numbers with this syntax.
Just tried something else
ALTER TABLE tablename add COLUMN stattime timestamp generated always AS ( Cast(EXTRACT(HOUR FROM statdate) as text) || ':' || cast(EXTRACT(MINUTE FROM statdate) as text) || ':' || cast(EXTRACT(SECOND FROM statdate) as text) ) stored; -- ERROR: generation expression is not immutable
I'm using the hours and minutes for a graph and I can't get in the middle of the Chartkick. Could do it in High Charts, but think it will be simpler to create the view chart and use that. The Rails/Chartkick looks like
<%= line_chart TableName.where(statdate: start..current_date).pluck(:statdate, :y_axis) %>
and can't break that apart. So will go with creating a View Table.
What's the right way to do this? I've looked here and at the postgresql docs and not having much luck.
Following comments, the solution
CREATE OR REPLACE VIEW public.view_bp_with_time AS
SELECT
id,
statdate,
statdate :: time AS stattime,
y-axis
FROM table_name
ORDER BY statdate
Now to bring into Rails. Not as straightforward as I thought. And I'm off the computer for the next week.
Per here:
https://www.postgresql.org/docs/current/sql-createtable.html
GENERATED ALWAYS AS ( generation_expr ) STORED
This clause creates the column as a generated column. The column cannot be written to, and when read the result of the specified expression will be returned.
The keyword STORED is required to signify that the column will be computed on write and will be stored on disk.
The generation expression can refer to other columns in the table, but not other generated columns. Any functions and operators used must be immutable. References to other tables are not allowed.
Basically the cast from timestamptz to timestamp is not immutable as there are time zones involved.
For more information see:
https://www.postgresql.org/docs/14/xfunc-volatility.html
Either:
Create a view that does the conversion.
Include it in your query as you show for the pgAdmin4 example.
Create a timestamp field on the table and either add the value to that field as part of INSERT\UPDATE or add a trigger that does that.
As with all my questions at the moment, I'm working with the "Advantage Database Server" on a remote machine, which is slow, and clumsy.
It would be great if I could write a quick script to dump changes made to the "live" system into a nice PostgreSQL database.
The existing database is made up of about 30 tables, however only about 7 of these are actively updated.
I have the ones I want copied defined as models already.
The ADS tables all have a pseudo-column of "ROWID" which should stay the same within the existing database (according to the documentation) ... this is also often used as the "Primary Key" on the ADS tables except for the fact that it isn't indexed!
I'm proposing to create a new table in PostgreSQL with a copy of this data, including the pseudo-column ROWID (not a PostgreSQL reserved word, I believe), and then doing a comparison of the live ADS data to the PostgreSQL equivalent.
class Determinand << AisBase
self.table_name = 'DETS'
self.sequence_name = :autogenerated
self.primary_key = 'DET'
end
class PgDeterminand << PostgresBase
self.sequence_name = :autogenerated
self.primary_key = 'DET'
end
livet = Determinand.select("ROWID").map(&:ROWID)
devt = PgDeterminand.select("ROWID").map(&:ROWID)
new_dets = Determinand.find_by(ROWID: livet - devt)
# or maybe
(livet - devt).map do |rid|
Determinand.find_by(ROWID: rid)
end
and then loop through the new_dets to create new PgDeterminand rows ...
the reading is very slow:
puts Benchmark.measure { livet=Determinand.select("ROWID").map(&:ROWID) }
0.196957 0.098432 0.295389 ( 26.503560)
livet.count
=> 6136
and this is not a big table ...
can anyone think of a clearer way to look at doing this?
-- EDIT --
Okay, I've copied all the existing models to an "Ads" folder, created new objects in the Postgres (based on the existing schema.rb file), removed all the belongs_to from the models (no referential integrity on the AIS LIMS tables!) and I can quickly and easily copy the data to the new tables like follows:
def force_utf8 (hsh)
hsh.each_with_object({}) do |(i,j),a|
a[i]= j.present? && j.is_a?(String) ? j.encode("utf-8", invalid: :replace, undef: :replace, replace: '?') : j
end
end
Ads::Determinand.all.as_json.each do |d|
Determinand.create(force_utf8(d))
end
this isn't an incremental yet, but using the ROWID from the existing table, I should be able to work from there
-- EDIT 2 --
ROWID appears to be essentially sequential for each table ... except that it uses the order '[A-Za-z0-9+/]' ... awesome!
I was hoping to do just a "greater than last stored ROWID" for new data in the "Live" system:
Ads::Determinand.where(Ads::Determinand.arel_table['ROWID'].gt(Determinand.maximum(:ROWID))).as_json.each do |d|
Determinand.create(force_utf8(d))
end
but this obviously doesn't cope with ROWIDs after an ending "zz":
CFTquNARAXIFAAAezz is greater than CFTquNARAXIFAAAe+D
Okay, I have this mostly sorted now:
Schema Initialisation
first I moved all my models to an "Ads" directory (adding in "module Ads" to each model), set up 2 databases in my project and gathered the "existing" schema using rake db:schema:dump
then I created new models (e.g.):
rails g model Determinand
I then copied the existing model from the ads_schema.rb to the rails migration, and rake db:migrate:postgres
Initial Data Dump
I then did an initial data export/import.
On smaller tables, I was able to use the following:
Ads::Client.all.as_json.each do |c|
Client.create(c)
end
but on larger tables I had to use a CSV export from the ADS, and a pgloader script to bring in the data:
load CSV
from 'RESULTS.csv'
having fields
(
SAMPNUM, DET, JOB, GLTHAN, INPUT, OUTPUT, RESULT, ERROR, GENFLAG,
SPECFLAG, STATFLAG, COMPFLAG, REPEAT, DETORDER, ANALYST, DETDATE [date format 'DD/MM/YYYY'],
DETTIME, LOGDATE [date format 'DD/MM/YYYY'], APPROVED [date format 'DD/MM/YYYY'], APPROVEDBY, INSTRUMENT, FILENAME, LINE_NO,
TEXTRESULT, DATATYPE, SUITE, TEST, SECTION, UKAS, MCERTS, ACCRED, DEVIATING,
PRINT_1, PRINT_1_BY, PRINT_1_AT, PRINT_2, PRINT_2_BY, PRINT_2_AT, LABEL, LABLOCN
)
into postgresql://$user:$password#localhost/ads_project
TARGET TABLE results
TARGET COLUMNS
(
'SAMPNUM', 'DET', 'JOB', 'GLTHAN', 'INPUT', 'OUTPUT', 'RESULT', 'ERROR', 'GENFLAG',
'SPECFLAG', 'STATFLAG', 'COMPFLAG', 'REPEAT', 'DETORDER', 'ANALYST', 'DETDATE',
'DETTIME', 'LOGDATE', 'APPROVED', 'APPROVEDBY', 'INSTRUMENT', 'FILENAME', 'LINE_NO',
'TEXTRESULT', 'DATATYPE', 'SUITE', 'TEST', 'SECTION', 'UKAS', 'MCERTS', 'ACCRED', 'DEVIATING',
'PRINT_1', 'PRINT_1_BY', 'PRINT_1_AT', 'PRINT_2', 'PRINT_2_BY', 'PRINT_2_AT', 'LABEL', 'LABLOCN'
)
with csv header,
fields optionally enclosed by '"',
fields terminated by ',',
drop indexes
before load do
$ alter table results alter column created_at drop not null, alter column updated_at drop not null; $$
after load do
$ update results set created_at = "DETDATE", updated_at=NOW() where created_at is null and updated_at is null; $,
$ alter table results alter column created_at set not null, alter column updated_at set not null; $
;
Incremental Updates
for the incremental updates I have to do something like the following:
On smaller tables (~ <1000 rows):
Ads::DetLimit.where.not(ROWID: DetLimit.pluck(:ROWID)).as_json.each do |d|
DetLimit.create(force_utf8(d))
end
On Larger tables I need to use Ruby to limit the IDs that have changed (essentially white-list not black-list):
zzz = SuiteDet.pluck(:ROWID)
yyy = Ads::SuiteDet.pluck(:ROWID)
Ads::SuiteDet.where(ROWID: yyy-zzz).as_json.each do |d|
SuiteDet.create(force_utf8(d))
end
Deployment
I created a CopyTable script to run, so that I can batch it, with just the increments now, and it takes about 2 minutes to run, which is acceptable
I'm not familiar with ADS, but IMO it'll be a good start if you have access to modify the dsign by adding necessary indexes to it.
Also Determinand.pluck(:id) is always MUCH faster than Determinand.select("ROWID").map(&:ROWID)
I have a Rails application that holds user data (in an aptly named user_data object). I want to display a summary table that shows me the count of total users and the count of users who are still active (status = 'Active'), created each month for the past 12 months.
In SQL against my Postgres database, I can get the result I want with the following query (the date I use in there is calculated by the application, so you can ignore that aspect):
SELECT total.creation_month,
total.user_count AS total_count,
active.user_count AS active_count
FROM
(SELECT date_trunc('month',"creationDate") AS creation_month,
COUNT("userId") AS user_count
FROM user_data
WHERE "creationDate" >= to_date('2015 12 21', 'YYYY MM DD')
GROUP BY creation_month) AS total
LEFT JOIN
(SELECT date_trunc('month',"creationDate") AS creation_month,
COUNT("userId") AS user_count
FROM user_data
WHERE "creationDate" >= to_date('2015 12 21', 'YYYY MM DD')
AND status = 'Active'
GROUP BY creation_month) AS active
ON total.creation_month = active.creation_month
ORDER BY creation_month ASC
How do I write this query with ActiveRecord?
I previously had just the total user count grouped by month in my display, but I am struggling with how to add in the additional column of active user counts.
My application is on Ruby 2.1.4 and Rails 4.1.6.
I gave up on trying to do this the ActiveRecord way. Instead I just constructed my query into a string and passed the string into
ActiveRecord::Base.connection.execute(sql_string)
This had the side effect that my result set came out as a array instead of a set of objects. So getting at the values went from a syntax (where user_data is the name assigned to a single record from the result set) like
user_data.total_count
to
user_data['total_count']
But that's a minor issue. Not worth the hassle.
In my Rails app I've run into an issue a couple times that I'd like to know how other people solve:
I have certain records where a value is optional, so some records have a value and some are null for that column.
If I order by that column on some databases the nulls sort first and on some databases the nulls sort last.
For instance, I have Photos which may or may not belong to a Collection, ie there are some Photos where collection_id=nil and some where collection_id=1 etc.
If I do Photo.order('collection_id desc) then on SQLite I get the nulls last but on PostgreSQL I get the nulls first.
Is there a nice, standard Rails way to handle this and get consistent performance across any database?
I'm no expert at SQL, but why not just sort by if something is null first then sort by how you wanted to sort it.
Photo.order('collection_id IS NULL, collection_id DESC') # Null's last
Photo.order('collection_id IS NOT NULL, collection_id DESC') # Null's first
If you are only using PostgreSQL, you can also do this
Photo.order('collection_id DESC NULLS LAST') #Null's Last
Photo.order('collection_id DESC NULLS FIRST') #Null's First
If you want something universal (like you're using the same query across several databases, you can use (courtesy of #philT)
Photo.order('CASE WHEN collection_id IS NULL THEN 1 ELSE 0 END, collection_id')
Even though it's 2017 now, there is still yet to be a consensus on whether NULLs should take precedence. Without you being explicit about it, your results are going to vary depending on the DBMS.
The standard doesn't specify how NULLs should be ordered in comparison with non-NULL values, except that any two NULLs are to be considered equally ordered, and that NULLs should sort either above or below all non-NULL values.
source, comparison of most DBMSs
To illustrate the problem, I compiled a list of a few most popular cases when it comes to Rails development:
PostgreSQL
NULLs have the highest value.
By default, null values sort as if larger than any non-null value.
source: PostgreSQL documentation
MySQL
NULLs have the lowest value.
When doing an ORDER BY, NULL values are presented first if you do ORDER BY ... ASC and last if you do ORDER BY ... DESC.
source: MySQL documentation
SQLite
NULLs have the lowest value.
A row with a NULL value is higher than rows with regular values in ascending order, and it is reversed for descending order.
source
Solution
Unfortunately, Rails itself doesn't provide a solution for it yet.
PostgreSQL specific
For PostgreSQL you could quite intuitively use:
Photo.order('collection_id DESC NULLS LAST') # NULLs come last
MySQL specific
For MySQL, you could put the minus sign upfront, yet this feature seems to be undocumented. Appears to work not only with numerical values, but with dates as well.
Photo.order('-collection_id DESC') # NULLs come last
PostgreSQL and MySQL specific
To cover both of them, this appears to work:
Photo.order('collection_id IS NULL, collection_id DESC') # NULLs come last
Still, this one does not work in SQLite.
Universal solution
To provide cross-support for all DBMSs you'd have to write a query using CASE, already suggested by #PhilIT:
Photo.order('CASE WHEN collection_id IS NULL THEN 1 ELSE 0 END, collection_id')
which translates to first sorting each of the records first by CASE results (by default ascending order, which means NULL values will be the last ones), second by calculation_id.
Photo.order('collection_id DESC NULLS LAST')
I know this is an old one but I just found this snippet and it works for me.
Put minus sign in front of column_name and reverse the order direction. It works on mysql. More details
Product.order('something_date ASC') # NULLS came first
Product.order('-something_date DESC') # NULLS came last
Bit late to the show but there is a generic SQL way to do it. As usual, CASE to the rescue.
Photo.order('CASE WHEN collection_id IS NULL THEN 1 ELSE 0 END, collection_id')
The easiest way is to use:
.order('name nulls first')
For posterity's sake, I wanted to highlight an ActiveRecord error relating to NULLS FIRST.
If you try to call:
Model.scope_with_nulls_first.last
Rails will attempt to call reverse_order.first, and reverse_order is not compatible with NULLS LAST, as it tries to generate the invalid SQL:
PG::SyntaxError: ERROR: syntax error at or near "DESC"
LINE 1: ...dents" ORDER BY table_column DESC NULLS LAST DESC LIMIT...
This was referenced a few years ago in some still-open Rails issues (one, two, three). I was able to work around it by doing the following:
scope :nulls_first, -> { order("table_column IS NOT NULL") }
scope :meaningfully_ordered, -> { nulls_first.order("table_column ASC") }
It appears that by chaining the two orders together, valid SQL gets generated:
Model Load (12.0ms) SELECT "models".* FROM "models" ORDER BY table_column IS NULL DESC, table_column ASC LIMIT 1
The only downside is that this chaining has to be done for each scope.
Rails 6.1 adds nulls_first and nulls_last methods to Arel for PostgreSQL.
Example:
User.order(User.arel_table[:login_count].desc.nulls_last)
Source: https://www.bigbinary.com/blog/rails-6-1-adds-nulls-first-and-nulls-last-to-arel
Here are some Rails 6 solutions.
The answer by #Adam Sibik is a great summary about the difference between various database systems.
Unfortunately, though, some of the presented solutions, including "Universal solution" and "PostgreSQL and MySQL specific", would not work any more with Rails 6 (ActiveRecord 6) as a result of its changed specification of order() not accepting some raw SQLs (I confirm the "PostgreSQL specific" solution still works as of Rails 6.1.4). For the background of this change, see, for example,
"Updates for SQL Injection in Rails 6.1" by Justin.
To circumvent the problem, you can wrap around the SQL statements with Arel.sql as follows, where NULLs come last, providing you are 100% sure the SQL statements you give are safe.
Photo.order(Arel.sql('CASE WHEN collection_id IS NULL THEN 1 ELSE 0 END, collection_id'))
Just for reference, if you want to sort by a Boolean column (is_ok, as an example) in the order of [TRUE, FALSE, NULL] regardless of the database systems, either of these should work:
Photo.order(Arel.sql('CASE WHEN is_ok IS NULL THEN 1 ELSE 0 END, is_ok DESC'))
Photo.order(Arel.sql('CASE WHEN is_ok IS NULL THEN 1 WHEN is_ok IS TRUE THEN -1 ELSE 0 END'))
(n.b., SQLite does not have the Boolean type and so the former may be safer arguably, though it should not matter because Rails should guarantee the value is either 0 or 1 (or NULL).)
In my case I needed sort lines by start and end date by ASC, but in few cases end_date was null and that lines should be in above, I used
#invoice.invoice_lines.order('start_date ASC, end_date ASC NULLS FIRST')
Adding arrays together will preserve order:
#nonull = Photo.where("collection_id is not null").order("collection_id desc")
#yesnull = Photo.where("collection_id is null")
#wanted = #nonull+#yesnull
http://www.ruby-doc.org/core/classes/Array.html#M000271
It seems like you'd have to do it in Ruby if you want consistent results across database types, as the database itself interprets whether or not the NULLS go at the front or end of the list.
Photo.all.sort {|a, b| a.collection_id.to_i <=> b.collection_id.to_i}
But that is not very efficient.