I have been looking for a solution to this problem since couple of hours now. It seems very simple but for some reason I can't get it to work.
I have a model, lets call it MyModel. It has 3 columns, :id, :column_a, :relationship_id
:relationship_id is a foreign key with not null constraint.
I have 4 records in the database already like such:-
:id, :column_a, :relationship_id,
1, 1, 7
2, 2, 8
3, 3, 9
4, 4, 10
I want to change the values of :column_a, using the following data
data = [{"id"=>"1", "column_a"=>4},
{"id"=>"2", "column_a"=>3},
{"id"=>"3", "column_a"=>2},
{"id"=>"4", "column_a"=>1}]
But this data is missing :relationship_id column values which I want to keep unchanged.
When I do something like MyModel.upsert_all(data, update_only: :column_a) I get the following error
ActiveRecord::NotNullViolation: PG::NotNullViolation: ERROR: null value in column "relationship_id" of relation "my_models" violates not-null constraint
DETAIL: Failing row contains (1, 4, null).
I have tried update_all and raw SQL but the problem remains the same. The only way I can get it working is by individually updating each record which is not efficient.
Maybe I am doing something wrong? Any ideas?
Related
This issue started happening once I upgraded Rails from 6.0.3.7 to 6.1.5.
will_paginate 3.1.8
I have a query that looks for records and it's paginated.
When I do something like
forms = Form.paginate(:page => page).order('forms.id desc')
if I inspect the returned items I notice that a raw_rnum_ attribute is added to each record. I suppose this is will_paginate's doing since that attr doesn't exist in my db. The value in each record is a sequential integer. So if the query returns three records we'll have the values 1, 2 and 3 as that attribute's values.
forms.map(&:raw_rnum_) #=> [1, 2, 3)
When the other bits of the query are put together I end up with a longer query and a condition at the end that tells which records to load (I assume based on the page size)
...AND "FORMS"."ID" IN (1,2,3) ORDER BY forms.id desc
This is causing my query to fail, since the actual ids of the items are not 1, 2 and 3.
This only happens if I upgrade rails as mentioned. Tried upgrading will_paginate to its latest, but the problem remains.
Before the upgrade of rails the SQL produced was the same. Except that in that case the form ids IN (...) was correct with the actual ids of the returned records and the values were obfuscated from the SQL.
...AND "FORMS"."ID" IN (:a4, :a5, :a6) ORDER BY forms.id desc ...["id", 22], ["id", 21], ["id", 1]]
The issue happens when I further my query to something my complex. Then the records are no longer returned.
The actual query object seems to built correctly, the count attribute in the query object seems correct, but the actual results don't return correctly.
Any ideas?
I need to sort a postgres table by the "more recent of column A, fallback to column B"
If my table looks like this: id, reminder_at, updated_at
1, 01-11-2019, 12-01-2018
2, null, 01-04-2019
3, null, 01-02-2019
4, 01-01-2019, 01-04-2019
expected sorting output would be
4, 01-01-2019, 01-04-2019 # 01-01-2019 is soonest
3, null, 01-02-2019 # then 01-02-2019
2, null, 01-04-2019 # then 01-04-2019
1, 01-11-2019, 12-01-2018 # then 01-11-2019
I'm currently doing this with application code, and i'd prefer to do in SQL
For example if the reminder_at went to NULL for record 1, then it would immediately go to the top because the updated_at date is the oldest
Currently:
SELECT *
FROM "tasks"
WHERE completed_at IS NULL
ORDER by reminder_at, updated_at
EDIT with Correct Answer:
SELECT *
FROM "tasks"
WHERE completed_at IS NULL
ORDER by COALESCE(reminder_at, updated_at)
use coalesce. It chooses the first non null value.
select * from tab
order by coalesce(col1, col2)
if instead, you wanted to use the greater of the 2 dates.. then use greatest()
select * from tab
order by greatest(col1, col2)
I am interpreting your data format to be mm-dd. Should 3 and 2 be flipped in your output, the second coming after the fourth?
Does the following work?
select id,reminder_at,updated_at,greatest(coalesce(reminder_at,'1/1/1900'),coalesce(updated_at,'1/1/1900')) as testcolumn from test order by greatest(coalesce(reminder_at,'1/1/1900'),coalesce(updated_at,'1/1/1900'));
I have table with column position, which in some cases, for some collection of records can be nil. I have default order options like
order('positions ASC')
id| name | position
1 5 null
2 6 null
3 7 null
If for some collection that I sort (example above), all values have null in position column, in which order I will get this collection from db?
I'm suggestion I will get collection in order of ids (1,2,3). Am I correct?
Addition #1: DB - Postgresql
According Postgres manual, if no sorting clause the records are returned according with physical position at the disk. It says nothing for sorted records with equal values on sort fields. But, it uses b-tree and, like clasic db managers, it must return on the order stored at the b-tree. You must expect that each of this change on db reorganization.
At the end, there are no warranty on the order of records with same values on sort fields.
Note: using Postgres you can make the NULL values at the first or the last (it is detailed at the referrer link).
At this related question, I'm agree with #macek.
You can do something like this.
Cats:
id| name | position
1 5 null
2 6 null
3 7 not_null
nil = Cat.order("id ASC").where(position: nil) = [1, 2]
not_nil = Cat.order("id ASC").where("position is not null") = [3]
not_nil + nil = [3, 1, 2]
This preserves order.
I am getting this error in the pg production mode, but its working fine in sqlite3 development mode.
ActiveRecord::StatementInvalid in ManagementController#index
PG::Error: ERROR: column "estates.id" must appear in the GROUP BY clause or be used in an aggregate function
LINE 1: SELECT "estates".* FROM "estates" WHERE "estates"."Mgmt" = ...
^
: SELECT "estates".* FROM "estates" WHERE "estates"."Mgmt" = 'Mazzey' GROUP BY user_id
#myestate = Estate.where(:Mgmt => current_user.Company).group(:user_id).all
If user_id is the PRIMARY KEY then you need to upgrade PostgreSQL; newer versions will correctly handle grouping by the primary key.
If user_id is neither unique nor the primary key for the 'estates' relation in question, then this query doesn't make much sense, since PostgreSQL has no way to know which value to return for each column of estates where multiple rows share the same user_id. You must use an aggregate function that expresses what you want, like min, max, avg, string_agg, array_agg, etc or add the column(s) of interest to the GROUP BY.
Alternately you can rephrase the query to use DISTINCT ON and an ORDER BY if you really do want to pick a somewhat arbitrary row, though I really doubt it's possible to express that via ActiveRecord.
Some databases - including SQLite and MySQL - will just pick an arbitrary row. This is considered incorrect and unsafe by the PostgreSQL team, so PostgreSQL follows the SQL standard and considers such queries to be errors.
If you have:
col1 col2
fred 42
bob 9
fred 44
fred 99
and you do:
SELECT col1, col2 FROM mytable GROUP BY col1;
then it's obvious that you should get the row:
bob 9
but what about the result for fred? There is no single correct answer to pick, so the database will refuse to execute such unsafe queries. If you wanted the greatest col2 for any col1 you'd use the max aggregate:
SELECT col1, max(col2) AS max_col2 FROM mytable GROUP BY col1;
I recently moved from MySQL to PostgreSQL and encountered the same issue. Just for reference, the best approach I've found is to use DISTINCT ON as suggested in this SO answer:
Elegant PostgreSQL Group by for Ruby on Rails / ActiveRecord
This will let you get one record for each unique value in your chosen column that matches the other query conditions:
MyModel.where(:some_col => value).select("DISTINCT ON (unique_col) *")
I prefer DISTINCT ON because I can still get all the other column values in the row. DISTINCT alone will only return the value of that specific column.
After often receiving the error myself I realised that Rails (I am using rails 4) automatically adds an 'order by id' at the end of your grouping query. This often results in the error above. So make sure you append your own .order(:group_by_column) at the end of your Rails query. Hence you will have something like this:
#problems = Problem.select('problems.username, sum(problems.weight) as weight_sum').group('problems.username').order('problems.username')
#myestate1 = Estate.where(:Mgmt => current_user.Company)
#myestate = #myestate1.select("DISTINCT(user_id)")
this is what I did.
In my Rails app I've run into an issue a couple times that I'd like to know how other people solve:
I have certain records where a value is optional, so some records have a value and some are null for that column.
If I order by that column on some databases the nulls sort first and on some databases the nulls sort last.
For instance, I have Photos which may or may not belong to a Collection, ie there are some Photos where collection_id=nil and some where collection_id=1 etc.
If I do Photo.order('collection_id desc) then on SQLite I get the nulls last but on PostgreSQL I get the nulls first.
Is there a nice, standard Rails way to handle this and get consistent performance across any database?
I'm no expert at SQL, but why not just sort by if something is null first then sort by how you wanted to sort it.
Photo.order('collection_id IS NULL, collection_id DESC') # Null's last
Photo.order('collection_id IS NOT NULL, collection_id DESC') # Null's first
If you are only using PostgreSQL, you can also do this
Photo.order('collection_id DESC NULLS LAST') #Null's Last
Photo.order('collection_id DESC NULLS FIRST') #Null's First
If you want something universal (like you're using the same query across several databases, you can use (courtesy of #philT)
Photo.order('CASE WHEN collection_id IS NULL THEN 1 ELSE 0 END, collection_id')
Even though it's 2017 now, there is still yet to be a consensus on whether NULLs should take precedence. Without you being explicit about it, your results are going to vary depending on the DBMS.
The standard doesn't specify how NULLs should be ordered in comparison with non-NULL values, except that any two NULLs are to be considered equally ordered, and that NULLs should sort either above or below all non-NULL values.
source, comparison of most DBMSs
To illustrate the problem, I compiled a list of a few most popular cases when it comes to Rails development:
PostgreSQL
NULLs have the highest value.
By default, null values sort as if larger than any non-null value.
source: PostgreSQL documentation
MySQL
NULLs have the lowest value.
When doing an ORDER BY, NULL values are presented first if you do ORDER BY ... ASC and last if you do ORDER BY ... DESC.
source: MySQL documentation
SQLite
NULLs have the lowest value.
A row with a NULL value is higher than rows with regular values in ascending order, and it is reversed for descending order.
source
Solution
Unfortunately, Rails itself doesn't provide a solution for it yet.
PostgreSQL specific
For PostgreSQL you could quite intuitively use:
Photo.order('collection_id DESC NULLS LAST') # NULLs come last
MySQL specific
For MySQL, you could put the minus sign upfront, yet this feature seems to be undocumented. Appears to work not only with numerical values, but with dates as well.
Photo.order('-collection_id DESC') # NULLs come last
PostgreSQL and MySQL specific
To cover both of them, this appears to work:
Photo.order('collection_id IS NULL, collection_id DESC') # NULLs come last
Still, this one does not work in SQLite.
Universal solution
To provide cross-support for all DBMSs you'd have to write a query using CASE, already suggested by #PhilIT:
Photo.order('CASE WHEN collection_id IS NULL THEN 1 ELSE 0 END, collection_id')
which translates to first sorting each of the records first by CASE results (by default ascending order, which means NULL values will be the last ones), second by calculation_id.
Photo.order('collection_id DESC NULLS LAST')
I know this is an old one but I just found this snippet and it works for me.
Put minus sign in front of column_name and reverse the order direction. It works on mysql. More details
Product.order('something_date ASC') # NULLS came first
Product.order('-something_date DESC') # NULLS came last
Bit late to the show but there is a generic SQL way to do it. As usual, CASE to the rescue.
Photo.order('CASE WHEN collection_id IS NULL THEN 1 ELSE 0 END, collection_id')
The easiest way is to use:
.order('name nulls first')
For posterity's sake, I wanted to highlight an ActiveRecord error relating to NULLS FIRST.
If you try to call:
Model.scope_with_nulls_first.last
Rails will attempt to call reverse_order.first, and reverse_order is not compatible with NULLS LAST, as it tries to generate the invalid SQL:
PG::SyntaxError: ERROR: syntax error at or near "DESC"
LINE 1: ...dents" ORDER BY table_column DESC NULLS LAST DESC LIMIT...
This was referenced a few years ago in some still-open Rails issues (one, two, three). I was able to work around it by doing the following:
scope :nulls_first, -> { order("table_column IS NOT NULL") }
scope :meaningfully_ordered, -> { nulls_first.order("table_column ASC") }
It appears that by chaining the two orders together, valid SQL gets generated:
Model Load (12.0ms) SELECT "models".* FROM "models" ORDER BY table_column IS NULL DESC, table_column ASC LIMIT 1
The only downside is that this chaining has to be done for each scope.
Rails 6.1 adds nulls_first and nulls_last methods to Arel for PostgreSQL.
Example:
User.order(User.arel_table[:login_count].desc.nulls_last)
Source: https://www.bigbinary.com/blog/rails-6-1-adds-nulls-first-and-nulls-last-to-arel
Here are some Rails 6 solutions.
The answer by #Adam Sibik is a great summary about the difference between various database systems.
Unfortunately, though, some of the presented solutions, including "Universal solution" and "PostgreSQL and MySQL specific", would not work any more with Rails 6 (ActiveRecord 6) as a result of its changed specification of order() not accepting some raw SQLs (I confirm the "PostgreSQL specific" solution still works as of Rails 6.1.4). For the background of this change, see, for example,
"Updates for SQL Injection in Rails 6.1" by Justin.
To circumvent the problem, you can wrap around the SQL statements with Arel.sql as follows, where NULLs come last, providing you are 100% sure the SQL statements you give are safe.
Photo.order(Arel.sql('CASE WHEN collection_id IS NULL THEN 1 ELSE 0 END, collection_id'))
Just for reference, if you want to sort by a Boolean column (is_ok, as an example) in the order of [TRUE, FALSE, NULL] regardless of the database systems, either of these should work:
Photo.order(Arel.sql('CASE WHEN is_ok IS NULL THEN 1 ELSE 0 END, is_ok DESC'))
Photo.order(Arel.sql('CASE WHEN is_ok IS NULL THEN 1 WHEN is_ok IS TRUE THEN -1 ELSE 0 END'))
(n.b., SQLite does not have the Boolean type and so the former may be safer arguably, though it should not matter because Rails should guarantee the value is either 0 or 1 (or NULL).)
In my case I needed sort lines by start and end date by ASC, but in few cases end_date was null and that lines should be in above, I used
#invoice.invoice_lines.order('start_date ASC, end_date ASC NULLS FIRST')
Adding arrays together will preserve order:
#nonull = Photo.where("collection_id is not null").order("collection_id desc")
#yesnull = Photo.where("collection_id is null")
#wanted = #nonull+#yesnull
http://www.ruby-doc.org/core/classes/Array.html#M000271
It seems like you'd have to do it in Ruby if you want consistent results across database types, as the database itself interprets whether or not the NULLS go at the front or end of the list.
Photo.all.sort {|a, b| a.collection_id.to_i <=> b.collection_id.to_i}
But that is not very efficient.