I need to set a combined index on two fields from a JSONB in my PostgreSQL DB. I can set an index for a single field like so (using ActiveRecord in my Rails 6 application):
add_index :my_table,
"(content->'reference')",
using: :gin,
name: 'index_my_table_on_content_reference'
This one works as expected. However, when I try to set a combined index for two fields, I get the following error:
add_index :my_table,
["(content->'reference')", "(content->'ext_id')"],
using: :gin,
name: 'index_my_table_on_content_ref_and_ext_id'
ActiveRecord::StatementInvalid: PG::UndefinedColumn: ERROR: column "(content->'reference')" does not exist
What am I doing wrong and how can I create a combined index for multiple fields in a JSONB column?
And before you ask: Yes, each JSONB blob has a key named reference.
Using: Ruby 2.6.5, Rails 6.0, PostgreSQL 11
The error PG::UndefinedColumn: ERROR: column "(content->'reference')" does not exist means it's treating "(content->'reference')" as a column name.
Reproducing from SQL:
create index my_table_json_idx on my_table using gin(
(content->'reference'),
"(content->'ext_id')"
);
Note the quotes around the second expression.
It seems there is a problem with your ActiveRecord library and it is escaping your jsonb expression when it isn't desirable.
Either use plain SQL or try to make your ActiveRecord library not escape your expression.
After some Googling I think changing the array to string "(content->'reference'), (content->'ext_id')" might work.
Ya Rails doesn't know how to dump your jsonb index using ruby. So SQL would be the way to go. This was a pretty good article for me when I started messing with jsonb Rails & jsonb
def change
reversible do |dir|
dir.up do
execute <<-SQL
CREATE INDEX IF NOT EXISTS idx_my_table_on_content_references on my_table using gin ((content->>'reference'));
SQL
end
dir.down {execute "DROP INDEX IF EXISTS idx_my_table_on_content_references;"}
end
end
Related
Error : PG::FeatureNotSupported: ERROR: cannot use column reference in DEFAULT expression
LINE 1: ..._at" timestamp, "total" decimal DEFAULT (COALESCE(price, (0)...
class AddTotalToOrderLines < ActiveRecord::Migration[6.1]
def up
execute <<~SQL
ALTER TABLE order_lines
ADD COLUMN total numeric GENERATED ALWAYS AS (COALESCE(price, 0) *
COALESCE(quantity, 0)) STORED;
SQL
end
Maybe you have another migration that's actually generating the error. The one you're showing there isn't :)
I'm trying to find every items that has been set to a specifique status last year.
I'm using Rail 5 with audited so I created a specific Audit Model and I try to write a scope to return my condition :
Audit = Audited.audit_class
class Audit
scope :followupRebus, -> { where(auditable_type: 'Followup')
.where(action: 'update')
.where("audited_changes LIKE '%step_id:[#{Step::REBUS}%'")
}
end
the content of the audited text field in postgres looks like this when I take it and show it with a .to_s
{"step_id"=>[9, 4], "active"=>[false, true]}
How can I get all audit with step_id = 9 ?
EDIT
Great thanks to DRSE, I finally found a working solution :
changing the default TEXT type of the Column with the migration sent by DRSE
Change the request like this :
class Audit
scope :followupRebus, -> { where(auditable_type: 'Followup')
.where(action: 'update')
.where("((audited_changes -> 'step_id')::json->>0)::int = :step_id", step_id: Step::REBUS)
}
end
You need to use Postgres JSON functions to query the JSON column audited_changes instead of the LIKE operator.
To find audits where the step_id was changed, you can use
.where("(audited_changes -> 'step_id')::jsonb ? :step_id", step_id: '9')
Note the use of the named bind variable :step_id, instead of using active record question mark (?) replacement because Postgres uses the question mark as a JSON query operator.
The clause above will find any audits where step_id = 9, whether that was the value set in the previous version or the updated version of your model.
To find audits where step_id = 9 in the the previous version:
# Check if the first value (indexed by 0) in 'step_id' array is '9'
.where("(audited_changes -> 'step_id')::jsonb->0 ? :step_id", step_id: '9')
To find audits with step_id = 9 set in the updated version:
# Check if the second value (indexed by 1) in 'step_id' array is 9
.where("(audited_changes -> 'step_id')::jsonb->1 ? :step_id", step_id: '9')
(Note: you should not directly string interpolate your conditions for your where clause because you are opening your app to a SQL injection vulnerability. Use rails style conditions with sanitized inputs instead.)
EDIT
Since you have indicated that your audited_changes column type is TEXT and not a JSON type, you will need to either run a migration to change the column type or else cast the column in the query.
To cast the column to JSON at query execution time, use audited_changes::json, so the example would be like this:
.where("(audited_changes::json -> 'step_id')::json ? :step_id", step_id: '9')
To change the column to JSON, start with rails g migration ChangeAuditedChangesColumnToJSONB. Then in your migration file (db/migrate/{timestamp}_change_audited_changes_column_to_jsonb.rb) write:
class ChangeAuditedChangesColumnToJSONB < ActiveRecord::Migration[5.1]
def up
change_column :audits, :audited_changes, :jsonb, using: 'audited_changes::JSONB'
end
def down
change_column :audits, :audited_changes, :text
end
end
Then run rails db:migrate and you should be good to go.
For a new project using Audited, you can add a param to the install step to specify the use of JSON or JSONB type for the audited_changes column.
rails generate audited:install --audited-changes-column-type jsonb # or json
In Rails, how can I add an index in a migration to a Postgres database with a specific sort order?
Ultimately wanting to pull off this query:
CREATE INDEX index_apps_kind_release_date_rating ON apps(kind, itunes_release_date DESC, rating_count DESC);
But right now in my migration I have this:
add_index :apps, [:kind, 'itunes_release_date desc', 'rating_count desc'], name: 'index_apps_kind_release_date_rating'
Which spits out this:
CREATE INDEX "index_apps_kind_release_date_rating" ON "apps" ("kind", "itunes_release_date desc", "rating_count desc")
Which errors out:
PG::Error: ERROR: column "itunes_release_date desc" does not exist
Rails now supports specifying index order in migrations:
def change
add_index(:accounts, [:branch_id, :party_id, :surname], order: {branch_id: :desc, party_id: :asc})
end
From http://apidock.com/rails/ActiveRecord/ConnectionAdapters/SchemaStatements/add_index
You do not need to specify DESC in the index. It will give a small speed benefit for the queries, that use this particular ordering, but in general - an index can be used for any oredring of a column.
Looks like Rails doesn't support ordered indexes.
I suspect that you can safely remove the two desc, btw: kind is in your where clause based on your previous question, so PG should happily look up the index in reverse order.
I have a page which simply displays all of the Links in my database sorted by the voteCount. Here is the controller:
class PagesController < ApplicationController
def index
params[:per_page] ||= 5
params[:page] ||= 1
#links = Link.order('voteCount DESC').paginate(:page => params[:page], :per_page => params[:per_page])
end
end
I save query the database using the paginate plugin, and prepend it with:
.order('voteCount DESC')
When I run this command on my local server, it runs fine. However, as soon as I deploy it to heroku, it fails. This is the output I get when I check the logs/execute it in the console:
Link Load (2.0ms) SELECT "links".* FROM "links" ORDER BY voteCount DESC LIMIT 5 OFFSET 0
ActiveRecord::StatementInvalid: PG::Error: ERROR: column "votecount" does not exist
LINE 1: SELECT "links".* FROM "links" ORDER BY voteCount DESC LIMI...
^
I've checked using the console, the voteCount column is definitely there. This might be due to the fact that my local environment runs sqlite3 and heroku makes me use postgres ....
Any help would be really appreciated. Thanks.
You have a case sensitivity problem. When you say something like:
create_table :links do |t|
t.integer :voteCount
end
Rails will send SQL like this to PostgreSQL to create the column:
"voteCount" integer
The double quotes around the identifier make it case sensitive and that means that you have to refer to it as "voteCount" forevermore. If you say this:
.order('"voteCount" DESC')
everything will should work.
SQLite doesn't care about identifier case so you could say voteCount, votecount, VOTECOUNT, ... in SQLite and it wouldn't matter.
Rails always quotes the identifiers that it produces when talking to PostgreSQL. Normally this doesn't matter, the Rails convention is to use lower case identifiers and PostgreSQL folds unquoted identifiers to lower case before using them so everything works by default.
There are some relevant best practices here:
Develop and deploy on the same stack.
Use lower case column names with words separated by underscores with PostgreSQL. This also happens to match the usual Ruby conventions.
I'd recommend that you use vote_count as the column name instead.
your heroku app probably doesn't have that column migrated yet
heroku run rake db:migrate
I was debugging a Ruby on Rails engine which has problems when running on Sqlite, it has a problem in finding records that the app itself creates. When run on MySQL everything works but the same query on SQLite is failing.
I've tracked down the issue and I found that the problem is in a simple WHERE query which won't find the created record. Essentially the table structure has a column called key which stores some md5 hashes. The failing spec insert a record with a given hash then on the following instruction do a SELECT query for the same hash, but SQLite returns no record for the same key. I've extracted the generated database and the failing query from the app and this is a copy of the app database:
http://dl.dropbox.com/u/2289657/combustion_test.sqlite
Here is a transcript of the queries executed by the software (made with the command line utility):
# Here I'm selecting all the records from the table
# there is a single record in it, the key is the third field
$ sqlite3 combustion_test.sqlite 'SELECT * FROM tr8n_translation_keys'
1||b56c67d10759f8012aff28fc03f26cbf|Hello World|We must start with this sentence!||||en-US|0|2012-03-14 11:49:50.335322|2012-03-14 11:49:50.335322|
# Here I'm selecting the record with that key and it doesn't return anything
$ sqlite3 combustion_test.sqlite "SELECT * FROM tr8n_translation_keys WHERE key = 'b56c67d10759f8012aff28fc03f26cbf'"
# Here I'selecting the record with a LIKE clause and it finds the record
$ sqlite3 combustion_test.sqlite "SELECT * FROM tr8n_translation_keys WHERE key LIKE 'b56c67d10759f8012aff28fc03f26cbf'"
1||b56c67d10759f8012aff28fc03f26cbf|Hello World|We must start with this sentence!||||en-US|0|2012-03-14 11:49:50.335322|2012-03-14 11:49:50.335322|
Should I report this as a bug to SQLite site?
P.S. I've tried also on a different system with a different SQLite version, but the results are the same.
Update
Here is the table schema
sqlite> .schema tr8n_translation_keys
CREATE TABLE "tr8n_translation_keys" (
"id" INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
"type" varchar(255), "key" varchar(255) NOT NULL,
"label" text NOT NULL,
"description" text,
"verified_at" datetime,
"translation_count" integer,
"admin" boolean,
"locale" varchar(255),
"level" integer DEFAULT 0,
"created_at" datetime,
"updated_at" datetime,
"synced_at" datetime
);
CREATE UNIQUE INDEX "index_tr8n_translation_keys_on_key" ON "tr8n_translation_keys" ("key");
CREATE INDEX "index_tr8n_translation_keys_on_synced_at" ON "tr8n_translation_keys" ("synced_at");
Update 2
Here is the rails code which compute the key value inserted into the table (I've removed some code, full method is here)
def self.find_or_create(label, desc = "", options = {})
key = generate_key(label, desc).to_s
# IF I UNCOMMENT THIS LINE EVERYTHING WORKS
#key = 'b56c67d10759f8012aff28fc03f26cbf'
tkey = Tr8n::Cache.fetch("translation_key_#{key}") do
existing_key = where(:key => key).first ### THIS IS THE FAILING WHERE
existing_key ||= begin
new_tkey = create(:key => key.to_s,
:label => label,
:description => desc,
:locale => locale,
:level => level,
:admin => Tr8n::Config.block_options[:admin])
# rest of method...
And here is the generate_key method, the comment about sqlite is from author, not mine)
def self.generate_key(label, desc = "")
# TODO: there is something iffy going on with the strings from the hash
# without the extra ~ = the strings are not seen in the sqlite database - wtf?
"#{Digest::MD5.hexdigest("#{label};;;#{desc}")}"
end
This works:
SELECT * FROM tr8n_translation_keys WHERE LOWER(key)='b56c67d10759f8012aff28fc03f26cbf';
But this doesn't:
SELECT * FROM tr8n_translation_keys WHERE key='b56c67d10759f8012aff28fc03f26cbf' COLLATE NOCASE;
When I examine the database in SQLiteManager, it shows the key as this:
X'6235366336376431303735396638303132616666323866633033663236636266'
which implies it's treating the key as a BLOB (raw binary data) rather than TEXT. This is why the comparison fails. But LOWER(key) causes the field to be cast to text, hence the comparison succeeds.
So, we need to find out why the entry has been stored as a BLOB instead of TEXT. How were these values inserted into the database?
Following your update 2: I'm not a Ruby expert, but the value returned from generate_key is not being converted to a string in the way you expect. Try to_str instead of to_s when calling generate_key.
Based on the following Stack Overflow answer...
https://stackoverflow.com/a/6591427/18064
... you might want to update the generation of your key as follows:
def self.generate_key(label, desc = "")
# TODO: there is something iffy going on with the strings from the hash
# without the extra ~ = the strings are not seen in the sqlite database - wtf?
"#{Digest::MD5.hexdigest("#{label};;;#{desc}").encode('UTF-8')}"
end
Note the addition of .encode('UTF-8').
This worked for me when I had the same problem as yourself.