I have a rails app that uses postgresql 12. Recently, I wrote some tests and saw some strange behavior.
I have a countries table. Its schema looks like that:
qq2_test=# \d countries;
Table "public.countries"
Column | Type | Collation | Nullable | Default
-----------------------+--------------------------------+-----------+----------+---------------------------------------
id | bigint | | not null | nextval('countries_id_seq'::regclass)
domain | character varying | | not null | ''::character varying
root_city_id | bigint | | |
language_id | bigint | | not null |
currency_id | bigint | | not null |
google_tag_manager_id | character varying | | not null | ''::character varying
created_at | timestamp(6) without time zone | | not null |
updated_at | timestamp(6) without time zone | | not null |
Indexes:
"countries_pkey" PRIMARY KEY, btree (id)
"index_countries_on_domain" UNIQUE, btree (domain)
"index_countries_on_currency_id" btree (currency_id)
"index_countries_on_language_id" btree (language_id)
Foreign-key constraints:
"fk_rails_6f479b409c" FOREIGN KEY (language_id) REFERENCES languages(id)
"fk_rails_7cac1212c7" FOREIGN KEY (currency_id) REFERENCES currencies(id)
"fk_rails_beac36a0bd" FOREIGN KEY (root_city_id) REFERENCES cities(id)
Referenced by:
TABLE "country_translations" CONSTRAINT "fk_rails_0c4ee35f26" FOREIGN KEY (country_id) REFERENCES countries(id) ON DELETE CASCADE
TABLE "countries_languages" CONSTRAINT "fk_rails_556e7398aa" FOREIGN KEY (country_id) REFERENCES countries(id) ON DELETE CASCADE
TABLE "cities" CONSTRAINT "fk_rails_996e05be41" FOREIGN KEY (country_id) REFERENCES countries(id)
As you can see, I have foreign key constraints on both currency_id and language_id fields.
When I run my tests I see that there are records in that table:
qq2_test=# select * from countries;
id | domain | root_city_id | language_id | currency_id | google_tag_manager_id | created_at | updated_at
-----------+--------+--------------+-------------+-------------+-----------------------+----------------------------+----------------------------
665664142 | com | | 1019186233 | 432072940 | | 2020-10-23 06:20:49.288637 | 2020-10-23 06:20:49.288637
169150333 | by | | 1019186233 | 432072940 | | 2020-10-23 06:20:49.288637 | 2020-10-23 06:20:49.288637
(2 rows)
There are two my test records and they have language and currency references. But their tables are empty:
qq2_test=# select * from currencies;
id | name | symbol | created_at | updated_at
----+------+--------+------------+------------
(0 rows)
qq2_test=# select * from languages;
id | name | locale | image | created_at | updated_at
----+------+--------+-------+------------+------------
(0 rows)
Why does postgresql allow nonexistent references in countries table?
Ruby 2.7.1 MRI
Rails: 6.0.3.4
Postgresql 12.4
Ubuntu 20.04
Assuming that you're the only person using the database (since you are talking about small, 1-2 row tests), I would guess that your Rails app (or corresponding driver) is disabling triggers or foreign key checks. It's totally possible to bypass the foreign key checks like so:
edb=# show session_replication_role ;
session_replication_role
--------------------------
origin
(1 row)
edb=# create table city (id int primary key, name text);
CREATE TABLE
edb=# select * from city;
id | name
----+------
(0 rows)
edb=# create table person (id int, name text, city int references city(id));
CREATE TABLE
edb=# insert into person values (1,'foo',1);
ERROR: insert or update on table "person" violates foreign key constraint "person_city_fkey"
DETAIL: Key (city)=(1) is not present in table "city".
edb=# set session_replication_role to replica;
SET
edb=# insert into person values (1,'foo',1);
INSERT 0 1
edb=# select * from person;
id | name | city
----+------+------
1 | foo | 1
(1 row)
edb=# select * from city;
id | name
----+------
(0 rows)
I would suggest that you temporarily set log_statement = all and run your tests again--then see in your Postgres server logs (default should be /var/log/postgresql/postgresql-12-main.log for Ubuntu) what might be disabling your foreign key constraint checks, then address your findings accordingly.
There are only two options:
The foreign key constraint is NOT VALID.
Such constraints are cheched for new entries, but existing entries can violate them.
But that would show up in your \d output, so that is not the case.
You have data corruption.
Apart from hardware problems or software bugs, possible explanations are:
Someone set session_replication_role = replica so that triggers don't fire.
A superuser ran
ALTER TABLE countries DISABLE TRIGGER ALL;
Related
I used inheritance feature from Postgresql 9.6 to create some archive for huge amount of data.
Finally parent table looks like this
foo=# \d+ foo_data_sshcommand
Table "public.foo_data_sshcommand"
Column | Type | Modifiers | Storage | Stats target | Description
------------+--------------------------+-------------------------------------------------------------------+----------+--------------+-------------
id | integer | not null default nextval('foo_data_sshcommand_id_seq'::regclass) | plain | |
time | timestamp with time zone | not null | plain | |
command | text | not null | extended | |
success | boolean | not null | plain | |
session_id | integer | not null | plain | |
Indexes:
"foo_data_sshcommand_pkey" PRIMARY KEY, btree (id)
"foo_data_sshcommand_session_id_c8b38d68" btree (session_id)
Check constraints:
"partition_check" CHECK ("time" >= '2019-10-01 00:00:00+02'::timestamp with time zone) NO INHERIT
Foreign-key constraints:
"foo_data_sshcommand_session_id_c8b38d68_fk" FOREIGN KEY (session_id) REFERENCES foo_data_sshsession(id) DEFERRABLE INITIALLY DEFERRED
Child tables: arc_2019_01_foo_data_sshcommand,
arc_2019_02_foo_data_sshcommand,
arc_2019_03_foo_data_sshcommand,
arc_2019_04_foo_data_sshcommand,
arc_2019_05_foo_data_sshcommand,
arc_2019_06_foo_data_sshcommand,
arc_2019_07_foo_data_sshcommand,
arc_2019_08_foo_data_sshcommand,
arc_2019_09_foo_data_sshcommand,
arc_2019_foo_data_sshcommand
Every month it creates two more tables and it adds them to the archive. It is not nice process, because it must move overflowed data from parent table to new child table. But it is not the problem that I want to talk about. I am curious about question - How many tables can paricipate on inheritance of one parent table?
I have the following model:
Table "public.models"
Column | Type | Collation | Nullable | Default
----------------------+-----------------------------+-----------+----------+---------------------------------------------
id | bigint | | not null | nextval('models_id_seq'::regclass)
research_provider_id | bigint | | not null |
covered_company_id | bigint | | not null |
publication_date | timestamp without time zone | | not null |
created_at | timestamp without time zone | | not null |
updated_at | timestamp without time zone | | not null |
insights_id | bigint | | not null | nextval('models_insights_id_seq'::regclass)
Indexes:
"models_pkey" PRIMARY KEY, btree (id)
Foreign-key constraints:
"fk_rails_22d32db7ac" FOREIGN KEY (covered_company_id) REFERENCES companies(id)
"fk_rails_3a764bb9c1" FOREIGN KEY (research_provider_id) REFERENCES companies(id)
Referenced by:
TABLE "model_product_groups" CONSTRAINT "fk_rails_1866a14ba0" FOREIGN KEY (model_id) REFERENCES models(id)
TABLE "model_analysts" CONSTRAINT "fk_rails_c7730c705b" FOREIGN KEY (model_id) REFERENCES models(id)
And I'm creating the objects using ActiveRecord, with:
Model.new(
# insights_id:
research_provider_id: company.id,
covered_company_id: covered_company_id,
publication_date: Time.current - rand(1..20).day,
......
)
What value should I pass to insights_id to use the models_insights_id_seq sequece? Tried DEFAULT and not passing anything and both fail to use the sequence, ie, making activerecord-import to generate nextval('public.models_insights_id_seq')
Note: This question is about to instruct activerecord-import to generate nextval('public.models_insights_id_seq') for the insights_id column, and not about using ActiveRecord to get the sequence next value.
Faced with same issue today. Just used another way for records import. Instead of importing of collection of AR objects, build a collection of attributes hashes without sequence parameters and then import them through Model.import. This way works for me.
Example. Given I have position column with default sequence value:
position | integer | not null default nextval('file_exports.task_file_item_position_seq'::regclass)
In this code next sequence value will be set by Postgres automatically in
each FileExports::TaskFileItem record.
...
params = file_records.map do |file_record|
build_file_item_params(file_record)
end
def build_file_item_params(file_record)
{
name: "some_name",
link: file_record.file.url
}
end
FileExports::TaskFileItem.import!(params, validate: true)
I'm using rails 4, postgres 9.3 and devise, and at the moment of sign up an user, I'm getting the error of :
PG::NotNullViolation: ERROR: the null value for column « usuario_id » violates the constraint not null.
The primary key or ID of the table or model usuario is usuario_id and it's an integer.
I understand that it violates de constraint, but how can I autoincrement the ID without entering it through the sign up form?
I didn't do the migration 'cause it's unnecessary because I already have the tables in postgres. The only columns that I add are the devise ones. Anyway this is the table from postgresql.
Tabla «public.usuario»
Column | Type | Modifiers
------------------------+-----------------------------+---------------------------------
usuario_id | integer | not null
comuna_id | integer | not null
usuario_nombre | character(256) |
usuario_apellidopat | character(256) |
usuario_apellidomat | character(256) |
usuario_rut | character varying(1024) |
email | character varying(1024) |
usuario_nombre_usuario | character(256) |
password | character varying(128) |
usuario_vip | boolean |
usuario_calle | character varying(128) |
usuario_numero_calle | smallint |
usuario_villa | character varying(128) |
usuario_numero_depto | smallint |
usuario_bloque | smallint |
encrypted_password | character varying(255) | not null valor por omisión
reset_password_token | character varying(255) |
reset_password_sent_at | timestamp without time zone |
remember_created_at | timestamp without time zone |
sign_in_count | integer | not null valor por omisión 0
current_sign_in_at | timestamp without time zone |
last_sign_in_at | timestamp without time zone |
current_sign_in_ip | character varying(255) |
last_sign_in_ip | character varying(255) |
confirmation_token | character varying(255) |
confirmed_at | timestamp without time zone |
confirmation_sent_at | timestamp without time zone |
unconfirmed_email | character varying(255) |
Índexes:
"pk_usuario" PRIMARY KEY, btree (usuario_id)
"index_usuario_on_reset_password_token" UNIQUE, btree (reset_password_token)
"usuario_pk" UNIQUE, btree (usuario_id)
"relationship_34_fk" btree (comuna_id)
Foreign key constraints:
"fk_usuario_relations_comuna" FOREIGN KEY (comuna_id) REFERENCES comuna(comuna_id) ON UPDATE RESTRICT ON DELETE RESTRICT
Referenced by:
TABLE "compra_remate" CONSTRAINT "fk_compra_r_relations_usuario" FOREIGN KEY (usu_usuario_id) REFERENCES usuario(usuario_id) ON UPDATE RESTRICT ON DELETE RESTRICT
TABLE "compra_remate" CONSTRAINT "fk_compra_r_relations_usuario2" FOREIGN KEY (usuario_id) REFERENCES usuario(usuario_id) ON UPDATE RESTRICT ON DELETE RESTRICT
TABLE "compra_venta_especial" CONSTRAINT "fk_compra_v_relations_usuario" FOREIGN KEY (usu_usuario_id) REFERENCES usuario(usuario_id) ON UPDATE RESTRICT ON DELETE RESTRICT
TABLE "compra_venta_normal" CONSTRAINT "fk_compra_v_relations_usuario" FOREIGN KEY (usu_usuario_id) REFERENCES usuario(usuario_id) ON UPDATE RESTRICT ON DELETE RESTRICT
TABLE "compra_venta_especial" CONSTRAINT "fk_compra_v_relations_usuario2" FOREIGN KEY (usuario_id) REFERENCES usuario(usuario_id) ON UPDATE RESTRICT ON DELETE RESTRICT
TABLE "compra_venta_normal" CONSTRAINT "fk_compra_v_relations_usuario2" FOREIGN KEY (usuario_id) REFERENCES usuario(usuario_id) ON UPDATE RESTRICT ON DELETE RESTRICT
TABLE "notificacion" CONSTRAINT "fk_notifica_relations_usuario" FOREIGN KEY (usuario_id) REFERENCES usuario(usuario_id) ON UPDATE RESTRICT ON DELETE RESTRICT
TABLE "prod_of_nec" CONSTRAINT "fk_prod_of__relations_usuario" FOREIGN KEY (usuario_id) REFERENCES u:
First, in your User model migration, did change the primary key for the user table to be usuario_id? See the following Stackoverflow question:
How to change primary key in rails migration file?
Second, did you make sure that you overrode the naming convention for the primary key in your ActiveRecord models?
Something like this:
class User < ActiveRecord::Base
self.primary_key = "usuario_id"
end
See the following section in the Rails Guide for more details.
In my Grails 2.3.7 project, I have a Product domain class, like this:
class Product {
String code
String description
Map attributes
static constraints = {
code(unique: true)
}
static mapping = {
code index: 'Code_Idx'
attributes fetch: 'join'
cache true
id generator: 'hilo'
}
}
It translates into this database:
create table product (id bigint not null, version bigint not null, code varchar(255) not null unique, description varchar(255) not null, primary key (id));
create table product_attributes (attributes bigint, attributes_idx varchar(255), attributes_elt varchar(255) not null);
create index Code_Idx on product (code);
With some 4000 products in database, the scaffold listing shows them just fine.
Except, when I click "sort" on code - because there is no index - so my server does this:
explain select this_.id as id14_0_, this_.version as version14_0_, this_.code as code14_0_, this_.description as descript4_14_0_,
attributes2_.attributes as attributes14_2_, attributes2_.attributes_elt as attributes3_2_, attributes2_.attributes_idx as attributes2_2_
from product this_ left outer join product_attributes attributes2_
on this_.id=attributes2_.attributes
order by lower(this_.code) desc limit 90, 100
+----+-------------+--------------+------+---------------+------+---------+------+-------+---------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+--------------+------+---------------+------+---------+------+-------+---------------------------------+
| 1 | SIMPLE | this_ | ALL | NULL | NULL | NULL | NULL | 4086 | Using temporary; Using filesort |
| 1 | SIMPLE | attributes2_ | ALL | NULL | NULL | NULL | NULL | 43975 | |
+----+-------------+--------------+------+---------------+------+---------+------+-------+---------------------------------+
Obviously, this takes ages. I can manually add the index:
ALTER TABLE `product_attributes` ADD INDEX(`attributes`);
and then it works OK. I think it should have been created automatically in the first place - there is very little sense in this schema without the index - but OK, I can ping Gorm to do it. My question is - what can I put into the domain class to have Gorm add this index ?
Grails doesn't automatically create indexes on such columns as 'attributes' in your example. In order to create and manage these indexes I highly recommend using the Database Migration Plugin. The documentation is well written, and outlines how to use it.
First of all, let me say I understand relational theory and I'm as competent as anyone in MySQL but I'm a total PostgreSQL noob.
When I try to insert a new record into my service table - only in production - I get this:
ActiveRecord::RecordNotUnique (PGError: ERROR: duplicate key value violates unique constraint "service_pkey"
: INSERT INTO "service" ("created_at", "name", "salon_id", "updated_at") VALUES ('2011-02-28 02:34:20.054269', 'Manicure', 1, '2011-02-28 02:34:20.054269') RETURNING "id"):
app/controllers/services_controller.rb:46
app/controllers/services_controller.rb:45:in `create'
I don't understand why. Shouldn't it auto-increment the PK for me?
Here's the table definition:
snip=> \d service
Table "public.service"
Column | Type | Modifiers
------------+-----------------------------+------------------------------------------------------
id | integer | not null default nextval('service_id_seq'::regclass)
name | character varying(255) |
salon_id | integer |
created_at | timestamp without time zone |
updated_at | timestamp without time zone |
Indexes:
"service_pkey" PRIMARY KEY, btree (id)
And here's the definition for that same table in development, where it works fine:
snip_development=> \d service
Table "public.service"
Column | Type | Modifiers
------------+-----------------------------+------------------------------------------------------
id | integer | not null default nextval('service_id_seq'::regclass)
name | character varying(255) |
salon_id | integer |
created_at | timestamp without time zone |
updated_at | timestamp without time zone |
Indexes:
"service_pkey" PRIMARY KEY, btree (id)
Same thing! So what could it possibly be?
You probably loaded up the table with data but neglected to set the current value of service_id_seq to the necessary value. You can just SELECT * FROM service_id_seq to check the current values, and use the setval function to reset it.
For example, SELECT setval('service_id_seq'::regclass, MAX(id)) FROM service; should reset the sequence to the current maximum value from the table.