I have a rails app that uses postgresql 12. Recently, I wrote some tests and saw some strange behavior.
I have a countries table. Its schema looks like that:
qq2_test=# \d countries;
Table "public.countries"
Column | Type | Collation | Nullable | Default
-----------------------+--------------------------------+-----------+----------+---------------------------------------
id | bigint | | not null | nextval('countries_id_seq'::regclass)
domain | character varying | | not null | ''::character varying
root_city_id | bigint | | |
language_id | bigint | | not null |
currency_id | bigint | | not null |
google_tag_manager_id | character varying | | not null | ''::character varying
created_at | timestamp(6) without time zone | | not null |
updated_at | timestamp(6) without time zone | | not null |
Indexes:
"countries_pkey" PRIMARY KEY, btree (id)
"index_countries_on_domain" UNIQUE, btree (domain)
"index_countries_on_currency_id" btree (currency_id)
"index_countries_on_language_id" btree (language_id)
Foreign-key constraints:
"fk_rails_6f479b409c" FOREIGN KEY (language_id) REFERENCES languages(id)
"fk_rails_7cac1212c7" FOREIGN KEY (currency_id) REFERENCES currencies(id)
"fk_rails_beac36a0bd" FOREIGN KEY (root_city_id) REFERENCES cities(id)
Referenced by:
TABLE "country_translations" CONSTRAINT "fk_rails_0c4ee35f26" FOREIGN KEY (country_id) REFERENCES countries(id) ON DELETE CASCADE
TABLE "countries_languages" CONSTRAINT "fk_rails_556e7398aa" FOREIGN KEY (country_id) REFERENCES countries(id) ON DELETE CASCADE
TABLE "cities" CONSTRAINT "fk_rails_996e05be41" FOREIGN KEY (country_id) REFERENCES countries(id)
As you can see, I have foreign key constraints on both currency_id and language_id fields.
When I run my tests I see that there are records in that table:
qq2_test=# select * from countries;
id | domain | root_city_id | language_id | currency_id | google_tag_manager_id | created_at | updated_at
-----------+--------+--------------+-------------+-------------+-----------------------+----------------------------+----------------------------
665664142 | com | | 1019186233 | 432072940 | | 2020-10-23 06:20:49.288637 | 2020-10-23 06:20:49.288637
169150333 | by | | 1019186233 | 432072940 | | 2020-10-23 06:20:49.288637 | 2020-10-23 06:20:49.288637
(2 rows)
There are two my test records and they have language and currency references. But their tables are empty:
qq2_test=# select * from currencies;
id | name | symbol | created_at | updated_at
----+------+--------+------------+------------
(0 rows)
qq2_test=# select * from languages;
id | name | locale | image | created_at | updated_at
----+------+--------+-------+------------+------------
(0 rows)
Why does postgresql allow nonexistent references in countries table?
Ruby 2.7.1 MRI
Rails: 6.0.3.4
Postgresql 12.4
Ubuntu 20.04
Assuming that you're the only person using the database (since you are talking about small, 1-2 row tests), I would guess that your Rails app (or corresponding driver) is disabling triggers or foreign key checks. It's totally possible to bypass the foreign key checks like so:
edb=# show session_replication_role ;
session_replication_role
--------------------------
origin
(1 row)
edb=# create table city (id int primary key, name text);
CREATE TABLE
edb=# select * from city;
id | name
----+------
(0 rows)
edb=# create table person (id int, name text, city int references city(id));
CREATE TABLE
edb=# insert into person values (1,'foo',1);
ERROR: insert or update on table "person" violates foreign key constraint "person_city_fkey"
DETAIL: Key (city)=(1) is not present in table "city".
edb=# set session_replication_role to replica;
SET
edb=# insert into person values (1,'foo',1);
INSERT 0 1
edb=# select * from person;
id | name | city
----+------+------
1 | foo | 1
(1 row)
edb=# select * from city;
id | name
----+------
(0 rows)
I would suggest that you temporarily set log_statement = all and run your tests again--then see in your Postgres server logs (default should be /var/log/postgresql/postgresql-12-main.log for Ubuntu) what might be disabling your foreign key constraint checks, then address your findings accordingly.
There are only two options:
The foreign key constraint is NOT VALID.
Such constraints are cheched for new entries, but existing entries can violate them.
But that would show up in your \d output, so that is not the case.
You have data corruption.
Apart from hardware problems or software bugs, possible explanations are:
Someone set session_replication_role = replica so that triggers don't fire.
A superuser ran
ALTER TABLE countries DISABLE TRIGGER ALL;
I have the following model:
Table "public.models"
Column | Type | Collation | Nullable | Default
----------------------+-----------------------------+-----------+----------+---------------------------------------------
id | bigint | | not null | nextval('models_id_seq'::regclass)
research_provider_id | bigint | | not null |
covered_company_id | bigint | | not null |
publication_date | timestamp without time zone | | not null |
created_at | timestamp without time zone | | not null |
updated_at | timestamp without time zone | | not null |
insights_id | bigint | | not null | nextval('models_insights_id_seq'::regclass)
Indexes:
"models_pkey" PRIMARY KEY, btree (id)
Foreign-key constraints:
"fk_rails_22d32db7ac" FOREIGN KEY (covered_company_id) REFERENCES companies(id)
"fk_rails_3a764bb9c1" FOREIGN KEY (research_provider_id) REFERENCES companies(id)
Referenced by:
TABLE "model_product_groups" CONSTRAINT "fk_rails_1866a14ba0" FOREIGN KEY (model_id) REFERENCES models(id)
TABLE "model_analysts" CONSTRAINT "fk_rails_c7730c705b" FOREIGN KEY (model_id) REFERENCES models(id)
And I'm creating the objects using ActiveRecord, with:
Model.new(
# insights_id:
research_provider_id: company.id,
covered_company_id: covered_company_id,
publication_date: Time.current - rand(1..20).day,
......
)
What value should I pass to insights_id to use the models_insights_id_seq sequece? Tried DEFAULT and not passing anything and both fail to use the sequence, ie, making activerecord-import to generate nextval('public.models_insights_id_seq')
Note: This question is about to instruct activerecord-import to generate nextval('public.models_insights_id_seq') for the insights_id column, and not about using ActiveRecord to get the sequence next value.
Faced with same issue today. Just used another way for records import. Instead of importing of collection of AR objects, build a collection of attributes hashes without sequence parameters and then import them through Model.import. This way works for me.
Example. Given I have position column with default sequence value:
position | integer | not null default nextval('file_exports.task_file_item_position_seq'::regclass)
In this code next sequence value will be set by Postgres automatically in
each FileExports::TaskFileItem record.
...
params = file_records.map do |file_record|
build_file_item_params(file_record)
end
def build_file_item_params(file_record)
{
name: "some_name",
link: file_record.file.url
}
end
FileExports::TaskFileItem.import!(params, validate: true)
Given two postgres tables samples and sample_values
Table "public.samples"
Column | Type | Modifiers
------------+-----------------------------+------------------------------------------------------
id | integer | not null default nextval('samples_id_seq'::regclass)
created_at | timestamp without time zone | not null
updated_at | timestamp without time zone | not null
Indexes:
"samples_pkey" PRIMARY KEY, btree (id)
Referenced by:
TABLE "sample_values" CONSTRAINT "fk_rails_501ff3bcb6" FOREIGN KEY (sample_id) REFERENCES samples(id)
and
Table "public.sample_values"
Column | Type | Modifiers
------------+-----------------------------+------------------------------------------------------------
id | integer | not null default nextval('sample_values_id_seq'::regclass)
value | integer |
sample_id | integer |
created_at | timestamp without time zone | not null
updated_at | timestamp without time zone | not null
Indexes:
"sample_values_pkey" PRIMARY KEY, btree (id)
Foreign-key constraints:
"fk_rails_501ff3bcb6" FOREIGN KEY (sample_id) REFERENCES samples(id)
I want to query for instances of Sample where all the associated SampleValue rows only match a certain criteria.
Using a combination of AR and raw sql the query looks like:
ALLOWED_VALUES = [1,2,3]
Sample.joins(:sample_values).
group('samples.id').
having('SUM((sample_values.value in (?))::int) = COUNT(*)', ALLOWED_VALUES)
I'd like to get rid of the raw sql however, I wasn't able to construct the nodes for the having clause with Arel.
I tried SampleValues.arel_table[:value].in(ALLOWED_VALUES).sum.eq(Arel.star.count) but that raises the following error:
TypeError: no implicit conversion of Arel::Table into String
I’m using PostGres 9.5. I have the following set up in my table, in hopes of auto-generating IDs …
myproject=> \d my_object_times
Table "public.my_object_times"
Column | Type | Modifiers
-------------------+-----------------------------+-------------------------------------
…
time_in_ms | bigint |
created_at | timestamp without time zone | not null
updated_at | timestamp without time zone | not null
name | character varying |
age | integer |
city | character varying |
state_id | integer |
country_id | integer |
overall_rank | integer |
age_group_rank | integer |
gender_rank | integer |
races_id | integer |
event_id | character varying |
id | character varying | not null default uuid_generate_v4()
But when I try and run a bulk insert statement, the content of which looks something like
INSERT INTO "my_object_times" ("first_name","last_name","time_in_ms","created_at","updated_at","name","participant","age","city","state_id","country_id","overall_rank","age_group_rank","gender_rank","races_id","event_id","id","division_low_age","division_high_age") VALUES (NULL,NULL,1403000,'2016-10-12 15:36:42.766936','2016-10-12 15:36:42.767104','Terry Williamson',NULL,NULL,NULL,NULL,NULL,4,1,NULL,NULL,'0bf8c3bc-3577-4cab-bdd4-61e64655eaed',NULL,3,3),(NULL,NULL,1431000,'2016-10-12 15:36:42.766936','2016-10-12 15:36:42.767104','Casey Reinl',NULL,NULL,NULL,NULL,NULL,5,1,NULL,NULL,'0bf8c3bc-3577-4cab-bdd4-61e64655eaed',NULL,2,2),(NULL,NULL,1473000,'2016-10-12 15:36:42.766936
I get an error
Error during processing: PG::NotNullViolation: ERROR: null value in column "id" violates not-null constraint
DETAIL: Failing row contains (null, null, 1403000, 2016-10-12 15:36:42.766936, 2016-10-12 15:36:42.767104, Terry Williamson, null, null, null, null, null, 4, 1, null, null, 0bf8c3bc-3577-4cab-bdd4-61e64655eaed, null, 3, 3).
The SQL is being auto generated by the Rails 4.2.7 activerecord-import gem, which I’m invoking like so
MyObjectTime.import inserts
How do I get my IDs auto-generated for me if none is supplied?
Try this,
columns_without_id = YourModel.column_names.reject { |column| column == 'id' }
import(columns_without_id, records)
Ref: Github issue
I'm using rails 4, postgres 9.3 and devise, and at the moment of sign up an user, I'm getting the error of :
PG::NotNullViolation: ERROR: the null value for column « usuario_id » violates the constraint not null.
The primary key or ID of the table or model usuario is usuario_id and it's an integer.
I understand that it violates de constraint, but how can I autoincrement the ID without entering it through the sign up form?
I didn't do the migration 'cause it's unnecessary because I already have the tables in postgres. The only columns that I add are the devise ones. Anyway this is the table from postgresql.
Tabla «public.usuario»
Column | Type | Modifiers
------------------------+-----------------------------+---------------------------------
usuario_id | integer | not null
comuna_id | integer | not null
usuario_nombre | character(256) |
usuario_apellidopat | character(256) |
usuario_apellidomat | character(256) |
usuario_rut | character varying(1024) |
email | character varying(1024) |
usuario_nombre_usuario | character(256) |
password | character varying(128) |
usuario_vip | boolean |
usuario_calle | character varying(128) |
usuario_numero_calle | smallint |
usuario_villa | character varying(128) |
usuario_numero_depto | smallint |
usuario_bloque | smallint |
encrypted_password | character varying(255) | not null valor por omisión
reset_password_token | character varying(255) |
reset_password_sent_at | timestamp without time zone |
remember_created_at | timestamp without time zone |
sign_in_count | integer | not null valor por omisión 0
current_sign_in_at | timestamp without time zone |
last_sign_in_at | timestamp without time zone |
current_sign_in_ip | character varying(255) |
last_sign_in_ip | character varying(255) |
confirmation_token | character varying(255) |
confirmed_at | timestamp without time zone |
confirmation_sent_at | timestamp without time zone |
unconfirmed_email | character varying(255) |
Índexes:
"pk_usuario" PRIMARY KEY, btree (usuario_id)
"index_usuario_on_reset_password_token" UNIQUE, btree (reset_password_token)
"usuario_pk" UNIQUE, btree (usuario_id)
"relationship_34_fk" btree (comuna_id)
Foreign key constraints:
"fk_usuario_relations_comuna" FOREIGN KEY (comuna_id) REFERENCES comuna(comuna_id) ON UPDATE RESTRICT ON DELETE RESTRICT
Referenced by:
TABLE "compra_remate" CONSTRAINT "fk_compra_r_relations_usuario" FOREIGN KEY (usu_usuario_id) REFERENCES usuario(usuario_id) ON UPDATE RESTRICT ON DELETE RESTRICT
TABLE "compra_remate" CONSTRAINT "fk_compra_r_relations_usuario2" FOREIGN KEY (usuario_id) REFERENCES usuario(usuario_id) ON UPDATE RESTRICT ON DELETE RESTRICT
TABLE "compra_venta_especial" CONSTRAINT "fk_compra_v_relations_usuario" FOREIGN KEY (usu_usuario_id) REFERENCES usuario(usuario_id) ON UPDATE RESTRICT ON DELETE RESTRICT
TABLE "compra_venta_normal" CONSTRAINT "fk_compra_v_relations_usuario" FOREIGN KEY (usu_usuario_id) REFERENCES usuario(usuario_id) ON UPDATE RESTRICT ON DELETE RESTRICT
TABLE "compra_venta_especial" CONSTRAINT "fk_compra_v_relations_usuario2" FOREIGN KEY (usuario_id) REFERENCES usuario(usuario_id) ON UPDATE RESTRICT ON DELETE RESTRICT
TABLE "compra_venta_normal" CONSTRAINT "fk_compra_v_relations_usuario2" FOREIGN KEY (usuario_id) REFERENCES usuario(usuario_id) ON UPDATE RESTRICT ON DELETE RESTRICT
TABLE "notificacion" CONSTRAINT "fk_notifica_relations_usuario" FOREIGN KEY (usuario_id) REFERENCES usuario(usuario_id) ON UPDATE RESTRICT ON DELETE RESTRICT
TABLE "prod_of_nec" CONSTRAINT "fk_prod_of__relations_usuario" FOREIGN KEY (usuario_id) REFERENCES u:
First, in your User model migration, did change the primary key for the user table to be usuario_id? See the following Stackoverflow question:
How to change primary key in rails migration file?
Second, did you make sure that you overrode the naming convention for the primary key in your ActiveRecord models?
Something like this:
class User < ActiveRecord::Base
self.primary_key = "usuario_id"
end
See the following section in the Rails Guide for more details.