I used inheritance feature from Postgresql 9.6 to create some archive for huge amount of data.
Finally parent table looks like this
foo=# \d+ foo_data_sshcommand
Table "public.foo_data_sshcommand"
Column | Type | Modifiers | Storage | Stats target | Description
------------+--------------------------+-------------------------------------------------------------------+----------+--------------+-------------
id | integer | not null default nextval('foo_data_sshcommand_id_seq'::regclass) | plain | |
time | timestamp with time zone | not null | plain | |
command | text | not null | extended | |
success | boolean | not null | plain | |
session_id | integer | not null | plain | |
Indexes:
"foo_data_sshcommand_pkey" PRIMARY KEY, btree (id)
"foo_data_sshcommand_session_id_c8b38d68" btree (session_id)
Check constraints:
"partition_check" CHECK ("time" >= '2019-10-01 00:00:00+02'::timestamp with time zone) NO INHERIT
Foreign-key constraints:
"foo_data_sshcommand_session_id_c8b38d68_fk" FOREIGN KEY (session_id) REFERENCES foo_data_sshsession(id) DEFERRABLE INITIALLY DEFERRED
Child tables: arc_2019_01_foo_data_sshcommand,
arc_2019_02_foo_data_sshcommand,
arc_2019_03_foo_data_sshcommand,
arc_2019_04_foo_data_sshcommand,
arc_2019_05_foo_data_sshcommand,
arc_2019_06_foo_data_sshcommand,
arc_2019_07_foo_data_sshcommand,
arc_2019_08_foo_data_sshcommand,
arc_2019_09_foo_data_sshcommand,
arc_2019_foo_data_sshcommand
Every month it creates two more tables and it adds them to the archive. It is not nice process, because it must move overflowed data from parent table to new child table. But it is not the problem that I want to talk about. I am curious about question - How many tables can paricipate on inheritance of one parent table?
Related
I have a rails app that uses postgresql 12. Recently, I wrote some tests and saw some strange behavior.
I have a countries table. Its schema looks like that:
qq2_test=# \d countries;
Table "public.countries"
Column | Type | Collation | Nullable | Default
-----------------------+--------------------------------+-----------+----------+---------------------------------------
id | bigint | | not null | nextval('countries_id_seq'::regclass)
domain | character varying | | not null | ''::character varying
root_city_id | bigint | | |
language_id | bigint | | not null |
currency_id | bigint | | not null |
google_tag_manager_id | character varying | | not null | ''::character varying
created_at | timestamp(6) without time zone | | not null |
updated_at | timestamp(6) without time zone | | not null |
Indexes:
"countries_pkey" PRIMARY KEY, btree (id)
"index_countries_on_domain" UNIQUE, btree (domain)
"index_countries_on_currency_id" btree (currency_id)
"index_countries_on_language_id" btree (language_id)
Foreign-key constraints:
"fk_rails_6f479b409c" FOREIGN KEY (language_id) REFERENCES languages(id)
"fk_rails_7cac1212c7" FOREIGN KEY (currency_id) REFERENCES currencies(id)
"fk_rails_beac36a0bd" FOREIGN KEY (root_city_id) REFERENCES cities(id)
Referenced by:
TABLE "country_translations" CONSTRAINT "fk_rails_0c4ee35f26" FOREIGN KEY (country_id) REFERENCES countries(id) ON DELETE CASCADE
TABLE "countries_languages" CONSTRAINT "fk_rails_556e7398aa" FOREIGN KEY (country_id) REFERENCES countries(id) ON DELETE CASCADE
TABLE "cities" CONSTRAINT "fk_rails_996e05be41" FOREIGN KEY (country_id) REFERENCES countries(id)
As you can see, I have foreign key constraints on both currency_id and language_id fields.
When I run my tests I see that there are records in that table:
qq2_test=# select * from countries;
id | domain | root_city_id | language_id | currency_id | google_tag_manager_id | created_at | updated_at
-----------+--------+--------------+-------------+-------------+-----------------------+----------------------------+----------------------------
665664142 | com | | 1019186233 | 432072940 | | 2020-10-23 06:20:49.288637 | 2020-10-23 06:20:49.288637
169150333 | by | | 1019186233 | 432072940 | | 2020-10-23 06:20:49.288637 | 2020-10-23 06:20:49.288637
(2 rows)
There are two my test records and they have language and currency references. But their tables are empty:
qq2_test=# select * from currencies;
id | name | symbol | created_at | updated_at
----+------+--------+------------+------------
(0 rows)
qq2_test=# select * from languages;
id | name | locale | image | created_at | updated_at
----+------+--------+-------+------------+------------
(0 rows)
Why does postgresql allow nonexistent references in countries table?
Ruby 2.7.1 MRI
Rails: 6.0.3.4
Postgresql 12.4
Ubuntu 20.04
Assuming that you're the only person using the database (since you are talking about small, 1-2 row tests), I would guess that your Rails app (or corresponding driver) is disabling triggers or foreign key checks. It's totally possible to bypass the foreign key checks like so:
edb=# show session_replication_role ;
session_replication_role
--------------------------
origin
(1 row)
edb=# create table city (id int primary key, name text);
CREATE TABLE
edb=# select * from city;
id | name
----+------
(0 rows)
edb=# create table person (id int, name text, city int references city(id));
CREATE TABLE
edb=# insert into person values (1,'foo',1);
ERROR: insert or update on table "person" violates foreign key constraint "person_city_fkey"
DETAIL: Key (city)=(1) is not present in table "city".
edb=# set session_replication_role to replica;
SET
edb=# insert into person values (1,'foo',1);
INSERT 0 1
edb=# select * from person;
id | name | city
----+------+------
1 | foo | 1
(1 row)
edb=# select * from city;
id | name
----+------
(0 rows)
I would suggest that you temporarily set log_statement = all and run your tests again--then see in your Postgres server logs (default should be /var/log/postgresql/postgresql-12-main.log for Ubuntu) what might be disabling your foreign key constraint checks, then address your findings accordingly.
There are only two options:
The foreign key constraint is NOT VALID.
Such constraints are cheched for new entries, but existing entries can violate them.
But that would show up in your \d output, so that is not the case.
You have data corruption.
Apart from hardware problems or software bugs, possible explanations are:
Someone set session_replication_role = replica so that triggers don't fire.
A superuser ran
ALTER TABLE countries DISABLE TRIGGER ALL;
Rails v. 5.2.4
ActiveRecord v5.2.4.3
I have a Rails app with a MySQL database, and my app has a Skill model and a SkillAdjacency model. The SkillAdjacency model has the following attributes:
requested_skill_id, table_name: 'Skill'
adjacent_skill_id, table_name: 'Skill'
score, integer
SkillAdjacencies are used to determine how "similar" two instances of Skill are to each other.
One of the app's constraints is that you can't create more than one instance of SkillAdjacency for each combination of requested_skill and adjacent_skill, and I plan to enforce this both with ActiveModel validations and with a composite index which employs a uniqueness constraint. So far I have the following:
add_index :skill_adjacencies, [:requested_skill_id, :adjacent_skill_id], unique: true, name: 'index_adjacencies_on_requested_then_adjacent', using: :btree
However, I know that the order in which the composite columns are declared is important, so I'm considering adding this 2nd composite index to account for the other possible order:
add_index :skill_adjacencies, [:adjacent_skill_id, :requested_skill_id], unique: true, name: 'index_adjacencies_on_adjacent_then_requested', using: :btree
But because writing to an index isn't free, I only want to add the 2nd index if it will actually result in a performance benefit. The problem is, whether or not this 2nd index will be beneficial depends on whether ActiveRecord will start with adjacent_skill_id vs. requested_skill_id when searching for a composite index to search.
How can I determine what order ActiveRecord uses? Does it just use the same order that's specified in the query? For example, if I query SkillAdjacency.where(requested_skill: Skill.last, adjacent_skill: Skill.first), will it always search for a composite index composed of requested_skill 1st and adjacent_skill 2nd? If that's the case, should I cover all my bases by creating that additional composite index?
Alternately, is there some under-the-hood magic which determines if the relevant composite index exists regardless of the order provided in the query?
EDIT:
I ran EXPLAIN and saw the following:
irb(main):013:0> SkillAdjacency.where(requested_skill_id: 1, adjacent_skill_id: 200).explain
SkillAdjacency Load (0.3ms) SELECT `skill_adjacencies`.* FROM `skill_adjacencies` WHERE `skill_adjacencies`.`requested_skill_id` = 1 AND `skill_adjacencies`.`adjacent_skill_id` = 200
=> EXPLAIN for: SELECT `skill_adjacencies`.* FROM `skill_adjacencies` WHERE `skill_adjacencies`.`requested_skill_id` = 1 AND `skill_adjacencies`.`adjacent_skill_id` = 200
+----+-------------+-------------------+------------+-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------+---------+-------------+------+----------+-------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------------------+------------+-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------+---------+-------------+------+----------+-------+
| 1 | SIMPLE | skill_adjacencies | NULL | const | index_adjacencies_on_requested_then_adjacent,index_adjacencies_on_adjacent_then_requested,index_skill_adjacencies_on_requested_skill_id,index_skill_adjacencies_on_adjacent_skill_id | index_adjacencies_on_requested_then_adjacent | 10 | const,const | 1 | 100.0 | NULL |
+----+-------------+-------------------+------------+-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------+---------+-------------+------+----------+-------+
1 row in set (0.00 sec)
irb(main):014:0> SkillAdjacency.where(adjacent_skill_id: 200, requested_skill: 1).explain
SkillAdjacency Load (0.3ms) SELECT `skill_adjacencies`.* FROM `skill_adjacencies` WHERE `skill_adjacencies`.`adjacent_skill_id` = 200 AND `skill_adjacencies`.`requested_skill_id` = 1
=> EXPLAIN for: SELECT `skill_adjacencies`.* FROM `skill_adjacencies` WHERE `skill_adjacencies`.`adjacent_skill_id` = 200 AND `skill_adjacencies`.`requested_skill_id` = 1
+----+-------------+-------------------+------------+-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------+---------+-------------+------+----------+-------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------------------+------------+-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------+---------+-------------+------+----------+-------+
| 1 | SIMPLE | skill_adjacencies | NULL | const | index_adjacencies_on_requested_then_adjacent,index_adjacencies_on_adjacent_then_requested,index_skill_adjacencies_on_requested_skill_id,index_skill_adjacencies_on_adjacent_skill_id | index_adjacencies_on_requested_then_adjacent | 10 | const,const | 1 | 100.0 | NULL |
+----+-------------+-------------------+------------+-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------+---------+-------------+------+----------+-------+
1 row in set (0.00 sec)
In both cases, I see that the value in the key column is index_adjacencies_on_requested_then_adjacent, despite each query passing in a different order for the query params. Can I assume this means the order of those params doesn't matter?
I have the following model:
Table "public.models"
Column | Type | Collation | Nullable | Default
----------------------+-----------------------------+-----------+----------+---------------------------------------------
id | bigint | | not null | nextval('models_id_seq'::regclass)
research_provider_id | bigint | | not null |
covered_company_id | bigint | | not null |
publication_date | timestamp without time zone | | not null |
created_at | timestamp without time zone | | not null |
updated_at | timestamp without time zone | | not null |
insights_id | bigint | | not null | nextval('models_insights_id_seq'::regclass)
Indexes:
"models_pkey" PRIMARY KEY, btree (id)
Foreign-key constraints:
"fk_rails_22d32db7ac" FOREIGN KEY (covered_company_id) REFERENCES companies(id)
"fk_rails_3a764bb9c1" FOREIGN KEY (research_provider_id) REFERENCES companies(id)
Referenced by:
TABLE "model_product_groups" CONSTRAINT "fk_rails_1866a14ba0" FOREIGN KEY (model_id) REFERENCES models(id)
TABLE "model_analysts" CONSTRAINT "fk_rails_c7730c705b" FOREIGN KEY (model_id) REFERENCES models(id)
And I'm creating the objects using ActiveRecord, with:
Model.new(
# insights_id:
research_provider_id: company.id,
covered_company_id: covered_company_id,
publication_date: Time.current - rand(1..20).day,
......
)
What value should I pass to insights_id to use the models_insights_id_seq sequece? Tried DEFAULT and not passing anything and both fail to use the sequence, ie, making activerecord-import to generate nextval('public.models_insights_id_seq')
Note: This question is about to instruct activerecord-import to generate nextval('public.models_insights_id_seq') for the insights_id column, and not about using ActiveRecord to get the sequence next value.
Faced with same issue today. Just used another way for records import. Instead of importing of collection of AR objects, build a collection of attributes hashes without sequence parameters and then import them through Model.import. This way works for me.
Example. Given I have position column with default sequence value:
position | integer | not null default nextval('file_exports.task_file_item_position_seq'::regclass)
In this code next sequence value will be set by Postgres automatically in
each FileExports::TaskFileItem record.
...
params = file_records.map do |file_record|
build_file_item_params(file_record)
end
def build_file_item_params(file_record)
{
name: "some_name",
link: file_record.file.url
}
end
FileExports::TaskFileItem.import!(params, validate: true)
Given two postgres tables samples and sample_values
Table "public.samples"
Column | Type | Modifiers
------------+-----------------------------+------------------------------------------------------
id | integer | not null default nextval('samples_id_seq'::regclass)
created_at | timestamp without time zone | not null
updated_at | timestamp without time zone | not null
Indexes:
"samples_pkey" PRIMARY KEY, btree (id)
Referenced by:
TABLE "sample_values" CONSTRAINT "fk_rails_501ff3bcb6" FOREIGN KEY (sample_id) REFERENCES samples(id)
and
Table "public.sample_values"
Column | Type | Modifiers
------------+-----------------------------+------------------------------------------------------------
id | integer | not null default nextval('sample_values_id_seq'::regclass)
value | integer |
sample_id | integer |
created_at | timestamp without time zone | not null
updated_at | timestamp without time zone | not null
Indexes:
"sample_values_pkey" PRIMARY KEY, btree (id)
Foreign-key constraints:
"fk_rails_501ff3bcb6" FOREIGN KEY (sample_id) REFERENCES samples(id)
I want to query for instances of Sample where all the associated SampleValue rows only match a certain criteria.
Using a combination of AR and raw sql the query looks like:
ALLOWED_VALUES = [1,2,3]
Sample.joins(:sample_values).
group('samples.id').
having('SUM((sample_values.value in (?))::int) = COUNT(*)', ALLOWED_VALUES)
I'd like to get rid of the raw sql however, I wasn't able to construct the nodes for the having clause with Arel.
I tried SampleValues.arel_table[:value].in(ALLOWED_VALUES).sum.eq(Arel.star.count) but that raises the following error:
TypeError: no implicit conversion of Arel::Table into String
In my Grails 2.3.7 project, I have a Product domain class, like this:
class Product {
String code
String description
Map attributes
static constraints = {
code(unique: true)
}
static mapping = {
code index: 'Code_Idx'
attributes fetch: 'join'
cache true
id generator: 'hilo'
}
}
It translates into this database:
create table product (id bigint not null, version bigint not null, code varchar(255) not null unique, description varchar(255) not null, primary key (id));
create table product_attributes (attributes bigint, attributes_idx varchar(255), attributes_elt varchar(255) not null);
create index Code_Idx on product (code);
With some 4000 products in database, the scaffold listing shows them just fine.
Except, when I click "sort" on code - because there is no index - so my server does this:
explain select this_.id as id14_0_, this_.version as version14_0_, this_.code as code14_0_, this_.description as descript4_14_0_,
attributes2_.attributes as attributes14_2_, attributes2_.attributes_elt as attributes3_2_, attributes2_.attributes_idx as attributes2_2_
from product this_ left outer join product_attributes attributes2_
on this_.id=attributes2_.attributes
order by lower(this_.code) desc limit 90, 100
+----+-------------+--------------+------+---------------+------+---------+------+-------+---------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+--------------+------+---------------+------+---------+------+-------+---------------------------------+
| 1 | SIMPLE | this_ | ALL | NULL | NULL | NULL | NULL | 4086 | Using temporary; Using filesort |
| 1 | SIMPLE | attributes2_ | ALL | NULL | NULL | NULL | NULL | 43975 | |
+----+-------------+--------------+------+---------------+------+---------+------+-------+---------------------------------+
Obviously, this takes ages. I can manually add the index:
ALTER TABLE `product_attributes` ADD INDEX(`attributes`);
and then it works OK. I think it should have been created automatically in the first place - there is very little sense in this schema without the index - but OK, I can ping Gorm to do it. My question is - what can I put into the domain class to have Gorm add this index ?
Grails doesn't automatically create indexes on such columns as 'attributes' in your example. In order to create and manage these indexes I highly recommend using the Database Migration Plugin. The documentation is well written, and outlines how to use it.