My DB has a lot of tables (Say 400+), and I only remember part of the name of the one I am looking for.
I know \d would show all the tables, but that's too much to look at. Is there some command to list all the tables whose names match the given regex?
Thanks
It's built in to psql, you can use wildcards in \d, \dt, etc, eg:
craig=> \dt test*
List of relations
Schema | Name | Type | Owner
--------+-----------+-------+-------
public | test | table | craig
public | testtable | table | craig
public | testu | table | craig
public | testx | table | craig
(4 rows)
You'll want to use \dt since \d will display details for each table, not just list the table.
You can do this with schemas too, eg:
\dt *.sometable
will list all tables named sometable in any schema.
Much more convenient than writing queries against pg_class joined to pg_namespace, or querying information_schema.
The usual globbing syntax is accepted, where ? is any single character and * is zero or more characters. So \dt ???? would list all tables with four-character names.
Multiple wildcards are permitted, eg:
craig=> \dt public.*e?t*
List of relations
Schema | Name | Type | Owner
--------+--------------+-------+-------
public | exclude_test | table | craig
public | prep_test | table | craig
public | test | table | craig
public | testtable | table | craig
public | testu | table | craig
public | testx | table | craig
(6 rows)
Not very convenient unless you make it a proc, but;
SELECT * FROM pg_tables WHERE SUBSTRING(tablename FROM '<regex>') <> '';
To make it more convenient, you can create and call a proc as;
CREATE FUNCTION ft(TEXT) RETURNS SETOF pg_tables AS
'SELECT * FROM pg_tables WHERE SUBSTRING(tablename from $1) <> '''';'
LANGUAGE SQL;
SELECT * FROM ft('.*oc.*') -- Gets all tables matching `.*oc.*`
An SQLfiddle to test both with.
There is a table called pg_tables which has all table names in it.
Related
I have a rails app that uses postgresql 12. Recently, I wrote some tests and saw some strange behavior.
I have a countries table. Its schema looks like that:
qq2_test=# \d countries;
Table "public.countries"
Column | Type | Collation | Nullable | Default
-----------------------+--------------------------------+-----------+----------+---------------------------------------
id | bigint | | not null | nextval('countries_id_seq'::regclass)
domain | character varying | | not null | ''::character varying
root_city_id | bigint | | |
language_id | bigint | | not null |
currency_id | bigint | | not null |
google_tag_manager_id | character varying | | not null | ''::character varying
created_at | timestamp(6) without time zone | | not null |
updated_at | timestamp(6) without time zone | | not null |
Indexes:
"countries_pkey" PRIMARY KEY, btree (id)
"index_countries_on_domain" UNIQUE, btree (domain)
"index_countries_on_currency_id" btree (currency_id)
"index_countries_on_language_id" btree (language_id)
Foreign-key constraints:
"fk_rails_6f479b409c" FOREIGN KEY (language_id) REFERENCES languages(id)
"fk_rails_7cac1212c7" FOREIGN KEY (currency_id) REFERENCES currencies(id)
"fk_rails_beac36a0bd" FOREIGN KEY (root_city_id) REFERENCES cities(id)
Referenced by:
TABLE "country_translations" CONSTRAINT "fk_rails_0c4ee35f26" FOREIGN KEY (country_id) REFERENCES countries(id) ON DELETE CASCADE
TABLE "countries_languages" CONSTRAINT "fk_rails_556e7398aa" FOREIGN KEY (country_id) REFERENCES countries(id) ON DELETE CASCADE
TABLE "cities" CONSTRAINT "fk_rails_996e05be41" FOREIGN KEY (country_id) REFERENCES countries(id)
As you can see, I have foreign key constraints on both currency_id and language_id fields.
When I run my tests I see that there are records in that table:
qq2_test=# select * from countries;
id | domain | root_city_id | language_id | currency_id | google_tag_manager_id | created_at | updated_at
-----------+--------+--------------+-------------+-------------+-----------------------+----------------------------+----------------------------
665664142 | com | | 1019186233 | 432072940 | | 2020-10-23 06:20:49.288637 | 2020-10-23 06:20:49.288637
169150333 | by | | 1019186233 | 432072940 | | 2020-10-23 06:20:49.288637 | 2020-10-23 06:20:49.288637
(2 rows)
There are two my test records and they have language and currency references. But their tables are empty:
qq2_test=# select * from currencies;
id | name | symbol | created_at | updated_at
----+------+--------+------------+------------
(0 rows)
qq2_test=# select * from languages;
id | name | locale | image | created_at | updated_at
----+------+--------+-------+------------+------------
(0 rows)
Why does postgresql allow nonexistent references in countries table?
Ruby 2.7.1 MRI
Rails: 6.0.3.4
Postgresql 12.4
Ubuntu 20.04
Assuming that you're the only person using the database (since you are talking about small, 1-2 row tests), I would guess that your Rails app (or corresponding driver) is disabling triggers or foreign key checks. It's totally possible to bypass the foreign key checks like so:
edb=# show session_replication_role ;
session_replication_role
--------------------------
origin
(1 row)
edb=# create table city (id int primary key, name text);
CREATE TABLE
edb=# select * from city;
id | name
----+------
(0 rows)
edb=# create table person (id int, name text, city int references city(id));
CREATE TABLE
edb=# insert into person values (1,'foo',1);
ERROR: insert or update on table "person" violates foreign key constraint "person_city_fkey"
DETAIL: Key (city)=(1) is not present in table "city".
edb=# set session_replication_role to replica;
SET
edb=# insert into person values (1,'foo',1);
INSERT 0 1
edb=# select * from person;
id | name | city
----+------+------
1 | foo | 1
(1 row)
edb=# select * from city;
id | name
----+------
(0 rows)
I would suggest that you temporarily set log_statement = all and run your tests again--then see in your Postgres server logs (default should be /var/log/postgresql/postgresql-12-main.log for Ubuntu) what might be disabling your foreign key constraint checks, then address your findings accordingly.
There are only two options:
The foreign key constraint is NOT VALID.
Such constraints are cheched for new entries, but existing entries can violate them.
But that would show up in your \d output, so that is not the case.
You have data corruption.
Apart from hardware problems or software bugs, possible explanations are:
Someone set session_replication_role = replica so that triggers don't fire.
A superuser ran
ALTER TABLE countries DISABLE TRIGGER ALL;
I used inheritance feature from Postgresql 9.6 to create some archive for huge amount of data.
Finally parent table looks like this
foo=# \d+ foo_data_sshcommand
Table "public.foo_data_sshcommand"
Column | Type | Modifiers | Storage | Stats target | Description
------------+--------------------------+-------------------------------------------------------------------+----------+--------------+-------------
id | integer | not null default nextval('foo_data_sshcommand_id_seq'::regclass) | plain | |
time | timestamp with time zone | not null | plain | |
command | text | not null | extended | |
success | boolean | not null | plain | |
session_id | integer | not null | plain | |
Indexes:
"foo_data_sshcommand_pkey" PRIMARY KEY, btree (id)
"foo_data_sshcommand_session_id_c8b38d68" btree (session_id)
Check constraints:
"partition_check" CHECK ("time" >= '2019-10-01 00:00:00+02'::timestamp with time zone) NO INHERIT
Foreign-key constraints:
"foo_data_sshcommand_session_id_c8b38d68_fk" FOREIGN KEY (session_id) REFERENCES foo_data_sshsession(id) DEFERRABLE INITIALLY DEFERRED
Child tables: arc_2019_01_foo_data_sshcommand,
arc_2019_02_foo_data_sshcommand,
arc_2019_03_foo_data_sshcommand,
arc_2019_04_foo_data_sshcommand,
arc_2019_05_foo_data_sshcommand,
arc_2019_06_foo_data_sshcommand,
arc_2019_07_foo_data_sshcommand,
arc_2019_08_foo_data_sshcommand,
arc_2019_09_foo_data_sshcommand,
arc_2019_foo_data_sshcommand
Every month it creates two more tables and it adds them to the archive. It is not nice process, because it must move overflowed data from parent table to new child table. But it is not the problem that I want to talk about. I am curious about question - How many tables can paricipate on inheritance of one parent table?
I would like to display Snort IPS events in my dashboard app, these events are written into a database via barnyard2. I'm going threw rails for zombies and they say to access info from tables out of a database the table should be plural, and the model is singular. You would then expect the table with the IPS events to be events and not event.
Barnyard2 comes with a create_postgres schema script. Below are the tables created by the script. They are all singular.
csdashboard=# \dt
List of relations
Schema | Name | Type | Owner
--------+-------------------+-------+-------------
public | data | table | csdashboard
public | detail | table | csdashboard
public | encoding | table | csdashboard
public | event | table | csdashboard
public | icmphdr | table | csdashboard
public | iphdr | table | csdashboard
public | opt | table | csdashboard
public | reference | table | csdashboard
public | reference_system | table | csdashboard
public | schema | table | csdashboard
public | sensor | table | csdashboard
public | sig_class | table | csdashboard
public | sig_reference | table | csdashboard
public | signature | table | csdashboard
public | tcphdr | table | csdashboard
public | udphdr | table | csdashboard
I do not need or want my app to enter anything into these tables, just find and display the info.
My questions are;
Do I need to create a model per table that I will be getting info from?
Since they should be plural, will this create a problem?
You do not have to stick with all conventions and there is an easy way to alter this one.
You can set table name in your model:
class Event < ActiveRecord::Base
self.table_name = 'event'
end
Yes. Yes.
You need a model per table. Without that you'll have big problems with ActiveRecord.
You'll have some other problems if your table name is not a plural of your model, once again with ActiveRecord.
In my Grails 2.3.7 project, I have a Product domain class, like this:
class Product {
String code
String description
Map attributes
static constraints = {
code(unique: true)
}
static mapping = {
code index: 'Code_Idx'
attributes fetch: 'join'
cache true
id generator: 'hilo'
}
}
It translates into this database:
create table product (id bigint not null, version bigint not null, code varchar(255) not null unique, description varchar(255) not null, primary key (id));
create table product_attributes (attributes bigint, attributes_idx varchar(255), attributes_elt varchar(255) not null);
create index Code_Idx on product (code);
With some 4000 products in database, the scaffold listing shows them just fine.
Except, when I click "sort" on code - because there is no index - so my server does this:
explain select this_.id as id14_0_, this_.version as version14_0_, this_.code as code14_0_, this_.description as descript4_14_0_,
attributes2_.attributes as attributes14_2_, attributes2_.attributes_elt as attributes3_2_, attributes2_.attributes_idx as attributes2_2_
from product this_ left outer join product_attributes attributes2_
on this_.id=attributes2_.attributes
order by lower(this_.code) desc limit 90, 100
+----+-------------+--------------+------+---------------+------+---------+------+-------+---------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+--------------+------+---------------+------+---------+------+-------+---------------------------------+
| 1 | SIMPLE | this_ | ALL | NULL | NULL | NULL | NULL | 4086 | Using temporary; Using filesort |
| 1 | SIMPLE | attributes2_ | ALL | NULL | NULL | NULL | NULL | 43975 | |
+----+-------------+--------------+------+---------------+------+---------+------+-------+---------------------------------+
Obviously, this takes ages. I can manually add the index:
ALTER TABLE `product_attributes` ADD INDEX(`attributes`);
and then it works OK. I think it should have been created automatically in the first place - there is very little sense in this schema without the index - but OK, I can ping Gorm to do it. My question is - what can I put into the domain class to have Gorm add this index ?
Grails doesn't automatically create indexes on such columns as 'attributes' in your example. In order to create and manage these indexes I highly recommend using the Database Migration Plugin. The documentation is well written, and outlines how to use it.
I've got a table like this:
table: searches
+------------------------------+
| id | address | date |
+------------------------------+
| 1 | 123 foo st | 03/01/13 |
| 2 | 123 foo st | 03/02/13 |
| 3 | 456 foo st | 03/02/13 |
| 4 | 567 foo st | 03/01/13 |
| 5 | 456 foo st | 03/01/13 |
| 6 | 567 foo st | 03/01/13 |
+------------------------------+
And want a result set like this:
+------------------------------+
| id | address | date |
+------------------------------+
| 2 | 123 foo st | 03/02/13 |
| 3 | 456 foo st | 03/02/13 |
| 4 | 567 foo st | 03/01/13 |
+------------------------------+
But ActiveRecord seems unable to achieve this result. Here's what I'm trying:
Model has a 'most_recent' scope: scope :most_recent, order('date_searched DESC')
Model.most_recent.uniq returns the full set (SELECT DISTINCT "searches".* FROM "searches" ORDER BY date DESC) -- obviously the query is not going to do what I want, but neither is selecting only one column. I need all columns, but only rows where the address is unique in the result set.
I could do something like Model.select('distinct(address), date, id'), but that feels...wrong.
You could do a
select max(id), address, max(date) as latest
from searches
group by address
order by latest desc
According to sqlfiddle that does exactly what I think you want.
It's not quite the same as your requirement output, which doesn't seem to care about which ID is returned. Still, the query needs to specify something, which is here done by the "max" aggregate function.
I don't think you'll have any luck with ActiveRecord's autogenerated query methods for this case. So just add your own query method using that SQL to your model class. It's completely standard SQL that'll also run on basically any other RDBMS.
Edit: One big weakness of the query is that it doesn't necessarily return actual records. If the highest ID for a given address doesn't corellate with the highest date for that address, the resulting "record" will be different from the one actually stored in the DB. Depending on the use case that might matter or not. For Mysql simply changing max(id) to id would fix that problem, but IIRC Oracle has a problem with that.
To show unique addresses:
Searches.group(:address)
Then you can select columns if you want:
Searches.group(:address).select('id,date')