I'm trying to write (Ruby) script that will drop all the foreign key and unique constraints in my PostgreSQL DB and then re-add them.
The FK part seems to be working OK.
However, dropping and recreating unique constraints isn't working.
I think the reason is that when the unique constraint is created, PostgreSQL creates an index along with it, and that index doesn't get automatically dropped when the unique constraint is dropped. So then when the script tries to re-add the unique constraint, I get an error like...
PG::Error: ERROR: relation "unique_username" already exists
: ALTER TABLE users ADD CONSTRAINT unique_username UNIQUE (username)
And indeed when I look at the DB in the pgAdmin GUI utility, that index exists.
The question is, how do I find it in my script and drop it?
Here's my script...
manage_constraints.rake
namespace :journal_app do
desc 'Drop constraints'
task :constraints_drop => :environment do
sql = %Q|
SELECT
constraint_name, table_catalog, table_name
FROM
information_schema.table_constraints
WHERE
table_catalog = 'journal_app_#{Rails.env}'
AND
constraint_name NOT LIKE '%_pkey'
AND
constraint_name NOT LIKE '%_not_null';
|
results = execute_sql(sql)
results.each do |row|
puts "Dropping constraint #{row['constraint_name']} from table #{row['table_name']}."
execute_sql("ALTER TABLE #{row['table_name']} DROP CONSTRAINT #{row['constraint_name']}")
end
end
# --------------------------------------------------------------------------------------------------------------------
desc 'Drops constraints, then adds them'
task :constraints_add => :environment do
Rake::Task['journal_app:constraints_drop'].invoke
UNIQUE_KEYS = [
{
:name => 'unique_username',
:table => 'users',
:columns => ['username']
},
{
:name => 'unique_email',
:table => 'users',
:columns => ['email']
}
]
FKs = [
{
:name => 'fk_entries_users',
:parent_table => 'users',
:child_table => 'entries',
:on_delete => 'CASCADE'
},
{
:name => 'fk_entries_entry_tags',
:parent_table => 'entries',
:child_table => 'entry_tags',
:on_delete => 'CASCADE'
},
# etc...
]
UNIQUE_KEYS.each do |constraint|
sql = "ALTER TABLE #{constraint[:table]} ADD CONSTRAINT #{constraint[:name]} UNIQUE (#{constraint[:columns].join(', ')})"
puts "Adding unique constraint #{constraint[:name]} to table #{constraint[:table]}."
puts ' SQL:'
puts " #{sql}"
execute_sql(sql)
end
FKs.each do |fk|
sql = %Q|
ALTER TABLE #{fk[:child_table]} ADD CONSTRAINT #{fk[:name]} FOREIGN KEY (#{fk[:parent_table].singularize}_id)
REFERENCES #{fk[:parent_table]} (id)
ON UPDATE NO ACTION ON DELETE #{fk[:on_delete]}|.strip!
puts "Adding foreign key #{fk[:name]}."
puts ' SQL:'
puts " #{sql}"
execute_sql(sql)
end
end
end
def execute_sql(sql)
ActiveRecord::Base.connection.execute(sql)
end
First, why do such a thing? This has the feel of one of those "I've decided on solution Y to problem X, and am having a problem with solution Y that I'm asking about" - where the real answer is "use solution Z not solution Y to solve problem X". In other words, try explaining the underlying problem you are having, there might be a much better way to solve it.
If you must do it, query pg_catalog.pg_index inner join pg_class on pg_class.oid = pg_index.indexrelid for indexes that are not indisprimary and exclude anything with EXISTS (SELECT 1 FROM pg_constraint on pg_index.indrelid = pg_constraint.conindid).
eg:
SELECT pg_class.relname
FROM pg_index INNER JOIN pg_class ON (pg_class.oid = pg_index.indexrelid)
INNER JOIN pg_namespace ON (pg_class.relnamespace = pg_namespace.oid)
WHERE NOT EXISTS (
SELECT 1 FROM pg_constraint WHERE pg_index.indrelid = pg_constraint.conindid
)
AND pg_index.indisprimary = 'f'
AND pg_namespace.nspname NOT LIKE 'pg_%';
Be aware that such queries may break in any major version transition as the pg_catalog is not guaranteed to retain the same schema across versions. Query the Pg version and use version-specific queries if necessary. Sound painful? It is, but it shouldn't generally be necessary, you're just doing something kind of weird.
For most purposes the very stable information_schema is quite sufficient.
Related
I have a problem with Yii eager loading.
I open user profile page and use:
$model=User::model()->with('routes', 'likes', 'comments', 'questions', 'cityname')->findByPk($id);
Relations is:
public function relations()
{
return array(
'routes'=>array(self::HAS_MANY, 'Route', 'author_id', 'order'=>'routes.id DESC'),
'questions'=>array(self::HAS_MANY, 'Question', 'author_id', 'order'=>'questions.id DESC'),
'comments'=>array(self::HAS_MANY, 'Comment', 'author_id', 'order'=>'comments.id DESC',),
'likes'=>array(self::HAS_MANY, 'Like', 'author_id', 'order'=>'likes.id DESC'),
'cityname'=>array(self::BELONGS_TO, 'City', 'city'),
);
}
When i have around 70 (or more) comments in Comment table, i have error:
Fatal error: Out of memory (allocated 348651520) (tried to allocate 78 bytes) in /home/milk/kolyasya.ru/diplomyii/framework/db/CDbCommand.php on line 516
The interesting part of this problem, is if i comment any element of with(), for example:
$model=User::model()->with('routes', 'likes', 'comments', /* 'questions' */, 'cityname')->findByPk($id);
then all works as it should.
I checked all relations in all models and set ini_set('memory_limit', '512M'), but i can't find a source of the problem.
Maybe I need to use lazy loading?
You're suffering from exploding number of row combinations. Take a look at this question, it describes the same problem on a smaller scale. Basically, you are running a huge query with multiple one-to-many joins, similar to this:
SELECT ... FROM `User` `t`
LEFT JOIN `Route` routes ON t.id = routes.author_id
LEFT JOIN `Question` questions ON t.id = questions.author_id
LEFT JOIN `Comment` comments ON t.id = comments.author_id
LEFT JOIN `Like` likes ON t.id = likes.author_id
LEFT JOIN `City` city ON t.city = city.id
WHERE t.id = :id
ORDER BY routes.id DESC, questions.id DESC, comments.id DESC, likes.id DESC
You can take this query, modify it to SELECT COUNT(*) and run it in phpMyAdmin to see how many rows it returns. It will be equal to the number of routes multiplied by the number of questions multiplied by the number of comments multiplied by the number of likes created by this user.
In this situation, it would be a lot more efficient to fetch each HAS_MANY relation in a separate query. Yii can do that:
$model=User::model()
->with(array(
'routes' => array('together' => false),
'likes' => array('together' => false),
'comments' => array('together' => false),
'questions' => array('together' => false),
'cityname' => array(),
))
->findByPk($id);
If you do so, Yii will instead produce multiple SQL queries with less memory usage, similar to the following:
SELECT ... FROM `User` `t`
LEFT JOIN `City` `city` ON `t`.`city` = `city`.`id`
WHERE `t`.`id` = :id;
SELECT ... FROM `Route` `routes`
WHERE `author_id` = :id
ORDER by `routes`.`id` DESC;
SELECT ... FROM `Question` `questions`
WHERE `author_id` = :id
ORDER BY `questions`.`id` DESC;
SELECT ... FROM `Comment` `comments`
WHERE `author_id` = :id
ORDER BY `comments`.`id` DESC;
SELECT ... FROM `Like` `likes`
WHERE `author_id` = :id
ORDER BY `likes`.`id` DESC;
The results will be aggregated and returned to your code just like before.
On Mysql it works correctly.
PG::Error: ERROR: for SELECT DISTINCT, ORDER BY expressions must
appear in select list LINE 1: ) ORDER BY
programs.rating DESC, program_sc... ^ :
Query:
SELECT DISTINCT "programs".* FROM "programs" INNER JOIN
"program_schedules" ON "program_schedules"."program_id" =
"programs"."id" WHERE (programs.rating >= 5 AND
program_schedules.start >= '2012-11-03 23:14:43.457659') AND (ptype =
'movie') ORDER BY programs.rating DESC,
program_schedules.start DESC
Rails code:
#data =
Program.joins(:program_schedules).where('programs.rating >= ?
AND program_schedules.start >= ?',5,
Time.now).order('programs.rating DESC,
program_schedules.start DESC').uniq
I have tried with
Program.select("programs.*,
program_schedules.*).joins(:program_schedules).where(...
but, in this way, when I'm going to read
#data.program_schedules
I get a nil value (When I know there are no nil values)
PostgreSQL 9.2 (Heroku), Ruby 1.9.2
Some info about my DB:
class Program < ActiveRecord::Base
has_many :program_schedules
end
class ProgramSchedule < ActiveRecord::Base
belongs_to :program
end
Schema.db
create_table "program_schedules", :force => true do |t|
t.integer "program_id"
t.datetime "start"
end
create_table "programs", :force => true do |t|
t.string "title",
t.string "ptype"
end
EDIT:
I don't need to order "programs_schedules" because I need all programs_schedules in my array related to that program.
You query is ambiguous in two ways:
ptype is not table-qualified and you did not disclose the table definitions. So the query is ambiguous.
More importantly, you want to:
ORDER BY programs.rating DESC, program_schedules.start DESC
At the same time, however, you instruct PostgreSQL to give you DISTINCT rows from programs. If there are multiple matching rows in program_schedules, how would Postgres know which one to pick for the ORDER BY clause? The first? Last? Earliest, latest, greenest? It's just undefined.
Generally, the ORDER BY clause cannot disagree with the DISTINCT clause, that's what the error message tells you.
Based on a few assumptions, filling in for missing information, your query could look like this:
SELECT p.*
FROM programs p
JOIN program_schedules ps ON ps.program_id = p.id
WHERE p.rating >= 5
AND ps.start >= '2012-11-03 23:14:43.457659'
AND p. ptype = 'movie' -- assuming ptype is from programs (?)
GROUP BY p.id -- assuming it's the primary key
ORDER BY p.rating DESC, min(ps.start) DESC; -- assuming smallest start
Also assuming you have PostgreSQL 9.0 or later, which is required for this to work. (Primary key covers whole table in GROUP BY.)
As for:
On Mysql it works correctly.
No it doesn't. It "works", but in mysterious ways rather than "correctly". MySQL allows for all sorts of weird mistakes and goes out of its way (and the SQL standard) to avoid having to throw exceptions - which is a very unfortunate way to deal with errors. It regularly comes back to haunt you later. Demo on Youtube.
Question update
I need all programs_schedules in my array related to that program.
You might want to add:
SELECT p.*, array_agg(ps.start ORDER BY ps.start)
Piggy backing off another question I posted, I have a complex find() that changes whether or not a certain id is nil or not. See here:
if self.id.nil?
blocks = AppointmentBlock.find(:first,
:conditions => ['appointment_blocks.employee_id = ? and ' +
'(time_slots.start_at between ? and ? or time_slots.end_at between ? and ?)',
self.employee_id, self.time_slot.start_at, self.time_slot.end_at,
self.time_slot.start_at, self.time_slot.end_at],
:joins => 'join time_slots on time_slots.time_slot_role_id = appointment_blocks.id')
else
blocks = AppointmentBlock.find(:first,
:conditions => ['appointment_blocks.id != ? and ' +
'appointment_blocks.employee_id = ? and ' +
'(time_slots.start_at between ? and ? or time_slots.end_at between ? and ?)',
self.id, self.employee_id, self.time_slot.start_at, self.time_slot.end_at,
self.time_slot.start_at, self.time_slot.end_at],
:joins => 'join time_slots on time_slots.time_slot_role_id = appointment_blocks.id')
end
I'm wondering if there is a gem out there that lets me pass in :first and :conditions type stuff as a block of code. I saw ez_where on github but wasn't sure if it was abandoned or not since its had no activity lately (although that could mean its very solid with no bugs) Any ideas?
You can also have a look at Arel which:
Arel is a Relational Algebra for Ruby.
It 1) simplifies the generation
complex of SQL queries
The approach is built into Rails 3 as well. Provides really elegant support for building complex chains of scopes and queries.
When I have a complicated sql query involving a lot of joins, UNION, etc, I create a view in mysql. Then I create a ActiveRecord model and am able to directly read the results via ActiveRecord.
Another advantage of this technique is that the complicated SQL and its execution plan will be calculated only once by the dbms (when you create the view) rather than on each execution.
Depending on the situation, I use either rake or a migration to create/re-create the views.
Note that you'll usually want to re-create the views if any of the underlying tables' schemas are changed.
Added:
Example
class SessionViews2 < ActiveRecord::Migration
def self.up
execute "DROP View if exists room_usages" # Dropping both view and table
execute "DROP Table if exists room_usages" # has helped in some corner cases
execute "CREATE VIEW room_usages AS
SELECT
CONCAT(rooms.id,schedules.id) as id
, rooms.id as room_id
, rooms.conference_id as conference_id
, rooms.popular_room as room_popular_room
, schedules.id as schedule_id
, schedules.timetable_id as schedule_timetable_id
, schedules.display_start_time as schedule_display_start_time
, schedules.display_end_time as schedule_display_end_time
, sess.id as session_id
, sess.title as title
, sess.subtitle as subtitle
FROM
rooms
INNER JOIN schedule_rooms ON rooms.id = schedule_rooms.room_id
INNER JOIN schedules ON schedule_rooms.schedule_id = schedules.id
INNER JOIN sessions as sess ON schedules.session_id = sess.id
WHERE
schedule_rooms.cancelled_time IS NULL
AND schedules.cancelled_time IS NULL"
end
def self.down
execute "DROP View if exists room_usages"
execute "DROP Table if exists room_usages"
end
end
Model room_usage.rb
class RoomUsage < ActiveRecord::Base
############################################################
# #
# This is a VIEW! No DB table, Read-Only #
# #
############################################################
belongs_to :conference
belongs_to :timetable, :foreign_key => :schedule_timetable_id
belongs_to :session
end
I have a named scopes like so...
named_scope :gender, lambda { |gender| { :joins => {:survey_session => :profile }, :conditions => { :survey_sessions => { :profiles => { :gender => gender } } } } }
and when I call it everything works fine.
I also have this average method I call...
Answer.average(:rating, :include => {:survey_session => :profile}, :group => "profiles.career")
which also works fine if I call it like that.
However if I were to call it like so...
Answer.gender('m').average(:rating, :include => {:survey_session => :profile}, :group => "profiles.career")
I get...
ActiveRecord::StatementInvalid: PGError: ERROR: table name "profiles" specified more than once
: SELECT avg("answers".rating) AS avg_rating, profiles.career AS profiles_career FROM "answers" LEFT OUTER JOIN "survey_sessions" survey_sessions_answers ON "survey_sessions_answers".id = "answers".survey_session_id LEFT OUTER JOIN "profiles" ON "profiles".id = "survey_sessions_answers".profile_id INNER JOIN "survey_sessions" ON "survey_sessions".id = "answers".survey_session_id INNER JOIN "profiles" ON "profiles".id = "survey_sessions".profile_id WHERE ("profiles"."gender" = E'm') GROUP BY profiles.career
Which is a little hard to read but says I'm including the table profiles twice.
If I were to just remove the include from average it works but it isn't really practical because average is actually being called inside a method which gets passed the scoped. So there is some times gender or average might get called with out each other and if either was missing the profile include it wouldn't work.
So either I need to know how to fix this apparent bug in Rails or figure out a way to know what scopes were applied to a ActiveRecord::NamedScope::Scope object so that I could check to see if they have been applied and if not add the include for average.
Looks like ActiveRecord is generating some bad SQL:
SELECT avg("answers".rating) AS avg_rating,
profiles.career AS profiles_career
FROM "answers"
LEFT OUTER JOIN "survey_sessions" survey_sessions_answers
ON "survey_sessions_answers".id = "answers".survey_session_id
LEFT OUTER JOIN "profiles"
ON "profiles".id = "survey_sessions_answers".profile_id
INNER JOIN "survey_sessions"
ON "survey_sessions".id = "answers".survey_session_id
INNER JOIN "profiles"
ON "profiles".id = "survey_sessions".profile_id
WHERE ("profiles"."gender" = E'm')
GROUP BY profiles.career
Presumably it's generated the left joins as part of getting the projected property, and the inner joins as part of getting the criteria: this wouldn't be invalid (just inefficient) if it assigned aliases to those tables, but it doesn't. Is there a way to specify an alias name from your app?
In my online store, an order is ready to ship if it in the "authorized" state and doesn't already have any associated shipments. Right now I'm doing this:
class Order < ActiveRecord::Base
has_many :shipments, :dependent => :destroy
def self.ready_to_ship
unshipped_orders = Array.new
Order.all(:conditions => 'state = "authorized"', :include => :shipments).each do |o|
unshipped_orders << o if o.shipments.empty?
end
unshipped_orders
end
end
Is there a better way?
In Rails 3 using AREL
Order.includes('shipments').where(['orders.state = ?', 'authorized']).where('shipments.id IS NULL')
You can also query on the association using the normal find syntax:
Order.find(:all, :include => "shipments", :conditions => ["orders.state = ? AND shipments.id IS NULL", "authorized"])
One option is to put a shipment_count on Order, where it will be automatically updated with the number of shipments you attach to it. Then you just
Order.all(:conditions => [:state => "authorized", :shipment_count => 0])
Alternatively, you can get your hands dirty with some SQL:
Order.find_by_sql("SELECT * FROM
(SELECT orders.*, count(shipments) AS shipment_count FROM orders
LEFT JOIN shipments ON orders.id = shipments.order_id
WHERE orders.status = 'authorized' GROUP BY orders.id)
AS order WHERE shipment_count = 0")
Test that prior to using it, as SQL isn't exactly my bag, but I think it's close to right. I got it to work for similar arrangements of objects on my production DB, which is MySQL.
Note that if you don't have an index on orders.status I'd strongly advise it!
What the query does: the subquery grabs all the order counts for all orders which are in authorized status. The outer query filters that list down to only the ones which have shipment counts equal to zero.
There's probably another way you could do it, a little counterintuitively:
"SELECT DISTINCT orders.* FROM orders
LEFT JOIN shipments ON orders.id = shipments.order_id
WHERE orders.status = 'authorized' AND shipments.id IS NULL"
Grab all orders which are authorized and don't have an entry in the shipments table ;)
This is going to work just fine if you're using Rails 6.1 or newer:
Order.where(state: 'authorized').where.missing(:shipments)