My project uses a few custom PostgreSQL stored functions for some features that would be a pain in raw SQL or ActiveRecord. Every now and then I will run the RSpec test suite, and find that all my stored functions have been blown away. Re-running the migrations to create them fixes the problem, but "rake db:structure:load" does NOT.
I am deeply confused. I never drop either the dev or test database unless this happens, but my functions are like Schrodinger's PL/pgSQL. I am REALLY hoping this never happens in production.
Here is an example of a failing test and my attempts to fix it:
ActiveRecord::StatementInvalid:
PG::UndefinedFunction: ERROR: function round_half_down(numeric) does not exist
# Damn. We have to drop the database so we can reload structure.sql:
$ RAILS_ENV=test rake db:drop
$ RAILS_ENV=test rake db:create
# load structure.sql instead of schema.rb:
$ RAILS_ENV=test rake db:structure:load
# Not fixed:
ActiveRecord::StatementInvalid:
PG::UndefinedFunction: ERROR: function round_half_down(numeric) does not exist
$ RAILS_ENV=test rake db:migrate:redo VERSION=20160421184708
== 20171002190107 CreateRoundHalfDownFunction: reverting ======================
-- execute("DROP FUNCTION IF EXISTS round_half_down(numeric)")
-> 0.0004s
== 20171002190107 CreateRoundHalfDownFunction: reverted (0.0005s) =============
== 20171002190107 CreateRoundHalfDownFunction: migrating ======================
-- execute("CREATE OR REPLACE FUNCTION ROUND_HALF_DOWN(NUMERIC)\n RETURNS NUMERIC LANGUAGE SQL AS\n$FUNC$\n SELECT CASE WHEN ($1%1) < 0.6 THEN FLOOR($1) ELSE CEIL($1) END;\n$FUNC$\n")
-> 0.0014s
== 20171002190107 CreateRoundHalfDownFunction: migrated (0.0014s) =============
Now it is fixed!
Yes, I verified that the function is present in structure.sql:
--
-- Name: round_half_down(numeric); Type: FUNCTION; Schema: public; Owner: -
--
CREATE FUNCTION round_half_down(numeric) RETURNS numeric
LANGUAGE sql
AS $_$
SELECT CASE WHEN ($1%1) < 0.6 THEN FLOOR($1) ELSE CEIL($1) END;
$_$;
For the record, this stopped happening to me with newer versions of the pg gem and PostgreSQL itself. I also added config.active_record.schema_format = :sql to application.rb, because my application makes heavy use of Postgres-specific features, and a number of stored functions.
Related
I'm attempting to use Heroku's CI to run my Rails application's tests but it's running into a problem when attempting to load my structure.sql file.
-----> Preparing test database
Running: rake db:schema:load_if_ruby
db:schema:load_if_ruby completed (3.24s)
Running: rake db:structure:load_if_sql
psql:/app/db/structure.sql:28: ERROR: must be owner of extension plpgsql
rake aborted!
failed to execute:
psql -v ON_ERROR_STOP=1 -q -f /app/db/structure.sql d767koa0m1kne1
Please check the output above for any errors and make sure that `psql` is installed in your PATH and has proper permissions.
/app/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.1/lib/active_record/tasks/postgresql_database_tasks.rb:108:in `run_cmd'
/app/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.1/lib/active_record/tasks/postgresql_database_tasks.rb:80:in `structure_load'
/app/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.1/lib/active_record/tasks/database_tasks.rb:223:in `structure_load'
/app/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.1/lib/active_record/tasks/database_tasks.rb:236:in `load_schema'
/app/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.1/lib/active_record/tasks/database_tasks.rb:255:in `block in load_schema_current'
/app/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.1/lib/active_record/tasks/database_tasks.rb:304:in `block in each_current_configuration'
/app/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.1/lib/active_record/tasks/database_tasks.rb:303:in `each'
/app/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.1/lib/active_record/tasks/database_tasks.rb:303:in `each_current_configuration'
/app/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.1/lib/active_record/tasks/database_tasks.rb:254:in `load_schema_current'
/app/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.1/lib/active_record/railties/databases.rake:290:in `block (3 levels) in <top (required)>'
/app/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.1/lib/active_record/railties/databases.rake:294:in `block (3 levels) in <top (required)>'
/app/vendor/bundle/ruby/2.4.0/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => db:structure:load
(See full trace by running task with --trace)
!
! Could not prepare database for test
!
The relevant line here is:
psql:/app/db/structure.sql:28: ERROR: must be owner of extension plpgsql
rake aborted!
Structure.sql contains this line:
COMMENT ON EXTENSION plpgsql IS 'PL/pgSQL procedural language';
Any ideas on how to get this working on Heroku's CI?
Ended up overriding db:structure:dump to remove the COMMENT ON ... statements:
namespace :db do
namespace :structure do
task dump: [:environment, :load_config] do
filename = ENV["SCHEMA"] || File.join(ActiveRecord::Tasks::DatabaseTasks.db_dir, "structure.sql")
sql = File.read(filename).each_line.grep_v(/\ACOMMENT ON EXTENSION.+/).join
File.write(filename, sql)
end
end
end
Another workaround would be to add something like
if Rails.env.development?
ActiveRecord::Tasks::DatabaseTasks.structure_load_flags = ["-v", "ON_ERROR_STOP=0"]
end
anywhere in the initialisation / tasks pipeline before the db:structure:load is executed.
If Kyle's solution isn't enough and the errors aren't caused only by comments on extensions, but actual extensions installations, you can still go the hard way and add this to an initializer:
# This is a temporary workaround for the Rails issue #29049.
# It could be safely removed when the PR #29110 got merged and released
# to use instead IGNORE_PG_LOAD_ERRORS=1.
module ActiveRecord
module Tasks
class PostgreSQLDatabaseTasks
ON_ERROR_STOP_1 = 'ON_ERROR_STOP=0'.freeze
end
end
end
Note: This isn't specific to Heroku but a broader Rails 5.1 issue
There are two solutions to this problem. First, as it was previously noted, is disabling the ON_ERROR_STOP feature. It'd help regardless of the environment. Custom rake task:
namespace :db do
namespace :structure do
# This little task is a workaround for a problem introduced in Rails5. Most specificaly here
# https://github.com/rails/rails/blob/5-1-stable/activerecord/lib/active_record/tasks/postgresql_database_tasks.rb#L77
# When psql encounters an error during loading of the structure it exits at once with error code 1.
# And this happens on heroku. It renders review apps and heroku CI unusable if you use a structure instead of a schema.
# Why?
# Our `db/structure.sql` contains entries like `CREATE EXTENSION` or `COMMENT ON EXTENSION`.
# Zylion of extensions on heroku are loaded in template0, so "our" db also has them, but because of that
# only a superuser (or owner of template0) has access to them - not our heroku db user. For that reason
# we can neither create an extension (it already exists, but that is not a problem, because dump contains IF NOT EXIST)
# nor comment on it (and comments don't have IF NOT EXIST directive). And that's an error which could be safely ignored
# but which stops loading of the rest of the structure.
desc "Disable exit-on-error behaviour when loading db structure in postgresql"
task disable_errors: :environment do
ActiveRecord::Tasks::DatabaseTasks.structure_load_flags = ["-v", "ON_ERROR_STOP=0"]
end
end
end
# And use it like so:
bin/rails db:structure:disable_errors db:structure:load
Another option, in my opinion, superior if it comes only to Heroku, would be using PostgreSQL in in-dyno plan (https://devcenter.heroku.com/articles/heroku-ci-in-dyno-databases), which is basically a DB instance sitting within dyno, thus we have full access to it. Also, the test suite should be significantly faster because we use a localhost connection, not over the wire. To enable it, change your app.json content to have entries like so:
{
"environments": {
"test": {
"addons": [
"heroku-postgresql:in-dyno"
]
}
}
}
I am running my app on heroku, using the heroku pg stack, and when I run db:migrate it reverts my previous migrations instead of moving forward.
rake db:version shows the current migration
Current version: 20160516172744
rake db:migrate:status shows all migrations as up
......(more above all up)
up 20160512175053 Create spree roles permissions.spree admin roles and access
up 20160512175054 Add editable is default and index on editable is default and name to spree roles.spree admin roles and access
up 20160513135317 Add indexes for speed
up 20160513140704 Add filter search params to spree product
up 20160516172744 Add tsvector colums to spree products
But when I hit heroku run rake db:migrate this is the output:
Migrating to AddTsvectorColumsToSpreeProducts (20160516172744)
== 20160516172744 AddTsvectorColumsToSpreeProducts: reverting =================
-- execute(" DROP FUNCTION IF EXISTS spree_products_tsv_trigger() CASCADE;\n")
-> 0.0031s
-- remove_index(:spree_products, :tsv)
-> 0.0058s
-- remove_column(:spree_products, :tsv)
-> 0.0025s
== 20160516172744 AddTsvectorColumsToSpreeProducts: reverted (0.0122s) ========
Migrating to AddFilterSearchParamsToSpreeProduct (20160513140704)
== 20160513140704 AddFilterSearchParamsToSpreeProduct: reverting ==============
-- remove_column(:spree_products, :designer_id)
-> 0.0062s
-- remove_column(:spree_products, :main_taxon_id)
-> 0.0024s
-- remove_column(:spree_products, :colour_id)
-> 0.0025s
-- remove_column(:spree_products, :size_id)
-> 0.0045s
-- remove_column(:spree_products, :condition_id)
-> 0.0023s
-- remove_column(:spree_products, :on_site)
-> 0.0023s
-- remove_column(:spree_products, :sgd_price)
-> 0.0025s
-- remove_column(:spree_products, :search_designer)
-> 0.0022s
-- remove_column(:spree_products, :search_category)
-> 0.0021s
-- remove_column(:spree_products, :search_sku)
-> 0.0022s
== 20160513140704 AddFilterSearchParamsToSpreeProduct: reverted (0.0341s) =====
.....(cont.)
Any ideas?
-Dan
rake db:migrate will, without any arguments, only migrate up to the latest version. If you ever see it migrate down, you most likely have the VERSION environment variable set to some value (see the docs for explanation of how that variable works), and Rails is trying to migrate to that version.
In your case, per the comments, it looks like you'd set VERSION=v3 in your environment. Rails is likely casting v3 to 0 and trying to migrate all migrations down.
I'm new to rake and I'm trying to find my way in automating some tasks. So I wrote my first rake task and failed:
namespace :app do
desc "Leaves application like new"
task :reset => :environment do
Rake::Task['db:drop:all'].invoke
Rake::Task['db:create:all'].invoke
Rake::Task['db:migrate'].invoke
Rake::Task['db:seed'].invoke
end
end
I'd like to know why this isn't working. After calling:
rake app:reset
everything runs fine, I can see the migration messages on screen, like this:
== CreateGalerias: migrating =================================================
-- create_table(:galerias)
NOTICE: CREATE TABLE will create implicit sequence "galerias_id_seq" for serial column "galerias.id"
NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "galerias_pkey" for table "galerias"
-> 0.1191s
== CreateGalerias: migrated (0.1194s) ========================================
But, at the end I get this message:
You have 11 pending migrations:
20110704052637 CreatePersonas
20110709100632 CreateOrganizaciones
20110709100646 CreateEventos
20110816102451 CreateMembresias
20110816155851 CreateCelebraciones
20110822135820 ActsAsTaggableOnMigration
20120410063100 CreateDocumentos
20120507200516 CreateUsuarios
20120515214226 ActivaUnnaccent
20120516091228 CreateGalerias
20120517004708 SetupHstore
Run `rake db:migrate` to update your database then try again.
Didn't it just migrated the database? why is it complaining about it?
Keep in mind that db:drop:all and db:create:all operate on all environments, and db:migrate and db:seed do not, so you are probably migrating in an unintended environment. Try changing db:drop:all to db:drop and db:create:all to db:create, and run the task specifying a particular environment like:
rake RAILS_ENV=production app:reset
I am getting the following uuid error while running a rails app with postgres as backend. Can someone help me out with which dependency is needed.
[root#localhost webapp]# rake db:migrate
(in /root/mysite/webapp)
== CreateContributors: migrating =============================================
-- create_table(:contributors, {:id=>false})
-> 0.0121s
-- execute("alter table contributors add primary key (id)")
NOTICE: ALTER TABLE / ADD PRIMARY KEY will create implicit index "contributors_pkey" for table "contributors"
-> 0.0797s
-- execute("alter table contributors alter column id set default uuid_generate_v1()::varchar")
rake aborted!
An error has occurred, this and all later migrations canceled:
PGError: ERROR: function uuid_generate_v1() does not exist
HINT: No function matches the given name and argument types. You might need to add explicit type casts.
: alter table contributors alter column id set default uuid_generate_v1()::varchar
The uuid_generate_v1() function is part of the uuid-ossp package and you have to install that into PostgreSQL before you can use it. You should have a file called uuid-ossp.sql in your PostgreSQL contrib directory. You can install the package with:
$ psql -d your_database < /the/path/to/uuid-ossp.sql
You'll probably want to run that as the database super user.
I have a rake task that pulls some data from a MS SQL db, and recently I noticed that some special characters get encoded as a question mark '?'. While trying to dig into this issue, I realized that I could only repo it as a rake task, but not from the console.
Here is a snippet of what the code looks like:
require 'dbi'
namespace :db do
task :test => :environment do
db2 = DBI.connect("DBI:ODBC:DRIVER=FreeTDS;SERVER=x.x.x.x;PORT=1433;DATABASE=MyDB;TDS_VERSION=8.0;UID=user_id;PWD=pass")
rows = db2.execute('select * from Topic where id = 123')
rows.each { |r| puts r['name'] }
rows.finish
end
end
when I run this as:
rake RAILS_ENV=production db:test
it produces:
The Devil?s Tail
But when I run the exact same commands using:
/> script/console production
I get
The Devil`s Tail
Notice the back tick? Anyone have any idea why may be causing this? I double checked the ENV variables, and they both have LANG: en_US.UTF-8
EDIT
Forgot to mention I am using ruby 1.8.7p72 and Rails 2.3.4