Heroku made a change to the way postgressql extensions gets installed
This is screwing up new rails review apps in heroku with the following error.
ERROR: extension "btree_gist" must be installed in schema "heroku_ext"
This is screwing up things as I need to drop existing extensions and re-enable with heroku_ext schema. I use bin/rails db:structure:load which is before running a migration.
Also the structure.sql is going to diverge as heroku add the schema in review app and we need to run the creation manually in local dev machine.
Does anybody came across this issue?
I have developed a monkey-patch solution that is, well, definitely a hack, but better than pre-dating migrations, or deleting lines from schema.rb.
Put this in config/initializers. I called mine _enable_extension_hack.rb to make sure it gets loaded early.
module EnableExtensionHerokuMonkeypatch
# Earl was here
def self.apply_patch!
adapter_const = begin
Kernel.const_get('ActiveRecord::ConnectionAdapters::PostgreSQLAdapter')
rescue NameError => ne
puts "#{self.name} -- #{ne}"
end
if adapter_const
patched_method = adapter_const.instance_method(:enable_extension)
# Only patch this method if it's method signature matches what we're expecting
if 1 == patched_method&.arity
adapter_const.prepend InstanceMethods
end
end
end
module InstanceMethods
def enable_extension(name)
name_override = name
if schema_exists?('heroku_ext')
puts "enable_extension -- Adding SCHEMA heroku_ext"
name_override = "#{name}\" SCHEMA heroku_ext -- Ignore trailing double quote"
end
super name_override
end
end
end
EnableExtensionHerokuMonkeypatch.apply_patch!
What does it do? It monkey-patches the enable_extension call that's causing the problem in db/schema.rb. Specifically, if it finds that there is a schema called "heroku_ext", it will decorate the parameter given to enable_extension so that the SQL that gets executed specifies the "heroku_ext" schema. Like this:
CREATE EXTENSION IF NOT EXISTS "pg_stat_statements" SCHEMA heroku_ext -- Ignore trailing double quote"
Without this hack, the generated SQL looks like this:
CREATE EXTENSION IF NOT EXISTS "pg_stat_statements"
That's cleaner way of patching schema.rb
# config/initializers/patch_enable_extension.rb
require 'active_record/connection_adapters/postgresql_adapter'
# NOTE: patch for https://devcenter.heroku.com/changelog-items/2446
module EnableExtensionHerokuPatch
def enable_extension(name, **)
return super unless schema_exists?("heroku_ext")
exec_query("CREATE EXTENSION IF NOT EXISTS \"#{name}\" SCHEMA heroku_ext").tap { reload_type_map }
end
end
module ActiveRecord
module ConnectionAdapters
class PostgreSQLAdapter
prepend EnableExtensionHerokuPatch
end
end
end
EDIT: Here’s the official changelog as a reference https://devcenter.heroku.com/changelog-items/2446
You'll need to do the unthinkable because of the recent heroku changes and modify past migration files for your review apps to work with the new Heroku system for extensions.
Add a predated, or at the top of the first migration file connection.execute 'CREATE SCHEMA IF NOT EXISTS heroku_ext'
Potentially also add to the database.yml a schema_search_path that includes heroku_ext (or set it to public,heroku_ext if you hadn't customized it)
Grep all enable_extension('extension_name')
Replace them all by a connection.execute('CREATE EXTENSION IF NOT EXISTS "extension_name" WITH SCHEMA "heroku_ext")
Pray that is enough to fix the problem
After making those changes we still had to contact heroku support because of, in order:
a pgaudit stack is not empty error
The fix here was to run a maintenance twice (because the postgres add-on which was scheduled for maintenance pre-dated the schema/extension changes change)
a ERROR: function pg_stat_statements_reset(oid, oid, bigint) does not exist error
The fix here was a manual intervention from heroku on the databases. It was caused by heroku trying to run a pg_stat_statements_reset each time a schema is created.
This hack seems to be working for me. The following script runs in the postdeploy step in app.json.
#!/bin/bash -xue
# Create extensions in the schema where Heroku requires them to be created
# The plpgsql extension has already been created before this script is run
heroku pg:psql -a $HEROKU_APP_NAME -c 'create extension if not exists citext schema heroku_ext'
heroku pg:psql -a $HEROKU_APP_NAME -c 'create extension if not exists pg_stat_statements schema heroku_ext'
# Remove enable_extension statements from schema.rb before loading it, since
# even 'create extension if not exists' fails when the schema is not heroku_ext
mv db/schema.rb{,.orig}
grep -v enable_extension db/schema.rb.orig > db/schema.rb
rails db:schema:load
As of Tuesday 16th August 2022, we've found that a patch no longer seems to be needed and enable_extension appears to be working seamlessly as normal without the need to explicitly specify a schema.
When I mentioned this in our ongoing thread with Heroku Support, they said that some of their fixes around the heroku_ext change may have been released at the time of resolving https://status.heroku.com/incidents/2450, but that there are still further fixes in development and yet to be released - so my guess is that this particular issue is one on those that has been fixed!
The answer supplied by dzirtusss got me most of the way there, the last thing I had to do to get this working for my Rails application was to edit the search paths in config/database.yml to include heroku_ext. This is what I appended to my default configuration.
# config/database.yml
default:
schema_search_path: public,heroku_ext
After adding the initialization script and the database.yml edit, I was able to successfully run a heroku pg:copy from one application to another with no errors and no unexpected behavior.
Heroku has introduced an --extensions flag recently, which helps as a workaround. See this FAQ.
For example, if you're using the btree_gist extension, you can use heroku run pg:reset --extensions 'btree_gist' --app YOUR_APP_NAME before running your migrations, they should then work without any changes in your codebase.
Use the Heroku datastore durability tool to create a backup on the
source database or heroku pg:backups:capture -a <SOURCE_APP>.
Determine which pg extensions the database uses (can check from psql
with \dx)
create a comma-separated string of the extensions (e.g.:fuzzystrmatch,pg_stat_statements,pg_trgm,pgcrypto,plpgsql,unaccent,uuid-ossp')
Make sure your Heroku CLI is updated to at least version 7.63.0 (use
heroku update to update)
Run this:
heroku pg:backups:restore $(heroku pg:backups public-url -a <SOURCE_APP>) DATABASE_URL --extensions '<EXTENSIONS>' -a <TARGET_APP>
Reset dynos on the TARGET_APP
We have a legacy PostgreSQL database that is perfectly accessible from a Rails app except that when the vendor provides an update one of the consistency checks they perform against the database fails because Rails has built a "new" table there, the schema migrations table. Is it possible to direct creation of this table elsewhere? Or is it possible to use the schema cache in Rails 4 to effect this? The release notes section 3.3 on General things says "Schema cache dump (commit) - To improve Rails boot time, instead of loading the schema directly from the database, load the schema from a dump file."
I found an old blog post about this last time I tried it here. Copying the relevant parts:
To make a dump of your schema, execute the following rake task:
RAILS_ENV=production bundle exec rake db:schema:cache:dump
This will generate a file db/schema_cache.dump, that Rails will use to load the internal state of the SchemaCache instance.
To disable the schema cache dump, add the following to your config/production.rb file:
config.active_record.use_schema_cache_dump = false
If you would like to clear the schema cache, execute:
RAILS_ENV=production bundle exec rake db:schema:cache:clear
I would like to export the database into a single sql-file. I have taken on a project built by a developer who didn't seem to back it up before leaving the project. I need to take a copy of the database structure so that i can work on it in my local test-environment basically.
I have noticed the rake db:structure:dump -command, wich in the terminal gives me the following respons: rake aborted!
Don't know how to build task 'dataset:export'
Is there a gem I need to install or something, or how do you simply export the database into a single file?
You can do the following:
a) make a rails version of the schema with rake db:schema.dump
The generated file is db/schema.rb
you can load then it locally with db:schema:load
b) use the mysql tools to get real sql dump. See for example
http://dev.mysql.com/doc/refman/5.0/en/mysqldump.html
c) try to debug why the rake task fails. a problem may be that you need to set RAILS_ENV first, for example on the Unix-bash with
RAILS_ENV=production; export RAILS_ENV
But if you do not find a solution fast, try a) or b), because having the database structure is important before changing code, updating gems, etc.
I am migrating an app from rails 2.3.8 to 3.2.3. The 'rake db:schema:dump' works fine in rails 2.3
but generates only the tables names with no column names in rails 3.2.
Even for the app to connect successfully through the console, I had to change config/application.rb to include
ActiveRecord::Base.table_name_prefix = 'dbo.'
Do I need to do something different for the rake task to pick up these prefixes? Or is something else causing the missing column names problem?
Further clarification:
I am looking for rake db:schema:dump because the programmers on site stopped using migrations and started making changes to the db directly. Now I am trying to restart using migrations. The first step recommended in the process is to use the schema dump as the starting point. Also, (and I am not sure) it is needed for the tests to rebuild the test db from the development db.
Short answer:
db:schema:dump isn't the right thing to use, but you can add a few lines of code to your Rakefile to get the outcome you want.
Longer answer:
The scuttlebutt is that the task db:schema:dump is actually not supposed to dump anything more than the structure. (I know, it's a misnomer.) It's analagous to db:structure:dump, except that the one gives you an .rb file, and the other gives you a .sql file.
You can create your own dumping rake task by appending the following code to your Rakefile:
For SQL 2008
task :mydump do
ActiveRecord::Base.connection.execute(
"dbcc traceon(2544, -1) \n go \n dbcc traceon(2546, -1) \n go \n dbcc stackdump"
)
end
Using the SQL server itself to create the dump (which is what the forgoing code does) limits you because the dump will always go to your log directory; you cannot specify otherwise.
If you use SqlDumper or some other utility, you will have more freedom. You can call such a utility from your rake task by executing it as through from the command line, using the system method. (See the example for MySQL below, which uses the mysqldump utility.)
(I have not tested the forgoing code, not having an installation of SQL 2008, myself, but the raw SQL code for creating a dump from with the SQL server is explained on this blog.)
Running the rake task
Then on your command line, call rake mydump or rake mydump RAILS_ENV=production.
For MySQL
You could do something similar for MySQL with the following:
task :mydump do
config = Rails.configuration.database_configuration[Rails.env]
system "mysqldump -h #{config["host"]} -u #{config["username"]} -p#{config["password"]} #{config["database"]} > db/dump.sql"
end
I've been using mysql forever. never really needed anything fancier. But I'm using heroku a lot and while I'm working, I like free search, so I'm using the acts_as_tsearch plugin. If you go to the git repository, it tells you:
* Preparing your PostgreSQL database
Add a text search configuration 'default':
CREATE TEXT SEARCH CONFIGURATION public.default ( COPY = pg_catalog.english )
So guess what? I
changed from mysql to postgresql in my rails config
ran that "CREATE TEXT" code in the sql pain of pgAdmin (a gui for postgres)
noticed that now my development DB has something called an "FTS configuration"
tried the search functionality and it works GREAT
But I'm having trouble getting that configuration to show up in the schema. When I did rake db:dump it doesn't make it in there. I know I can add this line to the schema.rb:
execute 'CREATE TEXT SEARCH CONFIGURATION public.default ( COPY = pg_catalog.english )'
and that works, but how can I get that configuration into the schema without my having to hand-add it? Can I create a file that is also loaded after schema.rb when someone types rake db:load?
And for the postgres people, a question: What does that CREATE TEXT SEARCH CONFIGURATION... do?
Why don't you try adding it to a migration file and rake that against the heroku db?