How does one load a schema into a secondary database? It seems that the ability to set a secondary database connection while maintaining the main ActiveRecord::Base.connection is not supported in Rails.
Domain Definition
We have models using a secondary database. Our primary database is MySQL, the secondary database is PostgreSQL. To use the ActiveRecord documentation's example:
|
+-- Book
| |
| +-- ScaryBook
| +-- GoodBook
+-- Author
+-- BankAccount
Where Book is abstract and uses establish_connection to connect to Postgres.
When dealing with the database, we can either use ActiveRecord::Base.connection or Book.connection.
Schema dump
To wit: Rails database tasks in the schema namespace allow us to dump the schema as so:
ActiveRecord::SchemaDumper.dump(ActiveRecord::Base.connection, file)
Which would allow me to do the following:
ActiveRecord::SchemaDumper.dump(Book.connection, file)
Problem: Schema Load
However, the load task holds no such ability. It merely evaluates the schema file as a whole:
desc 'Load a schema.rb file into the database'
task :load => :environment do
file = ENV['SCHEMA'] || "#{Rails.root}/db/schema.rb"
if File.exists?(file)
load(file)
else
abort %{#{file} doesn't exist yet. Run "rake db:migrate" to create it then try again. If you do not intend to use a database, you should instead alter #{Rails.root}/config/application.rb to limit the frameworks that will be loaded}
end
end
Where the schema file runs ActiveRecord::Schema.define without the connection definition. (Noting that the define method runs in the context of "the current connection adapter").
How do I change that "current connection adapter" without doing an ad-hoc ActiveRecord::Base.establish_connection, which is not what I want to do? I essentially want to run ActiveRecord::Schema.define in the context of Book.connection.
Edit: I'll note that I need a programmatic solution outside of running rake tasks, which is why I'm looking within the rake tasks to see what they're actually doing.
How to dump schema:
ActiveRecord::Base.establish_connection "custom_db_#{Rails.env}".to_sym
File.open Rails.root.join('db/schema_custom_db.rb'), 'w:utf-8' do |file|
ActiveRecord::SchemaDumper.dump ActiveRecord::Base.connection, file
end
How to load schema:
ActiveRecord::Tasks::DatabaseTasks.load_schema_current :ruby, Rails.root.join('db/schema_custom_db.rb'), "custom_db_#{Rails.env}"
I've been struggling recently with trying to just use ActiveRecord. I can load migrations and dump schema but when it comes to loading a schema I run into a similar problem and it's not a Rails or Rake one. Just FYI.
I know you don't want do this standalone but I'm trying.
Please note see this post:
rake db:schema:load vs. migrations
I'm still obsessed with trying to do it on my own without Rails or Rake help so that I gain a better understanding of how ActiveRecord works.
I made a very minimal Rails structure just to generate migrations.
file structure:
Rakefile
Gemfile
/bin
bundle
rake
rails
/config
application.rb
boot.rb
database.yml
environment.rb
/db
/migrate
development.sqlite3
schema.rb
bin/rails generate migration CreateSystemSettings
and I get a file 20150207170924_create_system_settings.rb in the db/migrate folder.
bin/rake db:migrate
== 20150204070000 DropArticles: migrating
== 20150204070000 DropArticles: migrated (0.0001s)
== 20150207163123 AddPartNumberToProducts: migrating
== 20150207163123 AddPartNumberToProducts: migrated (0.0000s)
== 20150207163909 ChangeSystemSettings: migrating
== 20150207163909 ChangeSystemSettings: migrated (0.0000s)
== 20150207170924 CreateSystemSettings: migrating
-- create_table(:system_settings)
-> 0.0085s
== 20150207170924 CreateSystemSettings: migrated (0.0199s)
so that works
Update pending
Related
Heroku made a change to the way postgressql extensions gets installed
This is screwing up new rails review apps in heroku with the following error.
ERROR: extension "btree_gist" must be installed in schema "heroku_ext"
This is screwing up things as I need to drop existing extensions and re-enable with heroku_ext schema. I use bin/rails db:structure:load which is before running a migration.
Also the structure.sql is going to diverge as heroku add the schema in review app and we need to run the creation manually in local dev machine.
Does anybody came across this issue?
I have developed a monkey-patch solution that is, well, definitely a hack, but better than pre-dating migrations, or deleting lines from schema.rb.
Put this in config/initializers. I called mine _enable_extension_hack.rb to make sure it gets loaded early.
module EnableExtensionHerokuMonkeypatch
# Earl was here
def self.apply_patch!
adapter_const = begin
Kernel.const_get('ActiveRecord::ConnectionAdapters::PostgreSQLAdapter')
rescue NameError => ne
puts "#{self.name} -- #{ne}"
end
if adapter_const
patched_method = adapter_const.instance_method(:enable_extension)
# Only patch this method if it's method signature matches what we're expecting
if 1 == patched_method&.arity
adapter_const.prepend InstanceMethods
end
end
end
module InstanceMethods
def enable_extension(name)
name_override = name
if schema_exists?('heroku_ext')
puts "enable_extension -- Adding SCHEMA heroku_ext"
name_override = "#{name}\" SCHEMA heroku_ext -- Ignore trailing double quote"
end
super name_override
end
end
end
EnableExtensionHerokuMonkeypatch.apply_patch!
What does it do? It monkey-patches the enable_extension call that's causing the problem in db/schema.rb. Specifically, if it finds that there is a schema called "heroku_ext", it will decorate the parameter given to enable_extension so that the SQL that gets executed specifies the "heroku_ext" schema. Like this:
CREATE EXTENSION IF NOT EXISTS "pg_stat_statements" SCHEMA heroku_ext -- Ignore trailing double quote"
Without this hack, the generated SQL looks like this:
CREATE EXTENSION IF NOT EXISTS "pg_stat_statements"
That's cleaner way of patching schema.rb
# config/initializers/patch_enable_extension.rb
require 'active_record/connection_adapters/postgresql_adapter'
# NOTE: patch for https://devcenter.heroku.com/changelog-items/2446
module EnableExtensionHerokuPatch
def enable_extension(name, **)
return super unless schema_exists?("heroku_ext")
exec_query("CREATE EXTENSION IF NOT EXISTS \"#{name}\" SCHEMA heroku_ext").tap { reload_type_map }
end
end
module ActiveRecord
module ConnectionAdapters
class PostgreSQLAdapter
prepend EnableExtensionHerokuPatch
end
end
end
EDIT: Here’s the official changelog as a reference https://devcenter.heroku.com/changelog-items/2446
You'll need to do the unthinkable because of the recent heroku changes and modify past migration files for your review apps to work with the new Heroku system for extensions.
Add a predated, or at the top of the first migration file connection.execute 'CREATE SCHEMA IF NOT EXISTS heroku_ext'
Potentially also add to the database.yml a schema_search_path that includes heroku_ext (or set it to public,heroku_ext if you hadn't customized it)
Grep all enable_extension('extension_name')
Replace them all by a connection.execute('CREATE EXTENSION IF NOT EXISTS "extension_name" WITH SCHEMA "heroku_ext")
Pray that is enough to fix the problem
After making those changes we still had to contact heroku support because of, in order:
a pgaudit stack is not empty error
The fix here was to run a maintenance twice (because the postgres add-on which was scheduled for maintenance pre-dated the schema/extension changes change)
a ERROR: function pg_stat_statements_reset(oid, oid, bigint) does not exist error
The fix here was a manual intervention from heroku on the databases. It was caused by heroku trying to run a pg_stat_statements_reset each time a schema is created.
This hack seems to be working for me. The following script runs in the postdeploy step in app.json.
#!/bin/bash -xue
# Create extensions in the schema where Heroku requires them to be created
# The plpgsql extension has already been created before this script is run
heroku pg:psql -a $HEROKU_APP_NAME -c 'create extension if not exists citext schema heroku_ext'
heroku pg:psql -a $HEROKU_APP_NAME -c 'create extension if not exists pg_stat_statements schema heroku_ext'
# Remove enable_extension statements from schema.rb before loading it, since
# even 'create extension if not exists' fails when the schema is not heroku_ext
mv db/schema.rb{,.orig}
grep -v enable_extension db/schema.rb.orig > db/schema.rb
rails db:schema:load
As of Tuesday 16th August 2022, we've found that a patch no longer seems to be needed and enable_extension appears to be working seamlessly as normal without the need to explicitly specify a schema.
When I mentioned this in our ongoing thread with Heroku Support, they said that some of their fixes around the heroku_ext change may have been released at the time of resolving https://status.heroku.com/incidents/2450, but that there are still further fixes in development and yet to be released - so my guess is that this particular issue is one on those that has been fixed!
The answer supplied by dzirtusss got me most of the way there, the last thing I had to do to get this working for my Rails application was to edit the search paths in config/database.yml to include heroku_ext. This is what I appended to my default configuration.
# config/database.yml
default:
schema_search_path: public,heroku_ext
After adding the initialization script and the database.yml edit, I was able to successfully run a heroku pg:copy from one application to another with no errors and no unexpected behavior.
Heroku has introduced an --extensions flag recently, which helps as a workaround. See this FAQ.
For example, if you're using the btree_gist extension, you can use heroku run pg:reset --extensions 'btree_gist' --app YOUR_APP_NAME before running your migrations, they should then work without any changes in your codebase.
Use the Heroku datastore durability tool to create a backup on the
source database or heroku pg:backups:capture -a <SOURCE_APP>.
Determine which pg extensions the database uses (can check from psql
with \dx)
create a comma-separated string of the extensions (e.g.:fuzzystrmatch,pg_stat_statements,pg_trgm,pgcrypto,plpgsql,unaccent,uuid-ossp')
Make sure your Heroku CLI is updated to at least version 7.63.0 (use
heroku update to update)
Run this:
heroku pg:backups:restore $(heroku pg:backups public-url -a <SOURCE_APP>) DATABASE_URL --extensions '<EXTENSIONS>' -a <TARGET_APP>
Reset dynos on the TARGET_APP
I am trying to find the correct (any) method to create an application in Ruby on Rails having an existing database (PostgreSQL) with data and fresh app made with:
rails new --database=postgresql -J --skip-coffee .
I found https://github.com/frenesim/schema_to_scaffold but first I need to have a file with a database structure: schema.rb. I’m looking for a way to do it automatically.
In result of rake db:schema:dumpfile schema.rb is generated, but only with content like that:
ActiveRecord::Schema.define(version: 0) do
enable_extension "plpgsql"
end
And I stuck here. Why that file is empty? Why are there no tables here?
I have a connection with DB and no errors. I did rake db:create before to test. Creation of bases described in database.yml is successful.
At the beginning I used Docker containers and this is my goal. But to exclude the probability of error, I installed the environment in the system (macOS Mojave) based on the socket. And I’ve got the same effect.
How to generate schema.rb with structure of existing database? Or is there different way to build RoR app based on the existing data structure?
Update: Connection with the new database I only did for testing purposes. To verify configuration.
Here's what else I did:
Dump existing structure with
pg_dump --schema-only app_development > db/structure.sql
I changed name in database.yml to have fresh place to import.
rake db:setup created new DB
rake db:structure:load create tables from db/structure.sql file in DB correctly.
But rake db:schema:dump still generate empty file as earlier.
If you have set proper db config you can use rake db:migrate to regenerate the schema file.
edit:
Ok so lets check if I understood correctly:
you have an existing db with tables and data in it
you have brand new rails app
you want to reflect db structure in you schema.rb file
Is that correct? If yes then like I wrote before - without adding any new migrations to your codebase, run rake db:migrate. That task not only applies changes from the migration file but also updates your schema file to be in sync with the actual database.
I've got it! Two days of my life.
File used to import PostgreSQL database has at the beginning:
CREATE SCHEMA employees;
-- and later
CREATE TABLE employees.department;
I thought that since Rails generates database by rake db:structure:load , the file's syntax is correct.
But when I create manually table users in new empty database and then pg_dump that new base I don't have CREATE SCHEMA query there.
And finally rake db:schema:dump fills schema.rb with tables as I want:
create_table "users", id: :serial, force: :cascade do |t|
t.text "name"
end
Because that fresh pg_dumped file has CREATE TABLE public.users query. public.
I think the key is in comments in database.yml file:
# Schema search path. The server defaults to $user,public
#schema_search_path: myapp,sharedapp,public
One picture is more valuable than a thousand words: that's the differences Table users on the right goes to schema.rb after rake db:schema:dump
Thanks guys for the comments. It's made sure me that I do not make a terrible mistake.
It sounds like you did rake db:create which creates a new database for you and then did rake db:schema:dump which generated a schema.rb file for you from the newly created (empty) database.
If you have an existing database that you want to use you will need to modify your database.yml file to connect to it.
If you want to create a new database you will need to generate Active Record database migrations e.g.) rails generate migration CreateProducts name:string part_number:string and then run them rake db:migrate to update your database and generate your schema.rb.
for a rails app, I'm using seeds.rb to populate the database with records and associated image-uploads. The seeds.rb gets all records data from a given YAML-file and grabs image-files from a folder to upload them. This works well in development-environment:
Folder Structure:
rails_app/
db/seeds.rb
...
data/
images1/
image1.jpg
image2.jpg
images2/
...
data.yml
data.yml:
item1:
description: Some description
filepath: images1/image1.jpg
item2:
description: ...
seeds.rb:
items = YAML.load_file(File.join(Rails.root, '..', 'data', 'data.yml'))
items.each do |item, details|
# create items with file-uploads, etc.
...
end
As all database-content is ready for production like this, we want to seed the production database via rake db:seed and access my local YAML-file and image-folder to create the records with their associated file-uploads.
To deploy, I'm using Capistrano and already found a task to seed data to production...
# Add this in config/deploy.rb
# and run 'cap production deploy:seed' to seed your database
desc 'Runs rake db:seed'
task :seed => [:set_rails_env] do
on primary fetch(:migration_role) do
within release_path do
with rails_env: fetch(:rails_env) do
execute :rake, "db:seed"
end
end
end
end
...Unfortunately, this task only works with the seeds.rb on the production server and thus can not find the YAML or images on my local machine.
How can I write a task for Capistrano to access my local YAML and files and db:seed them to the database?
(Appearantly it's not a common practice to seed the production database, but it worked well to get a YAML from the client-side with all files and already use this "proper" data for development/design)
Thanks!
At a high level, you'll want to create a task that is a prerequisite of deploy:seed. That will ensure your task is run first, just before the seed script is executed.
In terms of that task you create, you want it to upload certain files to the same server and relative to the same directory as where the seed task will be run. Looking at the seed task you pasted in your post, we can see the directory is release_path and the server is primary fetch(:migration_role).
Therefore, I suggest writing a task like this:
task :upload_seed_data do
on primary fetch(:migration_role) do
execute :mkdir, "-p", release_path.join("../data")
upload! "../data/data.yml", release_path.join("../data/data.yml")
# ... and so on for all files you want to upload
end
end
# Register the prerequisite
before "deploy:seed", :upload_seed_data
We have a legacy PostgreSQL database that is perfectly accessible from a Rails app except that when the vendor provides an update one of the consistency checks they perform against the database fails because Rails has built a "new" table there, the schema migrations table. Is it possible to direct creation of this table elsewhere? Or is it possible to use the schema cache in Rails 4 to effect this? The release notes section 3.3 on General things says "Schema cache dump (commit) - To improve Rails boot time, instead of loading the schema directly from the database, load the schema from a dump file."
I found an old blog post about this last time I tried it here. Copying the relevant parts:
To make a dump of your schema, execute the following rake task:
RAILS_ENV=production bundle exec rake db:schema:cache:dump
This will generate a file db/schema_cache.dump, that Rails will use to load the internal state of the SchemaCache instance.
To disable the schema cache dump, add the following to your config/production.rb file:
config.active_record.use_schema_cache_dump = false
If you would like to clear the schema cache, execute:
RAILS_ENV=production bundle exec rake db:schema:cache:clear
I'm using PostgreSQL v9.0.1 with Rails (and it's deps) # v2.3.8, owing to the use of the fulltext capability of postgres, I have a table which is defined as:
CREATE TABLE affiliate_products (
id integer NOT NULL,
name character varying(255),
model character varying(255),
description text,
price numeric(9,2),
created_at timestamp without time zone,
updated_at timestamp without time zone,
textsearch_vector tsvector,
);
Note the last line, this ensures that active record isn't able to process it with the standard schema dumper, so I have to set config.active_record.schema_format = :sql in ./config/environment.rb; and use rake db:test:clone_structure instead of rake db:test:clone.
None of this is too remarkable, only inconvenient - however rake db:test:clone_structure fails with the error:
ERROR: must be owner of language plpgsql
Because of line #16 in my resulting ./db/development_schema.sql:
CREATE OR REPLACE PROCEDURAL LANGUAGE plpgsql;
Under PostgreSQL v9.0+ the language plpsql is installed by the superuser, to the initial template, which is then available to the newly created schema.
I cannot run tests on this project without resolving this, and even editing ./db/development_schema.sql manually is futile as it is regenerated every time I run rake db:test:clone_structure (and ignored by rake db:test:clone).
I hope someone can shed some light on this?
Note: I have used both the pg 0.9.0 adapter gem, and the postgres gem at version 0.7.9.2008.01.28 - both display identical behaviour.
My teammates run PostgreSQL v8.4 where the language installation is a manual step.
I had the same problem. I fixed my template with the commands below
psql template1
template1=# alter role my_user_name with superuser;
read more at http://gilesbowkett.blogspot.com/2011/07/error-must-be-owner-of-language-plpgsql.html
For new readers, I read this older post after having run into this error in one of my own projects. I strongly feel that giving the app's PostgreSQL a superuser role is a terrible idea and changing the template is not ideal either. Since the referenced PSQL commands that are added by db:structure:dump are not needed by the Rails app's database, I have written a custom rake task that comments out the problematic lines in structure.sql. I have shared that code publicly on Github as a Gist at https://gist.github.com/rietta/7898366.
The solution was as follows:
On my installation, there are standard templates template0 and template1 - at least as I understand it postgres will look for the highest numbered templateN when creating a new database, unless the template is specified.
In this instance, as template0 included plpgsql, so did template1… the idea being that you will customise template1 to suite your site specific default needs, and in the case that you blow everything up, you would restore template1 from template0.
As my site specific requirement was to install plpgsql as part of the automated build of my web application (a step we had to keep to maintain 8.4 compatibility) - the solution was easy: remove plpgsql from template1 and the warning/error went away.
In the case that the site-specific defaults would change, and we should need to go back to the default behaviour, we would simply remove template1 and recreate it (which would use template0)
I encountered this error while attempting to do RAILS_ENV=development bundle exec rake db:reset. I was able to accomplish the same thing (for my purposes) by doing RAILS_ENV=development bundle exec rake db:drop db:create db:migrate instead.
I just filter the plpgsql extension statements from the structure.sql file post-dump:
# lib/tasks/db.rake
namespace :db do
desc "Fix 'ERROR: must be owner of extension plpgsql' complaints from Postgresql"
task :fix_psql_dump do |task|
filename = ENV['DB_STRUCTURE'] || File.join(ActiveRecord::Tasks::DatabaseTasks.db_dir, "structure.sql")
sql = File.read(filename)
sql.sub!(/(CREATE EXTENSION IF NOT EXISTS plpgsql)/, '-- \1')
sql.sub!(/(COMMENT ON EXTENSION plpgsql)/, '-- \1')
File.open(filename, 'w') do |f|
f.write(sql)
end
task.reenable
end
end
Rake::Task["db:structure:dump"].enhance do
Rake::Task["db:fix_psql_dump"].invoke
end