When running rake db:schema:dump on an app that uses a postgres schema (i.e. schema_name.users), it looks like it's only dumping tables for the first schema in the db user's search path. Is there a way to include tables from more than one schema?
To state the problem differently:
createdb myapp
psql myapp -U postgres -c "create table stuff"
#=> creates table "stuff" in the public schema
psql myapp -U postgres -c "create schema specific_thing"
psql myapp -U postgres -c 'create table "specific_thing".users(id int)'
createuser -U postgres -S -D -R special_user
psql myapp -U postgres -c "grant all on schema specific_thing to special_user"
psql myapp -U postgres -c "ALTER USER special_user SET search_path TO specific_thing,public"
In database.yml:
...
development:
adapter: postgresql
database: stuff
username: special_user
password:
host: localhost
...
Running: rake db:schema:dump
Only dumps the users talbe from the specific_thing schema and ignores everything in the public schema.
So here's what I found. I don't have the code in front of me so I can't point to anything specifically, but there is some code in PostgresAdapter that identifies what the adapter considers as its available 'tables'. The adapter assumes it's always going to deal with the 'public' schema (public is actually hardcoded in the query that gets a list of tables), so my solution to the problem involved subclassing PostgresAdapter to have a different concept of what it's 'tables' are. This got me most of the way there. There were some other obstacles to overcome but everything was much more clear after fitting this piece into the puzzle. Definitely involved spending some time in the Rails source though.
Related
I'm trying to setup a local production environment for a Ruby on Rails web application. I can run the application with rails server command, which gives the development environment.
The production environment I'm trying to set up is purely local and I've followed this tutorial for setting it up with apache 2: https://www.digitalocean.com/community/tutorials/how-to-setup-a-rails-4-app-with-apache-and-passenger-on-centos-6
However when I go to the page of my application I get the following error:
PG::InsufficientPrivilege: ERROR: permission denied for relation schema_migrations : SELECT "schema_migrations".* FROM "schema_migrations"
in my database.yml I have these settings for development and production:
adapter: postgresql
database: whiteboard
username:
password:
pool: 5
timeout: 5000
I'm not allowed to change these settings, no matter what.
Is there any way to fix this? (if yes, step by step please)
It seems you have to create a DB user with all needed privileges on your DB.
For example I think you could do the trick by log in your DB console then do something like:
CREATE USER your_new_username WITH PASSWORD 'your_new_password';
CREATE DATABASE whiteboard;
GRANT ALL PRIVILEGES ON DATABASE whiteboard to your_new_username;
ALTER DATABASE whiteboard OWNER TO your_new_username;
Then update you database.yml like this:
adapter: postgresql
database: whiteboard
username: your_new_username
password: your_new_password
pool: 5
timeout: 5000
Hope it helps!
I was using dbmate which also creates a table called schema_migrations on startup, and thus fails when a full dump is applied that also has the same table. Here are a few approaches
PostgreSQL doesn't support creating dump with IF NOT EXISTS, unlike mysql:
See Can pg_dump be instructed to create tables with "IF NOT EXISTS"?
However, PostgreSQL 9.1 and newer supports CREATE TABLE IF NOT EXISTS -syntax, so you could string replace the dump for example by using sed:
sed -i 's/CREATE TABLE/CREATE TABLE IF NOT EXISTS/g' dump.sql
But in practice, I ended to delete the lines from the dump with schema_migrations on it, and +1-2 lines following:
sed -i '/CREATE TABLE public.schema_migrations (/,+2 d' dump.sql
sed -i '/ALTER TABLE ONLY public.schema_migrations/,+1 d' dump.sql
I have db named "mydb". Now How can I create a new db with name "mydb_test" with only schema of "mydb"
Tried the following link PostgreSQL how to create a copy of a database or schema?
As mentioned over there I tried the following command,
createdb -T olddb newdb
This copies including the data into newdb
If I try the second option mentioned in the above link I get following error,
# pg_dump -Cs -U postgres my_test_db > dump_schema_file
# psql -U postgres naggappan_my_test_db < dump_schema_file
psql: FATAL: database "naggappan_my_test_db" does not exist
How can I take only schema
You may create a new dump without data of required schema and then restore from it into a new DB:
# pg_dump --dbname=source_db_name --username=postgres --encoding=UTF8 --schema=db_schema --schema-only --file=path_to_filename.dump
# psql --dbname=target_db_name --username=postgres --file=path_to_filename.dump
If you have an existing dump with schema and/or data in binary format, you can restore only the schema from it using pg_restore:
# pg_restore --dbname=target_db_name --username=postgres --schema-only path_to_filename.dump
I'm learning Rails and my final app will be hosted on Heroku, which uses postgres, so I figured it'd be smart to work with postgres in development too as I'm building what is supposed to be a rather simple search function and want to avoid as many problems as possible actually deploying it.
Sadly, I'm using Ubuntu 14.04 so naturally the steps will be harder than on for example Windows.
Here's what I've done so far, which is a rather comical enterprise into a world that gives me nothing but problems at every step:
Actually installed postgresql. sudo apt-get install postgresql-9.4 as per the official website of course didn't work so I had to find a workaround (as always) but it should be installed now. I ran sudo apt-get install -y postgresql postgresql-contrib to get it working.
Tried logging in per some instructions with su postgres, but even after setting a password for su or using sudo su postgres that didn't work. Ended up creating a user with sudo -u postgres createuser -P my_user matching the name of my app. Created a database too.
Tried creating a new rails project with rails new my_app --database=postgresql. Didn't work as it complained lacking a pg gem (sorry for not pre-emptively making a Gemfile for you?) so I gave that up and just created it without specifying a database.
Removed the sqlite gem and added gem 'pg' in the Gemfile. Ran bundle install, but it didn't work. Had to run sudo apt-get install libpq-dev to install something I'm not sure what it is and then it worked.
Modified the database.yml as per some instructions and ran rake db:setup. Rails gave this error: FATAL: Peer authentication failed for user "my_user". Well, that's cool.
Not quite sure why, but I added a database here called my_app_development for it with the owner my_user but then db:setup instead complained that it lacked permissions to create a database (but I just created it FOR you?).
I ran chmod -R 0666 my_app as someone highly upvoted on SO suggested but holy shit that was bad as it didn't even give me permissions to enter the folder myself! Reverted that quickly and tried something else.
Someone suggested running psql -U my_user postgres but that only gives me the error psql: FATAL: Peer authentication failed for user "my_user"
Experimented logging in via psql postgres (I don't know what psql is, I'm just following suggestions) and tried ALTER ROLE my_user CREATEDB; but it only returns a permission denied error.
Officially gave up and came here.
Can anyone help me with the actual steps to follow from the beginning? It shouldn't be THIS hard, right?
By the way, this is what my database.yml looks like:
default: &default
adapter: sqlite3
pool: 5
timeout: 5000
development:
adapter: postgresql
encoding: unicode
database: my_app_development
host: localhost
pool: 5
username: my_user
password: my_password
test:
<<: *default
database: db/test.sqlite3
production:
<<: *default
database: db/production.sqlite3
Edit: Thanks alot to Ajay for walking me through how to setup postgres. If anyone comes across this thread, as frustrated as I am with postgres, here are a few pointers:
PG::InsufficientPrivilege: ERROR: permission denied to create database means the user doesn't have the right privileges. Log in via sudo -u postgres psql and you should see postgres=# before everything you type in the terminal. While there, type ALTER ROLE my_user CREATEDB; and it should work. I don't know why it didn't the first time I used that, perhaps I forgot sudo?
FATAL: Peer authentication failed for user "my_user" means you need to change some things in a file as per the instructions in one of the answers. Make sure to change it for both local and postgres. I have it set to md5 for everything but local and it works.
Login via sudo -u postgres psql and type `select * from pg_catalog.pg_user;' to check your current users. Good way to see if you created the user correctly and what privileges it has.
default: &default
adapter: sqlite3
pool: 5
timeout: 5000
Above adapter: sqlite3 is causing the error
Please try this:
default: &default
adapter: postgresql
pool: 5
timeout: 5000
development:
<<: *default
database: my_app_development
username: psql #postgres username
password: your_password #password
After you entered the valid postgres credentials(username/password) here. Try following in your terminal :
$ rake db:create #this will create your my_app_development database.
$ rake db:migrate #migrate your database.
5. Modified the database.yml as per some instructions and ran rake
db:setup. Rails gave this error: FATAL: Peer authentication failed
for user "my_user". Well, that's cool.
you need to open your pg_hba.conf (probally located at /etc/postgresql/9.4/main/pg_hba.conf) and change the authentication method from "peer" to "md5" (which will asks for password) or to "trust" (which will unsecuritly allow access without password).
To know where your pg_hba is located, execute this on your terminal (terminal of the machine where the postgresql are running):
ps ax | grep postgresql.conf
it should return something like:
8803 ? S 0:00 /usr/lib/postgresql/9.4/bin/postgres -D /var/lib/postgresql/9.4/main -c config_file=/etc/postgresql/9.4/main/postgresql.conf
look the folder where config_file is located. In this case is /etc/postgresql/9.4/main/. Inside this folder there's another configuration file called pg_hba.conf (the permissions file). Edit it (with super user):
sudo nano /etc/postgresql/9.4/main/pg_hba.conf
on the lasts lines you will see something like that:
# DO NOT DISABLE!
# If you change this first entry you will need to make sure that the
# database superuser can access the database using some other method.
# Noninteractive access to all databases is required during automatic
# maintenance (custom daily cronjobs, replication, and similar tasks).
#
# Database administrative login by Unix domain socket
local all postgres trust
# TYPE DATABASE USER ADDRESS METHOD
# "local" is for Unix domain socket connections only
local all all trust
# IPv4 local connections:
host all all 127.0.0.1/32 trust
# IPv6 local connections:
host all all ::1/128 md5
# Allow replication connections from localhost, by a user with the
# replication privilege.
#local replication postgres peer
#host replication postgres 127.0.0.1/32 md5
#host replication postgres ::1/128 md5
You see the "trust" references? In your default pg_hba.conf they should be "peer". In my example, I had changed to "trust" (i.e, doesn't ask for passwords) all local connections, because my postgresql server not accept outside connections. But you can change to "md5", which will permit access when the user provide the correct password.
After change this, save and exit (in nano is Ctrl+O, Enter to confirm, Ctrl+X to exit). Then, restart postgresql (sudo /etc/init.d/postgresql restart - maybe works with just a reload)
UPDATE:
DISCLAIMER:
although trusting your local connections will not create a hole security (unless, of course you are sharing the machine with anothers users), do it only for testing purposes - to discover where the problem is (if is a permission/pg_hba problem or not). After discover where the problem are, its more concise to have one specific user to your project and use an authentication for it ("md5", "peer").
Using one single user for all your projects on the machine (e.g. the "postgres" user), and/or not use an authentication ("trust"), is like create a Rails project and use just one generic controller, instead having a controller for each table/group of logic.
Can I add the migration file, but tell rails not to run it locally? I need the file for others to setup the application locally.
If you have a file:
db/migrate/20121010100909_modify_table_x.rb
You can go into your database and run the following SQL.
MySQL:
INSERT INTO 'schema_migrations' VALUES ('20121010100909');
PostgreSQL:
INSERT INTO schema_migrations VALUES ('20121010100909');
And it will then ignore that migration.
Edit - How to "go into your database"
Using the parameters from config/database.yml in Rails, connect to the database you are using.
You will need to use the command-line tool of whatever database software you're using. E.g.
For PostgreSQL:
psql -d <database_name> -U <username>
For MySQL:
mysql -u <username> <databasename>#localhost -p
Type in your password if required.
Then type in and execute the SQL above.
You could insert the proper timestamp into the schema_migrations table locally.
I'm having problems getting testing to work with Postgresql and Rails 3.
Both development and production databases I can get to work fine, however the test database throws the following errors when I run rake or db:test:prepare, etc.
PGError: ERROR: source database "template1" is being accessed by other users
Update
Googling around, it seems that one should use template0 instead of template1 when using createdb to create a new database in Postgres. In typical “So I’ll remove the cause. But not the symptom” fashion, I found vendor/rails/railities/lib/task/databases.rake and changed line 109 to read:
createdb #{enc_option} \
-U "#{abcs["test"]["username"]}" \
-T template0 #{abcs["test"]["database"]}
But I don't really wanna do that, as I'm using Rails as a GEM, any one know of another work around or fix?
database.yml:
development:
adapter: postgresql
encoding: unicode
database: test1234_development
pool: 5
username: holden
password: postgres
test:
adapter: postgresql
encoding: unicode
database: test1234_test
pool: 5
username: holden
password: postgres
Full error:
NOTICE: database "test1234_test" does not exist, skipping
PGError: ERROR: source database "template1" is being accessed by other users
DETAIL: There are 1 other session(s) using the database.
: CREATE DATABASE "test1234_test" ENCODING = 'unicode'
Short story: CREATE DATABASE works by copying an existing database. PostgreSQL won't let you copy a database if another session is connected to it. If template1 is being accessed by other users, CREATE DATABASE will fail.
The question you need to answer: Why are other sessions connected to template1?
The difference between template0 and template1
At the point you initialize a database cluster, template0 and template1 are the same. Any location-specific stuff you want to make available to every database you create by using CREATE DATABASE should go into template1. So, for example, if you add the procedural langauge PL/python to template1, every database you create later will include PL/python.
The database template0 is intended to be a "virgin" template. It should contain only standard database objects--the ones created by initializing the cluster. As a "virgin" template, it should never be changed. Never.
If you need to specify encoding and locale settings (collation), then you can do that by copying template0. You can't do that by copying template1.
This problem occur when you had logged(psql template1 or psql template0) in template1 and template0 database and exit using below command.
Ctrl + z
Better way exist from db use below postgres command then problem will not create:
\q + enter
There are 2 solutions, If have problem.
Solution - 1
Restart posgres service like.
sudo service postgresql restart
Solution - 2
sudo ps aux | grep template1
Make sure don't delete this processes
postgres 8363 0.0 0.0 111760 7832 pts/11 T 09:49 0:00 /usr/lib/postgresql/9.5/bin/psql template1
ankit 18119 0.0 0.0 14224 976 pts/14 S+ 12:33 0:00 grep --color=auto template1
rest of process should be kill using below command.
sudo kill -9
Now try to create db again.
Hope this help you.
Ankit H Gandhi.
Just restart the service of database.
I restarted my system and the error was still showing. However, I followed the steps below to sort it out.
Stop all processes using the postgres port 5432 by doing this in command prompt (Admin): Type netstat -ano in command prompt. Find the pid with Local Address of 0.0.0.0:5432. Then use taskkill /pid {pid} /f to kill the task.
Start the postgres service in windows services.
I also got this error while trying to reset the database while I had the default Ruby on Rails server WEBrick running:
$ bin/rake db:reset
PG::Error: ERROR: database "dev" is being accessed by other users
DETAIL: There is 1 other session using the database.
: DROP DATABASE IF EXISTS "dev"
The other user here was the running Rails app. After shutting down the server with CTRL + c, I was able to re-run the database reset command without any problems.
It makes sense too. You can't drop the database if someone else is currently connected to it, as Mike Sherrill also points out.
Solution for me was to delete old server and create a new one from Postgresql administration web interface. Could now create new database without this error.
I was also stuck setting up postgres on ruby on rails project, ensure that you have installed pg locally and created a user with its password then on your database.yml should have:- host: localhost, password: (set password) then run:
$ rails db:create
$ rails db:migrate