Using Heroku with the Postgresql add-on. Upon reviewing my logs, it seems postgresql is logging every ... single ... transaction. I understand you can set the log level by doing something like (https://www.postgresql.org/docs/9.1/static/runtime-config-logging.html)
ALTER DATABASE my_database SET log_statement=error;
However, Heroku says
ERROR: permission denied to set parameter "log_statement"
Note that might be a possible duplicate of how to turn off Heroku SQL logs from postgres but they never addressed how to get around the Heroku permissions. (Should I start a bounty on that ticket or keep this one?)
How do I modify the Heroku Postgresql logging levels?
Update
Folowing Get DB owner's name in PostgreSql
pg_catalog.pg_get_userbyid(d.datdba) as "Owner"
FROM pg_catalog.pg_database d
ORDER BY 1;
I get the following result
Name Owner
d7c0sfp134dmml u2mqn7a68c982v
postgres postgres
template0 postgres
template1 postgres
ALTER DATABASE d7c0sfp134dmml SET log_statement = error;
Results in
ERROR: permission denied to set parameter "log_statement" Query
failed PostgreSQL said: permission denied to set parameter
"log_statement"
As of August 8th 2017, Heroku Postgres now offers the ability to change this via the PGSettings feature. For example the command to log all statements on a standard-0 or above (no hobby plans) would be:
heroku pg:settings:log-statement all postgresql-large-1234 -a sushi
ref: https://devcenter.heroku.com/articles/heroku-postgres-settings#log-statement
You interpreted documentation wrongly. The log_statement parameter doesn't allow the error value. Possible values are instead: none, ddl, mod, all (and default pseudo-value).
You may also want to change log_min_messages parameter or log_min_error_statement parameter. They allow these values: DEBUG5, DEBUG4, DEBUG3, DEBUG2, DEBUG1, LOG, NOTICE, WARNING, ERROR, FATAL and PANIC. Set PANIC for minimal logging.
The fact is that for all three aforementioned parameters documentation says:
Only superusers can change this setting.
So, unless you can connect to PostgreSQL database as the postgres user or edit PostgreSQL configuration files directly, you probably cannot change logging options.
You may ask Heroku support about some workarounds. But I don't see how you can change logging options, because they probably won't give you superuser access. Maybe you can ask them to do that for you (i.e. change logging options) or you can use some special tools to filter logs (without changing logging options), I don't know.
Related
I recently downloaded and install PostgreSQL in Windows. Now how to use PostgreSQL in Ruby on Rails? I search this on the internet but couldn't find any step by step easy solution. I don't understand what do.
.Step 1. Download and install PostgreSQL v9.0.4-1 from here because here said only 9.0.x would be supported on windows 7. I kept all the default options and just used 'secret' as the password when prompted by the Postgres installer for one (again not entirely sure what the consequences of sharing that info on the internet is... will soon find out I'm sure). You'll need this password in step 3.
.Step 2. Change environment variables such that Path (for system, not user (I'm not sure if this is significant or not)) is: C:\Program Files\PostgreSQL\9.0\bin
(n.b. I'm on 64-bit windows hence it not being installed for 32-bit in 'C:\Program Files (x86)\PostgreS...')
Don't forget to change access rights to folder PostgreSQL\9.0 and remove any default readonly rights on the folder or content. (You may also need to restart your computer for these to take effect - thanks #Gavin -although not likely).
.Step 3. Test Postgres installation by trying to create a new database: From command line: createdb -U postgres mydb_as_postgres. You should be prompted to enter the password now, if you're not it may be that you need to start the server first (I can't remember whether I needed to do this or not). The easiest way is through pgAdmin III, which should be 'pgAdmin3.exe' in a folder somewhere like C:\Program Files\PostgreSQL\9.0\bin. Once you've started pgAdmin III there should be a panel on the left called 'Object Browser'. In this there should be a tree with:
Server Groups > Servers > PostgreSQL 9.0 (localhost:5432)
Right click on 'PostgreSQL 9.0 (localhost:5432)' and select 'Connect'.
The createdb -U postgres mydb_as_postgres command should create a new databse called 'mydb_as_postgres' which you can check by firing up pgAdmin III and double clicking on 'PostgreSQL 9.0 (localhost:5432)'. Under this there should be:
Databases (2) which should list 2 databases called mydb_as_postgres and postgres
I called it _as_postgres because the -U postgres part of the command tells Postgres to create the database with the postgres user as it's owner, which you need to specify when you're not signed in as the postgres user. I have all of my files stored as 'AJames' user though so if you're the same and want to keep developing your app when signed in as a different user you need to create a Postgres 'role' for that user now (see step 4).
.Step 4. Through pgAdmin III. Right-click on Login Roles (which for me is in):
Object Browser > Server Groups > Servers > PostgreSQL 9.0 (localhost:5432) > Login Roles
Right-click on Login Roles and select 'New Login Role...' in Role name, put in your operating system user name, which for me is AJames, and fill in your password under the 'Role Privileges' tab, I checked all the boxes, but an experienced postgres user would likely strongly recommend to only check the 'inherits rights from parent roles' and the 'can create database objects' But I'm not an experienced user and just want to debug Rails SQL calls in Postgres so I also checked the 'Superuser' and 'Can create roles', just in case.
.Step 5. You should now be able to create a new database without being signed in as the postgres user. Try typing:
createdb mydb_as_user
Hopefully this should work for you.
.Step 6. Okay, so you've got a development.sqlite3 file in your rails 'db/' directory. Initially I was going to set the next test as converting this from sqlite3 to psql.
I couldn't get this to work though but I left my attempts here as the solution I used required having the data in a Rails app on Heroku.com (see instead the solution from step 7 onwards). For those who only have a local app and no data in Heroku, they can't use the same approach, so they might need to explore something like this:
x6.1 First, test 'psql' by trying a command from your command line like:
psql mydb_as_user
this should display something like below (after you've typed in your password):
C:>psql mydb_as_user
Password:
psql (9.0.4)
WARNING: Console code page (850) differs from Windows code page (1252)
8-bit characters might not work correctly. See psql reference
page "Notes for Windows users" for details.
Type "help" for help.
mydb5=#
x6.2 try entering:
CREATE TABLE users_table (id integer, "name" text);
It should display:
CREATE TABLE
mydb5=#
If you check in pgAdmin III, you should see the table there under:
Object Browser > Server Groups > Servers > PostgreSQL 9.0 (localhost:5432) > Databases > mydb_as_user > Schemas > public > Tables > users_table >
x6.3 Okay, next to try the conversion. Downloaded sqlite-shell precompiled binary for windows.
x6.4 Create a new directory, I used 'C:\temp' and put the sqlite3.exe and your development.sqlite3 files in it.
x6.5 Use the following commands (which are from here) to dump the development.sqlite3 database into Postgres.
sqlite3 development .dump | psql development2
you might get an error like:
psql: FATAL: database "development2" does not exist
x6.6 so I went into pgAdmin III and made a development 2 database, tried the command again and got:
ERROR: syntax error at or near "PRAGMA"
LINE 1: PRAGMA foreign_keys=OFF;
^
BEGIN
COMMIT
Like I said, I couldn't get it to work. I'm sure there's a way of getting round that error but I thought of a different way and so I instead used this solution (which requires a Heroku account to have your data and does the conversion from sqlite3 to psql using the Taps gem (I believe):
.Step 7. in pgAdmin III I created another database. Under the properties tab I set name: 'development', owner: 'AJames' (replace this with your own Windows user name). And under the privileges tab, set role: 'public' and checked the ALL option (thought this resets to unchecked so I'm not sure that's necessary).
.Step 8. add gem 'pg', '0.11.0' to your gem file. You'll probably also want to remove the: gem 'sqlite3' at this point too.
.Step 9. set database.yml as suggested here to:
development:
adapter: postgresql
database: db/development
username: AJames # replace this with your own user name
password: secret # replace this with your own password
host: localhost
encoding: UTF8
pool: 5
timeout: 5000
If you are working on an open source project and don't want your password to be made publicly available, have a look at some of the answers to Securely providing the database password in a Rails app.
.Step 10. from command line in your rails app's root directory run: rake db:migrate This will create the new schema and all the tables in the Postgres database.
.Step 11. run heroku db:pull from your command line (again from in the root directory of your rails app) to pull all your data down and into your new empty Postgres database. I think at this point your taps gem will be doing this work for you.
.Step 12. Hopefully there is no step 12! ...and it should now be working for you. Happy RoR PostgreSQL debugging! Please edit, or let me know, if there are any errors in this.
Good Luck
Apparently when using the Octopus gem to do Postgres replication everything should be plug and play. However I can't seem to find what I'm doing wrong.
This is my config/shards.yml
octopus:
environments:
- development
replicated: true
development:
slave1:
adapter: postgresql
host: localhost
database: slaveapp_development
username: pguser
password: pgpass
The AR model Provider(I create the exact same tables in each app via Rake tasks) I'd like to sync/replicate to my slave:
class Provider < ActiveRecord::Base
has_many :products
replicated_model()
end
I boot both apps via Rails server and enter Masterapp's console and from there:
> Provider.using(:slave1).create({provider_params...})
#=> works! I get a new record in slave1's DB.
> Provider.using(:master).create({provider_params...})
#=> works partly. Creates record only in master's DB.
The problem is that when calling Provider.using(:master)... I'm expecting:
1 - Create record at master's DB.
2 - Replicate same record at slave1's DB. <--- This is NOT happening.
That is not the purpose of the Octopus gem.
It redirects the database requests, depending on whether it is a read or write operation. That way you can use your Rails models without thinking about the current database connection and if it fits the intended operation.
It does not copy the data to the slaves.
To perform the actual replication, i.e. the data transfer from the master to the slaves, you have to set it up yourself.
There are several options with PostgreSQL. If you are using a newer version (9.1+) you can use the integrated streaming replication in "hot standby" mode. There are tutorials on how to set it up, e.g.
http://www.rassoc.com/gregr/weblog/2013/02/16/zero-to-postgresql-streaming-replication-in-10-mins/
http://prongs.org/blog/postgresql-replication
If you are stuck with an older version of PostgreSQL have a look at the alternatives.
Try adding fully_replicated: false after replicated: true in your config.
When i run rails db i get the below error.
psql: FATAL: role "ubuntu" does not exist
First i want to know the exact meaning of what it says before solving it.
I have tried http://stackoverflow.com/questions/11919391/postgresql-error-fatal-role-username-does-not-exist this link as well but nothing seems to workout for me ..
I don't have a Rails-specific answer for you, but I can tell you some possibly helpful facts about PostgreSQL authentication:
PostgreSQL uses the term "role" to mean "user" or "login", so the error message means you are trying to connect to the PostgreSQL database with a username of ubuntu, but no such user exists.
If you don't explicitly tell PostgreSQL what user you want to log in as, it will use your operating system credentials. That is, if you're logged in to Linux as the user ubuntu, PostgreSQL will try to log you in as a PostgreSQL user named ubuntu.
PostgreSQL authentication is configured in a file called pg_hba.conf. See http://www.postgresql.org/docs/current/static/auth-pg-hba-conf.html. However, if you're using Rails, I suspect there's a way for you to control the PostgreSQL authentication setup without editing the pg_hba.conf file manually.
When I execute heroku db:pull it finds my local dev db at:
postgres://127.0.0.1/myapp_development?encoding=utf8
once I confirm though, it fails with:
Sequel::DatabaseConnectionError -> PGError: fe_sendauth: no password supplied
I tried running the pull with the local db specified, e.g.
heroku db:pull postgres://root:#localhost/db_name
which gives the same no password supplied error.
I thought I may need to change root: to myname: because thats the user I granted superuser rights to when I setup postgres but neither root: or myname: works
My database.yml has username: and password: blank for all databases specified.
From the command line as myname#ubuntu I can type psql myapp_development and connect fine and run selects.
What am I missing here?
Is it related to my pg_hba.conf settings? I had a look inside that and it says:
# Database administrative login by UNIX sockets
local all postgres ident
# TYPE DATABASE USER CIDR-ADDRESS METHOD
# "local" is for Unix domain socket connections only
local all all ident
# IPv4 local connections:
host all all 127.0.0.1/32 md5
# IPv6 local connections:
host all all ::1/128 md5
Should 'trusted' be used there? And if so if I edit that file do I need to restart postgres? And if 'trust' is necessary then how come rails and the psql command line tools work without passwords when logged in as my myname user?
Thank you!
Authentication method trust might do the trick, but as you are undoubtedly aware, this is not secure.
After editing pg_hba.conf, you don't have to restart. A reload is enough (quoting the manual):
The pg_hba.conf file is read on start-up and when the main server
process receives a SIGHUP signal. If you edit the file on an active
system, you will need to signal the postmaster (using pg_ctl reload or
kill -HUP) to make it re-read the file.
pg_ctl reload
See the fine manual. You might need the manual for version for 8.3. Shared db on heroku currently runs on PostgreSQL 8.3. (Also, I doubt you have access to pg_ctl on heroku.)
Be aware of this:
If no password has been set up for a user, the stored password is null
and password authentication will always fail for that user.
Emphasis mine. You might be able to log in locally, because the auth-methods ident or peer allow for that. But for your purpose you may need a password!
Currently I get the following error:
PGError (FATAL: password authentication failed for user "postgres" ):
when my app tries to access the database.
I wanted to test my SQL calls against postgres as my app regular breaks when pushed up to production on Heroku because of the stricter requirements of postgres (which I think is probably a healthy thing) over sqlite 3. So I have a similar request to this for a not out of date tutorial for installing postgres for use with ruby on rails on windows (7) please. I thought it'd be a quick 20 minutes of downloading and installing but 2 and a bit hours later and I don't think I'm very close yet. So far I've:
Downloaded and installed Postgres Version 8.4.8-1 from here
Set my environment variables such that Path (for User) is: C:\Ruby192\bin;C:\Program Files (x86)\PostgreSQL\8.4\bin
Gemfile:
gem 'pg', '0.11.0' # instead of gem 'sqlite3', '1.3.3'
ran bundle install for my rails app seemingly successfully, but haven't found easy way to validate installation yet.
set database.yml as suggested here to:
development:
adapter: postgresql
database: db/development
username: postgres
password: secret
host: localhost
encoding: UTF8
pool: 5
timeout: 5000
I know I need to set up a user name and password for Postgres, maybe also start the postgres server, connect to it(?) and put in my local IP address I'll connect to it on into a config files somewhere and then edit one of the other .conf files in 'C:\Program Files (x86)\PostgreSQL\8.4\data' etc...
I think Rails has made me soft, am I over thinking things or is it actually fairly tricky to set up and I should just go back to Sqlite3, for which there's also the awesomely useful SQLite Manager, Firefox plugin?
I'm still searching for a beginners guide to installing and using Postgres for rails but so far have only been confused by most of the stuff I look at / tried following like this, this, this, this, this(for Snow Leopard), this(linux).
Any pointers would be much appreciated. Thanks!
James
An approach to installing Postgres on windows 7 for use as PostgreSQL database for a rails 3 (3.0.7) project.
Preamble (you can skip this bit)
So the first thing to point out is that Postgres is not just a different file extension from .sqlite3, it's a whole mechanism for managing your databases. As such it has a client/server model, of which you'll need to set up both to use Postgres as the database for your rails app.
Motivation for going through considerable pain of Postgres setup versus almost effortless sqlite setup: if you're deploying to Heroku, they're currently using Postgres so some of your SQL calls that are fine on sqlite3 will break when used with Postgres. It's much easier to debug postgres locally rather than when it's on Heroku's servers.
So I did the following things:
(Disclaimer: I may have forgotten to include some of the things I did... it took me over 48 hours of on and off pain to get it to work... if the following advice doesn't work for you then the huge (2300 pages!!) but very thorough Postgres documentation should help. I'd recommend downloading this anyway if you're serious about using Postgres as it has a lot of material that I've only just begun to understand the significance of.)
(Second disclaimer: I have almost certainly broken 20 sensible Postgres guidelines and exposed security holes in the Postgres database whilst doing so. If there any obvious things an experienced Postgres user disagrees with, please edit my post.)
.Step 1. Download and install PostgreSQL v9.0.4-1 from here because here said only 9.0.x would be supported on windows 7. I kept all the default options and just used 'secret' as the password when prompted by the Postgres installer for one (again not entirely sure what the consequences of sharing that info on the internet is... will soon find out I'm sure). You'll need this password in step 3.
.Step 2. Change environment variables such that Path (for system, not user (I'm not sure if this is significant or not)) is: C:\Program Files\PostgreSQL\9.0\bin
(n.b. I'm on 64-bit windows hence it not being installed for 32-bit in 'C:\Program Files (x86)\PostgreS...')
Don't forget to change access rights to folder PostgreSQL\9.0 and remove any default readonly rights on the folder or content.
(You may also need to restart your computer for these to take effect - thanks #Gavin -although not likely).
.Step 3. Test Postgres installation by trying to create a new database:
From command line: createdb -U postgres mydb_as_postgres.
You should be prompted to enter the password now, if you're not it may be that you need to start the server first (I can't remember whether I needed to do this or not). The easiest way is through pgAdmin III, which should be 'pgAdmin3.exe' in a folder somewhere like C:\Program Files\PostgreSQL\9.0\bin. Once you've started pgAdmin III there should be a panel on the left called 'Object Browser'. In this there should be a tree with:
Server Groups > Servers > PostgreSQL 9.0 (localhost:5432)
Right click on 'PostgreSQL 9.0 (localhost:5432)' and select 'Connect'.
The createdb -U postgres mydb_as_postgres command should create a new databse called 'mydb_as_postgres' which you can check by firing up pgAdmin III and double clicking on 'PostgreSQL 9.0 (localhost:5432)'. Under this there should be:
Databases (2) which should list 2 databases called mydb_as_postgres and postgres
I called it _as_postgres because the -U postgres part of the command tells Postgres to create the database with the postgres user as it's owner, which you need to specify when you're not signed in as the postgres user. I have all of my files stored as 'AJames' user though so if you're the same and want to keep developing your app when signed in as a different user you need to create a Postgres 'role' for that user now (see step 4).
.Step 4. Through pgAdmin III. Right-click on Login Roles (which for me is in):
Object Browser > Server Groups > Servers > PostgreSQL 9.0 (localhost:5432) > Login Roles
Right-click on Login Roles and select 'New Login Role...'
in Role name, put in your operating system user name, which for me is AJames,
and fill in your password under the 'Role Privileges' tab, I checked all the boxes, but an experienced postgres user would likely strongly recommend to only check the 'inherits rights from parent roles' and the 'can create database objects' But I'm not an experienced user and just want to debug Rails SQL calls in Postgres so I also checked the 'Superuser' and 'Can create roles', just in case.
.Step 5. You should now be able to create a new database without being signed in as the postgres user. Try typing:
createdb mydb_as_user
Hopefully this should work for you.
.Step 6. Okay, so you've got a development.sqlite3 file in your rails 'db/' directory. Initially I was going to set the next test as converting this from sqlite3 to psql.
I couldn't get this to work though but I left my attempts here as the solution I used required having the data in a Rails app on Heroku.com (see instead the solution from step 7 onwards). For those who only have a local app and no data in Heroku, they can't use the same approach, so they might need to explore something like this:
x6.1 First, test 'psql' by trying a command from your command line like:
psql mydb_as_user
this should display something like below (after you've typed in your password):
C:>psql mydb_as_user
Password:
psql (9.0.4)
WARNING: Console code page (850) differs from Windows code page (1252)
8-bit characters might not work correctly. See psql reference
page "Notes for Windows users" for details.
Type "help" for help.
mydb5=#
x6.2 try entering:
CREATE TABLE users_table (id integer, "name" text);
It should display:
CREATE TABLE
mydb5=#
If you check in pgAdmin III, you should see the table there under:
Object Browser > Server Groups > Servers > PostgreSQL 9.0 (localhost:5432) > Databases > mydb_as_user > Schemas > public > Tables > users_table >
x6.3 Okay, next to try the conversion. Downloaded sqlite-shell precompiled binary for windows.
x6.4 Create a new directory, I used 'C:\temp' and put the sqlite3.exe and your development.sqlite3 files in it.
x6.5 Use the following commands (which are from here) to dump the development.sqlite3 database into Postgres.
sqlite3 development .dump | psql development2
you might get an error like:
psql: FATAL: database "development2" does not exist
x6.6 so I went into pgAdmin III and made a development 2 database, tried the command again and got:
ERROR: syntax error at or near "PRAGMA"
LINE 1: PRAGMA foreign_keys=OFF;
^
BEGIN
COMMIT
Like I said, I couldn't get it to work. I'm sure there's a way of getting round that error but I thought of a different way and so I instead used this solution (which requires a Heroku account to have your data and does the conversion from sqlite3 to psql using the Taps gem (I believe):
.Step 7. in pgAdmin III I created another database. Under the properties tab I set name: 'development', owner: 'AJames' (replace this with your own Windows user name). And under the privileges tab, set role: 'public' and checked the ALL option (thought this resets to unchecked so I'm not sure that's necessary).
.Step 8. add
gem 'pg', '0.11.0'
to your gem file. You'll probably also want to remove the:
gem 'sqlite3'
at this point too.
.Step 9. set database.yml as suggested here to:
development:
adapter: postgresql
database: db/development
username: AJames # replace this with your own user name
password: secret # replace this with your own password
host: localhost
encoding: UTF8
pool: 5
timeout: 5000
If you are working on an open source project and don't want your password to be made publicly available, have a look at some of the answers to Securely providing the database password in a Rails app.
.Step 10. from command line in your rails app's root directory run:
rake db:migrate
This will create the new schema and all the tables in the Postgres database.
.Step 11. run heroku db:pull from your command line (again from in the root directory of your rails app) to pull all your data down and into your new empty Postgres database. I think at this point your taps gem will be doing this work for you.
.Step 12. Hopefully there is no step 12! ...and it should now be working for you. Happy RoR PostgreSQL debugging! Please edit, or let me know, if there are any errors in this.
Also, here's a list of additional stuff that might be interesting/useful:
This is a blog post about Postgres passwords, what they're for,
why you need them, how to change them etc.
This, under 'Creating a spatial database', is useful for newbies to
understand what pg_hba.conf is about and the second link that
Reno gave above, under the 'Using pgAdmin III GUI' is useful to
testing to see if postgres is actually working, before trying to fit
it with Rails (i.e. try creating a database and putting a table and
some data into it).
In the huge but very comprehensive Postgres documentation, I'd
start on page 58, 'I. tutorial'. Then on pdf page 431(!) there's
'Chapter 17. Server setup and operation' that I also found useful.
the answer from AJP is the correct one with just a small re-config.
The line
database: db/development
does not work for me.
I have to change it to
database: development
I use this for ruby on rails for the command
rake db:create
and
rake db:migrate
to work
None of your links appeared to be the (arguably) most useful documentation - the official postgresql docs. I recently configure a Linux Mint box with postgresql and django using a combination of those documents and these, though the latter are specific to Linux.
I'd worry more about validating the postgresql side of things, less about rails. That is to say, your question should be "How can I set up and test a postgresql server on a Windows 7 box", moreso than you need to know how to get it locked into rails.
Edit:
Maybe this could also be of use to you - official postgresql wiki with detailed installation guides.
Good luck!