I'm trying to migrate an InfluxDB database to TimescaleDB with the help of outflux:
$ outflux migrate database --input-server https://xxx:443 --input-user xxx --input-pass xxx --output-conn "dbname=tutorial user=postgres"
2019/10/18 17:06:42 All pipelines scheduled
2019/10/18 17:06:42 All pipelines finished
2019/10/18 17:06:42 Migration execution time: 0.000 seconds
$
As the execution of the command above does not ask for the PostgreSQL password, outflux obviously does not connect to PostgreSQL.
But why it does not return an appropriate error message?
How can I properly set the host, port, database, user and password of my PostgreSQL database so that outflux connects to it?
Related
I'm trying to migrate users from one system to another. Each system has its own database and different classes.
My plan is to connect to one database, read some info from one database via SQL commands:
ActiveRecord::Base.connection.execute(sql_command)
do something with the data and then write some results on the new database using normal models.
I plan on doing that inside Sidekiq job but I'm trying to do some testing using a Rails console on Heroku.
This should be pretty straightforward, but this proves ridiculously difficult.
When I launched a Rails console on Heroku. I'm connecting to DATABASE_URL, which is ok, but when I try to connect to the old database and execute a command, like this:
ActiveRecord::Base.establish_connection(adapter: "postgresql", encoding: "unicode", pool: 10, url: "postgres://...info...")
ActiveRecord::Base.connection.execute("select count(*) from users")
I end up with:
PG::ConnectionBad (could not connect to server: No such file or directory)
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
I can, however, connect to this old database by launching my rails console on heroku using DATABASE_URL as the env variable:
$ heroku run 'DATABASE_URL=postgres://...info... rails console' -a console' -a app
BUT I don't know how to switch back to my new database so I can update things.
How does one switch databases at run time when using rails console on heroku?
Why try to runtime switch the database? Why not have both connected at the same time and specify at the model level which database they read/write from? Rails supports connecting multiple databases and specifying in individual models what database connection to use: https://guides.rubyonrails.org/active_record_multiple_databases.html
The problem was that using url: didn't work and all parameters needed to be specified.
config = {"adapter"=>"postgresql", "encoding"=>"unicode", "pool"=>10, "username"=>"u...", "password"=>"p...", "port"=>5432, "database"=>"d...", "host"=>"ec2..."}
If you go for a 3tier database yml, you can use this:
config = ActiveRecord::Base.configurations["production"]["seconddb"]
Then, you can use establish_connection
ActiveRecord::Base.establish_connection(config)
ActiveRecord::Base.connection.execute("select count(*) from users")
Once I started specifying username, password, port, database and host, it all worked like a charm.
To work with both databases at the same time, a good way is to create a class
class OtherDB < ActiveRecord::Base
establish_connection(ActiveRecord::Base.configurations["production"]["seconddb"])
end
Then you can call things that way
OtherDB.table_name = "table_name"
OtherDB.first
ref (Establish a connection to another database only in a block?)
And to run SQL commands:
OtherDB.connection.execute("select count(*) from users")
I recently created a basic level database for my rails app and I noticed while I was attempting to run:
heroku run rake db:migrate
from the command line, that it was connecting to my old database. How do you control which database heroku connects to from the heroku run command?
I don't think it's possible to specify the database when running Heroku Rake commands, but you could set a specific database as primary (and thus automatically connecting to it when running said commands).
Run heroku pg:info, which should output something like (two databases should be listed in your case):
=== HEROKU_POSTGRESQL_MAROON_URL (DATABASE_URL)
Plan: Ronin
Status: Available
Data Size: 26.1 MB
Tables: 5
PG Version: 9.5.3
Connections: 2
Fork/Follow: Available
Rollback: Unsupported
Created: 2012-05-02 21:54 UTC
Maintenance: not required (Mondays 23:00 to Tuesdays 03:00 UTC)
Infrastructure: Legacy
You can then choose a database to promote:
heroku pg:promote HEROKU_POSTGRESQL_MAROON_URL
You can read more in this Heroku article.
I am pretty new to heroku & postgresql. I am trying to dump my production database into my local development database in my local machine.
I dumped like this (my actual info redacted):
pg_dump --host=<myhost> --port=<port> --username=<username> --password --dbname=app_production > output.sql
Then I imported to my local app_development database like this:
psql -d app_development -f output.sql
but now, when I start my server I get this:
PG::InsufficientPrivilege: ERROR: permission denied for relation schema_migrations : SELECT "schema_migrations".* FROM "schema_migrations"
I also use Navicat to see the local database, and now I can't open any of my tables. Each time I try I get
ERROR: permission denied for relation <nameofwhatevertable>
How can I reset permissions for my app_development database when I've dumped my app_production database.
It seems like your db permissions have been mangled. Do you have PgAdmin III installed? http://www.pgadmin.org/download/
PgAdmin III will let you reset all db and user parameters. It will also allow you to see what is going on in your database(s) and tables so you can debug them.
It's a PostGreSQL development tool that is very much like MySQL Workbench. Free download from PostGreSQL. Runs on multiple platforms. Easy to set up.
It is most handy in cases like this, when you're trying to track down an intractable error. Highly recommended tool. Takes the guessing out of pg work.
I am having Postgres database dump file of 150 GB on Amazon EC2 instance. While dumping the data on RDS from EC2 server I am getting error.
The output of the command is as given below. It's giving error
1. psql connection not open
2. connection to server was lost
on RDS dump postgres copy command
Command Output:
SET
SET
SET
SET
SET
SET
SET
SET
ALTER TABLE
ALTER TABLE
ALTER SEQUENCE
ALTER TABLE
psql:filename.sql:1396266: connection not open
psql:filename.sql:1396266: connection to server was lost
Application Configuration:
Ruby 1.9.3
Rails 3
PostgreSQL 9.3
Please help why it is breaking while copying the data. As the connection was established and command is running. Suddenly while executing the copy command it is breaking.
Update Findings
Command/script that I am using to dump data the data is below:
psql -h instance.id.region-2.rds.amazonaws.com -p 5432 -U username -W -d database_name -f filename.sql
Workaround for finding issues as below:
I took head 100 rows and tail 200 rows and made 1 file from the big 150 GB file it converted to 56KB. when I run this command, it's dumped successfully.
So the file size is causing problem. The same command is running for small size file for dumping data on RDS.
How can I resolve this issue?
I think your connect it's disconnect in tcp connection level.
these is any limit in your env from your client to RDS?
like netflow control, idle session kill and so on...
I'm having problems getting testing to work with Postgresql and Rails 3.
Both development and production databases I can get to work fine, however the test database throws the following errors when I run rake or db:test:prepare, etc.
PGError: ERROR: source database "template1" is being accessed by other users
Update
Googling around, it seems that one should use template0 instead of template1 when using createdb to create a new database in Postgres. In typical “So I’ll remove the cause. But not the symptom” fashion, I found vendor/rails/railities/lib/task/databases.rake and changed line 109 to read:
createdb #{enc_option} \
-U "#{abcs["test"]["username"]}" \
-T template0 #{abcs["test"]["database"]}
But I don't really wanna do that, as I'm using Rails as a GEM, any one know of another work around or fix?
database.yml:
development:
adapter: postgresql
encoding: unicode
database: test1234_development
pool: 5
username: holden
password: postgres
test:
adapter: postgresql
encoding: unicode
database: test1234_test
pool: 5
username: holden
password: postgres
Full error:
NOTICE: database "test1234_test" does not exist, skipping
PGError: ERROR: source database "template1" is being accessed by other users
DETAIL: There are 1 other session(s) using the database.
: CREATE DATABASE "test1234_test" ENCODING = 'unicode'
Short story: CREATE DATABASE works by copying an existing database. PostgreSQL won't let you copy a database if another session is connected to it. If template1 is being accessed by other users, CREATE DATABASE will fail.
The question you need to answer: Why are other sessions connected to template1?
The difference between template0 and template1
At the point you initialize a database cluster, template0 and template1 are the same. Any location-specific stuff you want to make available to every database you create by using CREATE DATABASE should go into template1. So, for example, if you add the procedural langauge PL/python to template1, every database you create later will include PL/python.
The database template0 is intended to be a "virgin" template. It should contain only standard database objects--the ones created by initializing the cluster. As a "virgin" template, it should never be changed. Never.
If you need to specify encoding and locale settings (collation), then you can do that by copying template0. You can't do that by copying template1.
This problem occur when you had logged(psql template1 or psql template0) in template1 and template0 database and exit using below command.
Ctrl + z
Better way exist from db use below postgres command then problem will not create:
\q + enter
There are 2 solutions, If have problem.
Solution - 1
Restart posgres service like.
sudo service postgresql restart
Solution - 2
sudo ps aux | grep template1
Make sure don't delete this processes
postgres 8363 0.0 0.0 111760 7832 pts/11 T 09:49 0:00 /usr/lib/postgresql/9.5/bin/psql template1
ankit 18119 0.0 0.0 14224 976 pts/14 S+ 12:33 0:00 grep --color=auto template1
rest of process should be kill using below command.
sudo kill -9
Now try to create db again.
Hope this help you.
Ankit H Gandhi.
Just restart the service of database.
I restarted my system and the error was still showing. However, I followed the steps below to sort it out.
Stop all processes using the postgres port 5432 by doing this in command prompt (Admin): Type netstat -ano in command prompt. Find the pid with Local Address of 0.0.0.0:5432. Then use taskkill /pid {pid} /f to kill the task.
Start the postgres service in windows services.
I also got this error while trying to reset the database while I had the default Ruby on Rails server WEBrick running:
$ bin/rake db:reset
PG::Error: ERROR: database "dev" is being accessed by other users
DETAIL: There is 1 other session using the database.
: DROP DATABASE IF EXISTS "dev"
The other user here was the running Rails app. After shutting down the server with CTRL + c, I was able to re-run the database reset command without any problems.
It makes sense too. You can't drop the database if someone else is currently connected to it, as Mike Sherrill also points out.
Solution for me was to delete old server and create a new one from Postgresql administration web interface. Could now create new database without this error.
I was also stuck setting up postgres on ruby on rails project, ensure that you have installed pg locally and created a user with its password then on your database.yml should have:- host: localhost, password: (set password) then run:
$ rails db:create
$ rails db:migrate