Inheriting from postgres docker container - doesn't keep the daemon alive? - docker

I'm writing a Dockerfile that pulls in an external dump, then loads it - https://github.com/scala-eveapi/postgres-sde/commit/95d2ed70dff8326c9acc75c56c9a7b8c8f6bbc73 - docker build works fine. When running it, it restored the DB, but after running the .sql, it just exits, instead of keeping the postgres server alive.
The file:
FROM postgres:latest
ADD https://www.fuzzwork.co.uk/dump/latest/postgres-20161114-TRANQUILITY.dmp.bz2 sde.bz2
# ADD postgres-20161114-TRANQUILITY.dmp.bz2 sde.bz2
RUN bunzip2 sde.bz2
COPY load-sde.sh /docker-entrypoint-initdb.d/01-load-sde.sh
COPY add-constraints.sql /docker-entrypoint-initdb.d/02-add-constraints.sql
The other two files are:
#!/bin/bash
set -e
pg_restore -d "${POSTGRES_DB:-$POSTGRES_USER}" -U "$POSTGRES_USER" sde
And the SQL:
alter table "mapSolarSystems"
alter column "solarSystemName" set not null;
alter table "invTypes"
alter column "typeName" set not null;
alter table "staStations"
alter column "stationName" set not null;
alter table "staStations"
alter column "solarSystemID" set not null;
Logs:
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ...
WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
ok
Success. You can now start the database server using:
pg_ctl -D /var/lib/postgresql/data -l logfile start
****************************************************
WARNING: No password has been set for the database.
This will allow anyone with access to the
Postgres port to access your database. In
Docker's default configuration, this is
effectively any other container on the same
system.
Use "-e POSTGRES_PASSWORD=password" to set
it in "docker run".
****************************************************
waiting for server to start....LOG: database system was shut down at 2016-12-06 12:05:18 UTC
LOG: MultiXact member wraparound protections are now enabled
LOG: database system is ready to accept connections
LOG: autovacuum launcher started
done
server started
ALTER ROLE
/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/01-load-sde.sh
ERROR: role "yaml" does not exist
STATEMENT: ALTER TABLE "agtAgentTypes" OWNER TO yaml;
[...] pg_restore errors
WARNING: errors ignored on restore: 89

The errors caused it to stop apparently. I added the yaml role and now it works properly.

Look at the last three lines on the initial Dockerfile (the one you're inheriting from: https://github.com/docker-library/postgres/blob/edd455e5b1dbfddc280beb244228054374f2f3dd/9.6/Dockerfile):
ENTRYPOINT ["/docker-entrypoint.sh"]
EXPOSE 5432
CMD ["postgres"]
You're extending that Dockerfile, but you're not setting the command to run... So yea, the container stops and it'll be marked as exited.

Related

Running a Chainlink Node - Can't connect to database

Using docker-desktop on macOS.
I'm trying to run a node following the instructions on this page.
The database name is node, which is the same as the username: node. The user has access to the database and can log in using psql client.
Connection strings I've tried in the .env file:
postgresql://node#localhost/node
postgresql://node:password#localhost/node
postgresql://node:password#localhost:5432/node
postgresql://node:password#127.0.0.1:5432/node
postgresql://node:password#127.0.0.1/node
When I run the start command: cd ~/.chainlink-kovan && docker run -p 6688:6688 -v ~/.chainlink-kovan:/chainlink -it --env-file=.env smartcontract/chainlink local n , using docker-desktop on macOS, I get the following stack trace:
2020-09-15T14:24:41Z [INFO] Starting Chainlink Node 0.8.15 at commit a904730bd62c7174b80a2c4ccf885de3e78e3971 cmd/local_client.go:50
2020-09-15T14:24:41Z [INFO] SGX enclave *NOT* loaded cmd/enclave.go:11
2020-09-15T14:24:41Z [INFO] This version of chainlink was not built with support for SGX tasks cmd/enclave.go:12
2020-09-15T14:24:41Z [INFO] Locking postgres for exclusive access with 500ms timeout orm/orm.go:69
2020-09-15T14:24:41Z [ERROR] unable to lock ORM: dial tcp 127.0.0.1:5432: connect: connection refused logger/default.go:139 stacktrace=github.com/smartcontractkit/chainlink/core/logger.Error
/chainlink/core/logger/default.go:117
...
Does anyone know how I can resolve this?
The problem probably caused by the fact that your chainlink database has been locked with Exclusive Lock and before stopping node that locks never removed.
What you do in this situation (as what works for me) is use PgAdmin Ui or similar way to find all Locks then find the Exclusive Lock that is held on the chainlink database and note down its Process id or ids (if multiple exclusive locks there are on chainlink DB)
Log in to your pg client and run SELECT pg_terminate_backend(<pid>) or SELECT pg_cancel_backend(<pid>); Enter PID of those locks here without quotes and meanwhile keep refreshing on pg admin URL to see if those processes stopped If stopped then rerun your chainlink node.
The problem is with docker networking.
Add --network host to the docker run command so that it is:
cd ~/.chainlink-kovan && docker run -p 6688:6688 -v ~/.chainlink-kovan:/chainlink -it --env-file=.env smartcontract/chainlink --network host local n
This fixes the issue.

How to connect some graphical sql tool to database via ssh AND docker

I would like to connect my GUI database tool to my SQL Database via ssh and docker.
Currently a can connect do database via terminal using ssh user#host and some docker-compose exec mydb ... command. But then, it's of course a command line access to db.
My needs / my question
Is there a way to have my GUI database tool to connect to that DB that way ?
To be more explicit, i would like to connect to that db without any server change of any kind (really important point). So with only local configuration change. Maybe we can use the same way i can use by hand ?
What i tried
I already tried to use some ssh configuration in my config file like ProxyCommand in my ssh config file, but these command are executed in my computer...so i don't find any way to success with this.
I also searched many times anyone with the same will without success.
Somebody with great idea ?
There are some things to consider here.
make sure the port mapping is valid in the docker-compose.yaml file, it should be something like
services:
...
database:
...
ports:
- "3306:3306" # 3306 is the default Non-SSL port for MySQL database images
make sure the machine you are running the database container on has no firewall rule to block incoming traffic
make sure the in the database, the user you are trying to connect with has a whitelisted location.
for example, if you want to be able to connect as root from anywhere, you need check the user table in the mysql database, and it should look something like:
mysql> select User, Host from user;
+------------------+-----------+
| User | Host |
+------------------+-----------+
| root | % |
| mysql.infoschema | localhost |
| mysql.session | localhost |
| mysql.sys | localhost |
| root | localhost |
+------------------+-----------+
5 rows in set (0.00 sec)
mysql>
In this case, the root user can connect to this particular DB from anywhere, because of the wildcard (%) host.

Docker container (Kubernetes): Mysql user access denied

Hi I have followed some k8s tutorials on how to get going with setting up a local db + WordPress installation, but user can't connect to mysql within my cluster.
(everything else seems ok - in Kubernetes Dashboard Web UI)
Error: [15:40:55][~]#kubectl logs -f website-56677747c7-c7lb6
[21-Nov-2019
11:07:17 UTC] PHP Warning: mysqli::__construct():
php_network_getaddresses: getaddrinfo failed: Name or service not
known in Standard input code on line 22 [21-Nov-2019 11:07:17 UTC] PHP
Warning: mysqli::__construct(): (HY000/2002):
php_network_getaddresses: getaddrinfo failed: Name or service not
known in Standard input code on line 22
MySQL Connection Error: (2002) php_network_getaddresses: getaddrinfo
failed: Name or service not known [21-Nov-2019 11:07:20 UTC] PHP
Warning: mysqli::__construct(): (HY000/1045): Access denied for user
'websiteu5er'#'10.1.0.35' (using password: YES) in Standard input code
on line 22
MySQL Connection Error: (1045) Access denied for user
'websiteu5er'#'10.1.0.35' (using password: YES)
MySQL Connection Error: (1045) Access denied for user
'websiteu5er'#'10.1.0.35' (using password: YES)
MySQL Connection Error: (1045) Access denied for user
'websiteu5er'#'10.1.0.35' (using password: YES)
My Dockerfile (which I used to create the image pushed to docker hub then pulled into k8s service + deployment):
FROM mysql:5.7
# This should create the following default root + user?
ENV MYSQL_ROOT_PASSWORD=hello123
ENV MYSQL_DATABASE=website
ENV MYSQL_USER=websiteu5er
ENV MYSQL_PASSWORD=hello123
RUN /etc/init.d/mysql start \
&& mysql -u root --password='hello123' -e "GRANT ALL PRIVILEGES ON *.* TO 'websiteu5er'#'%' IDENTIFIED BY 'hello123';"
FROM wordpress:5.2.4-php7.3-apache
# Copy wp-config file over
COPY configs/wp-config.php .
RUN chown -R www-data:www-data *
COPY ./src/wp-content/themes/bam /var/www/html/wp-content/themes/bam
The standard Docker Hub mysql image has the ability to run arbitrary SQL scripts on the very first startup of the database only. It can also set up an initial database user with a known password, again on the first startup only. Details are in the linked Docker Hub page.
In a Kubernetes context I’d use just the environment variables, and specify them in my pod spec.
containers:
- name: mysql
image: mysql:5.7 # not a custom image
env:
- name: MYSQL_USER
value: websiteu5er
- name: MYSQL_PASSWORD
value: hello123
If you did need more involved setup, I’d create a ConfigMap that contained SQL scripts, and then mount that into the container in /docker-entrypoint-initdb.d.
There’s two things going on in your Dockerfile. One is that, when you have multiple FROM lines, you’re actually executing a multi-stage build; the image you get out at the end is only the Wordpress image, and the MySQL parts before it get skipped. The second is that you can’t actually create an image FROM mysql that contains any database-level configuration or content, so the image that comes out of the first stage has the environment variables set but won’t actually have executed your GRANT PRIVILEGES statement.
I’d just delete everything before the last FROM line and not try to build a derived MySQL image; use the /docker-entrypoint-initdb.d mechanism at startup time instead.
I see you are trying to start mysql database and then grant privileges for your user but you are doing it all wrong.
After runing:
RUN /etc/init.d/mysql start \
&& mysql -u root --password='hello123' -e "GRANT ALL PRIVILEGES ON *.* TO 'websiteu5er'#'%' IDENTIFIED BY 'hello123';"
mysql starts but your query after && never runs so your user doesn't get its privileges. It will get run only if mysql exits successfully.
Look here for explanation how && works in shell.
What you want to do is run this command after mysql starts and you can do it in several ways but probably the best in your case would be using PostStart hook in kubernetes:
spec:
containers:
- name: test
image: someimage
lifecycle:
postStart:
exec:
command: ["/bin/sh", "-c", "mysql -u root --password='hello123' -e \"GRANT ALL PRIVILEGES ON *.* TO 'websiteu5er'#'%' IDENTIFIED BY 'hello123';\""]
You may also want to add few second sleep command before you run the query to make sure the server actually started before you connect to it.
Also take a look at kubernetes documentation and read more about lifecycle hooks.
Let me know if it was helpful.

Is the server running on host "localhost" (::1) and accepting TCP/IP connections on port 5432?

Before anything, please note that I have found several similar questions on Stack Overflow and articles all over the web, but none of those helped me fix my issue:
PG Error could not connect to server: Connection refused Is the server running on port 5432?
PG::ConnectionBad - could not connect to server: Connection refused
psql: could not connect to server: Connection refused
Now, here is the issue:
I have a Rails app that works like a charm.
With my collaborator, we use GitHub to work together.
We have a master and an mvp branches.
I recently updated my git version with Homebrew (Mac).
We use Foreman to start our app locally.
Now, when I try to launch the app locally, I get the following error:
PG::ConnectionBad at /
could not connect to server: Connection refused
Is the server running on host "localhost" (::1) and accepting
TCP/IP connections on port 5432?
could not connect to server: Connection refused
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 5432?
I tried to reboot my computers several times.
I also checked the content of /usr/local/var/postgres:
PG_VERSION pg_dynshmem pg_multixact pg_snapshots pg_tblspc postgresql.conf
base pg_hba.conf pg_notify pg_stat pg_twophase postmaster.opts
global pg_ident.conf pg_replslot pg_stat_tmp pg_xlog server.log
pg_clog pg_logical pg_serial pg_subtrans postgresql.auto.conf
As you can see, there is no postmaster.pid file in there.
Any idea how I could fix this?
run postgres -D /usr/local/var/postgres and you should see something like:
FATAL: lock file "postmaster.pid" already exists
HINT: Is another postmaster (PID 379) running in data directory "/usr/local/var/postgres"?
Then run kill -9 PID in HINT
And you should be good to go.
You most likely ran out of battery and your postgresql server didn't shutdown correctly.
The easiest workaround is to download the official postgresql app and launch it: it will force the server to start (http://postgresapp.com/)
Most likely it's because your system shutdown unexpectedly
Try
postgres -D /usr/local/var/postgres
You might see
FATAL: lock file "postmaster.pid" already exists
HINT: Is another postmaster (PID 449) running in data directory "/usr/local/var/postgres"?
Then try
kill -9 PID
example
kill -9 419
And it should start postgres normally
The postgresql server might be down and the solution might be as simple as running:
sudo service postgresql start
which fixed the issue for me.
This could be caused by the pid file created for postgres which has not been deleted due to unexpected shutdown. To fix this, remove this pid file.
Find the postgres data directory. On a MAC using homebrew it is /usr/local/var/postgres/, other systems it might be /usr/var/postgres/
Remove pid file by running:
rm postmaster.pid
Restart postgress. On Mac, run:
brew services restart postgresql
I had almost just as same error with my Ruby on Rails application running postgresql(mac). This worked for me:
brew services restart postgresql
This worked in my case:
brew uninstall postgresql
rm -fr /usr/local/var/postgres/
brew install postgresql
In my case PostgreSQL updates from version 13.4 to 14 in background, so it fixes by:
brew postgresql-upgrade-database
In other case the problem fixed by:
rm -rf /usr/local/var/postgres/postmaster.pid
or
rm -rf /opt/homebrew/var/postgres/postmaster.pid
Restart service postgresql:
brew services restart postgresql
PS:
How can you understand what is the problem?
For first see what service is not correct started:
brew services list
For second show file postgres.log, where will be the error:
tail -f /usr/local/var/log/postgres.log
or
tail -f /opt/homebrew/var/log/postgres*
And so find answer by this error's text
I resolved the issue via this command
pg_ctl -D /usr/local/var/postgres start
At times, you might get this error
pg_ctl: another server might be running; trying to start server anyway
So, try running the following command and then run the first command given above.
pg_ctl -D /usr/local/var/postgres stop
Step 1:
cd /etc/postgresql/12/main/
open file named postgresql.conf
sudo nano postgresql.conf
add this line to that file
listen_addresses = '*'
then open file named pg_hba.conf
sudo nano pg_hba.conf
and add this line to that file
host all all 0.0.0.0/0 md5
It allows access to all databases for all users with an encrypted password
restart your server
sudo /etc/init.d/postgresql restart
This is how I solved my problem:
see the status of services
brew services list
and the output was :
Name Status User Plist
postgresql error myuser /Users/myuser/Library/LaunchAgents/homebrew.mxcl.postgresql.plist
unbound stopped
I change the file name in this path, you can also remove it
mv /Users/myuser/Library/LaunchAgents/homebrew.mxcl.postgresql.plist /Users/myuser/Library/LaunchAgents/homebrew.mxcl.postgresql.plist_temp
and then reboot the os
sudo reboot
after booting I started the postgresql and it worked.
brew services start postgresql
Find postgresql#10 service directory
$ ls /usr/local/var/postgresql#10
Find file postmaster.pid and delete that file
$ rm -f postmaster.pid
Restart postgres service using
$ brew services restart postgresql#10
This worked for me:
run
sudo lsof -i :<port_number>
after that it will display the PID which is currently attached to the process.
After that run sudo kill -9 <PID>
if that doesn't work,
try the solution offered by user8376606 it would definitely work!
If you want to restart Postgresql on Linux, then you have to use the following command.
/etc/init.d/postgresql restart
In my case
I have changed the port in postgresql.conf file
and restart postgresql services in
Run => service.msc => Restart
now retry
First I tried
lsof -wni tcp:5432
but it doesn't show any PID number.
Second I tried
Postgres -D /usr/local/var/postgres
and it showed that server is listening.
So I just restarted my mac to restore all ports back and it worked for me.
For Docker users: In my case it was caused by excessive docker image size. You can remove unused data using prune command:
docker system prune --all --force --volumes
Warning: as per manual (docker system prune --help):
This will remove:
all stopped containers
all networks not used by at least one container
all dangling images
all dangling build cache
I encountered a similar problem when I was trying to connect my Django application to PostgreSQL database.
I wrote my Dockerfile with instructions to setup the Django project followed by instructions to install PostgreSQL and run Django server in my docker-compose.yml.
I defined two services in my docker-compose-yml.
services:
postgres:
image: "postgres:latest"
environment:
- POSTGRES_DB=abc
- POSTGRES_USER=abc
- POSTGRES_PASSWORD=abc
volumes:
- pg_data:/var/lib/postgresql/data/
django:
build: .
command: python /code/manage.py runserver 0.0.0.0:8004
volumes:
- .:/app
ports:
- 8004:8004
depends_on:
- postgres
Unfortunately whenever I used to run docker-compose up then same err. used to pop up.
And this is how my database was defined in Django settings.py.
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': 'abc',
'USER': 'abc',
'PASSWORD': 'abc',
'HOST': '127.0.0.1',
'PORT': '5432',
'OPTIONS': {
'client_encoding': 'UTF8',
},
}
}
So, In the end I made use of docker-compose networking which means if I change the host of my database to postgres which is defined as a service in docker-compose.yml will do the wonders.
So, Replacing 'HOST': '127.0.0.1' => 'HOST': 'postgres' did wonders for me.
After replacement this is how your Database config in settings.py will look like.
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': 'abc',
'USER': 'abc',
'PASSWORD': 'abc',
'HOST': 'postgres',
'PORT': '5432',
'OPTIONS': {
'client_encoding': 'UTF8',
},
}
}
I often encounter this problem on windows,the way I solved the problem is Service - Click PostgreSQL Database Server 8.3 - Click the second tab "log in" - choose the first line "the local system account".
It also gives the same error if you just stop your PostgreSQL app. You just need to start it again. (PostgreSQL 11)
I have faced the same issue and I was unable to start the postgresql server and was unable to access my db even after giving password, and I have been doing all the possible ways.
This solution worked for me,
For the Ubuntu users:
Through command line, type the following commands:
1.service --status-all (which gives list of all services and their status. where "+" refers to running and "-" refers that the service is no longer running)
check for postgresql status, if its "-" then type the following command
2.systemctl start postgresql (starts the server again)
refresh the postgresql page in browser, and it works
For the Windows users:
Search for services, where we can see list of services and the right click on postgresql, click on start and server works perfectly fine.
In my case I forgot to change the database from postgres (on my production) back to sqlite3 which I was using for development.
It's working for me >>Node.Js App
user#MacBook-Pro % sudo lsof -i :5430
Output
COMMAND PID user FD TYPE DEVICE SIZE/OFF NODE NAME
node 7885 user 21u IPv6 0x2e7d89f6118f95b9 0t0 TCP *:radec-corp (LISTEN)
Kill the PID
user#MacBook-Pro % sudo kill -9 7885
One more test
user#MacBook-Pro % sudo lsof -i :5430
user#MacBook-Pro % "No more running PID for the port 5430"
In my case, on a Ruby on Rails project, I removed a .pid file from the folder tmp/pids and restart the system.
Had the same issue. I checked that my database.yml file, (dev mode) host was pointing to 5433. I updated it to 5432 and it worked.
Just in case someone needs this for windows, read on.
On windows hit the Windows button + R
then enter services.msc and look for postgresql-x64-14, Right click it and click start
Then go back to your PgAdmin4 for windows and then enter your master password if asked.
From here, you should be able to proceed as usual with viewing of the db schemas.
Also, for Django, restart your server with CTR+C then python manage.py runserver (assuming you're working inside a virtual env)
Good luck
ps -ef|grep postgres
Then kill the process with PID
sudo kill -9 PID
Then start the postgresql
sudo service postgresql start
In my case I when it happens to me I need to do the following steps:
1º Step
Log in postgres user:
#sudo su postgres
2º Step
Run the following steps: /opt/PostgreSQL/10/bin/postgres -D /opt/PostgreSQL/10/data -r /usr/local/var/postgres/server.log
Explanation:
We access the utility of postgres located at /opt/PostgreSQL/10/bin/ in your case could be another but identify where it's.
After this step we tell to the utility of postgres where the it's data folder is by using the option -D, this data folder contains all necessary configuration of postgres server.
The option -r we tell to postgres where to send stdout and stderr to given file, in my case the file that I used is /usr/local/var/postgres/server.log
Note:
I'm using Postgre 10
Linux Ubuntu

Postgres sh: /usr/local/var/postgres/server.log: No such file or directory

I recently had to reinstall homebrew and postgres and now postgres isn't working with rails.
First when trying to run rails s, I get this:
/Users/lee/.rvm/gems/ruby-1.9.3-p125#achievex/gems/activerecord-3.2.6/lib/active_record/connection_adapters/postgresql_adapter.rb:1206:in `initialize': could not connect to server: Connection refused (PG::Error)
So I try starting postgres and I get this:
pg_ctl -D /usr/local/var/postgres -l /usr/local/var/postgres/server.log start
server starting
sh: /usr/local/var/postgres/server.log: No such file or directory
It appears that postgres is running because of this:
ps aux | grep postgreslee 1099 0.3 0.0 2425480 188 s000 R+ 5:48PM 0:00.01 grep postgres
I'm totally at a loss here as to why I can't get it running.
First, ps | grep is in fact pulling your grep out of the process tree. Postgres is not running. This sort of thing happens when the directory to put the log in doesn't exist. So probably you are pointing it at a non-existant directory.
This could happen if you don't have a data directory yet.
Recommendations:
Start by making sure your data directory exists. search for pg_hba.conf and see if it is anywhere on your system. If it is then there is a good chance it is in your data directory (if it is in etc, then look for base/1/PG_VERSION This will be in your data directory one level up from base).
if it exists already change your postgresql.conf to set logging setting appropriately and drop the -l option.
If no data directory exists, use the initdb command to make one.

Resources