Getting Internal Server Error on prisma deploy - docker

Question

I have a Postgres database on Heroku, upon deploying the data model by doing prisma deploy often times the following error is produced.
ERROR: Whoops. Looks like an internal server error. Search your server logs for request ID: local:cjxrmcnpx00hq0692zuwttqwv
{
"data": {
"addProject": null
},
"errors": [
{
"message": "Whoops. Looks like an internal server error. Search your server logs for request ID: local:cjxrmcnpx00hq0692zuwttqwv",
"path": [
"addProject"
],
"locations": [
{
"line": 2,
"column": 9
}
],
"requestId": "local:cjxrmcnpx00hq0692zuwttqwv"
}
],
"status": 200
}
and on checking the Docker logs I am seeing this erorr:
Jul 14, 2019 12:18:34 PM org.postgresql.Driver connect
prisma_1 | SEVERE: Connection error:
prisma_1 | org.postgresql.util.PSQLException: FATAL: too many connections for role "bcueventxumaik"
prisma_1 | at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2433)
prisma_1 | at org.postgresql.core.v3.QueryExecutorImpl.readStartupMessages(QueryExecutorImpl.java:2566)
prisma_1 | at org.postgresql.core.v3.QueryExecutorImpl.<init>(QueryExecutorImpl.java:131)
prisma_1 | at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:210)
prisma_1 | at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
prisma_1 | at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
prisma_1 | at org.postgresql.Driver.makeConnection(Driver.java:452)
prisma_1 | at org.postgresql.Driver.connect(Driver.java:254)
prisma_1 | at slick.jdbc.DriverDataSource.getConnection(DriverDataSource.scala:101)
prisma_1 | at slick.jdbc.DataSourceJdbcDataSource.createConnection(JdbcDataSource.scala:68)
prisma_1 | at slick.jdbc.JdbcBackend$BaseSession.<init>(JdbcBackend.scala:453)
prisma_1 | at slick.jdbc.JdbcBackend$DatabaseDef.createSession(JdbcBackend.scala:46)
prisma_1 | at slick.jdbc.JdbcBackend$DatabaseDef.createSession(JdbcBackend.scala:37)
prisma_1 | at slick.basic.BasicBackend$DatabaseDef.acquireSession(BasicBackend.scala:249)
prisma_1 | at slick.basic.BasicBackend$DatabaseDef.acquireSession$(BasicBackend.scala:248)
prisma_1 | at slick.jdbc.JdbcBackend$DatabaseDef.acquireSession(JdbcBackend.scala:37)
prisma_1 | at slick.basic.BasicBackend$DatabaseDef$$anon$2.run(BasicBackend.scala:274)
prisma_1 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
prisma_1 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
prisma_1 | at java.lang.Thread.run(Thread.java:748)
prisma_1 |
prisma_1 | Exception in thread "main" org.postgresql.util.PSQLException: FATAL: too many connections
prisma_1 | at org.postgresql.core.v3.QueryExecutorImpl.readStartupMessages(QueryExecutorImpl.java:2566)prisma_1 | at org.postgresql.core.v3.QueryExecutorImpl.<init>(QueryExecutorImpl.java:131)prisma_1 | at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:210)
prisma_1 | at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)prisma_1 | at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
prisma_1 | at org.postgresql.Driver.makeConnection(Driver.java:452)
prisma_1 | at org.postgresql.Driver.connect(Driver.java:254)prisma_1 | at slick.jdbc.DriverDataSource.getConnection(DriverDataSource.scala:101)
prisma_1 | at slick.jdbc.DataSourceJdbcDataSource.createConnection(JdbcDataSource.scala:68)
prisma_1 | at slick.jdbc.JdbcBackend$BaseSession.<init>(JdbcBackend.scala:453)
prisma_1 | at slick.jdbc.JdbcBackend$DatabaseDef.createSession(JdbcBackend.scala:46)
prisma_1 | at slick.jdbc.JdbcBackend$DatabaseDef.createSession(JdbcBackend.scala:37)
prisma_1 | at slick.basic.BasicBackend$DatabaseDef.acquireSession(BasicBackend.scala:249)
prisma_1 | at slick.basic.BasicBackend$DatabaseDef.acquireSession$(BasicBackend.scala:248)
prisma_1 | at slick.jdbc.JdbcBackend$DatabaseDef.acquireSession(JdbcBackend.scala:37)
prisma_1 | at slick.basic.BasicBackend$DatabaseDef$$anon$2.run(BasicBackend.scala:274)
prisma_1 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
prisma_1 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
prisma_1 | at java.lang.Thread.run(Thread.java:748)
prisma_prisma_1 exited with code 1
The error is saying too many connections but I am firing prisma deploy from only one terminal and at the same time I am able to connect to the database using PgAdmin4. Moreover, the database seems to be perfectly reachable as I am able to ping the database from inside the container.
PS. Updated the docker logs as earlier on running docker logs -f processid I was getting older logs but now upon building the container again using docker-compose up I got the lastest logs

Answer 1

As the error clearly states there are too many connections to the database. So we need to investigate how many connections there are, who is creating them and why they are created. In order to either limit the consumers or increase the amount of available connections.
First we can start by using the heroku CLI to check the amount of used and available connections:
$ heroku pg:info
=== DATABASE_URL
Plan: Private 2
Status: Available
HA Status: Available
Data Size: 2.23 GB
Tables: 83
PG Version: 10.1
Connections: 26/400
Connection Pooling: Available
For more information on how to investigate heroku postgres databases see: https://devcenter.heroku.com/articles/heroku-postgresql#pg-info
To further investigate who is connected to your database you can either use psql or pgAdmin. If using pgAdmin you can select the database, click on the dashboard tab and select the server activity panel on the bottom of the page revealing all connected sessions. If using psql you could write a select like this:
SELECT pid as process_id,
usename as username,
datname as database_name,
client_addr as client_address,
application_name,
backend_start,
state,
FROM pg_stat_activity;
For a more detailed how to see: https://dataedo.com/kb/query/postgresql/list-database-sessions
By now you probably identified who is creating the connections to your database and can limit the client to use less (or increase the amount of available database connections).
One possible consumer of database connections is the prisma server itself of course. The prisma config luckily provides a setting to limit database connections.
The connectionLimit property in PRISMA_CONFIG determines the number of
database connections a Prisma service is going to use.
You can read more about it here: https://www.prisma.io/docs/prisma-server/database-connector-POSTGRES-jgfr/#managing-database-connections
If you are using heroku to run the docker container with your prisma server a PRISMA_CONFIG could look like this:
port: $PORT
managementApiSecret: ${PRISMA_MANAGEMENT_API_SECRET}
databases:
default:
connector: postgres
migrations: true
connectionLimit: 2
uri: ${DATABASE_URL}?ssl=1
I hope this structured approach helped. Let me know if you need more clarification. If so please provide details regarding the nature of the existing database connections.

Answer 2

run this command
docker logs <YOUR_PRISMA_CONTAINER_NAME>

Answer 3

use pooling:
import dotenv from 'dotenv'
dotenv.config()
import { PrismaClient } from '#prisma/client'
// add prisma to the NodeJS global type
interface CustomNodeJsGlobal extends NodeJS.Global {
prisma: PrismaClient
}
// Prevent multiple instances of Prisma Client in development
declare const global: CustomNodeJsGlobal
const prisma = global.prisma || new PrismaClient()
if (process.env.NODE_ENV === 'development') global.prisma = prisma
export default prisma
plus use:
await prisma.$disconnect()