Keycloak and Docker OutOfMemoryError - process/resource limits reached - docker

Question

I have rented a virtual ubuntu server. Various applications run on it in Docker containers and natively:
Plesk
Wordpress
Flarum
MySQL
Wiki.js (in Docker container)
Keycloak (in Docker container)
MariaDB (in Docker container)
I use Keycloak as SSO for Wordpress, Wiki.js and Flarum. Now I have the problem that Keycloak simply crashes after a while and I can't restarted it in Docker. I get the following error message:
keycloak_1 | 17:22:06,447 DEBUG [org.jboss.as.config] (MSC service thread 1-3) VM Arguments: -D[Standalone] -Xms512m -Xmx2048m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -XX:+UseAdaptiveSizePolicy -XX:MaxMetaspaceSize=1024m -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true-Djava.net.preferIPv4Stack=true --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED -Dorg.jboss.boot.log.file=/opt/jboss/keycloak/standalone/log/server.log -Dlogging.configuration=file:/opt/jboss/keycloak/standalone/configuration/logging.properties
keycloak_1 | 17:22:19,493 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([
keycloak_1 | ("subsystem" => "infinispan"),
keycloak_1 | ("cache-container" => "keycloak"),
keycloak_1 | ("thread-pool" => "transport")
keycloak_1 | ]) - failure description: {"WFLYCTL0080: Failed services" => {"org.wildfly.clustering.infinispan.cache-container.keycloak" => "org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheException: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
keycloak_1 | Caused by: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheException: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
keycloak_1 | Caused by: org.infinispan.commons.CacheException: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
keycloak_1 | Caused by: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached"}}
keycloak_1 | 17:22:19,505 INFO [org.jboss.as.server] (ServerService Thread Pool -- 46) WFLYSRV0010: Deployed "keycloak-server.war" (runtime-name : "keycloak-server.war")
keycloak_1 | 17:22:19,507 INFO [org.jboss.as.controller] (Controller Boot Thread) WFLYCTL0183: Service status report
keycloak_1 | WFLYCTL0186: Services which failed to start: service org.wildfly.clustering.infinispan.cache.ejb.http-remoting-connector: org.infinispan.commons.CacheConfigurationException: Error starting component org.infinispan.expiration.impl.InternalExpirationManager
keycloak_1 | service org.wildfly.clustering.infinispan.cache-container.keycloak: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheException: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
keycloak_1 | WFLYCTL0448: 32 additional services are down due to their dependencies being missing or failed
keycloak_1 | 17:22:19,599 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server
keycloak_1 | 17:22:19,606 ERROR [org.jboss.as] (Controller Boot Thread) WFLYSRV0026: Keycloak 12.0.4 (WildFly Core 13.0.3.Final) started (with errors) in 15455ms - Started 558 of 926 services (44 services failed or missing dependencies, 684 services are lazy, passive or on-demand)
keycloak_1 | 17:22:19,614 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://127.0.0.1:9990/management
keycloak_1 | 17:22:19,614 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://127.0.0.1:9990
The critical mistake should be the following:
keycloak_1 | 17:48:15,196 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 60) MSC000001: Failed to start service org.wildfly.clustering.infinispan.cache-container.keycloak: org.jboss.msc.service.StartException in service org.wildfly.clustering.infinispan.cache-container.keycloak: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheException: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
At first time I thought that Keycloak with Docker need more memory. Unfortunately, the change didn't bring the desired success. After some research, I read that sometime there are some problems with the threads on virtual servers. Unfortunately, I don't know that much about this topic. I hope someone can help me. :)
Am I right that it can be due to the thread limit of the virtual server?
Attached is my docker-compose file:
version: '3'
services:
mariadb:
image: mariadb:latest
restart: always
environment:
MYSQL_ROOT_PASSWORD: ******
MYSQL_DATABASE: app_keycloak
MYSQL_USER: ******
MYSQL_PASSWORD: ******
ports:
- 3308:3306
# Copy-pasted from https://github.com/docker-library/mariadb/issues/94
healthcheck:
test: ["CMD", "mysqladmin", "ping", "--silent"]
keycloak:
image: jboss/keycloak:latest
restart: always
environment:
DB_VENDOR: mariadb
DB_ADDR: mariadb
DB_DATABASE: ******
DB_USER: ******
DB_PASSWORD: ******
KEYCLOAK_USER: ******
KEYCLOAK_PASSWORD: ******
JGROUPS_DISCOVERY_PROTOCOL: JDBC_PING
JAVA_OPTS: "-server -Xms512m -Xmx2048m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -XX:+UseAdaptiveSizePolicy -XX:MaxMetaspaceSize=1024m -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.head$t.headless=true-Djava.net.preferIPv4Stack=true"
ports:
- 8080:8080
depends_on:
- mariadb
Update 1:
It does not seem to be due to the thread limit.
systemctl show --property=DefaultTasksMax
I looked to see if there was a limit. I read that Ubuntu set DefaultTasksMax to 15%.
cat /proc/user_beancounters
Overall I have by provider a limit of 700 threads.
Additionally, I looked at how many threads were using the current services. Docker in particular.
systemctl status *.service | grep -e Tasks
systemctl status docker.service | grep -e Tasks --> 75
With the findings I set DefaultTasksMax to 200.
nano /etc/systemd/system.conf
systemctl daemon-reload
In the end, I restarted the Docker Compose.
docker-compose down
docker-compose up
Unfortunately, I still get the same error. :(
Update 2:
An update to version 13 of Keycloak has apparently fixed the problem. I will continue to monitor the behavior.