Keycloak Docker container fails to start after restarting the container - docker

Question

I have a Keycloak installation running as docker container in a docker-compose environment. Every night, my backup stops relevant containers, performs a DB and volume backup and restarts the containers again. For most it works, but Keycloak seems to have a problem with it and does not come up again afterwards. Looking at the logs, the error message is:
The batch failed with the following error: :
keycloak | WFLYCTL0062: Composite operation failed and was rolled back. Steps that failed:
keycloak | Step: step-9
keycloak | Operation: /subsystem=datasources/jdbc-driver=postgresql:add(driver-name=postgresql, driver-module-name=org.postgresql.jdbc, driver-xa-datasource-class-name=org.postgresql.xa.PGXADataSource)
keycloak | Failure: WFLYCTL0212: Duplicate resource [
keycloak | ("subsystem" => "datasources"),
keycloak | ("jdbc-driver" => "postgresql")
keycloak | ]
...
The batch failed with the following error: :
keycloak | WFLYCTL0062: Composite operation failed and was rolled back. Steps that failed:
keycloak | Step: step-9
keycloak | Operation: /subsystem=datasources/jdbc-driver=postgresql:add(driver-name=postgresql, driver-module-name=org.postgresql.jdbc, driver-xa-datasource-class-name=org.postgresql.xa.PGXADataSource)
keycloak | Failure: WFLYCTL0212: Duplicate resource [
keycloak | ("subsystem" => "datasources"),
keycloak | ("jdbc-driver" => "postgresql")
keycloak | ]
The docker-compose.yml entry for Keycloak looks as follows, important data obviously removed
keycloak:
image: jboss/keycloak:8.0.1
container_name: keycloak
environment:
- PROXY_ADDRESS_FORWARDING=true
- DB_VENDOR=postgres
- DB_ADDR=db
- DB_DATABASE=keycloak
- DB_USER=keycloak
- DB_PASSWORD=<password>
- VIRTUAL_HOST=<url>
- VIRTUAL_PORT=8080
- LETSENCRYPT_HOST=<url>
volumes:
- /opt/docker/keycloak-startup:/opt/jboss/startup-scripts
The volume I'm mapping is there to make some changes to WildFly to make sure it behaves well with the reverse proxy:
embed-server --std-out=echo
# Enable https listener for the new security realm
/subsystem=undertow/ \
server=default-server/ \
http-listener=default \
:write-attribute(name=proxy-address-forwarding, \
value=true)
# Create new socket binding with proxy https port
/socket-binding-group=standard-sockets/ \
socket-binding=proxy-https \
:add(port=443)
# Enable https listener for the new security realm
/subsystem=undertow/ \
server=default-server/ \
http-listener=default \
:write-attribute(name=redirect-socket, \
value="proxy-https")
After stopping the container, its not starting anymore with the messages shown above. Removing the container and re-creating it works fine however. I tried to remove the volume after the initial start, this doesn't really make a difference either. I already learned that I have to remove the KEYCLOAK_USER=admin and KEYCLOAK_PASSWORD environment variables after the initial boot as otherwise the container complains that the user already exists and doesn't start anymore. Any idea how to fix that?

Answer 1

Update on 23rd of May 2021:
The issue has been resolved on RedHats Jira, it seems to be resolved in version 12. The related GitHub pull request can be found here: https://github.com/keycloak/keycloak-containers/pull/286
According to RedHat support, this is a known "issue" and not supposed to be fixed. They want to concentrate on a workflow where a container is removed and recreated, not started and stopped. They agreed with the general problem, but stated that currently there are no resources available. Stopping and starting the container is a operation which is currently not supported.
See for example https://issues.redhat.com/browse/KEYCLOAK-13094?jql=project%20%3D%20KEYCLOAK%20AND%20text%20~%20%22docker%20restart%22 for reference

Answer 2

A legitimate use case for restarting is to add debug logging. For example to debug authentication with an external identity provider.
I ended up creating a shell script that does:
docker stop [container]
docker rm [container]
recreates the image i want with changes to the logging configuration
docker run [options] [container]
However a nice feature of docker is the ability to restart a stopped container automatically, decreasing downtime. This Keycloak bug takes that feature away.

Answer 3

I had the same problem here, and my solution was:
Export docker container to a .tar file:
docker export CONTAINER_NAME > latest.tar
2- Create a new volume in a docker
docker volume create VOLUME_NAME
3 - Start a new docker container mapping the volume created to a container db path, something like this:
docker run --name keycloak2 -v keycloak_db:/opt/jboss/keycloak/standalone/data/ -p 8080:8080 -e PROXY_ADDRESS_FORWARDING=true -e KEYCLOAK_USER=admin -e KEYCLOAK_PASSWORD=root jboss/keycloak
4 - Stop the container
5 - Unpack the tar file and find the database path, something like this:
tar unpack path: /opt/jboss/keycloak/standalone/data
6 - Move the path content to docker volume, if you dont know where is the physical path use docker inspect volume VOLUME_NAME to find the path
7 - Start the stoped container
This works for me, I hope its so helpfull to the next person to fix this problem.