Confluence on Docker runs setup assistent on existing installation after update - docker

A few days ago, my watchtower updated Confluence on Docker with the 6.15.1-alpine tag. It's hosted using Atlassians official image. Since those update, Confluence shows the setup screen. Haven't any chance to get inside the admin panel. When continue the wizard end entering server credentials of the existing installation, it gave an error that an installation already exists that would be overwritten if continued.
It was a re-push of the exact version tag 6.15.1 tag, not a regular version update. So there seems no possibility to use the old, working image. Also other versions seems re-pushed. Tried some older ones and also a new one, without success.
docker-compose.yml
version: "2"
volumes:
confluence-home:
services:
confluence:
container_name: confluence
image: atlassian/confluence-server:6.15.1-alpine
#restart: always
mem_limit: 6g
volumes:
- confluence-home:/var/atlassian/application-data/confluence
- ./confluence.cfg.xml:/var/atlassian/application-data/confluence/confluence.cfg.xml
- ./server.xml:/opt/atlassian/confluence/conf/server.xml
- ./mysql-connector-java-5.1.42-bin.jar:/opt/atlassian/confluence/lib/mysql-connector-java-5.1.42-bin.jar
networks:
- traefik
environment:
- "TZ=Europe/Berlin"
- JVM_MINIMUM_MEMORY=4096m
- JVM_MAXIMUM_MEMORY=4096m
labels:
- "traefik.port=8090"
- "traefik.backend=confluence"
- "traefik.frontend.rule=Host:confluence.my-domain.com"
networks:
traefik:
external: true

I found out that there were the following changes on the images:
Ownership
The logs throwed errors about not beinng able to write on log files because nearly the entire home directory was owned by an user called bin:
root#8ac38faa94f1:/var/atlassian/application-data/confluence# ls -l
total 108
drwx------ 2 bin bin 4096 Aug 19 00:03 analytics-logs
drwx------ 3 bin bin 4096 Jun 15 2017 attachments
drwx------ 2 bin bin 24576 Jan 12 2019 backups
[...]
This could be fixed by executing a chown:
docker exec -it confluence bash
chown confluence:confluence -R /var/atlassian/application-data/confluence
Moutings inside mount
My docker-compose.yml mounts a volume to /var/atlassian/application-data/confluence and inside those volume, the confluence.cfg.xml file was mounted from current directory. This approach is a bit older and should seperate the user data in the volume from configuration files like docker-compose.yml and also the application itself as confluence.cfg.xml.
Seems not properly working any more on using Docker 17.05 and Docker-Compose 1.8.0 (at least in combination with Confluence), so I simply removed that second mount and placed the configuration file inside the volume.
Atlassian creates config files now dynamically
It was noticeable that my mounted configuration files like confluence.cfg.xml and server.xml were overwritten by Atlassians container. Their source code shows that they now use Jina2, a common Python template engine used in e.g. Ansible. A python script parse those files on startup and create Confluences configuration files, without properly checking on all of those files if they already exists.
Mounting them as read only caused the app to crash because this is also not handled in their Python script. By analyzing their templates, I learned that they replaced nearly every config item by environment variables. Not a bad approach, so I specified my server.xml parameters by env variables instead of mouting the entire file.
In my case, Confluence is behind a Traefik reverse proxy and it's required to tell Confluence it's final application url for end users:
environment:
- ATL_proxyName=confluence.my-domain.com
- ATL_proxyPort=443
- ATL_tomcat_scheme=https
Final working docker-compose.yml
By applying all modifications above, accessing the existing installation works again using the following docker-compose.yml file:
version: "2"
volumes:
confluence-home:
services:
confluence:
container_name: confluence
image: atlassian/confluence-server:6.15.1
#restart: always
mem_limit: 6g
volumes:
- confluence-home:/var/atlassian/application-data/confluence
- ./mysql-connector-java-5.1.42-bin.jar:/opt/atlassian/confluence/lib/mysql-connector-java-5.1.42-bin.jar
networks:
- traefik
environment:
- "TZ=Europe/Berlin"
- JVM_MINIMUM_MEMORY=4096m
- JVM_MAXIMUM_MEMORY=4096m
- ATL_proxyName=confluence.my-domain.com
- ATL_proxyPort=443
- ATL_tomcat_scheme=https
labels:
- "traefik.port=8090"
- "traefik.backend=confluence"
- "traefik.frontend.rule=Host:confluence.my-domain.com"
networks:
traefik:
external: true

Related

ERROR Disk error while locking directory /var/kafka-logs in 3.10 Kafka

I am using Kafka 3.1.0, Portainer 2.9.0 and docker 20.10.11 to build a 1 broker, 1 consumer and 1 producer cluster.
I am trying to map the log dirs via the docker-compose from the container to the host machine in order to persist the content of that directory (because if the container falls that information will be lost). I know it is recommended to have more than 1 broker, but since I am just testing this feature, I don't want to overcomplicate myself.
The problem I get is
ERROR Disk error while locking directory /var/kafka-logs (kafka.server.LogDirFailureChannel)
java.nio.file.AccessDeniedException: /var/kafka-logs/.lock
[2022-03-31 12:00:53,986] ERROR [KafkaServer id=1] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
I have checked and the user that executes the broker has all permissions (since I created that directory with my Dockerfile).
RUN mkdir /var/kafka-logs \
&& chown -R kafka:kafka /var/kafka-logs \
&& chmod -R 777 /var/kafka-logs
I have seen that this problem was a thing in the 3.0 version and was fixed in the 3.1, and also that it only happened in Windows, so I don't know the source of this problem.
Edit: I have checked and even without the mapping it still prints that error. It must be a problem of changing the log.dirs property to a non /tmp directory, because if I leave the default configuration it works just fine.
By default I mean the following:
log.dirs=/tmp/kafka-logs
My docker-compose:
version: "3.8"
networks:
net:
external: true
services:
kafka-broker1:
image: registry.gitlab.com/repo/kafka:2.13_3.1.0_v0.1
volumes:
- /var/volumes/kafka/config/server1.properties:/opt/kafka/config/server.properties
networks:
- net
kafka-producer:
image: registry.gitlab.com/repo/kafka:2.13_3.1.0_v0.1
stdin_open: true
tty: true
networks:
- net
kafka-consumer:
image: registry.gitlab.com/repo/kafka:2.13_3.1.0_v0.1
stdin_open: true
tty: true
networks:
- net
The problem was that I have been creating a few docker images and the container with the same name and it didn't picked the newest image.
Once I erased the rest of images and the container picked the lastest it all worked just fine, so it was basically a problem of not having enough permissions to get the lock of that directory.

Apache Nifi (on docker): only one of the HTTP and HTTPS connectors can be configured at one time error

Have a problem adding authentication due to a new needs while using Apache NiFi (NiFi) without SSL processing it in a container.
The image version is apache/nifi:1.13.0
It's said that SSL is unconditionally required to add authentication. It's recommended to use tls-toolkit in the NiFi image to add SSL. Worked on the following process:
Except for environment variable nifi.web.http.port for HTTP communication, and executed up the standalone mode container with nifi.web.https.port=9443
docker-compose up
Joined to the container and run the tls-toolkit script in the nifi-toolkit.
cd /opt/nifi/nifi-toolkit-1.13.0/bin &&\
sh tls-toolkit.sh standalone \
-n 'localhost' \
-C 'CN=yangeok,OU=nifi' \
-O -o $NIFI_HOME/conf
Attempt 1
Organized files in directory $NIFI_HOME/conf. Three files keystore.jks, truststore.jsk, and nifi.properties were created in folder localhost that entered the value of the option -n of the tls-toolkit script.
cd $NIFI_HOME/conf &&
cp localhost/*.jks .
The file $NIFI_HOME/conf/localhost/nifi.properties was not overwritten as it is, but only the following properties were imported as a file $NIFI_HOME/conf/nifi.properties:
nifi.web.http.host=
nifi.web.http.port=
nifiweb.https.host=localhost
nifiweb.https.port=9443
Restarted container
docker-compose restart
The container died with below error log:
Only one of the HTTP and HTTPS connectors can be configured at one time
Attempt 2
After executing the tls-toolkit script, all files a were overwritten, including file nifi.properties
cd $NIFI_HOME/conf &&
cp localhost/* .
Restarted container
docker-compose restart
The container died with the same error log
Hint
The dead container volume was also accessible, so copied and checked file nifi.properties, and when did docker-compose up or restart, it changed as follows:
The part I overwritten or modified:
nifi.web.http.host=
nifi.web.http.port=
nifi.web.http.network.interface.default=
#############################################
nifi.web.https.host=localhost
nifi.web.https.port=9443
The changed part after re-executing the container:
nifi.web.http.host=a8e283ab9421
nifi.web.http.port=9443
nifi.web.http.network.interface.default=
#############################################
nifi.web.https.host=a8e283ab9421
nifi.web.https.port=9443
I'd like to know how to execute the container with http.host, http.port empty. docker-compose.yml file is as follows:
version: '3'
services:
nifi:
build:
context: .
args:
NIFI_VERSION: ${NIFI_VERSION}
container_name: nifi
user: root
restart: unless-stopped
network_mode: bridge
ports:
- ${NIFI_HTTP_PORT}:8080/tcp
- ${NIFI_HTTPS_PORT}:9443/tcp
volumes:
- ./drivers:/opt/nifi/nifi-current/drivers
- ./templates:/opt/nifi/nifi-current/templates
- ./data:/opt/nifi/nifi-current/data
environment:
TZ: 'Asia/Seoul'
########## JVM ##########
NIFI_JVM_HEAP_INIT: ${NIFI_HEAP_INIT} # The initial JVM heap size.
NIFI_JVM_HEAP_MAX: ${NIFI_HEAP_MAX} # The maximum JVM heap size.
########## Web ##########
# NIFI_WEB_HTTP_HOST: ${NIFI_HTTP_HOST} # nifi.web.http.host
# NIFI_WEB_HTTP_PORT: ${NIFI_HTTP_PORT} # nifi.web.http.port
NIFI_WEB_HTTPS_HOST: ${NIFI_HTTPS_HOST} # nifi.web.https.host
NIFI_WEB_HTTP_PORT: ${NIFI_HTTPS_PORT} # nifi.web.https.port
Thank you

How to restore redis dump from version 3.2 (without docker) to 6.0 (running with bitnami docker image)

Initial goal
Currently we have an old version of redis running on a single physical server(redis v3.2.1). The goal is to have a redis/slave topology with sentinel for HA.
A way to accomplish this, is to create a slave of this single instance with version 6. Unfortunately, there is an icompatibility between version 3 and 6.
What I try to do
I tried to restore the dump to version 4.0.2-r0. It works fine.
Then I tried to restore to version 4.0.2-r1 and it failed and the
dump.rdb was totally replaced by an empty file automatically
How to reproduce
Install a single instance of redis with :
apt install -y redis-server redis-tools
Create a redis container with this docker-compose.yml file
version: '2'
services:
redis-master:
#image: 'bitnami/redis:6.0'
#image: 'bitnami/redis:4.0.2-r1'
image: 'bitnami/redis:4.0.2-r0'
container_name: redis-master
volumes:
- 'redis_data_master:/bitnami/redis/data'
ports:
- '6379:6379'
environment:
- REDIS_REPLICATION_MODE=master
- ALLOW_EMPTY_PASSWORD=yes
- REDIS_DISABLE_COMMANDS=FLUSHDB,FLUSHALL
volumes:
redis_data_master:
driver: local
docker-compose up -d
docker stop redis-master
# Replace test_redis_data_master by the right folder
sudo cp dump.rdb /var/lib/docker/volumes/test_redis_data_master/_data/dump.rdb
With this docker compose file, it work with v4.0.2-r0 but not with v4.0.2-r1.
cf logs here:
v4.0.2-r0
30:M 01 Dec 13:52:55.993 # Server initialized 30:M 01 Dec
13:52:58.049 * DB loaded from disk: 2.056 seconds 30:M 01 Dec
13:52:58.049 * Ready to accept connections
The database is loaded successfully
v4.0.2-r1
20:M 01 Dec 13:49:05.472 # Server initialized 20:M 01 Dec
13:49:05.472 * Ready to accept connections
The database is empty
Conclusion
Can you please help with this issue. Did someone faced the same issue or the same situation ?
Thanks in advance for your help :D
Have you tried to upgrade directly from 4.0 to 5.0?
I don't think upgrade revision by revision is a good idea since we release a new revision every day, and sometimes, they have bugs that are patched with another revision.
Apart from that, I recommend you to follow the official documentation to upgrade your databases. A major version usually means incompatible changes, so I am pretty sure you will find some issues in the way. It is highly probable that just changing the image tag will not work.

Graylog in Docker persistent

I'm trying to make a Graylog Docker Container persistent.
Meaning that after restarting (docker-compose down; docker-compose up) the logs will still be there alongside the configuration.
I've used the documentation at https://docs.graylog.org/en/3.1/pages/installation/docker.html I created a yml file with the content under the topic "Persisting data".
I only edited the line "GRAYLOG_HTTP_EXTERNAL_URI=http://127.0.0.1:9000/" to not use localhost but the external ip the machine is using.
Docker works, i can create an input and collect logfiles. What does not work is the data being persistent. Also every time i restart the node id changes, so i have to reconfigure the input. Running docker volume ls lists five volumes 3 of which are the ones created in the yml file.
I don't understand why data is not persistent. Can anybody help?
I had the same problem and I'd been struggling for a while before I found a solution. I'm on 3.2 and also had issues with node persistence. The documentation doesn't seem to directly state that there is one more configuration folder you need to persist, which is:
/usr/share/graylog/data/config
They actually mention it in the Custom configuration files section and when I took a look via CLI in that directory, it turns out that it's where the graylog.conf and node-id (the file Graylog uses to store information about its nodes) are stored as well!
Here's my docker-compose.override.yml section with the necessary changes (marked with '# ADDED' comments)
services:
graylog:
environment:
# CHANGE ME (must be at least 16 characters)!
- GRAYLOG_PASSWORD_SECRET=somepasswordpepper
# Password: admin
- GRAYLOG_ROOT_PASSWORD_SHA2=8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
- GRAYLOG_HTTP_EXTERNAL_URI=http://127.0.0.1:9000/
- GRAYLOG_IS_MASTER=true
#- GRAYLOG_NODE_ID_FILE=/usr/share/graylog/data/config/node-id
ports:
# Graylog web interface and REST API
- 9000:9000
# Syslog TCP
- 1514:1514
# Syslog UDP
- 1514:1514/udp
# GELF TCP
- 12201:12201
# GELF UDP
- 12201:12201/udp
volumes:
- "graylogjournal:/usr/share/graylog/data/journal"
- "graylogconfig:/usr/share/graylog/data/config" # ADDED
volumes:
graylogjournal:
driver: local
graylogconfig: # ADDED
driver: local # ADDED
Hope this helps
You can add into daemon.json file these lines ;
{
"log-driver": "gelf",
"log-opts": {
"gelf-address": "udp://1.2.3.4:12201"
}
}
https://docs.docker.com/config/containers/logging/gelf/

How do you mount the docker socket on Windows?

I'm trying to make an application work on Windows that's been developed only on Unices. It's all dockerized and it uses the traefik load balancer. The volumes for the docker for running traefik looks like this:
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro,delegated
- ${PWD}/load_balancer/traefik.toml:/etc/traefik/traefik.toml:ro,delegated
The first volume works fine on Mac or Linux, but does it on Windows? The application is failing (the load balancer is giving a 404) and it might be related to that volume. When I start the image, the socket looks like a socket:
/ # ls -laF /var/run/docker.sock
srw-rw---- 1 root root 0 Sep 2 11:04 /var/run/docker.sock=
Is this working? Any way to test it? What's the correct way of doing this?
Trying to figure this out, I tried replacing it with this:
volumes:
- //./pipe/docker_engine:/var/run/docker.sock
based on various articles and bug reports I found online. The docker image starts but it fails in the same way and now in the docker container it looks like a directory:
/ # ls -laF /var/run/docker.sock
total 4
drwxr-xr-x 2 root root 40 Sep 3 14:52 ./
drwxr-xr-x 1 root root 4096 Sep 3 14:57 ../
Following Marc ABOUCHACRA's answers, I tried:
volumes:
- type: npipe
source: ////./pipe/docker_engine
target: /var/run/docker.sock
consistency: delegated
but that also looks like a directory:
/ # ls -laF /var/run/docker.sock
total 4
drwxr-xr-x 2 root root 40 Sep 3 14:52 ./
drwxr-xr-x 1 root root 4096 Sep 3 14:57 ../
I also tried this:
volumes:
- npipe:////./pipe/docker_engine:/var/run/docker.sock:ro,delegated
but that fails with this error:
ERROR: Volume npipe:////./pipe/docker_engine:/var/run/docker.sock:ro,delegated has incorrect format, should be external:internal[:mode]
The whole docker-compose.yml section looks like this:
lb:
image: load-balancer
build: ${WORKSPACE}/go-home/load_balancer
ports:
- 80:80
- 443:443
links:
- wifi-ui-dev
- wifi-ui-prod
- portal
- wifi-api
env_file:
- .env
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro,delegated
- ${PWD}/load_balancer/traefik.toml:/etc/traefik/traefik.toml:ro,delegated
My question is specifically about running this docker image, which is a Linux, on a Windows host, running Docker for Windows. I understand that I can run it on a Linux host by installing Linux on another machine or a VM on the Windows machine, it's equivalent. Running Windows guests is not what I'm after either in case there's a way of exposing sockets from Windows to Windows only.
If you cannot nor want use network sockets, then you can use named pipes.
The syntax depends whether you run Linux or Windows containers and on the shell you use.
Linux containers
If you run Linux containers on a Windows machine, this seems to work using Powershell or bash:
docker run --rm -it -v "//var/run/docker.sock://var/run/docker.sock" image_with_docker docker version
Please note the extra / in front of /var/run/docker.sock, both for the source and destination volumes.
Windows containers
If you run Windows containers on a Windows machine, this seems to work using Powershell or bash:
docker run -v "//./pipe/docker_engine://./pipe/docker_engine" --rm -it image-with-docker docker version
Note that this works only in Powershell:
docker run -v "\\.\pipe\docker_engine:\\.\pipe\docker_engine" --rm -it image-with-docker docker version
Therefore, it's better to use the version with /.
Extra - docker-compose.yml
If you use a docker-compose.yaml file, this works with Windows containers.
version: '3.7'
services:
docker:
image: image-with-docker
command:
- docker
- version
volumes:
- type: npipe
source: \\.\pipe\docker_engine
target: \\.\pipe\docker_engine
With Linux containers, you can use the shortened form:
docker:
image: image-with-docker
command:
- docker
- version
volumes:
- //var/run/docker.sock://var/run/docker.sock
Extra - Kubernetes
If you are running Windows containers on a Windows node in Kubernetes, this seems to work:
apiVersion: v1
kind: Pod
spec:
containers:
- name: docker
image: image-with-docker
command:
- powershell
args:
- Start-Sleep
- "999999"
volumeMounts:
- mountPath: \\.\pipe\docker_engine
name: dockersock
volumes:
- name: dockersock
hostPath:
path: \\.\pipe\docker_engine
type: null
nodeSelector:
kubernetes.io/os: windows
In this case, beside using the \, please note the type: null in the definition of the dockersock volume: if you don't set it, it will not work.
Notes
Everything was tested on docker 19.03 and on Kubernetes 1.18.
Client:
Version: 19.03.3
API version: 1.40
Go version: go1.12.10
Git commit: 2355349d-
Built: 10/14/2019 16:41:26
OS/Arch: windows/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.8
API version: 1.40 (minimum version 1.24)
Go version: go1.12.17
Git commit: afacb8b
Built: Wed Mar 11 01:37:20 2020
OS/Arch: windows/amd64
Experimental: false
If you encounter the following error on windows:
cannot create container for service portainer: Unrecognised volume spec: file '\.\pipe\docker_engine' cannot be mapped. Only directories can be mapped on this platform
ERROR: Encountered errors while bringing up the project.
Try adding an extra slash to it, resulting in following volumes section:
volumes:
- source: \\.\pipe\docker_engine\
target: \\.\pipe\docker_engine\
type: npipe
Tested with compose 3.7 and docker CE 19.03.12
Using short syntax with the type of the bind mount is not possible : npipe:////./pipe/docker_engine:/var/run/docker.sock:ro,delegated
You need to use the long syntax in your compose file :
volumes:
- type: npipe
source: ////./pipe/docker_engine
target: /var/run/docker.sock
consistency: delegated
You can find some documentation about the long syntax in the official documentation. This syntaxe is from v3.2
Also keep in mind what #lucas-ramage said about using windows container only when using npipe.
In 2022, on Windows 11, what worked for me is below volume configuration (Note the doube slash (//) on the host side of configruation. Not sure why docker_enginer variant does not work for me.
-v "//var/run/docker.sock:/var/run/docker.sock"
Per the Docker for Windows FAQ,
On Docker Desktop for Windows, clients can connect to the Docker Engine through a named pipe: npipe:////./pipe/docker_engine
See also this issue on GitHub,
The windows version of docker doesn't use unix socket (/var/run/docker.sock) but npipe (npipe:////./pipe/docker_engine). So you have either
to switch to linux container which runs docker in a full virtualized linux with unix socket
to pass the npipe instead of the unix socket to the container (windows container only)
to use a network socket (should work with linux and windows container)
However, since this is a Linux container, your options are either A) Run docker in a virtual machine (first choice above), or B) Use a network socket (the third choice).

Resources