I want to make my nifi data volume and configuration persist means even if I delete container and docker compose up again I would like to keep what I built so far in my nifi. I try to mount volumes as follows in my docker compose file in volumes section nevertheless it doesn't work and my nifi processors are not saved. How can I do it correctly? Below my docker-compose.yaml file.
version: "3.7"
services:
nifi:
image: koroslak/nifi:latest
container_name: nifi
restart: always
environment:
- NIFI_HOME=/opt/nifi/nifi-current
- NIFI_LOG_DIR=/opt/nifi/nifi-current/logs
- NIFI_PID_DIR=/opt/nifi/nifi-current/run
- NIFI_BASE_DIR=/opt/nifi
- NIFI_WEB_HTTP_PORT=8080
ports:
- 9000:8080
depends_on:
- openldap
volumes:
- ./volume/nifi-current/state:/opt/nifi/nifi-current/state
- ./volume/database/database_repository:/opt/nifi/nifi-current/repositories/database_repository
- ./volume/flow_storage/flowfile_repository:/opt/nifi/nifi-current/repositories/flowfile_repository
- ./volume/nifi-current/content_repository:/opt/nifi/nifi-current/repositories/content_repository
- ./volume/nifi-current/provenance_repository:/opt/nifi/nifi-current/repositories/provenance_repository
- ./volume/log:/opt/nifi/nifi-current/logs
#- ./volume/conf:/opt/nifi/nifi-current/conf
postgres:
image: koroslak/postgres:latest
container_name: postgres
restart: always
environment:
- POSTGRES_PASSWORD=secret123
ports:
- 6000:5432
volumes:
- postgres:/var/lib/postgresql/data
pgadmin:
container_name: pgadmin
image: dpage/pgadmin4:4.18
restart: always
environment:
- PGADMIN_DEFAULT_EMAIL=admin
- PGADMIN_DEFAULT_PASSWORD=admin
ports:
- 8090:80
metabase:
container_name: metabase
image: metabase/metabase:v0.34.2
restart: always
environment:
MB_DB_TYPE: postgres
MB_DB_DBNAME: metabase
MB_DB_PORT: 5432
MB_DB_USER: metabase_admin
MB_DB_PASS: secret123
MB_DB_HOST: postgres
ports:
- 3000:3000
depends_on:
- postgres
openldap:
image: osixia/openldap:1.3.0
container_name: openldap
restart: always
ports:
- 38999:389
# Mocked source systems
jira-api:
image: danielgtaylor/apisprout:latest
container_name: jira-api
restart: always
ports:
- 8000:8000
command: https://raw.githubusercontent.com/mvrabel/nifi-postgres-metabase/master/api_examples/jira-api.json
pipedrive-api:
image: danielgtaylor/apisprout:latest
container_name: pipedrive-api
restart: always
ports:
- 8100:8000
command: https://raw.githubusercontent.com/mvrabel/nifi-postgres-metabase/master/api_examples/pipedrive-api.yaml
restcountries-api:
image: danielgtaylor/apisprout:latest
container_name: restcountries-api
restart: always
ports:
- 8200:8000
command: https://raw.githubusercontent.com/mvrabel/nifi-postgres-metabase/master/api_examples/restcountries-api.json
volumes:
postgres:
nifi:
openldap:
metabase:
pgadmin:
Using Registry you can achieve that all changes you are doing or your nifi are committed to git. I.e. if you change some processor configuration, it will be reflected in your git repo.
As for flow files, you may need to fix volumes mappings.
Related
I'm having a problem persisting data with docker-compose.
I want my service chatmysql to persist data I put inside a database, but everytime i run docker-compose down it all vanishes.
I checked directory /var/lib/docker/volumes to see if it stores data there when containers are running and the volume was completely empty.
I didn't have that issue when I was running containers with docker run command so I guess its fault of my docker-compose.yaml file. Can someone help me?
I'm running this on Ubuntu 20.04.
version: '3'
services:
chatmysql:
image: mysql/mysql-server
container_name: chatmysql
hostname: db
user: root
networks:
- chatnet
ports:
- 3307:3306
volumes:
- chatmysqlvolume:/lib/var/mysql
chatbackend:
depends_on:
- chatmysql
build:
context: backend/src
container_name: chatbackend
hostname: backend
networks:
- chatnet
ports:
- 8080:8080
environment:
- MYSQLUSERNAME=${MYSQLUSERNAME:-user}
- MYSQLPASSWORD=${MYSQLPASSWORD:?database password not set}
- MYSQLHOST=${MYSQLHOST:-db}
- MYSQLPORT=${MYSQLPORT:-3306}
- MYSQLDBNAME=${MYSQLDBNAME:-test}
restart: always
deploy:
restart_policy:
condition: on-failure
chatfrontend:
build: frontend
container_name: chatfrontend
hostname: front
networks:
- chatnet
ports:
- 3000:3000
volumes:
chatmysqlvolume:
networks:
chatnet:
driver: bridge
You need to change the mounted volume, try this :
version: '3.7'
services:
chatmysql:
image: mysql/mysql-server
container_name: chatmysql
hostname: db
user: root
networks:
- chatnet
ports:
- 3307:3306
volumes:
- chatmysqlvolume:/var/lib/mysql
chatbackend:
depends_on:
- chatmysql
build:
context: backend/src
container_name: chatbackend
hostname: backend
networks:
- chatnet
ports:
- 8080:8080
environment:
- MYSQLUSERNAME=${MYSQLUSERNAME:-user}
- MYSQLPASSWORD=${MYSQLPASSWORD:?database password not set}
- MYSQLHOST=${MYSQLHOST:-db}
- MYSQLPORT=${MYSQLPORT:-3306}
- MYSQLDBNAME=${MYSQLDBNAME:-test}
restart: always
deploy:
restart_policy:
condition: on-failure
chatfrontend:
build: frontend
container_name: chatfrontend
hostname: front
networks:
- chatnet
ports:
- 3000:3000
volumes:
chatmysqlvolume:
networks:
chatnet:
driver: bridge
I am using Docker 20.10.7 on Mac OS and docker-compose to run multiple docker containers.
When I start it for the first time, all the docker images are properly labeled and appear as the following.
However, after subsequent runs (docker-compose up, docker-compose down), suddenly all the image names are changed to sha256 and start to look like this
Please advise how to avoid this behavior. Thank you.
UPDATE #1
This is the docker-compose file I use to start containers.
Initially the old displayed with properly labeled image names.
However, not even if I run a docker system prune command it continues to label them as sha256:...
version: '3.8'
services:
influxdb:
image: influxdb:1.8
container_name: influxdb
ports:
- "8083:8083"
- "8086:8086"
- "8090:8090"
- "2003:2003"
env_file:
- 'env.influxdb.properties'
volumes:
- /Users/user1/Docker/influxdb/data:/var/lib/influxdb
restart: unless-stopped
telegraf:
image: telegraf:latest
container_name: telegraf
links:
- db
volumes:
- ./telegraf.conf:/etc/telegraf/telegraf.conf:ro
- /var/run/docker.sock:/var/run/docker.sock
restart: unless-stopped
grafana:
image: grafana/grafana:latest
container_name: grafana
ports:
- "3000:3000"
env_file:
- 'env.grafana.properties'
links:
- influxdb
volumes:
- /Users/user1/Docker/grafana/data:/var/lib/grafana
restart: unless-stopped
db:
image: mysql
container_name: db-container
command: --default-authentication-plugin=mysql_native_password
ports:
- '3306:3306'
environment:
MYSQL_ROOT_PASSWORD: P#ssw0rd
MYSQL_USER: root
MYSQL_PASSWORD: P#ssw0rd
MYSQL_DATABASE: db1
volumes:
- /Users/user1/Docker/mysql/data:/var/lib/mysql
- "../sql/schema.sql:/docker-entrypoint-initdb.d/1.sql"
healthcheck:
test: "/usr/bin/mysql --user=root --password=P#ssw0rd --execute \"SHOW DATABASES;\""
interval: 2s
timeout: 20s
retries: 10
restart: always
adminer:
image: adminer
container_name: adminer
restart: always
ports:
- 8081:8080
redis:
image: bitnami/redis
container_name: redis
environment:
- ALLOW_EMPTY_PASSWORD=yes
#- REDIS_DISABLE_COMMANDS=FLUSHDB,FLUSHALL
ports:
- '6379:6379'
volumes:
- '/Users/user1/Docker/redis/data:/bitnami/redis/data'
- ./redis.conf:/opt/bitnami/redis/mounted-etc/redis.conf
I was trying to create a PHP development environment with PHP, MariaDB, and a tutorial suggested to use Adminer for database management. So I generate my docker-compose.yml file like this:
version : '3.1'
services:
php:
build:
context: .
dockerfile: Dockerfile
ports:
- 80:80
volumes:
- ./src:/var/www/html/
db:
image: mariadb
restart: always
environment:
MYSQL_ROOT_PASSWORD: example
volumes:
- ./mariadb-data:/var/lib/mysql
adminer:
image: adminer
environment:
ADMINER_DEFAULT_SERVER: db
restart: always
ports:
- 8080:8080
But when I set the volumes for MariaDB, I got an error in the Adminer login page. When I don't set them it seems to work well.
version : '3.1'
services:
php:
build:
context: .
dockerfile: Dockerfile
ports:
- 80:80
volumes:
- ./src:/var/www/html/
db:
image: mariadb
restart: always
environment:
MYSQL_ROOT_PASSWORD: example
volumes:
- ./mariadb-data:/var/lib/mysql
adminer:
image: adminer
environment:
ADMINER_DEFAULT_SERVER: db
restart: always
ports:
- 8080:8080
links:
- php
- db
I am trying to run a WordPress site inside of a docker container on Ubuntu VPS using Nginx-Proxy.
I created the following docker-compose.yml file
version: '3.4'
services:
nginx-proxy:
image: jwilder/nginx-proxy
container_name: nginx-proxy
ports:
- 80:80
- 443:443
restart: always
networks:
- nginx-proxy
volumes:
- /var/run/docker.sock:/tmp/docker.sock:ro
- /etc/nginx/vhost.d:/etc/nginx/vhost.d:ro
- /etc/certificates:/etc/nginx/certs
wordpress:
image: wordpress
container_name: wordpress
restart: always
ports:
- 8080:80
environment:
- VIRTUAL_HOST=wordpress.domain.com
- VIRTUAL_PORT=5500
- WORDPRESS_DB_HOST=db
- WORDPRESS_DB_USER=db_username
- WORDPRESS_DB_PASSWORD=db_password
- WORDPRESS_DB_NAME=db_name
depends_on:
- nginx-proxy
- db
networks:
- nginx-proxy
volumes:
- wordpress:/var/www/html
ports:
- 5500:5500
expose:
- 5500
db:
image: mysql:latest
container_name: db
restart: always
environment:
MYSQL_DATABASE: db_name
MYSQL_USER: db_username
MYSQL_PASSWORD: db_password
MySQL_RANDOM_ROOT_PASSWORD: '1'
depends_on:
- nginx-proxy
networks:
- nginx-proxy
volumes:
- db:/var/lib/mysql
ports:
- 5600:5600
expose:
- 5600
volumes:
wordpress:
db:
Every time I run docker-compose up I get the following error
Service "nginx-proxy" uses an undefined network "nginx-proxy"
I created a network using the following command
docker network create nginx-proxy
Here is the output of docker network ls
Why do I get that error? How can I fix it?
Anything you name in a per-service networks: block needs to be declared in a top-level networks: block.
version: '3.4'
services:
nginx-proxy:
networks:
- nginx-proxy # <-- matches below
volumes: { ... }
networks:
nginx-proxy: # <-- matches above
# may be empty, but this block is required
If you don't declare any networks: at all, Compose creates a network named default and attaches containers to it. For almost all uses this is what you need. So it may be simpler to just delete the networks: blocks entirely.
version: '3.4'
services:
nginx-proxy:
image: jwilder/nginx-proxy
# No networks:; just use automatic [default]
(You similarly do not need to manually provide a container_name:, or to expose: ports at the Compose level.)
I'm trying to setup a docker to run mysql Mosquitto and node red but keep getting the unsupported config option errors..
Services:
mysql:
image: mysql
container_name: mysql
restart: always
ports:
- “6603:3306”
Environment:
MYSQL_ROOT_PASSWORD: “abcd1234”
volumes:
- mysql-data
node-red:
image: nodered/node-red:latest
restart: always
container_name: nodered
environment:
-TZ=Europe/London
depends_on:
- mysql
ports:
- “1880:1880”
links:
- mysql:mysql
- mosquitto:mosquitto
volumes:
- node-red-data
mosquitto:
image: eclipse-mosquitto
hostname: mosquitto
container_name: mosquitto
restart: always
ports:
- "1883:1883"
volumes:
mysql-data:
node-red-data:
Any thoughts on why im getting these errors?
Unsupported config option for Services: 'mosquitto'
Unsupported config option for volumes: 'mysql-data'