I'm facing the same problem as here - I have set up a private Docker Registry with TLS certification (certificates generated via Certbot), and I can interact with it directly via curl etc. (thus proving that the certificate is correct), but the Docker Plugin in my Drone flow gives an error x509: certificate signed by unknown authority.
As per this StackOverflow answer, I believe that putting the certificate at /etc/docker/certs.d/<my_registry_address:port>/ca.crt should fix this problem, but it doesn't appear to (neither does adding the certificate into the standard /etc/ssl/certs/ca-certificates.crt location)
Demonstration that the certificates work as-expected, having already built the Docker Drone Plugin locally as per https://github.com/drone-plugins/drone-docker:
$ docker run --rm -v <path_to_directory_containing_pems>:/custom-certs -it --entrypoint /bin/sh plugins/docker
/ # ls /custom-certs
accounts archive csr keys live renewal renewal-hooks
/ # apk add curl
...
OK: 28 MiB in 56 packages
/ # curl https://docker-registry.scubbo.org:8843/v2/_catalog
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
/ # curl https://docker-registry.scubbo.org:8843/v2/_catalog --cacert /custom-certs/live/docker-registry.scubbo.org/fullchain.pem
{"repositories":[...]}
/ # cat /custom-certs/live/docker-registry.scubbo.org/fullchain.pem >> /etc/ssl/certs/ca-certificates.crt
/ # curl https://docker-registry.scubbo.org:8843/v2/_catalog
{"repositories":[...]}
Here's my .drone.yml, for a Runner instantiated with --env=DRONE_RUNNER_VOLUMES=/var/run/docker.sock:/var/run/docker.sock,<path_to_directory_containing_pems>:/custom-certs:
kind: pipeline
name: hello-world
type: docker
platform:
os: linux
arch: arm64
steps:
- name: copy-cert-into-place
image: busybox
volumes:
- name: docker-cert-persistence
path: /etc/docker/certs.d/
commands:
# https://stackoverflow.com/a/56410355/1040915
# Note that we need to mount the whole `custom-certs` directory into the workflow and then copy the file to `/etc/...`,
# rather than mounting the file directly into `/etc/...`, because the original file is a symlink and it's not possible (AFAIK)
# to instruct Docker to "mount the eventual-target-of this symlink into <location>"
- mkdir -p /etc/docker/certs.d/docker-registry.scubbo.org:8843
- cp -L /custom-certs/live/docker-registry.scubbo.org/fullchain.pem /etc/docker/certs.d/docker-registry.scubbo.org:8843/ca.crt
- name: check-cert-persists-between-stages
image: alpine
volumes:
- name: docker-cert-persistence
path: /etc/docker/certs.d/
commands:
- apk add curl
# The command below would fail if the cert was unavailable or invalid
- curl https://docker-registry.scubbo.org:8843/v2/_catalog --cacert /etc/docker/certs.d/docker-registry.scubbo.org:8843/ca.crt
- name: build-image
# ...contents irrelevant to this question...
- name: push-built-image
image: plugins/docker
volumes:
- name: docker-cert-persistence
path: /etc/docker/certs.d/
settings:
repo: docker-registry.scubbo.org:8843/scubbo/blog_nginx
tags: built_in_ci
debug: true
launch_debug: true
volumes:
- name: docker-cert-persistence
temp: {}
giving these logs from push-built-image step - ending in...
+ /usr/local/bin/docker tag 472d41d9c03ee60fe9c1965ad9cfd36a1cdb6cbf docker-registry.scubbo.org:8843/scubbo/blog_nginx:built_in_ci
+ /usr/local/bin/docker push docker-registry.scubbo.org:8843/scubbo/blog_nginx:built_in_ci
The push refers to repository [docker-registry.scubbo.org:8843/scubbo/blog_nginx]
Get "https://docker-registry.scubbo.org:8843/v2/": x509: certificate signed by unknown authority
exit status 1
How should I go about providing the CA Certificate to my Drone Docker Plugin step to permit it to communicate over TLS with a secure Docker registry? This answer suggests simply reverting to insecure integration, which works but is unsatisfactory.
EDIT: After re-reading this documentation, I extended the copy-cert-into-place commands to copy all 3 certificate-related files:
commands:
- mkdir -p /etc/docker/certs.d/docker-registry.scubbo.org:8843
- cp -L /custom-certs/live/docker-registry.scubbo.org/fullchain.pem /etc/docker/certs.d/docker-registry.scubbo.org:8843/ca.crt
- cp -L /custom-certs/live/docker-registry.scubbo.org/privkey.pem /etc/docker/certs.d/docker-registry.scubbo.org:8843/client.key
- cp -L /custom-certs/live/docker-registry.scubbo.org/cert.pem /etc/docker/certs.d/docker-registry.scubbo.org:8843/client.cert
but that did not resolve the problem - same x509: certificate signed by unknown authority error.
EDIT2: I directly confirmed (directly on a host, outside the context of a plugin or docker container) that adding the certificate to the path used above is sufficient to permit interaction with the registry:
$ docker pull docker-registry.scubbo.org:8843/scubbo/blog_nginx:built_in_ci
Error response from daemon: Get "https://docker-registry.scubbo.org:8843/v2/": x509: certificate signed by unknown authority
$ sudo cp -L <path_to_directory_containing_pems>/live/docker-registry.scubbo.org/chain.pem /etc/docker/certs.d/docker-registry.scubbo.org\:8843/ca.crt
$ docker pull docker-registry.scubbo.org:8843/scubbo/blog_nginx:built_in_ci
built_in_ci: Pulling from scubbo/blog_nginx
Digest: sha256:3a17f86f23050303d94443f24318b49fb1a5e2d0cc9228270678c8aa55b4d2c2
Status: Image is up to date for docker-registry.scubbo.org:8843/scubbo/blog_nginx:built_in_ci
docker-registry.scubbo.org:8843/scubbo/blog_nginx:built_in_ci
This isn't a complete answer, but I was able to get secure registry access working by switching from mounting a directory, to mounting the file directly:
I changed the docker run option to --env=DRONE_RUNNER_VOLUMES=/var/run/docker.sock:/var/run/docker.sock,$(readlink -f <path_to_directory_containing_pems>/live/docker-registry.scubbo.org/chain.pem):/registry_cert.crt
I changed the commands in copy-cert-into-place to:
- mkdir -p /etc/docker/certs.d/docker-registry.scubbo.org:8843
- cp /registry_cert.crt /etc/docker/certs.d/docker-registry.scubbo.org:8843/ca.crt
I don't consider this a complete answer (and would love further input or advice!), because:
I don't know why copying the file out of the mounted directory into /etc/docker/... (as in the original question) didn't work, but mounting the file directly from the host filesystem worked. (Note that the check-cert-persists-between-stages stage confirms that the certificate is correct, so it's not a mistake of copying a wrong or empty file)
I don't know how to mount the file directly into an in-stage path that contains a colon - this answer indicates how to mount a path containing a colon directly into a container, but in this case we're passing the path to DRONE_RUNNER_VOLUMES
I have Jenkins locally running on port 8081 on a linux machine that is setup in office .
I have a public IP that I am trying to use to make Jenkins publicly available.
I have entered the public IP with port in Manage Jenkins -> Configure System -> Jenkins
URL like: http://182.156.xxx.xx:8081/
Now if I direct to http://182.156.xxx.xx:8081/ , it gives me HTTP 404 error(screenshot attached).
Note: I have setup the Jenkins in Ubuntu with below commands:
wget -q -O - https://pkg.jenkins.io/debian-stable/jenkins.io.key | sudo apt-key add -
sudo sh -c 'echo deb https://pkg.jenkins.io/debian-stable binary/ >
/etc/apt/sources.list.d/jenkins.list'
sudo apt-get update
sudo apt-get install jenkins
etc/default/jenkins file:
# defaults for Jenkins automation server
# pulled in from the init script; makes things easier.
NAME=jenkins
# arguments to pass to java
# Allow graphs etc. to work even when an X server is present
JAVA_ARGS="-Djava.awt.headless=true"
#JAVA_ARGS="-Xmx256m"
# make jenkins listen on IPv4 address
#JAVA_ARGS="-Djava.net.preferIPv4Stack=true"
PIDFILE=/var/run/$NAME/$NAME.pid
# user and group to be invoked as (default to jenkins)
JENKINS_USER=$NAME
JENKINS_GROUP=$NAME
# location of the jenkins war file
JENKINS_WAR=/usr/share/$NAME/$NAME.war
# jenkins home location
JENKINS_HOME=/var/lib/$NAME
# set this to false if you don't want Jenkins to run by itself
# in this set up, you are expected to provide a servlet container
# to host jenkins.
RUN_STANDALONE=true
# log location. this may be a syslog facility.priority
JENKINS_LOG=/var/log/$NAME/$NAME.log
#JENKINS_LOG=daemon.info
# Whether to enable web access logging or not.
# Set to "yes" to enable logging to /var/log/$NAME/access_log
JENKINS_ENABLE_ACCESS_LOG="no"
# OS LIMITS SETUP
# comment this out to observe /etc/security/limits.conf
# this is on by default because http://github.com/jenkinsci/jenkins/commit/2fb288474e980d0e7ff9c4a3b768874835a3e92e
# reported that Ubuntu's PAM configuration doesn't include pam_limits.so, and as a result the # of file
# descriptors are forced to 1024 regardless of /etc/security/limits.conf
MAXOPENFILES=8192
# set the umask to control permission bits of files that Jenkins creates.
# 027 makes files read-only for group and inaccessible for others, which some security sensitive users
# might consider benefitial, especially if Jenkins runs in a box that's used for multiple purposes.
# Beware that 027 permission would interfere with sudo scripts that run on the master (JENKINS-25065.)
#
# Note also that the particularly sensitive part of $JENKINS_HOME (such as credentials) are always
# written without 'others' access. So the umask values only affect job configuration, build records,
# that sort of things.
#
# If commented out, the value from the OS is inherited, which is normally 022 (as of Ubuntu 12.04,
# by default umask comes from pam_umask(8) and /etc/login.defs
# UMASK=027
# port for HTTP connector (default 8080; disable with -1)
HTTP_PORT=8081
# servlet context, important if you want to use apache proxying
PREFIX=/$NAME
# arguments to pass to jenkins.
# --javahome=$JAVA_HOME
# --httpListenAddress=$HTTP_HOST (default 0.0.0.0)
# --httpPort=$HTTP_PORT (default 8080; disable with -1)
# --httpsPort=$HTTP_PORT
# --argumentsRealm.passwd.$ADMIN_USER=[password]
# --argumentsRealm.roles.$ADMIN_USER=admin
# --webroot=~/.jenkins/war
# --prefix=$PREFIX
JENKINS_ARGS="--webroot=/var/cache/$NAME/war --httpPort=$HTTP_PORT"
In this jenkins file, I have only changed the HTTP PORT from 8080 to 8081. As on port 8080, the jenkins is already running with the same public IP.
Jenkins version : 2.289.2
Java version : 8
Ubuntu version : 20.04
jenkins_error_screenshot
I want to secure my NiFi with HTTPS using the tls-toolkit in standalone mode inside a Docker container, on a remote virtual machine running RHEL 8 (so actually using Podman instead of Docker but using a podman-docker module, I can treat podman as a Docker). I want to use the port 19443 now, but eventually I will be using the 9443.
I have created my simple testing Dockerfile:
FROM apache/nifi:latest
WORKDIR /opt/nifi/nifi-current
RUN /opt/nifi/nifi-toolkit-current/bin/tls-toolkit.sh standalone -n "localhost" -C "CN=user_1, OU=NiFi"
RUN ls localhost/
RUN cp -fv /opt/nifi/nifi-current/localhost/* /opt/nifi/nifi-current/conf/ # <- first problem, see build
RUN ls conf/
RUN /opt/nifi/nifi-current/bin/nifi.sh start
EXPOSE 19443
USER nifi
HTTP Works
I have pulled the apache/nifi image and using the command:
docker run --name my_nifi -p 19443:19443 -d -e NIFI_WEB_HTTP_PORT='19443' my_nifi
where the last my_nifi is the image tag that I have created from the Dockerfile.
With this container I can connect to
http://<the remote IP address>:19443/nifi
and it works, showing the NiFi page.
Dockerfile build
docker build -t my_nifi --no-cache .
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
STEP 1: FROM apache/nifi:latest
STEP 2: WORKDIR /opt/nifi/nifi-current
c6788497ae98d998a561aab162f1cded42f17026abe3745e61021826858ff6db
STEP 3: RUN /opt/nifi/nifi-toolkit-current/bin/tls-toolkit.sh standalone -n "localhost" -C "CN=user_1, OU=NiFi"
2020/12/30 08:38:15 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandaloneCommandLine: No nifiPropertiesFile specified, using embedded one.
2020/12/30 08:38:16 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone: Running standalone certificate generation with output directory ../nifi-current
2020/12/30 08:38:16 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone: Generated new CA certificate ../nifi-current/nifi-cert.pem and key ../nifi-current/nifi-key.key
2020/12/30 08:38:16 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone: Writing new ssl configuration to ../nifi-current/localhost
2020/12/30 08:38:16 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone: Successfully generated TLS configuration for localhost 1 in ../nifi-current/localhost
2020/12/30 08:38:16 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone: Generating new client certificate ../nifi-current/CN=user_1_OU=NiFi.p12
2020/12/30 08:38:17 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone: Successfully generated client certificate ../nifi-current/CN=user_1_OU=NiFi.p12
2020/12/30 08:38:17 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone: tls-toolkit standalone completed successfully
0ce5790c026b4650615a6dc8e5745dece2fe6374104825cf4a9ecdc8dfbbdf46
STEP 4: RUN ls localhost/
keystore.jks nifi.properties truststore.jks
85710975c4ed5f1029ad9e7c70b7516e7cf63a9b568e20844d7cf74f8b33f648
STEP 5: RUN cp -fv /opt/nifi/nifi-current/localhost/* /opt/nifi/nifi-current/conf/
'/opt/nifi/nifi-current/localhost/keystore.jks' -> '/opt/nifi/nifi-current/conf/keystore.jks'
'/opt/nifi/nifi-current/localhost/nifi.properties' -> '/opt/nifi/nifi-current/conf/nifi.properties'
'/opt/nifi/nifi-current/localhost/truststore.jks' -> '/opt/nifi/nifi-current/conf/truststore.jks'
a2b99978024840cc4d2702b31f8f2346398673f31ace9d776af112b1aa3d45ac
STEP 6: RUN ls conf/
authorizers.xml login-identity-providers.xml
bootstrap-notification-services.xml nifi.properties
bootstrap.conf state-management.xml
logback.xml zookeeper.properties
0adb1c26826936d08f7edd6df604a0689c23cb9e3db47be06f1c9b4ce935a50d
STEP 7: RUN /opt/nifi/nifi-current/bin/nifi.sh start
Java home: /usr/local/openjdk-8
NiFi home: /opt/nifi/nifi-current
Bootstrap Config File: /opt/nifi/nifi-current/conf/bootstrap.conf
7146d8dc7f891643f42dfd2efef446cedf7b98cf2ecad90ebf6b5de335408b4e
STEP 8: EXPOSE 19443
72f941725ac0c9a66d2c2e0a21286b6db52b3a039c721dccd70234f75dfdd9fe
STEP 9: USER nifi
STEP 10: COMMIT my_nifi
77cf9574d75af00aeed7c6dbacbb853badad82e12f9f448a94f6162df2c1df44
In STEP 3 I use the NiFi tls-toolkit to create the jks keys and the new nifi.properties file, but:
in STEP 5-6, I see the problem that even though the cp command
says that the files have been copied in the conf/ folder, they are
not if I just list the content of that folder.
after the build, I ran a new container (docker run --name my_nifi -p 19443:19443 -d my_nifi and even adding -e NIFI_WEB_HTTPS_PORT='19443' is the same) and tried to enter it and manually cp
the files:
keystore.jks
nifi.properties
truststore.jks
into the conf/ folder and it did copied.
But at the restart of this second container I get this ERROR:
2020-12-30 08:50:33,022 INFO [main] org.eclipse.jetty.util.log Logging initialized #7671ms to org.eclipse.jetty.util.log.Slf4jLog
2020-12-30 08:50:33,066 WARN [main] org.apache.nifi.web.server.JettyServer Both the HTTP and HTTPS connectors are configured in nifi.properties. Only one of these connectors should be configured. See the NiFi Admin Guide for more details
2020-12-30 08:50:33,066 WARN [main] org.apache.nifi.web.server.JettyServer HTTP connector: http://8eafc1fa77d0:8080
2020-12-30 08:50:33,066 WARN [main] org.apache.nifi.web.server.JettyServer HTTPS connector: https://localhost:9443
2020-12-30 08:50:33,066 ERROR [main] org.apache.nifi.web.server.JettyServer NiFi only supports one mode of HTTP or HTTPS operation, not both simultaneously. Check the nifi.properties file and ensure that either the HTTP hostname and port or the HTTPS hostname and port are empty
2020-12-30 08:50:33,068 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down.
java.lang.IllegalStateException: Only one of the HTTP and HTTPS connectors can be configured at one time
at org.apache.nifi.web.server.JettyServer.configureConnectors(JettyServer.java:825)
at org.apache.nifi.web.server.JettyServer.<init>(JettyServer.java:178)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.nifi.NiFi.<init>(NiFi.java:151)
at org.apache.nifi.NiFi.<init>(NiFi.java:72)
at org.apache.nifi.NiFi.main(NiFi.java:301)
2020-12-30 08:50:33,068 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server...
2020-12-30 08:50:33,069 INFO [Thread-1] org.apache.nifi.NiFi Jetty web server shutdown completed (nicely or otherwise).
But the nifi.properties that has been copied is this one below, which does not have the http values filled:
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Core Properties #
nifi.flow.configuration.file=./conf/flow.xml.gz
nifi.flow.configuration.archive.enabled=true
nifi.flow.configuration.archive.dir=./conf/archive/
nifi.flow.configuration.archive.max.time=30 days
nifi.flow.configuration.archive.max.storage=500 MB
nifi.flow.configuration.archive.max.count=
nifi.flowcontroller.autoResumeState=true
nifi.flowcontroller.graceful.shutdown.period=10 sec
nifi.flowservice.writedelay.interval=500 ms
nifi.administrative.yield.duration=30 sec
# If a component has no work to do (is "bored"), how long should we wait before checking again for work?
nifi.bored.yield.duration=10 millis
nifi.queue.backpressure.count=10000
nifi.queue.backpressure.size=1 GB
nifi.authorizer.configuration.file=./conf/authorizers.xml
nifi.login.identity.provider.configuration.file=./conf/login-identity-providers.xml
nifi.templates.directory=./conf/templates
nifi.ui.banner.text=
nifi.ui.autorefresh.interval=30 sec
nifi.nar.library.directory=./lib
nifi.nar.library.autoload.directory=./extensions
nifi.nar.working.directory=./work/nar/
nifi.documentation.working.directory=./work/docs/components
####################
# State Management #
####################
nifi.state.management.configuration.file=./conf/state-management.xml
# The ID of the local state provider
nifi.state.management.provider.local=local-provider
# The ID of the cluster-wide state provider. This will be ignored if NiFi is not clustered but must be populated if running in a cluster.
nifi.state.management.provider.cluster=zk-provider
# Specifies whether or not this instance of NiFi should run an embedded ZooKeeper server
nifi.state.management.embedded.zookeeper.start=false
# Properties file that provides the ZooKeeper properties to use if <nifi.state.management.embedded.zookeeper.start> is set to true
nifi.state.management.embedded.zookeeper.properties=./conf/zookeeper.properties
# H2 Settings
nifi.database.directory=./database_repository
nifi.h2.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE
# FlowFile Repository
nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
nifi.flowfile.repository.wal.implementation=org.apache.nifi.wali.SequentialAccessWriteAheadLog
nifi.flowfile.repository.directory=./flowfile_repository
nifi.flowfile.repository.checkpoint.interval=20 secs
nifi.flowfile.repository.always.sync=false
nifi.flowfile.repository.encryption.key.provider.implementation=
nifi.flowfile.repository.encryption.key.provider.location=
nifi.flowfile.repository.encryption.key.id=
nifi.flowfile.repository.encryption.key=
nifi.flowfile.repository.retain.orphaned.flowfiles=true
nifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager
nifi.queue.swap.threshold=20000
# Content Repository
nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
nifi.content.claim.max.appendable.size=1 MB
nifi.content.repository.directory.default=./content_repository
nifi.content.repository.archive.max.retention.period=7 days
nifi.content.repository.archive.max.usage.percentage=50%
nifi.content.repository.archive.enabled=true
nifi.content.repository.always.sync=false
nifi.content.viewer.url=../nifi-content-viewer/
nifi.content.repository.encryption.key.provider.implementation=
nifi.content.repository.encryption.key.provider.location=
nifi.content.repository.encryption.key.id=
nifi.content.repository.encryption.key=
# Provenance Repository Properties
nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository
nifi.provenance.repository.encryption.key.provider.implementation=
nifi.provenance.repository.encryption.key.provider.location=
nifi.provenance.repository.encryption.key.id=
nifi.provenance.repository.encryption.key=
# Persistent Provenance Repository Properties
nifi.provenance.repository.directory.default=./provenance_repository
nifi.provenance.repository.max.storage.time=30 days
nifi.provenance.repository.max.storage.size=10 GB
nifi.provenance.repository.rollover.time=10 mins
nifi.provenance.repository.rollover.size=100 MB
nifi.provenance.repository.query.threads=2
nifi.provenance.repository.index.threads=2
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.always.sync=false
# Comma-separated list of fields. Fields that are not indexed will not be searchable. Valid fields are:
# EventType, FlowFileUUID, Filename, TransitURI, ProcessorID, AlternateIdentifierURI, Relationship, Details
nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, ProcessorID, Relationship
# FlowFile Attributes that should be indexed and made searchable. Some examples to consider are filename, uuid, mime.type
nifi.provenance.repository.indexed.attributes=
# Large values for the shard size will result in more Java heap usage when searching the Provenance Repository
# but should provide better performance
nifi.provenance.repository.index.shard.size=500 MB
# Indicates the maximum length that a FlowFile attribute can be when retrieving a Provenance Event from
# the repository. If the length of any attribute exceeds this value, it will be truncated when the event is retrieved.
nifi.provenance.repository.max.attribute.length=65536
nifi.provenance.repository.concurrent.merge.threads=2
# Volatile Provenance Repository Properties
nifi.provenance.repository.buffer.size=100000
# Component Status Repository
nifi.components.status.repository.implementation=org.apache.nifi.controller.status.history.VolatileComponentStatusRepository
nifi.components.status.repository.buffer.size=1440
nifi.components.status.snapshot.frequency=1 min
# Site to Site properties
nifi.remote.input.host=localhost
nifi.remote.input.secure=true
nifi.remote.input.socket.port=10443
nifi.remote.input.http.enabled=true
nifi.remote.input.http.transaction.ttl=30 sec
nifi.remote.contents.cache.expiration=30 secs
# web properties #
nifi.web.http.host=
nifi.web.http.port=
nifi.web.http.network.interface.default=
nifi.web.https.host=localhost
nifi.web.https.port=9443
nifi.web.https.network.interface.default=
nifi.web.jetty.working.directory=./work/jetty
nifi.web.jetty.threads=200
nifi.web.max.header.size=16 KB
nifi.web.proxy.context.path=
nifi.web.proxy.host=
nifi.web.max.content.size=
nifi.web.max.requests.per.second=30000
nifi.web.should.send.server.version=true
# security properties #
nifi.sensitive.props.key=
nifi.sensitive.props.key.protected=
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
nifi.sensitive.props.provider=BC
nifi.sensitive.props.additional.keys=
nifi.security.keystore=./conf/keystore.jks
nifi.security.keystoreType=jks
nifi.security.keystorePasswd=U/lgE52hjoAhCa0w9KD2XWZeVp1gyNPT5sAY9I0Kyng
nifi.security.keyPasswd=U/lgE52hjoAhCa0w9KD2XWZeVp1gyNPT5sAY9I0Kyng
nifi.security.truststore=./conf/truststore.jks
nifi.security.truststoreType=jks
nifi.security.truststorePasswd=EvHdoccmVKi8dQj51ohiOIYIuR/J/SaMWb176qBIVrY
nifi.security.user.authorizer=managed-authorizer
nifi.security.allow.anonymous.authentication=false
nifi.security.user.login.identity.provider=
nifi.security.ocsp.responder.url=
nifi.security.ocsp.responder.certificate=
# OpenId Connect SSO Properties #
nifi.security.user.oidc.discovery.url=
nifi.security.user.oidc.connect.timeout=5 secs
nifi.security.user.oidc.read.timeout=5 secs
nifi.security.user.oidc.client.id=
nifi.security.user.oidc.client.secret=
nifi.security.user.oidc.preferred.jwsalgorithm=
nifi.security.user.oidc.additional.scopes=
nifi.security.user.oidc.claim.identifying.user=
# Apache Knox SSO Properties #
nifi.security.user.knox.url=
nifi.security.user.knox.publicKey=
nifi.security.user.knox.cookieName=hadoop-jwt
nifi.security.user.knox.audiences=
# Identity Mapping Properties #
# These properties allow normalizing user identities such that identities coming from different identity providers
# (certificates, LDAP, Kerberos) can be treated the same internally in NiFi. The following example demonstrates normalizing
# DNs from certificates and principals from Kerberos into a common identity string:
#
# nifi.security.identity.mapping.pattern.dn=^CN=(.*?), OU=(.*?), O=(.*?), L=(.*?), ST=(.*?), C=(.*?)$
# nifi.security.identity.mapping.value.dn=$1#$2
# nifi.security.identity.mapping.transform.dn=NONE
# nifi.security.identity.mapping.pattern.kerb=^(.*?)/instance#(.*?)$
# nifi.security.identity.mapping.value.kerb=$1#$2
# nifi.security.identity.mapping.transform.kerb=UPPER
# Group Mapping Properties #
# These properties allow normalizing group names coming from external sources like LDAP. The following example
# lowercases any group name.
#
# nifi.security.group.mapping.pattern.anygroup=^(.*)$
# nifi.security.group.mapping.value.anygroup=$1
# nifi.security.group.mapping.transform.anygroup=LOWER
# cluster common properties (all nodes must have same values) #
nifi.cluster.protocol.heartbeat.interval=5 sec
nifi.cluster.protocol.heartbeat.missable.max=8
nifi.cluster.protocol.is.secure=true
# cluster node properties (only configure for cluster nodes) #
nifi.cluster.is.node=false
nifi.cluster.node.address=localhost
nifi.cluster.node.protocol.port=11443
nifi.cluster.node.protocol.threads=10
nifi.cluster.node.protocol.max.threads=50
nifi.cluster.node.event.history.size=25
nifi.cluster.node.connection.timeout=5 sec
nifi.cluster.node.read.timeout=5 sec
nifi.cluster.node.max.concurrent.requests=100
nifi.cluster.firewall.file=
nifi.cluster.flow.election.max.wait.time=5 mins
nifi.cluster.flow.election.max.candidates=
# cluster load balancing properties #
nifi.cluster.load.balance.host=
nifi.cluster.load.balance.port=6342
nifi.cluster.load.balance.connections.per.node=1
nifi.cluster.load.balance.max.thread.count=8
nifi.cluster.load.balance.comms.timeout=30 sec
# zookeeper properties, used for cluster management #
nifi.zookeeper.connect.string=
nifi.zookeeper.connect.timeout=10 secs
nifi.zookeeper.session.timeout=10 secs
nifi.zookeeper.root.node=/nifi
# Zookeeper properties for the authentication scheme used when creating acls on znodes used for cluster management
# Values supported for nifi.zookeeper.auth.type are "default", which will apply world/anyone rights on znodes
# and "sasl" which will give rights to the sasl/kerberos identity used to authenticate the nifi node
# The identity is determined using the value in nifi.kerberos.service.principal and the removeHostFromPrincipal
# and removeRealmFromPrincipal values (which should align with the kerberos.removeHostFromPrincipal and kerberos.removeRealmFromPrincipal
# values configured on the zookeeper server).
nifi.zookeeper.auth.type=
nifi.zookeeper.kerberos.removeHostFromPrincipal=
nifi.zookeeper.kerberos.removeRealmFromPrincipal=
# kerberos #
nifi.kerberos.krb5.file=
# kerberos service principal #
nifi.kerberos.service.principal=
nifi.kerberos.service.keytab.location=
# kerberos spnego principal #
nifi.kerberos.spnego.principal=
nifi.kerberos.spnego.keytab.location=
nifi.kerberos.spnego.authentication.expiration=12 hours
# external properties files for variable registry
# supports a comma delimited list of file locations
nifi.variable.registry.properties=
# analytics properties #
nifi.analytics.predict.enabled=false
nifi.analytics.predict.interval=3 mins
nifi.analytics.query.interval=5 mins
nifi.analytics.connection.model.implementation=org.apache.nifi.controller.status.analytics.models.OrdinaryLeastSquares
nifi.analytics.connection.model.score.name=rSquared
nifi.analytics.connection.model.score.threshold=.90
How do I solve this?
According to the documentation of the nifi image , you should add specific variables to your docker run command if you want to go https. I will try that, by providing external keystore and truststore.
docker run --name nifi \
-v /User/dreynolds/certs/localhost:/opt/certs \
-p 8443:8443 \
-e AUTH=tls \
-e KEYSTORE_PATH=/opt/certs/keystore.jks \
-e KEYSTORE_TYPE=JKS \
-e KEYSTORE_PASSWORD=QKZv1hSWAFQYZ+WU1jjF5ank+l4igeOfQRp+OSbkkrs \
-e TRUSTSTORE_PATH=/opt/certs/truststore.jks \
-e TRUSTSTORE_PASSWORD=rHkWR1gDNW3R9hgbeRsT3OM3Ue0zwGtQqcFKJD2EXWE \
-e TRUSTSTORE_TYPE=JKS \
-e INITIAL_ADMIN_IDENTITY='CN=Random User, O=Apache, OU=NiFi, C=US' \
-d \
apache/nifi:latest
You could also try to build the image from scratch (ie by downloading nifi from the Dockerfile etc...)
I created docker container with this command:
docker run --name artifactory -d -p 8081:8081 \
-v /jfrog/artifactory:/var/opt/jfrog/artifactory \
-e EXTRA_JAVA_OPTIONS='-Xms128M -Xmx512M -Xss256k -XX:+UseG1GC' \
docker.bintray.io/jfrog/artifactory-oss:latest
and started artifactory, but the response I get is 404 - not found
If u access http://99.79.191.172:8081/artifactory u see it
If you follow the Artifactory Docker install documentation, you'll see you also need to expose port 8082 for the new JFrog Router, which is now handling the traffic coming in to the UI (and other services as needed).
This new architecture is from Artifactory 7.x. By setting latest as the repository tag, you don't have full control of what version you are running...
So your command should look like
docker run --name artifactory -p 8081:8081 -d -p 8082:8082 \
-v /jfrog/artifactory:/var/opt/jfrog/artifactory \
docker.bintray.io/jfrog/artifactory-oss:latest
For controlling the configuration (like the Java options you want), it's recommended to use the Artifactory system.yaml configuration. This file is the best way to control all aspects of the Artifactory system configuration.
I start my instance with
sudo groupadd -g 1030 artifactory
sudo useradd -u 1030 -g artifactory artifactory
sudo chown artifactory:artifactory /daten/jfrog -R
docker run \
-d \
--name artifactory \
-v /daten/jfrog/artifactory:/var/opt/jfrog/artifactory \
--user "$(id -u artifactory):$(id -g artifactory)" \
--restart always \
-p 8084:8081 -p 9082:8082 releases-docker.jfrog.io/jfrog/artifactory-oss:latest
This is my /daten/jfrog/artifactory/etc/system.yaml (I changed nothing manually)
## #formatter:off
## JFROG ARTIFACTORY SYSTEM CONFIGURATION FILE
## HOW TO USE: comment-out any field and keep the correct yaml indentation by deleting only the leading '#' character.
configVersion: 1
## NOTE: JFROG_HOME is a place holder for the JFrog root directory containing the deployed product, the home directory for all JFrog products.
## Replace JFROG_HOME with the real path! For example, in RPM install, JFROG_HOME=/opt/jfrog
## NOTE: Sensitive information such as passwords and join key are encrypted on first read.
## NOTE: The provided commented key and value is the default.
## SHARED CONFIGURATIONS
## A shared section for keys across all services in this config
shared:
## Java 11 distribution to use
#javaHome: "JFROG_HOME/artifactory/app/third-party/java"
## Extra Java options to pass to the JVM. These values add to or override the defaults.
#extraJavaOpts: "-Xms512m -Xmx2g"
## Security Configuration
security:
## Join key value for joining the cluster (takes precedence over 'joinKeyFile')
#joinKey: "<Your joinKey>"
## Join key file location
#joinKeyFile: "<For example: JFROG_HOME/artifactory/var/etc/security/join.key>"
## Master key file location
## Generated by the product on first startup if not provided
#masterKeyFile: "<For example: JFROG_HOME/artifactory/var/etc/security/master.key>"
## Maximum time to wait for key files (master.key and join.key)
#bootstrapKeysReadTimeoutSecs: 120
## Node Settings
node:
## A unique id to identify this node.
## Default auto generated at startup.
#id: "art1"
## Default auto resolved by startup script
#ip:
## Sets this node as primary in HA installation
#primary: true
## Sets this node as part of HA installation
#haEnabled: true
## Database Configuration
database:
## One of mysql, oracle, mssql, postgresql, mariadb
## Default Embedded derby
## Example for postgresql
#type: postgresql
#driver: org.postgresql.Driver
#url: "jdbc:postgresql://<your db url, for example: localhost:5432>/artifactory"
#username: artifactory
#password: password
I see this in router-request.log
{"BackendAddr":"localhost:8040","ClientAddr":"127.0.0.1:43740","DownstreamContentSize":95,"DownstreamStatus":404,"Duration":3608608,"RequestMethod":"GET","RequestPath":"/access/api/v1/users/jffe#000?expand=groups","StartUTC":"2021-12-30T11:49:19.56803042Z","level":"info","msg":"","request_Uber-Trace-Id":"664d23ea1941d9b0:410817c2c69f2849:31b50a1adccb9846:0","request_User-Agent":"JFrog Access Java Client/7.29.9 72909900 Artifactory/7.29.8 72908900","time":"2021-12-30T11:49:19Z"}
{"BackendAddr":"localhost:8040","ClientAddr":"127.0.0.1:43734","DownstreamContentSize":95,"DownstreamStatus":404,"Duration":4000683,"RequestMethod":"GET","RequestPath":"/access/api/v1/users/jffe#000?expand=groups","StartUTC":"2021-12-30T11:49:19.567751867Z","level":"info","msg":"","request_Uber-Trace-Id":"23967a8743252dd8:436e2a5407b66e64:31cfc496ccc260fa:0","request_User-Agent":"JFrog Access Java Client/7.29.9 72909900 Artifactory/7.29.8 72908900","time":"2021-12-30T11:49:19Z"}
{"BackendAddr":"localhost:8040","ClientAddr":"127.0.0.1:43736","DownstreamContentSize":95,"DownstreamStatus":404,"Duration":4021195,"RequestMethod":"GET","RequestPath":"/access/api/v1/users/jffe#000?expand=groups","StartUTC":"2021-12-30T11:49:19.567751873Z","level":"info","msg":"","request_Uber-Trace-Id":"28300761ec7b6cd5:36588fa084ee7105:10fbdaadbc39b21e:0","request_User-Agent":"JFrog Access Java Client/7.29.9 72909900 Artifactory/7.29.8 72908900","time":"2021-12-30T11:49:19Z"}
{"BackendAddr":"localhost:8040","ClientAddr":"127.0.0.1:43622","DownstreamContentSize":95,"DownstreamStatus":404,"Duration":3918873,"RequestMethod":"GET","RequestPath":"/access/api/v1/users/jffe#000?expand=groups","StartUTC":"2021-12-30T11:49:19.567751891Z","level":"info","msg":"","request_Uber-Trace-Id":"6d57920d087f4d0f:26b9120411520de2:49b0e61895e17734:0","request_User-Agent":"JFrog Access Java Client/7.29.9 72909900 Artifactory/7.29.8 72908900","time":"2021-12-30T11:49:19Z"}
{"BackendAddr":"localhost:8040","ClientAddr":"127.0.0.1:43742","DownstreamContentSize":95,"DownstreamStatus":404,"Duration":2552815,"RequestMethod":"GET","RequestPath":"/access/api/v1/users/jffe#000?expand=groups","StartUTC":"2021-12-30T11:49:19.569112324Z","level":"info","msg":"","request_Uber-Trace-Id":"d4a7bb216cf31eb:5c783ae80b95778f:fd11882b03eb63f:0","request_User-Agent":"JFrog Access Java Client/7.29.9 72909900 Artifactory/7.29.8 72908900","time":"2021-12-30T11:49:19Z"}
{"BackendAddr":"localhost:8081","ClientAddr":"127.0.0.1:43730","DownstreamContentSize":45,"DownstreamStatus":200,"Duration":18106757,"RequestMethod":"POST","RequestPath":"/artifactory/api/auth/loginRelatedData","StartUTC":"2021-12-30T11:49:19.557661286Z","level":"info","msg":"","request_Uber-Trace-Id":"d4a7bb216cf31eb:640bf3bca741e43b:28f0abcfc40f203:0","request_User-Agent":"JFrog-Frontend/1.29.6","time":"2021-12-30T11:49:19Z"}
{"BackendAddr":"localhost:8081","ClientAddr":"127.0.0.1:43726","DownstreamContentSize":169,"DownstreamStatus":200,"Duration":19111069,"RequestMethod":"GET","RequestPath":"/artifactory/api/crowd","StartUTC":"2021-12-30T11:49:19.557426794Z","level":"info","msg":"","request_Uber-Trace-Id":"664d23ea1941d9b0:417647e0e0fd0911:55e80b7f7ab0724e:0","request_User-Agent":"JFrog-Frontend/1.29.6","time":"2021-12-30T11:49:19Z"}
{"BackendAddr":"localhost:8081","ClientAddr":"127.0.0.1:43724","DownstreamContentSize":496,"DownstreamStatus":200,"Duration":19308753,"RequestMethod":"GET","RequestPath":"/artifactory/api/securityconfig","StartUTC":"2021-12-30T11:49:19.557346739Z","level":"info","msg":"","request_Uber-Trace-Id":"6d57920d087f4d0f:7bdba564c07f8bc5:71b1b99e1e406d5f:0","request_User-Agent":"JFrog-Frontend/1.29.6","time":"2021-12-30T11:49:19Z"}
{"BackendAddr":"localhost:8081","ClientAddr":"127.0.0.1:43728","DownstreamContentSize":2,"DownstreamStatus":200,"Duration":19140699,"RequestMethod":"GET","RequestPath":"/artifactory/api/saml/config","StartUTC":"2021-12-30T11:49:19.557516365Z","level":"info","msg":"","request_Uber-Trace-Id":"23967a8743252dd8:2f9035e56dd9f0c5:4315ec00a6b32eb4:0","request_User-Agent":"JFrog-Frontend/1.29.6","time":"2021-12-30T11:49:19Z"}
{"BackendAddr":"localhost:8081","ClientAddr":"127.0.0.1:43732","DownstreamContentSize":148,"DownstreamStatus":200,"Duration":18907203,"RequestMethod":"GET","RequestPath":"/artifactory/api/httpsso","StartUTC":"2021-12-30T11:49:19.557786692Z","level":"info","msg":"","request_Uber-Trace-Id":"28300761ec7b6cd5:2767cf480f6ebd73:2c013715cb58b384:0","request_User-Agent":"JFrog-Frontend/1.29.6","time":"2021-12-30T11:49:19Z"}
I've to change the port to 8084 (it's already occupied) But I run into 404 as well.
Who knows how to solve it ?
I have installed and started jenkins successfully. The problem is that after working for a while, the web interface returns "refused to connect".
After lsof -i -P -n | grep LISTEN, I don't see anything listening to the port 8080.
The weird thing is cat /var/log/jenkins/jenkins.log returns nothing.
When I run service jenkins start nothing happens. service jenkins restart on the other hand gets things back to normal.
Here is the contents of /etc/default/jenkins:
# defaults for Jenkins automation server
# pulled in from the init script; makes things easier.
NAME=jenkins
# arguments to pass to java
# Allow graphs etc. to work even when an X server is present
JAVA_ARGS="-Djava.awt.headless=true"
#JAVA_ARGS="-Xmx1024m"
# make jenkins listen on IPv4 address
#JAVA_ARGS="-Djava.net.preferIPv4Stack=true"
PIDFILE=/var/run/$NAME/$NAME.pid
# user and group to be invoked as (default to jenkins)
JENKINS_USER=$NAME
JENKINS_GROUP=$NAME
# location of the jenkins war file
JENKINS_WAR=/usr/share/$NAME/$NAME.war
# jenkins home location
JENKINS_HOME=/var/lib/$NAME
# set this to false if you don't want Jenkins to run by itself
# in this set up, you are expected to provide a servlet container
# to host jenkins.
RUN_STANDALONE=true
# log location. this may be a syslog facility.priority
JENKINS_LOG=/var/log/$NAME/$NAME.log
#JENKINS_LOG=daemon.info
# Whether to enable web access logging or not.
# Set to "yes" to enable logging to /var/log/$NAME/access_log
JENKINS_ENABLE_ACCESS_LOG="no"
# OS LIMITS SETUP
# comment this out to observe /etc/security/limits.conf
# this is on by default because http://github.com/jenkinsci/jenkins/commit/2fb288474e980d0e7ff9c4a3b768874835a3e92e
# reported that Ubuntu's PAM configuration doesn't include pam_limits.so, and as a result the # of file
# descriptors are forced to 1024 regardless of /etc/security/limits.conf
MAXOPENFILES=8192
# set the umask to control permission bits of files that Jenkins creates.
# 027 makes files read-only for group and inaccessible for others, which some security sensitive users
# might consider benefitial, especially if Jenkins runs in a box that's used for multiple purposes.
# Beware that 027 permission would interfere with sudo scripts that run on the master (JENKINS-25065.)
#
# Note also that the particularly sensitive part of $JENKINS_HOME (such as credentials) are always
# written without 'others' access. So the umask values only affect job configuration, build records,
# that sort of things.
#
# If commented out, the value from the OS is inherited, which is normally 022 (as of Ubuntu 12.04,
# by default umask comes from pam_umask(8) and /etc/login.defs
# UMASK=027
# port for HTTP connector (default 8080; disable with -1)
HTTP_PORT=8080
# servlet context, important if you want to use apache proxying
PREFIX=/$NAME
# arguments to pass to jenkins.
# --javahome=$JAVA_HOME
# --httpListenAddress=$HTTP_HOST (default 0.0.0.0)
# --httpPort=$HTTP_PORT (default 8080; disable with -1)
# --httpsPort=$HTTP_PORT
# --argumentsRealm.passwd.$ADMIN_USER=[password]
# --argumentsRealm.roles.$ADMIN_USER=admin
# --webroot=~/.jenkins/war
# --prefix=$PREFIX
JENKINS_ARGS="--webroot=/var/cache/$NAME/war --httpPort=$HTTP_PORT"
I got the same issue. I'm deploying jenkins on vultr vps with limited ram memory. I believe the limited memory is the cause. I fixed this by following this link Jenkins build throwing an out of memory error
sudo fallocate -l 1G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
Check again:
free -m