NGINX Reverse Proxy throws HTTP 503 on JIRA - docker

Question

I'm setting up a internal server for JIRA, Jenkins etc. For simplification i use an Reverse Proxy and docker-compose.
My docker-compose.yml has following configuration:
version: '3.7'
services:
proxy:
container_name: proxy
image: nginx:alpine
init: true
restart: unless-stopped
ports:
- 80:80
- 443:443
networks:
proxynet:
ipv4_address: 172.20.0.2
depends_on:
- jenkins
- nexus
volumes:
- ./reverse-proxy/data/app.conf:/etc/nginx/conf.d
- ./reverse-proxy/logs:/var/logs/nginx/
...
jira:
container_name: jira
image: cptactionhank/atlassian-jira:latest
init: true
restart: unless-stopped
networks:
proxynet:
ipv4_address: 172.20.0.5
volumes:
- ./jira/webapp:/var/atlassian/jira
- ./jira/logs:/opt/atlassian/jira/logs
environment:
- X_PROXY_PORT=8080
- X_PROXY_SCHEME="http"
volumes:
data:
networks:
proxynet:
driver: bridge
ipam:
config:
- subnet: 172.20.0.0/16
reverse-proxy/data/app.conf
server {
listen 80;
server_name 192.123.456.789;
...
location /jira {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://172.20.0.5:8080/jira;
proxy_ssl_verify off;
proxy_read_timeout 90;
}
}
If i visit the 192.123.456.789/jira, i get an HTTP ERROR 503.
NGINX access log:
[04/Sep/2019:16:49:33 +0000] "GET /jira HTTP/1.1" 503 36 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36" "-"
JIRA startup log:
`sMMMMMMMMMMMMMM+
MMMMMMMMMMMMMM
:sdMMMMMMMMMMM
MMMMMM
`sMMMMMMMMMMMMMM+ MMMMMM
MMMMMMMMMMMMMM +MMMMM
:sMMMMMMMMMMM MMMMM
MMMMMM `UOJ
`sMMMMMMMMMMMMM+ MMMMMM
MMMMMMMMMMMMMM +MMMMM
:sdMMMMMMMMMM MMMMM
MMMMMM `UOJ
MMMMMM
+MMMMM
MMMMM
`UOJ
Atlassian Jira
Version : 8.3.3
If you encounter issues starting or stopping JIRA, please see the Troubleshooting guide at https://docs.atlassian.com/jira/jadm-docs-083/Troubleshooting+installation
Using JIRA_HOME: /var/atlassian/jira
Server startup logs are located in /opt/atlassian/jira/logs/catalina.out
04-Sep-2019 17:03:57.136 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version: Apache Tomcat/8.5.40
04-Sep-2019 17:03:57.141 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built: Apr 10 2019 14:31:19 UTC
04-Sep-2019 17:03:57.142 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server number: 8.5.40.0
04-Sep-2019 17:03:57.142 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name: Linux
04-Sep-2019 17:03:57.142 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version: 4.15.0-54-generic
04-Sep-2019 17:03:57.143 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture: amd64
04-Sep-2019 17:03:57.143 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home: /usr/lib/jvm/java-1.8-openjdk/jre
04-Sep-2019 17:03:57.144 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version: 1.8.0_212-b04
04-Sep-2019 17:03:57.144 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor: IcedTea
04-Sep-2019 17:03:57.144 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE: /opt/atlassian/jira
04-Sep-2019 17:03:57.145 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME: /opt/atlassian/jira
04-Sep-2019 17:03:57.146 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=/opt/atlassian/jira/conf/logging.properties
04-Sep-2019 17:03:57.147 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
04-Sep-2019 17:03:57.147 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xms384m
04-Sep-2019 17:03:57.148 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xmx2048m
04-Sep-2019 17:03:57.148 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:InitialCodeCacheSize=32m
04-Sep-2019 17:03:57.148 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:ReservedCodeCacheSize=512m
04-Sep-2019 17:03:57.149 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.awt.headless=true
04-Sep-2019 17:03:57.149 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Datlassian.standalone=JIRA
04-Sep-2019 17:03:57.153 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true
04-Sep-2019 17:03:57.154 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dmail.mime.decodeparameters=true
04-Sep-2019 17:03:57.154 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.dom4j.factory=com.atlassian.core.xml.InterningDocumentFactory
04-Sep-2019 17:03:57.155 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:-OmitStackTraceInFastThrow
04-Sep-2019 17:03:57.155 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.locale.providers=COMPAT
04-Sep-2019 17:03:57.155 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djira.home=/var/atlassian/jira
04-Sep-2019 17:03:57.156 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Datlassian.plugins.startup.options=-fg
04-Sep-2019 17:03:57.156 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djdk.tls.ephemeralDHKeySize=2048
04-Sep-2019 17:03:57.156 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
04-Sep-2019 17:03:57.157 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
04-Sep-2019 17:03:57.157 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xloggc:/opt/atlassian/jira/logs/atlassian-jira-gc-%t.log
04-Sep-2019 17:03:57.158 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:+UseGCLogFileRotation
04-Sep-2019 17:03:57.158 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:NumberOfGCLogFiles=5
04-Sep-2019 17:03:57.158 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:GCLogFileSize=20M
04-Sep-2019 17:03:57.159 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:+PrintGCDetails
04-Sep-2019 17:03:57.159 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:+PrintGCDateStamps
04-Sep-2019 17:03:57.160 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:+PrintGCTimeStamps
04-Sep-2019 17:03:57.160 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:+PrintGCCause
04-Sep-2019 17:03:57.161 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dignore.endorsed.dirs=
04-Sep-2019 17:03:57.161 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=/opt/atlassian/jira
04-Sep-2019 17:03:57.163 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/opt/atlassian/jira
04-Sep-2019 17:03:57.163 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/opt/atlassian/jira/temp
04-Sep-2019 17:03:57.164 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64:/usr/lib/jvm/java-1.8-openjdk/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib]
04-Sep-2019 17:03:57.505 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"]
04-Sep-2019 17:03:57.555 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 1377 ms
04-Sep-2019 17:03:57.576 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
04-Sep-2019 17:03:57.578 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.5.40
2019-09-04 17:04:05,574 localhost-startStop-1 INFO [c.a.jira.startup.JiraHomeStartupCheck] The jira.home directory '/var/atlassian/jira' is validated and locked for exclusive use by this instance.
2019-09-04 17:04:05,783 JIRA-Bootstrap INFO [c.a.jira.startup.JiraStartupLogger]
****************
JIRA starting...
****************
2019-09-04 17:04:05,923 JIRA-Bootstrap INFO [c.a.jira.startup.JiraStartupLogger]
___ Environment _____________________________
JIRA Build : 8.3.3#803004-sha1:4d4040e0714d65b7fffa4801569d014c0b16eaa9
Build Date : Wed Aug 28 00:00:00 GMT 2019
JIRA Installation Type : Standalone
Application Server : Apache Tomcat/8.5.40 - Servlet API 3.1
Java Version : 1.8.0_212 - IcedTea
Current Working Directory : /var/atlassian/jira
Maximum Allowable Memory : 1979MB
Total Memory : 371MB
Free Memory : 330MB
Used Memory : 41MB
Memory Pool: Code Cache : Code Cache: init = 33554432(32768K) used = 8746176(8541K) committed = 33554432(32768K) max = 536870912(524288K)
Memory Pool: Metaspace : Metaspace: init = 0(0K) used = 21664984(21157K) committed = 22331392(21808K) max = -1(-1K)
Memory Pool: Compressed Class Space : Compressed Class Space: init = 0(0K) used = 2323808(2269K) committed = 2490368(2432K) max = 1073741824(1048576K)
Memory Pool: Eden Space : Eden Space: init = 107479040(104960K) used = 22147168(21628K) committed = 107544576(105024K) max = 572653568(559232K)
Memory Pool: Survivor Space : Survivor Space: init = 13369344(13056K) used = 0(0K) committed = 13369344(13056K) max = 71565312(69888K)
Memory Pool: Tenured Gen : Tenured Gen: init = 268435456(262144K) used = 21798312(21287K) committed = 268435456(262144K) max = 1431699456(1398144K)
JVM Input Arguments : -Djava.util.logging.config.file=/opt/atlassian/jira/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms384m -Xmx2048m -XX:InitialCodeCacheSize=32m -XX:ReservedCodeCacheSize=512m -Djava.awt.headless=true -Datlassian.standalone=JIRA -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true -Dmail.mime.decodeparameters=true -Dorg.dom4j.factory=com.atlassian.core.xml.InterningDocumentFactory -XX:-OmitStackTraceInFastThrow -Djava.locale.providers=COMPAT -Djira.home=/var/atlassian/jira -Datlassian.plugins.startup.options=-fg -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Xloggc:/opt/atlassian/jira/logs/atlassian-jira-gc-%t.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=20M -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintGCCause -Dignore.endorsed.dirs= -Dcatalina.base=/opt/atlassian/jira -Dcatalina.home=/opt/atlassian/jira -Djava.io.tmpdir=/opt/atlassian/jira/temp
Java Compatibility Information : JIRA version = 8.3.3, Java Version = 1.8.0_212
___ Java System Properties _________________
atlassian.plugins.startup.options : -fg
atlassian.standalone : JIRA
awt.toolkit : sun.awt.X11.XToolkit
catalina.base : /opt/atlassian/jira
catalina.home : /opt/atlassian/jira
catalina.useNaming : true
common.loader : "${catalina.base}/lib",
"${catalina.base}/lib/*.jar",
"${catalina.home}/lib",
"${catalina.home}/lib/*.jar"
file.encoding : UTF-8
file.encoding.pkg : sun.io
ignore.endorsed.dirs :
java.awt.graphicsenv : sun.awt.X11GraphicsEnvironment
java.awt.headless : true
java.awt.printerjob : sun.print.PSPrinterJob
java.class.version : 52.0
java.home : /usr/lib/jvm/java-1.8-openjdk/jre
java.io.tmpdir : /opt/atlassian/jira/temp
java.locale.providers : COMPAT
java.naming.factory.initial : org.apache.naming.java.javaURLContextFactory
java.naming.factory.url.pkgs : org.apache.naming
java.protocol.handler.pkgs : org.apache.catalina.webresources
java.runtime.name : OpenJDK Runtime Environment
java.runtime.version : 1.8.0_212-b04
java.specification.name : Java Platform API Specification
java.specification.vendor : Oracle Corporation
java.specification.version : 1.8
java.util.concurrent.ForkJoinPool.common.threadFactory : org.apache.catalina.startup.SafeForkJoinWorkerThreadFactory
java.util.logging.config.file : /opt/atlassian/jira/conf/logging.properties
java.util.logging.manager : org.apache.juli.ClassLoaderLogManager
java.vendor : IcedTea
java.vendor.url : https://icedtea.classpath.org
java.vendor.url.bug : https://icedtea.classpath.org/bugzilla
java.version : 1.8.0_212
java.vm.info : mixed mode
java.vm.name : OpenJDK 64-Bit Server VM
java.vm.specification.name : Java Virtual Machine Specification
java.vm.specification.vendor : Oracle Corporation
java.vm.specification.version : 1.8
java.vm.vendor : IcedTea
java.vm.version : 25.212-b04
jdk.tls.ephemeralDHKeySize : 2048
jira.home : /var/atlassian/jira
mail.mime.decodeparameters : true
org.apache.catalina.security.SecurityListener.UMASK : 0027
org.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER : true
org.dom4j.factory : com.atlassian.core.xml.InterningDocumentFactory
os.arch : amd64
os.name : Linux
os.version : 4.15.0-54-generic
package.access : sun.,
org.apache.catalina.,
org.apache.coyote.,
org.apache.jasper.,
org.apache.tomcat.
package.definition : sun.,
java.,
org.apache.catalina.,
org.apache.coyote.,
org.apache.jasper.,
org.apache.naming.,
org.apache.tomcat.
server.loader :
shared.loader :
sun.arch.data.model : 64
sun.boot.library.path : /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64
sun.cpu.endian : little
sun.cpu.isalist :
sun.io.unicode.encoding : UnicodeLittle
sun.java.command : org.apache.catalina.startup.Bootstrap start
sun.java.launcher : SUN_STANDARD
sun.jnu.encoding : UTF-8
sun.management.compiler : HotSpot 64-Bit Tiered Compilers
sun.os.patch.level : unknown
tomcat.util.buf.StringCache.byte.enabled : true
tomcat.util.scan.StandardJarScanFilter.jarsToScan : log4j-taglib*.jar,
log4j-web*.jar,
log4javascript*.jar,
slf4j-taglib*.jar
tomcat.util.scan.StandardJarScanFilter.jarsToSkip : annotations-api.jar,
ant-junit*.jar,
ant-launcher.jar,
ant.jar,
asm-*.jar,
aspectj*.jar,
bootstrap.jar,
catalina-ant.jar,
catalina-ha.jar,
catalina-jmx-remote.jar,
catalina-storeconfig.jar,
catalina-tribes.jar,
catalina-ws.jar,
catalina.jar,
cglib-*.jar,
cobertura-*.jar,
commons-beanutils*.jar,
commons-codec*.jar,
commons-collections*.jar,
commons-daemon.jar,
commons-dbcp*.jar,
commons-digester*.jar,
commons-fileupload*.jar,
commons-httpclient*.jar,
commons-io*.jar,
commons-lang*.jar,
commons-logging*.jar,
commons-math*.jar,
commons-pool*.jar,
dom4j-*.jar,
easymock-*.jar,
ecj-*.jar,
el-api.jar,
geronimo-spec-jaxrpc*.jar,
h2*.jar,
hamcrest-*.jar,
hibernate*.jar,
httpclient*.jar,
icu4j-*.jar,
jasper-el.jar,
jasper.jar,
jaspic-api.jar,
jaxb-*.jar,
jaxen-*.jar,
jdom-*.jar,
jetty-*.jar,
jmx-tools.jar,
jmx.jar,
jsp-api.jar,
jstl.jar,
jta*.jar,
junit-*.jar,
junit.jar,
log4j*.jar,
mail*.jar,
objenesis-*.jar,
oraclepki.jar,
oro-*.jar,
servlet-api-*.jar,
servlet-api.jar,
slf4j*.jar,
taglibs-standard-spec-*.jar,
tagsoup-*.jar,
tomcat-api.jar,
tomcat-coyote.jar,
tomcat-dbcp.jar,
tomcat-i18n-*.jar,
tomcat-jdbc.jar,
tomcat-jni.jar,
tomcat-juli-adapters.jar,
tomcat-juli.jar,
tomcat-util-scan.jar,
tomcat-util.jar,
tomcat-websocket.jar,
tools.jar,
websocket-api.jar,
wsdl4j*.jar,
xercesImpl.jar,
xml-apis.jar,
xmlParserAPIs-*.jar,
xmlParserAPIs.jar,
xom-*.jar
user.country : US
user.dir : /var/atlassian/jira
user.home : /sbin
user.language : en
user.name : daemon
user.timezone : GMT
2019-09-04 17:04:06,338 JIRA-Bootstrap INFO [c.a.j.config.database.SystemDatabaseConfigurationLoader] Reading database configuration from /var/atlassian/jira/dbconfig.xml
2019-09-04 17:04:06,391 JIRA-Bootstrap INFO [c.a.j.config.database.SystemDatabaseConfigurationLoader] Reading database configuration from /var/atlassian/jira/dbconfig.xml
2019-09-04 17:04:06,392 JIRA-Bootstrap INFO [c.a.j.config.database.SystemDatabaseConfigurationLoader] Reading database configuration from /var/atlassian/jira/dbconfig.xml
2019-09-04 17:04:08,155 JIRA-Bootstrap INFO [c.a.j.config.database.SystemDatabaseConfigurationLoader] Reading database configuration from /var/atlassian/jira/dbconfig.xml
2019-09-04 17:04:08,187 JIRA-Bootstrap INFO [c.a.jira.startup.JiraStartupLogger]
___ Starting the JIRA Plugin System _________________
2019-09-04 17:04:17,775 JIRA-Bootstrap INFO [c.a.j.config.database.SystemDatabaseConfigurationLoader] Reading database configuration from /var/atlassian/jira/dbconfig.xml
2019-09-04 17:04:17,931 JIRA-Bootstrap INFO [c.a.jira.startup.JiraStartupLogger] Running JIRA startup checks.
2019-09-04 17:04:17,933 JIRA-Bootstrap INFO [c.a.jira.startup.JiraStartupLogger] JIRA pre-database startup checks completed successfully.
2019-09-04 17:04:18,023 JIRA-Bootstrap INFO [c.a.j.config.database.DatabaseConfigurationManagerImpl] The database is not yet configured. Enqueuing Database Checklist Launcher on post-database-configured-but-pre-database-activated queue
2019-09-04 17:04:18,024 JIRA-Bootstrap INFO [c.a.j.config.database.DatabaseConfigurationManagerImpl] The database is not yet configured. Enqueuing Post database-configuration launchers on post-database-activated queue
2019-09-04 17:04:18,045 JIRA-Bootstrap INFO [c.a.jira.startup.LauncherContextListener] Memory Usage:
---------------------------------------------------------------------------------
Heap memory : Used: 73 MiB. Committed: 371 MiB. Max: 1980 MiB
Non-heap memory : Used: 68 MiB. Committed: 86 MiB. Max: 1536 MiB
---------------------------------------------------------------------------------
TOTAL : Used: 141 MiB. Committed: 458 MiB. Max: 3516 MiB
---------------------------------------------------------------------------------
04-Sep-2019 17:05:03.590 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
04-Sep-2019 17:05:03.618 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
04-Sep-2019 17:05:03.701 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 66145 ms
I also started JIRA without the X_PROXY_* environment variables, but still no success.
I'm a bit confused because jira has started successfully, but nginx can't access it apparently. What am i missing?

Answer 1

I think i have found your problem. In the nginx config you have this line
location /jira {
...
Change it to
location ^~ /jira {
add_header Channel-id "JIRA";
...