Apache Storm 1.1.1 - where to find my log files? - log4j2

Question

I have checked these questions:
Logging from a storm bolt - where is it going?
And the solution is not working anymore.
In theory, the system variable of storm.log.dir is set when we launch storm jar. As the solution suggests, you can use ps aux | grep storm.log.dir to search the argument's value.
It shows:
java -client -Ddaemon.name= -Dstorm.options= -Dstorm.home=/opt/apache-storm-1.1.1 -Dstorm.log.dir=/opt/apache-storm-1.1.1/logs .....
But, when I go there, I have:
[root#xxx ~]# ls -la /opt/apache-storm-1.1.1/logs
total 58272
drwxr-xr-x 3 root root 4096 Jan 15 01:21 .
drwxr-xr-x 14 root root 4096 Oct 31 10:09 ..
-rw-r--r-- 1 root root 0 Nov 3 10:46 access-logviewer.log
-rw-r--r-- 1 root root 0 Oct 31 10:09 access-nimbus.log
-rw-r--r-- 1 root root 0 Oct 31 10:09 access-supervisor.log
-rw-r--r-- 1 root root 0 Nov 23 14:36 access-ui.log
-rw-r--r-- 1 root root 8916 Nov 27 17:34 access-web-logviewer.log
-rw-r--r-- 1 root root 0 Oct 31 10:09 access-web-nimbus.log
-rw-r--r-- 1 root root 0 Oct 31 10:09 access-web-supervisor.log
-rw-r--r-- 1 root root 31661 Nov 27 17:40 access-web-ui.log
-rw-r--r-- 1 root root 2247 Nov 3 11:34 logviewer.log
-rw-r--r-- 1 root root 0 Nov 3 10:46 logviewer.log.metrics
-rw-r--r-- 1 root root 20690 Oct 31 10:09 nimbus.log
-rw-r--r-- 1 root root 0 Oct 31 10:09 nimbus.log.metrics
-rw-r--r-- 1 root root 46713727 Feb 6 17:13 supervisor.log
-rw-r--r-- 1 root root 3143062 Jan 10 08:23 supervisor.log.1.gz
-rw-r--r-- 1 root root 3104009 Jan 11 22:06 supervisor.log.2.gz
-rw-r--r-- 1 root root 3103550 Jan 13 11:43 supervisor.log.3.gz
-rw-r--r-- 1 root root 3103899 Jan 15 01:21 supervisor.log.4.gz
-rw-r--r-- 1 root root 0 Oct 31 10:09 supervisor.log.metrics
-rw-r--r-- 1 root root 401456 Nov 27 17:40 ui.log
-rw-r--r-- 1 root root 0 Nov 23 14:36 ui.log.metrics
drwxr-xr-x 48 root root 4096 Feb 12 18:08 workers-artifacts
Entering workers-artifacts, I have dirs named after nodes, and inside, I have worker.yaml.
[root#mq1-acustats-process KafkaStormRadius-1-1518455061]# cd 1027
[root#mq1-acustats-process 1027]# ls -la
total 4
drwxr-xr-x 2 root root 24 Feb 12 18:04 .
drwxr-xr-x 8 root root 72 Feb 12 18:04 ..
-rw-r--r-- 1 root root 109 Feb 12 18:04 worker.yaml
[root#mq1-acustats-process 1027]# cat worker.yaml
worker-id: 33e6e77d-8a7b-48b2-b2d4-1be0af726c52
logs.users: []
logs.groups: []
topology.submitter.user: root
So, no logs here. Where are they?
I have this worker.xml under {STORM_DIR}/log4j2:
<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<configuration monitorInterval="60" shutdownHook="disable">
<properties>
<property name="pattern">%d{yyyy-MM-dd HH:mm:ss.SSS} %c{1.} %t [%p] %msg%n</property>
<property name="patternNoTime">%msg%n</property>
<property name="patternMetrics">%d %-8r %m%n</property>
</properties>
<appenders>
<RollingFile name="A1"
fileName="${sys:workers.artifacts}/${sys:storm.id}/${sys:worker.port}/${sys:logfile.name}"
filePattern="${sys:workers.artifacts}/${sys:storm.id}/${sys:worker.port}/${sys:logfile.name}.%i.gz">
<PatternLayout>
<pattern>${pattern}</pattern>
</PatternLayout>
<Policies>
<SizeBasedTriggeringPolicy size="100 MB"/> <!-- Or every 100 MB -->
</Policies>
<DefaultRolloverStrategy max="9"/>
</RollingFile>
<RollingFile name="radius"
fileName="${sys:storm.log.dir}/radius.log"
filePattern="${sys:storm.log.dir}/radius.log.%i.gz">
<PatternLayout>
<pattern>${pattern}</pattern>
</PatternLayout>
<Policies>
<SizeBasedTriggeringPolicy size="50 MB"/> <!-- Or every 20 MB -->
</Policies>
<DefaultRolloverStrategy max="9"/>
</RollingFile>
<RollingFile name="STDOUT"
fileName="${sys:storm.log.dir}/radius.out"
filePattern="${sys:storm.log.dir}/radius.out.%i.gz">
<PatternLayout>
<pattern>${patternNoTime}</pattern>
</PatternLayout>
<Policies>
<SizeBasedTriggeringPolicy size="50 MB"/> <!-- Or every 100 MB -->
</Policies>
<DefaultRolloverStrategy max="4"/>
</RollingFile>
<RollingFile name="STDERR"
fileName="${sys:storm.log.dir}/radius.err"
filePattern="${sys:storm.log.dir}/radius.err.%i.gz">
<PatternLayout>
<pattern>${patternNoTime}</pattern>
</PatternLayout>
<Policies>
<SizeBasedTriggeringPolicy size="50 MB"/> <!-- Or every 100 MB -->
</Policies>
<DefaultRolloverStrategy max="4"/>
</RollingFile>
<RollingFile name="METRICS"
fileName="${sys:storm.log.dir}/radius.metrics"
filePattern="${sys:storm.log.dir}/radius.metrics.%i.gz">
<PatternLayout>
<pattern>${patternMetrics}</pattern>
</PatternLayout>
<Policies>
<SizeBasedTriggeringPolicy size="2 MB"/>
</Policies>
<DefaultRolloverStrategy max="9"/>
</RollingFile>
<Syslog name="syslog" format="RFC5424" charset="UTF-8" host="localhost" port="514"
protocol="UDP" appName="[${sys:storm.id}:${sys:worker.port}]" mdcId="mdc" includeMDC="true"
facility="LOCAL5" enterpriseNumber="18060" newLine="true" exceptionPattern="%rEx{full}"
messageId="[${sys:user.name}:${sys:logging.sensitivity}]" id="storm" immediateFail="true" immediateFlush="true"/>
</appenders>
<loggers>
<root level="info"> <!-- We log everything -->
<appender-ref ref="A1"/>
<appender-ref ref="syslog"/>
</root>
<Logger name="org.apache.storm.metric.LoggingMetricsConsumer" level="info" additivity="false">
<appender-ref ref="METRICS"/>
</Logger>
<Logger name="com.joestelmach.natty" level="error" additivity="false">
<appender-ref ref="radius"/>
</Logger>
<Logger name="STDERR" level="INFO">
<appender-ref ref="STDERR"/>
<appender-ref ref="syslog"/>
</Logger>
<Logger name="STDOUT" level="INFO">
<appender-ref ref="STDOUT"/>
<appender-ref ref="syslog"/>
</Logger>
</loggers>
</configuration>
But, I don't see radius.log created under ${sys:storm.log.dir}.
Why?
By the way:
difference between cluster.xml and worker.xml?

Answer 1

My guess would be that the storm.log.dir variable isn't being correctly set for the worker JVM. Remember that the JVM you start when running storm jar isn't the same JVM that will run your topology.
In order to set VM options for your workers, you can use the worker.childopts variable in storm.yaml to set options globally (you might want to make sure to copy over the defaults from https://github.com/apache/storm/blob/v1.1.1/conf/defaults.yaml#L171 if you do this), or topology.worker.childopts in your topology config to set them per topology.
For example, I get logs printed to E:\testLogs with the following config:
storm.yaml
worker.childopts: "-Dstorm.log.dir=E:\\testLogs"
worker.xml
...
<RollingFile name="A1"
fileName="${sys:storm.log.dir}/worker.log"
filePattern="${sys:workers.artifacts}/${sys:storm.id}/${sys:worker.port}/${sys:logfile.name}.%i.gz">
<PatternLayout>
<pattern>${pattern}</pattern>
</PatternLayout>
<Policies>
<SizeBasedTriggeringPolicy size="100 MB"/> <!-- Or every 100 MB -->
</Policies>
<DefaultRolloverStrategy max="9"/>
</RollingFile>
...
where the rest of the worker.xml is the default one shipping with Storm 1.1.1.
Regarding cluster.xml and worker.xml, cluster.xml is the logging configuration for the Storm daemons (Nimbus, Supervisor, UI etc.), and worker.xml is the logging configuration for the worker processes (the ones running your topology components)

Answer 2

I have read the documentation left by another team and found these lines.... I have a service of storm not launched.
systemctl start storm-supervisor
Checking the service definition under /etc/systemd/system/ (I am with CentOS so I am using systemd), and it reads:
[Unit]
Description=Nimbus Service
After=network.target
[Service]
Type=simple
Restart=always
RestartSec=0s
ExecStart=/opt/apache-storm/bin/storm supervisor
[Install]
WantedBy=multi-user.target
After launching that, I now see log files created under the correct dir.
Worth to mention that I also changed two other things:
Use slf4j instead of log4j.
Change the log4j2.properties to use an absolute path (under /tmp) instead of reading the system variable like {sys:storm.log.dir} in order to discard a variable factor.