Syslog from AKS cluster nodes - Using App Armor & Seccomp - azure-aks

Question

How can I retrieve /var/log/syslog logs from AKS cluster nodes?
Azure recommends the use of Linux security features App Armor and Seccomp. Both of them produce log entries at /var/log/syslog of each cluster node where a workload running has a profile attached.
I've run a test with a nginx container using both, and I can see the corresponding log entries in my node:
Apr 29 10:37:45 aks-agentpool-31529777-vmss000000 kernel: [ 6248.505152] audit: type=1326 audit(1651228665.090:23650): auid=4294967295 uid=1001 gid=2000 ses=4294967295 pid=2016 comm="docker-entrypoi" exe="/bin/dash" sig=0 arch=c000003e syscall=3 compat=0 ip=0x7f90c03fda67 code=0x7ffc0000
Apr 29 10:37:45 aks-agentpool-31529777-vmss000000 kernel: [ 6248.505154] audit: type=1326 audit(1651228665.090:23651): auid=4294967295 uid=1001 gid=2000 ses=4294967295 pid=2016 comm="docker-entrypoi" exe="/bin/dash" sig=0 arch=c000003e syscall=257 compat=0 ip=0x7f90c03fdb84 code=0x7ffc0000
Apr 29 10:42:46 aks-agentpool-31529777-vmss000000 kernel: [ 6550.189592] audit: type=1326 audit(1651228966.775:25032): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=9028 comm="runc:[2:INIT]" exe="/" sig=0 arch=c000003e syscall=257 compat=0 ip=0x561a3e78fa2a code=0x7ffc0000
Syslog's are supposed to be retrievable through configuration of the monitoring agent at the destination Log Analytics workspace, by enabling the syslog facility:
Agents Configuration.
I've also enabled Container Insights in the cluster and Diagnostic Settings for every log category available. Still, I have no way to see those logs without directly connecting to the node and the node VM appears as a not connected source for the Log Analytics workspace: Workspace Data Sources
Env:
Kubernetes v1.21.9
Node: Linux aks-agentpool-31529777-vmss000000 5.4.0-1074-azure #77~18.04.1-Ubuntu SMP
1 pod with one nginx container + two daemonsets and a configMap to deploy the profiles.
apiVersion: v1
kind: Namespace
metadata:
name: testing
---
apiVersion: v1
kind: ConfigMap
metadata:
name: seccomp-profiles-map
namespace: testing
data:
mynginx: |-
{
"defaultAction": "SCMP_ACT_LOG"
}
---
apiVersion: v1
kind: ConfigMap
metadata:
name: apparmor-profiles
namespace: testing
data:
mynginx: |-
# vim:syntax=apparmor
#include <tunables/global>
profile mynginx flags=(complain) {
#include <abstractions/base>
}
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: seccomp-profiles-loader
# Namespace must match that of the ConfigMap.
namespace: testing
labels:
daemon: seccomp-profiles-loader
spec:
selector:
matchLabels:
daemon: seccomp-profiles-loader
template:
metadata:
name: seccomp-profiles-loader
labels:
daemon: seccomp-profiles-loader
spec:
automountServiceAccountToken: false
initContainers:
- name: seccomp-profile-loader
image: busybox
command: ["/bin/sh"]
# args: ["/profiles/*", "/var/lib/kubelet/seccomp/profiles/"]
args: ["-c", "cp /profiles/* /var/lib/kubelet/seccomp/profiles/"]
securityContext:
privileged: true
volumeMounts:
- name: node-profiles-folder
mountPath: /var/lib/kubelet/seccomp/profiles
readOnly: false
- name: profiles
mountPath: /profiles
readOnly: true
containers:
- name: seccomp-profiles-pause
# https://github.com/kubernetes/kubernetes/tree/master/build/pause
image: gcr.io/google_containers/pause
securityContext:
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
volumes:
- name: node-profiles-folder
hostPath:
path: /var/lib/kubelet/seccomp/profiles
- name: profiles
configMap:
name: seccomp-profiles-map
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: apparmor-loader
# Namespace must match that of the ConfigMap.
namespace: testing
labels:
daemon: apparmor-loader
spec:
selector:
matchLabels:
daemon: apparmor-loader
template:
metadata:
name: apparmor-loader
labels:
daemon: apparmor-loader
spec:
containers:
- name: apparmor-loader
image: google/apparmor-loader:latest
args:
# Tell the loader to pull the /profiles directory every 30 seconds.
- -poll
- 60s
- /profiles
securityContext:
# The loader requires root permissions to actually load the profiles.
privileged: true
volumeMounts:
- name: sys
mountPath: /sys
readOnly: true
- name: apparmor-includes
mountPath: /etc/apparmor.d
readOnly: true
- name: profiles
mountPath: /profiles
readOnly: true
volumes:
# The /sys directory must be mounted to interact with the AppArmor module.
- name: sys
hostPath:
path: /sys
# The /etc/apparmor.d directory is required for most apparmor include templates.
- name: apparmor-includes
hostPath:
path: /etc/apparmor.d
# Map in the profile data.
- name: profiles
configMap:
name: apparmor-profiles
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
namespace: testing
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
annotations:
container.apparmor.security.beta.kubernetes.io/nginx: localhost/mynginx
spec:
automountServiceAccountToken: false
nodeSelector:
"kubernetes.io/os": linux
containers:
- name: nginx
securityContext:
seccompProfile:
type: Localhost
localhostProfile: profiles/mynginx
image: nginx
imagePullPolicy: Always
ports:
- containerPort: 3000
env:
- name: PORT
value: '3000'
---
apiVersion: v1
kind: Service
metadata:
name: nginx
annotations:
service.beta.kubernetes.io/azure-load-balancer-internal: "true"
spec:
type: LoadBalancer
ports:
- port: 3000
selector:
app: nginx