kube-apiserver - invalid bearer token, Token has been invalidated - kubeadm

Question

When I try to start kube-apiserver I get following logs:
I1215 14:18:23.130968 1 controller.go:83] Starting OpenAPI controller
I1215 14:18:23.131021 1 customresource_discovery_controller.go:208] Starting DiscoveryController
I1215 14:18:23.131047 1 naming_controller.go:288] Starting NamingConditionController
I1215 14:18:23.131067 1 establishing_controller.go:73] Starting EstablishingController
I1215 14:18:23.131084 1 nonstructuralschema_controller.go:191] Starting NonStructuralSchemaConditionController
I1215 14:18:23.149275 1 controller_utils.go:1036] Caches are synced for crd-autoregister controller
I1215 14:18:23.191491 1 client.go:354] parsed scheme: ""
I1215 14:18:23.191600 1 client.go:354] scheme "" not registered, fallback to default scheme
I1215 14:18:23.191681 1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{127.0.0.1:2379 0 <nil>}]
I1215 14:18:23.206235 1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
E1215 14:18:23.224439 1 authentication.go:65] Unable to authenticate the request due to an error: [invalid bearer token, Token has been invalidated]
I1215 14:18:23.232453 1 controller.go:606] quota admission added evaluator for: leases.coordination.k8s.io
I1215 14:18:23.234017 1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
I1215 14:18:23.279768 1 cache.go:39] Caches are synced for APIServiceRegistrationController controller
I1215 14:18:23.289645 1 cache.go:39] Caches are synced for autoregister controller
I1215 14:18:23.292993 1 client.go:354] parsed scheme: ""
I1215 14:18:23.293085 1 client.go:354] scheme "" not registered, fallback to default scheme
I1215 14:18:23.293264 1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{127.0.0.1:2379 0 <nil>}]
I1215 14:18:23.294432 1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
I1215 14:18:23.328179 1 cache.go:39] Caches are synced for AvailableConditionController controller
I1215 14:18:23.375123 1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
I1215 14:18:23.834251 1 controller.go:107] OpenAPI AggregationController: Processing item
I1215 14:18:23.834297 1 controller.go:130] OpenAPI AggregationController: action for item : Nothing (removed from the queue).
I1215 14:18:23.849807 1 storage_scheduling.go:128] all system priority classes are created successfully or already exist.
E1215 14:18:24.259439 1 authentication.go:65] Unable to authenticate the request due to an error: [invalid bearer token, Token has been invalidated]
I1215 14:18:24.943441 1 controller.go:130] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue).
E1215 14:18:25.269310 1 authentication.go:65] Unable to authenticate the request due to an error: [invalid bearer token, Token has been invalidated]
E1215 14:18:26.277415 1 authentication.go:65] Unable to authenticate the request due to an error: [invalid bearer token, Token has been invalidated]
E1215 14:18:27.283456 1 authentication.go:65] Unable to authenticate the request due to an error: [invalid bearer token, Token has been invalidated]
E1215 14:18:28.290149 1 authentication.go:65] Unable to authenticate the request due to an error: [invalid bearer token, Token has been invalidated]
E1215 14:18:29.297812 1 authentication.go:65] Unable to authenticate the request due to an error: [invalid bearer token, Token has been invalidated]
E1215 14:18:30.303821 1 authentication.go:65] Unable to authenticate the request due to an error: [invalid bearer token, Token has been invalidated]
E1215 14:18:31.312197 1 authentication.go:65] Unable to authenticate the request due to an error: [invalid bearer token, Token has been invalidated]
E1215 14:18:32.320085 1 authentication.go:65] Unable to authenticate the request due to an error: [invalid bearer token, Token has been invalidated]
E1215 14:18:33.331813 1 authentication.go:65] Unable to authenticate the request due to an error: [invalid bearer token, Token has been invalidated]
E1215 14:18:34.337389 1 authentication.go:65] Unable to authenticate the request due to an error: [invalid bearer token, Token has been invalidated]
E1215 14:18:35.345553 1 authentication.go:65] Unable to authenticate the request due to an error: [invalid bearer token, Token has been invalidated]
E1215 14:18:36.354106 1 authentication.go:65] Unable to authenticate the request due to an error: [invalid bearer token, Token has been invalidated]
I1215 14:18:36.916194 1 client.go:354] parsed scheme: ""
I1215 14:18:36.916270 1 client.go:354] scheme "" not registered, fallback to default scheme
I1215 14:18:36.916389 1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{127.0.0.1:2379 0 <nil>}]
I1215 14:18:36.916756 1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
I1215 14:18:36.933986 1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
I1215 14:18:36.944414 1 client.go:354] parsed scheme: ""
I1215 14:18:36.944474 1 client.go:354] scheme "" not registered, fallback to default scheme
[ more invalid token logs goes from here ]
When I try to run: kubectl get pods:
The connection to the server localhost:8080 was refused - did you specify the right host or port?
My api server config:
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
component: kube-apiserver
tier: control-plane
name: kube-apiserver
namespace: kube-system
spec:
containers:
- command:
- kube-apiserver
- --advertise-address=...
- --allow-privileged=true
- --authorization-mode=Node,RBAC
- --client-ca-file=/etc/kubernetes/pki/ca.crt
- --enable-admission-plugins=NodeRestriction
- --enable-bootstrap-token-auth=true
- --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
- --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
- --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
- --etcd-servers=https://127.0.0.1:2379
- --insecure-port=0
- --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
- --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
- --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
- --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
- --requestheader-allowed-names=front-proxy-client
- --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
- --requestheader-extra-headers-prefix=X-Remote-Extra-
- --requestheader-group-headers=X-Remote-Group
- --requestheader-username-headers=X-Remote-User
- --secure-port=6443
- --service-account-key-file=/etc/kubernetes/pki/sa.pub
- --service-cluster-ip-range=10.96.0.0/12
- --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
- --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
image: k8s.gcr.io/kube-apiserver:v1.15.0
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 8
httpGet:
host: ...
path: /healthz
port: 6443
scheme: HTTPS
initialDelaySeconds: 15
timeoutSeconds: 15
name: kube-apiserver
resources:
requests:
cpu: 250m
volumeMounts:
- mountPath: /etc/ssl/certs
name: ca-certs
readOnly: true
- mountPath: /etc/ca-certificates
name: etc-ca-certificates
readOnly: true
- mountPath: /etc/kubernetes/pki
name: k8s-certs
readOnly: true
- mountPath: /usr/local/share/ca-certificates
name: usr-local-share-ca-certificates
readOnly: true
- mountPath: /usr/share/ca-certificates
name: usr-share-ca-certificates
readOnly: true
hostNetwork: true
priorityClassName: system-cluster-critical
volumes:
- hostPath:
path: /etc/ssl/certs
type: DirectoryOrCreate
name: ca-certs
- hostPath:
path: /etc/ca-certificates
type: DirectoryOrCreate
name: etc-ca-certificates
- hostPath:
path: /etc/kubernetes/pki
type: DirectoryOrCreate
name: k8s-certs
- hostPath:
path: /usr/local/share/ca-certificates
type: DirectoryOrCreate
name: usr-local-share-ca-certificates
- hostPath:
path: /usr/share/ca-certificates
type: DirectoryOrCreate
name: usr-share-ca-certificates
status: {}
Kube controller config:
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
component: kube-controller-manager
tier: control-plane
name: kube-controller-manager
namespace: kube-system
spec:
containers:
- command:
- kube-controller-manager
- --allocate-node-cidrs=true
- --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
- --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
- --bind-address=127.0.0.1
- --client-ca-file=/etc/kubernetes/pki/ca.crt
- --cluster-cidr=10.244.0.0/16
- --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
- --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
- --controllers=*,bootstrapsigner,tokencleaner
- --kubeconfig=/etc/kubernetes/controller-manager.conf
- --leader-elect=true
- --node-cidr-mask-size=24
- --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
- --root-ca-file=/etc/kubernetes/pki/ca.crt
- --service-account-private-key-file=/etc/kubernetes/pki/sa.key
- --use-service-account-credentials=true
image: k8s.gcr.io/kube-controller-manager:v1.15.0
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 8
httpGet:
host: 127.0.0.1
path: /healthz
port: 10252
scheme: HTTP
initialDelaySeconds: 15
timeoutSeconds: 15
name: kube-controller-manager
resources:
requests:
cpu: 200m
volumeMounts:
- mountPath: /etc/ssl/certs
name: ca-certs
readOnly: true
- mountPath: /etc/ca-certificates
name: etc-ca-certificates
readOnly: true
- mountPath: /usr/libexec/kubernetes/kubelet-plugins/volume/exec
name: flexvolume-dir
- mountPath: /etc/kubernetes/pki
name: k8s-certs
readOnly: true
- mountPath: /etc/kubernetes/controller-manager.conf
name: kubeconfig
readOnly: true
- mountPath: /usr/local/share/ca-certificates
name: usr-local-share-ca-certificates
readOnly: true
- mountPath: /usr/share/ca-certificates
name: usr-share-ca-certificates
readOnly: true
hostNetwork: true
priorityClassName: system-cluster-critical
volumes:
- hostPath:
path: /etc/ssl/certs
type: DirectoryOrCreate
name: ca-certs
- hostPath:
path: /etc/ca-certificates
type: DirectoryOrCreate
name: etc-ca-certificates
- hostPath:
path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec
type: DirectoryOrCreate
name: flexvolume-dir
- hostPath:
path: /etc/kubernetes/pki
type: DirectoryOrCreate
name: k8s-certs
- hostPath:
path: /etc/kubernetes/controller-manager.conf
type: FileOrCreate
name: kubeconfig
- hostPath:
path: /usr/local/share/ca-certificates
type: DirectoryOrCreate
name: usr-local-share-ca-certificates
- hostPath:
path: /usr/share/ca-certificates
type: DirectoryOrCreate
name: usr-share-ca-certificates
status: {}
I use k8s in 1.15.0 version
I wonder what is the root cause - how I can refresh token in order to apiserver function correctly?