How to deploy a deployment in another namespace in Kubernetes? - jenkins

I'm using Jenkins deployed on Kubernetes. Jenkins pods are deployed in 'kubernetes-plugin' namespace, and uses service account 'jenkins', which is defined below:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: jenkins
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: jenkins
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get","list","watch"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
- apiGroups: ["", "extensions", "apps"]
resources: ["deployments", "replicasets", "pods"]
verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: jenkins
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: jenkins
subjects:
- kind: ServiceAccount
name: jenkins
But when I use kubectl apply -f web-api-deploy.yaml -n default in the jenkins pipeline, it report the following error:
deployments.extensions "news-app-web-api-dev" is forbidden: User "system:serviceaccount:kubernetes-plugin:jenkins" cannot get deployments.extensions in the namespace "default"
which means: you cannot deploy on namespace 'default' when using service account 'jenkins' in namespace 'kubernetes-plugin'
So is there a way to deploy a deployment in another namespace?? How.

So is there a way to deploy a deployment in another namespace?? How.
If I'm not mistaken, this github project gives steps to run in different namespace. It all boils down to this:
You need to craete ServiceAccount, Role and RoleBinding in different namespace and use it like noted in documentation. Here is relevant part:
Ensure you create the namespaces and roles with the following commands,
then run the tests in namespace kubernetes-plugin with the service account
jenkins (edit src/test/kubernetes/service-account.yml to use a different
service account)
kubectl create namespace kubernetes-plugin-test
kubectl create namespace kubernetes-plugin-test-overridden-namespace
kubectl create namespace kubernetes-plugin-test-overridden-namespace2
kubectl apply -n kubernetes-plugin-test -f src/main/kubernetes/service-account.yml
kubectl apply -n kubernetes-plugin-test-overridden-namespace -f src/main/kubernetes/service-account.yml
kubectl apply -n kubernetes-plugin-test-overridden-namespace2 -f src/main/kubernetes/service-account.yml
kubectl apply -n kubernetes-plugin-test -f src/test/kubernetes/service-account.yml
kubectl apply -n kubernetes-plugin-test-overridden-namespace -f src/test/kubernetes/service-account.yml
kubectl apply -n kubernetes-plugin-test-overridden-namespace2 -f src/test/kubernetes/service-account.yml
Also applicable to your situation is to create new Role and RoleBinding in default namespace referencing jenkins ServiceAccount from kubernetes-plugin namespace like so:
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: role-jenkins-default
namespace: default
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get","list","watch"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
- apiGroups: ["", "extensions", "apps"]
resources: ["deployments", "replicasets", "pods"]
verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: roleb-jenkins-default
namespace: default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: role-jenkins-default
subjects:
- kind: ServiceAccount
name: jenkins
namespace: kubernetes-plugin
Note that role- and roleb- prefixes as well as -deault suffix are added to name for clarity. Same goes for explicitly listing namespace default for easier bookkeeping and clarity.
This change should get you past by the error mentioned in your question.

Related

kubernetes plugin in jenkins is not working, Test connection is failing

Environment running:
My Jenkins master is running in the EC2 instance. Jenkins version:- Jenkins 2.249.1
My Kubernetes cluster is running in EKS
I have installed the Kubernetes plugin(1.27.7) and trying to create a Jenkins agent in k8's. I have created a service account, role, role binding. My Yaml file looks as below.
apiVersion: v1
kind: ServiceAccount
metadata:
name: jenkins-admin
namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: jenkins
namespace: default
labels:
"app.kubernetes.io/name": 'jenkins'
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get","list","watch"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: jenkins-role-binding
namespace: default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: jenkins
subjects:
- kind: ServiceAccount
name: jenkins-admin
namespace: default
In Place of the default namespace, I have given a specific namespace
Below is the attached screenshots of My configuration
I have added credentials as the secret text
Error it is throwing:
Error testing connection https://xxxxxxxx.us-east-1.eks.amazonaws.com: java.io.FileNotFoundException: /var/lib/jenkins/.kube/config (No such file or directory)

Not able to execute GitLab Runner in Kubernetes cluster: cannot create resource "secrets" in API group "" in the namespace "gitlab"

Currently I'm facing the issue:
ERROR: Job failed (system failure):
prepare environment:
setting up credentials:
secrets is forbidden:
User "system:serviceaccount:default:gitlab-runner" cannot create
resource "secrets" in API group "" in the namespace "gitlab"`
after following the official documentation on how to integrate the GitLab Runner.
I'm using the following runner-chart-values.yaml:
# The GitLab Server URL (with protocol) that want to register the runner against
# ref: https://docs.gitlab.com/runner/commands/README.html#gitlab-runner-register
#
gitlabUrl: http://example.domain/
# The Registration Token for adding new runners to the GitLab Server. This must
# be retrieved from your GitLab instance.
# ref: https://docs.gitlab.com/ce/ci/runners/README.html
#
runnerRegistrationToken: "<token>"
# For RBAC support:
rbac:
create: true
rules:
- apiGroups: ["*"]
# Run all containers with the privileged flag enabled
# This will allow the docker:dind image to run if you need to run Docker
# commands. Please read the docs before turning this on:
# ref: https://docs.gitlab.com/runner/executors/kubernetes.html#using-dockerdind
runners:
privileged: true
Any clues what's going on?
Many thanks!
For me adding all necessary roles was the only solution that actually helped.
Here the corresponding runner-chart-values.yaml file:
## GitLab Runner Image
gitlabUrl: http://example.domain/
runnerRegistrationToken: "<token>"
rbac:
create: true
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["list", "get", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get"]
- apiGroups: [""]
resources: ["pods/attach"]
verbs: ["list", "get", "create", "delete", "update"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["list", "get", "create", "delete", "update"]
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["list", "get", "create", "delete", "update"]
runners:
privileged: true
Looks like there is namespace mismatch however you can try this below option
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: gitlab-runner
namespace: gitlab-runner
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["list", "get", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get"]
make sure you are creating the service account of Role to proper namespace.
Command to create Role binding
kubectl create rolebinding --namespace=gitlab-runner gitlab-runner-binding --role=gitlab-runner --serviceaccount=gitlab-runner:default
here is nice documentation : https://medium.com/#ruben.laguna/installing-a-gitlab-runner-on-kubernetes-ac386c924bc8
Here is a complete solution using Helm, I copied the rights proposed by Richard in this answer.
Using the followin template (gitlab-rbac/templates) we could patch a given namespace using:
helm upgrade -i gitlab-rbac-name ./gitlab-rbac \
-n your-namespace-here --create-namespace
Once installed, you can check your current rights with:
kubectl auth can-i create secrets --as=system:serviceaccount:gitlab:default \
-n your-namespace-here
The template gitlab-rbac/templates/rbac.yaml contains the following:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: {{ .Release.Namespace }}-admin
namespace: {{ .Release.Namespace }}
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["list", "get", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get"]
- apiGroups: [""]
resources: ["pods/attach"]
verbs: ["list", "get", "create", "delete", "update"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["list", "get", "create", "delete", "update"]
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["list", "get", "create", "delete", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: gitlab-runner-{{ .Release.Namespace }}-admin
namespace: {{ .Release.Namespace }}
subjects:
- kind: ServiceAccount
name: default
namespace: gitlab
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: {{ .Release.Namespace }}-admin
Note that you might need a lot more rights for your runners, you will probably need to update the rules depending on your pipeline. For example if you allow your template to create namespaces, you will need to add a cluster wide role for this. This means adding the following in the template file:
# ... Role and RoleBinding templates
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: namespace-admin
rules:
- apiGroups: ['']
resources: ['namespaces']
verbs: ['create']
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: gitlab-runner-namespace-admin
subjects:
- kind: ServiceAccount
name: default
namespace: gitlab
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: namespace-admin
The value (gitlab-rbac/values.yaml) file is empty in this example as we only use the namespace argument.
I also got the same error. So I have used this method. It solved my error.
helm install --namespace <NAMESPACE> gitlab-runner -f values.yml --set rbac.create=true gitlab/gitlab-runner
Extending Harsh's answer: Please make sure that you're working under active 'gitlab-runner' namespace or using the key --namespace=gitlab-runner. To switch between active namespaces, please use the following command:
kubens gitlab-runner
So you don't need to use --namespace=gitlab-runner everytime.
JFYI, I've done that steps from the article on my k8s cluster and it works fine for me.
In addition to the other answers. Here is a link to the official documentation which lists what permissions are needed depending on the strategy used:
Link to documentation: https://docs.gitlab.com/runner/executors/kubernetes.html

Deploy Jenkins in EKS cluster

I have a running EKS cluster and deployed the web application in the default namespace, now I'm trying to install Jenkins using k8s manifest file.
Here is list of files which I deployed, when I'm trying to configure Kubernetes cloud in manage Jenkins - configured system I'm not able to validate the test connection.
Note: I'm trying to configure Jenkins using the service account method.
rbac.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: jenkins
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: jenkins
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get","list","watch"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["create","delete","get","list","patch","update"]
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["create","delete","get","list","patch","update"]
- apiGroups: ["apps"]
resources: ["deployments"]
verbs: ["create","delete","get","list","patch","update"]
- apiGroups: [""]
resources: ["services"]
verbs: ["create","delete","get","list","patch","update"]
- apiGroups: [""]
resources: ["ingresses"]
verbs: ["create","delete","get","list","patch","update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: jenkins
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: jenkins
subjects:
- kind: ServiceAccount
name: jenkins
namespace: jenkins
Can someone please help me?
For service account "system:serviceaccount:default:jenkins" to have access to list resource "pods" in API group "" in the namespace "jenkins", change your RoleBinding to:
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: jenkins
namespace: jenkins
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: jenkins
subjects:
- kind: ServiceAccount
name: jenkins
namespace: default
You can use kubectl auth can-i command to test if the service account has access to perform a required function after applying the RoleBinding.

Jenkins agent for connecting Amazone EKS not work

I try to configure Kubernetes agent in my Jenkins for deploy microservices using Jenkins pipeline.
I created Amazone EKS cluster using eksctl commande. After cluster creation a created kubeconfig file for configure secret file credential in Jenkins.
When i try to connect my kubernetes agent to my cluster I have an error :
Error testing connection https://<CLUSTER>.sk1.eu-west-3.eks.amazonaws.com: Failure executing: GET at: https://<CLUSTER>.sk1.eu-west-3.eks.amazonaws.com/api/v1/namespaces/default/pods. Message: pods is forbidden: User "system:anonymous" cannot list resource "pods" in API group "" in the namespace "default". Received status: Status(apiVersion=v1, code=403, details=StatusDetails(causes=[], group=null, kind=pods, name=null, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=pods is forbidden: User "system:anonymous" cannot list resource "pods" in API group "" in the namespace "default", metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Forbidden, status=Failure, additionalProperties={}).
Your config secret does not have enough permission to perform basic task. Please bind the below role in your service account who's token you have used in config secret. Please follow this one
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
namespace: default
name: jenkins-master
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get","list","watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["get","list","watch"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: jenkins-master
namespace: default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: jenkins-master
subjects:
- kind: ServiceAccount
name: jenkins-master //replace your service account name
For more details follow this article.

Kubernetes Cluster Context with Multiple Namespaces

I've a huge pipeline with different developer groups with several permission levels.(For using Jenkins Kubernetes Plugin .)
For example QA teams and Developer teams has different service accounts at kubernetes cluster.
So I need create some connection with kubernetes clusters but every connection I change context of cluster with namespace name .
I want to use multiple namespaces at kubernetes context .
That is my own kubernetes context file .
- context:
cluster: minikube
namespace: user3
user: minikube
How I can handle this problem with kubernetes api call or in yaml files ?
That is my example service account yaml file .
apiVersion: v1
kind: ServiceAccount
metadata:
name: dev
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: dev
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get","list","watch"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: dev
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: dev
subjects:
- kind: ServiceAccount
name: dev
If you want one jenkins to talk to kubernetes API with different service accounts you need to create multiple Jenkins "clouds" in configuration, each with different credentials. Then in your pipeline you set the "cloud" option to choose the right one

Resources