I was reading the documentation about kubernetes.io/dockerconfigjson
and I just have a question: Is there any security risk to public publish dockerconfigjson? For example:
data:
.dockerconfigjson: <base64>
Posted community wiki answer for better visibility. Feel free to expand it.
As suggested by David Maze's comment:
I'd expect that to usually contain credentials to access your Docker registry...so yes, it'd be a significant security exposure to publish it?
It's dangerous and not recommended because docker config.json imported to Kubernetes is mainly used for keeping credentials used for pulling images from private registry.
Even if it's saved in base64 format as in example from Kubernetes docs (in your example too) it can be easily decoded:
my-secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: myregistrykey
namespace: awesomeapps
data:
.dockerconfigjson: UmVhbGx5IHJlYWxseSByZWVlZWVlZWVlZWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWxsbGxsbGxsbGxsbGxsbGxsbGxsbGxsbGxsbGxsbGx5eXl5eXl5eXl5eXl5eXl5eXl5eSBsbGxsbGxsbGxsbGxsbG9vb29vb29vb29vb29vb29vb29vb29vb29vb25ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubmdnZ2dnZ2dnZ2dnZ2dnZ2dnZ2cgYXV0aCBrZXlzCg==
type: kubernetes.io/dockerconfigjson
Let's decode it:
user#shell:~/ $ cat my-secret.yaml | yq e '.data.".dockerconfigjson"' - | base64 -d
Really really reeeeeeeeeeaaaaaaaaaaaaaaaaaaaaaaaaaaalllllllllllllllllllllllllllllllyyyyyyyyyyyyyyyyyyyy llllllllllllllooooooooooooooooooooooooooonnnnnnnnnnnnnnnnnnnnnnnnggggggggggggggggggg auth keys
Related
I have created a bunch of secrets using the documented CLI method like so:
echo "ak_prod_4kj56hv24hkjcg56hj2c34k5j3hbj3k124v5h243c" | gcloud secrets versions add some-api-key --data-file=-
I have set my YAML to read them at start-up, this works because my app code will throw if no value is configured.
spec:
template:
spec:
- image:
env:
- name: Some__ApiKey
valueFrom:
secretKeyRef:
key: "1"
name: some-api-key
But my code doesn't work. It was working on Azure, so this isn't a problem with my code. When I call the API, my key is rejected. A key is configured, my code checks that and besides, Cloud Run fails if it cannot read its secrets.
The problem was due to whitespace at the end of the secret.
Somehow a single whitespace character had been introduced. Looking back over my CLI command history it could be trailing whitespace after the --data-file=-
Perhaps it's the space between the " | in Google's example.
The Google console GUI does not present the secret value in quotes and so it is almost impossible to tell this has happened.
One week just on this problem. One week. The cost of badly designed software/bad sample code.
It's actually the echo. You need echo -n.
echo -n "ak_prod_4kj56hv24hkjcg56hj2c34k5j3hbj3k124v5h243c" | gcloud secrets versions add some-api-key --data-file=-
Redhat Openshift autumatically creates a range of user ids that can be used in a given namespace, e.g.
$ oc describe namespace xyz-123
Name: xyz-123
Labels: <none>
Annotations: xx.com/accountID: id-testcluster-account
xx.com/type: System
openshift.io/sa.scc.mcs: s0:c25,c20
openshift.io/sa.scc.supplemental-groups: 1000640000/10000
openshift.io/sa.scc.uid-range: 1000640000/10000
Here is the problem:
While creating docker image, I am setting USER id in Dockerfile:
USER 1001121001:1001121001
I am specifying runAsUser in Helm charts to deploy this image:
runAsUser : 1001121001
When I try to create the deployment, the deployment fails. Because the user ID 1001121001 does not fall in the range above i.e. [1000640000, 1000640000+10000].
The deployment error:
$ oc get deployment abc-123 -n xyz-123 -o yaml
....
....
message: 'pods "abc-123-7f8fc74765-" is forbidden: unable to validate against any security context constraint: [spec.containers[0].securityContext.runAsUser: Invalid value: 1000321001: must be in the ranges: [1000660000, 1000669999]]'
....
....
Tried options - 1:
Using anyuid works as described here : https://www.openshift.com/blog/a-guide-to-openshift-and-uids
But the document says:
"Upon closer inspection of the “anyuid” SCC, it is clear that any user and any group can be used by the Pod launched by a ServiceAccount with access to the “anyuid” SCC. The “RunAsAny” strategy is effectively skipping the default OpenShift restrictions and authorization allowing the Pod to choose any ID."
Hence, I dont want to use this anyuid optiuon.
Tried option-2:
After creating a namespace get the range allowed for that namespace and select an id (say 1000660000) from that range and use that while deploying by setting that id for runAsUser: 1000660000.
All files/folders in the docker image will have the ownership/permissions set to USER 1001121001 and the container started with the id 1000660000 and hence there are issues running the container due to read/write/execute permissions.
To overcome this I need to give o+rwx permissions for all the files, which is risky.
Is there any other way to specify a USER in Dockerfile and use the same USER id during deployment in Redhat Openshift?
$ oc version
Client Version: 4.6.9
Server Version: 4.6.9
Kubernetes Version: v1.19.0+7070803
Solution:
The suggestion from Ritesh worked.
Created the namespace specifying the UID range and covering the specific USER ID. Then created the deployment in this namespace:
Created a namespace with predefined user id range (covering the specific USER id 1001121001) before deploying into the namespace.
apiVersion: v1
kind: Namespace
metadata:
name: xyz-123
annotations:
annotations:
openshift.io/sa.scc.mcs: 's0:c26,c5'
openshift.io/sa.scc.supplemental-groups: 1001120001/10000
openshift.io/sa.scc.uid-range: 1001120001/10000
If you are creating namespace while doing deployment or before then can use following option. Using this you can use runAsUser : 1001121001 (or any other user)
define the yaml file
apiVersion: v1
kind: Namespace
metadata:
name: dspx-dummy-runtimeid
annotations:
openshift.io/sa.scc.mcs: <>
openshift.io/sa.scc.supplemental-groups: <>
openshift.io/sa.scc.uid-range: <>
Use kubectl apply -f <namespace.yaml> or oc apply -f <namespace.yaml>.
I am trying to implement an example application for VerticalPodAutoscaler (VPA) and got this error
error: unable to recognize "foo.yaml": no matches for kind "VerticalPodAutoscaler" in version "autoscaling.k8s.io/v1beta2"
Source code refered: https://medium.com/infrastructure-adventures/vertical-pod-autoscaler-deep-dive-limitations-and-real-world-examples-9195f8422724
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
name: bar
spec:
targetRef:
apiVersion: "apps/v1"
kind: Deployment
name: foo
updatePolicy:
updateMode: "Off"
I also tried combinations of v1, vXbetaY but nothing was working.
Debugging Done:
I tried to search specific examples for Azure AKS VPA but did not find any relevant documentation.
I did this kubectl api-resources | grep autoscaling and ONLY HorizontalPodAutoscaler is present in this list
Any thing that I am missing for getting VPA working on AKS?
well, since its a custom resource, you first need to install it. https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler#installation
Docker image tags are mutable, in that image:latest and image:1.0 can both point to image#sha256:....., but when version 1.1 is released, image:latest stored within a registry can be pointed to an image with a different sha digest. Pulling an image with a particular tag now does not mean that an identical image will be pulled next time.
If a Kubernetes YAMl resource definition refers to an image by tag (not by digest), is there a means of determining what sha digest each image will actually resolve to, before the resource definition is deployed? Is this functionality supported using kustomize or kubectl?
Use case is wanting to determine what has actually been deployed in one environment before deploying to another (I'd like to take a hash of the resolved resource definition and could then use this to understand whether image:1.0 to be deployed to PROD refers to the same image:1.0 that was deployed to UAT).
Are there any tools that can be used to support this functionality?
For example, given the following YAML, is there a way of replacing all images with their resolved digests?
apiVersion: v1
kind: Pod
metadata:
name: example
spec:
containers:
- name: image1
image: image1:1.1
command:
- /bin/sh -c some command
- name: image2
image: image2:2.2
command:
- /bin/sh -c some other command
To get something like this:
apiVersion: v1
kind: Pod
metadata:
name: example
spec:
containers:
- name: image1
image: image1#sha:....
command:
- /bin/sh -c some command
- name: image2
image: image2#sha:....
command:
- /bin/sh -c some other command
I'd like to be able to do something like pipe yaml (that might come from cat, kustomize or kubectl ... --dry-run) through a tool and then pass to kubectl apply -f:
cat mydeployment.yaml | some-tool | kubectl apply -f -
EDIT:
The background to this is the need to be able to prove to auditors/regulators that what is about to be deployed to one env (PROD) is exactly what has been successfully deployed to another env (UAT). I'd like to use normal tags in the deployment template and at the time of deploying to UAT, take a snapshot of the template with the tags replaced with the digests of the resolved images. That snapshot will be what is deployed (via kubectl or similar). When deploying to PROD, that same snapshot will be used.
This tool is supporting exactly what you need...
kbld: https://get-kbld.io/
Resolves name-tag pair reference (nginx:1.17) into digest reference
(index.docker.io/library/nginx#sha256:2539d4344...)
Looks integrates quite well with templating tools like Kustomize or even Helm
You can all the containers used info with this command. This will list all namespaces, with pod names, with container image name and sha256 of the image.
kubectl get pods --all-namespaces -o=jsonpath='{range .items[*]}{"\n"}{.metadata.namespace}{","}{.metadata.name}{","}{range .status.containerStatuses[*]}{.image}{", "}{.imageID}{", "}{end}{end}' | sort
is there a means of determining what sha digest each image will actually resolve to, before the resource definition is deployed?
No, and in the case you describe, it can vary by node. The Deployment will create some number of Pods, each Pod will get scheduled on some Node, and the Kubelet there will only pull the image if it doesn’t have something with that tag already. If you have two replicas, and you’ve changed the image a tag points to, then on node A it could use the older image that was already there, but on node B where there isn’t an image, it will pull and get the newer version.
The best practice here is to avoid changing the image a tag points to. Give each build coming out of your CI system a unique tag (a datestamp or source control commit ID for example) and use that in your Kubernetes object specifications. That avoids this problem entirely.
A workaround is to set
imagePullPolicy: Always
in your pod specs, which will force the node to pull a newer version, but this is unnecessary overhead in most cases.
Here's another on - k8s-digester from google folks. It's a bit different in a sense than the main focus is on cluster-side changes(via Adm Controller) even though client-side KRM functions seems to also be possible.
Overall, kbld seems to be more about development experience and adoption with your cli/CICD/orchestration, while k8s-digester is more about administration on the cluster side.
I have installed spinnaker and kubernetes as suggested in the manual https://www.spinnaker.io/guides/tutorials/codelabs/kubernetes-source-to-prod/
Thing is, I cannot seem to be able to access my docker containers on Docker Hub via Spinnaker in step 3 in the manual.
Here is my spinnaker.yml (the relevant part):
kubernetes:
# For more information on configuring Kubernetes clusters (kubernetes), see
# http://www.spinnaker.io/v1.0/docs/target-deployment-setup#section-kubernetes-cluster-setup
# NOTE: enabling kubernetes also requires enabling dockerRegistry.
enabled: ${SPINNAKER_KUBERNETES_ENABLED:true}
primaryCredentials:
# These credentials use authentication information at ~/.kube/config
# by default.
name: euwest1.aws.crossense.io
dockerRegistryAccount: ${providers.dockerRegistry.primaryCredentials.name}
dockerRegistry:
# For more information on configuring Docker registries, see
# http://www.spinnaker.io/v1.0/docs/target-deployment-configuration#section-docker-registry
# NOTE: Enabling dockerRegistry is independent of other providers.
# However, for convienience, we tie docker and kubernetes together
# since kubernetes (and only kubernetes) depends on this docker provider
# configuration.
enabled: ${SPINNAKER_KUBERNETES_ENABLED:true}
primaryCredentials:
name: crossense
address: ${SPINNAKER_DOCKER_REGISTRY:https://index.docker.io/}
repository: ${SPINNAKER_DOCKER_REPOSITORY:crossense/gator}
username: crossense
# A path to a plain text file containing the user's password
password: password #${SPINNAKER_DOCKER_PASSWORD_FILE}
Thank you guys, in advance, for any and all of the help :)
I believe the issue is that the docker registry does not provide index services. Therefore you need to provide a list of all the images that you want to have available.
dockerRegistry:
enabled: true
accounts:
- name: spinnaker-dockerhub
requiredGroupMembership: []
address: https://index.docker.io
username: username
password: password
email: fake.email#spinnaker.io
cacheIntervalSeconds: 30
repositories:
- library/httpd
- library/python
- library/openjrd
- your-org/your-image
primaryAccount: spinnaker-dockerhub
The halyard commands to execute this is:
export ACCOUNT=spinnaker-dockerhub
hal config provider docker-registry account edit $ACCOUNT --repositories [library/httpd, library/python]
hal config provider docker-registry account edit $ACCOUNT --add-repository library/python
This will update your halyard config file, pending a deploy.
Note, if you do not have access to one of the images, the command will likely fail.
Spinnnaker is really tricky to configure. I have no idea what is your problem but I would recommend you to setup spinnaker using the helm chart, it abstract all the configuration and deployments for you
https://github.com/kubernetes/charts/tree/master/stable/spinnaker
It may be a copy/paste error, but your kubernetes.enabled and dockerRegistry.enabled looks to be mis-indented.