I am trying to create a Private Container Registry on DigitalOcean Kubernetes. And I want all data to be saved in the DigitalOcean Spaces. I am using this tutorial:
https://www.digitalocean.com/community/tutorials/how-to-set-up-a-private-docker-registry-on-top-of-digitalocean-spaces-and-use-it-with-digitalocean-kubernetes
Things and pod are running well, I am able to push or pull images and I would like to configure basic auth (htpasswd) on top on it, but when I add htpasswd attribute to my chart values file, I am getting error:
{"level":"fatal","msg":"configuring application: unable to configure authorization (htpasswd): no access controller registered with name: htpasswd","time":"2022-12-14T13:02:23.608Z"}
My chart_values.yaml file:
ingress:
enabled: true
hosts:
- cr.somedomain.com
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
nginx.ingress.kubernetes.io/proxy-body-size: "30720m"
args:
- --set controller.extraArgs.ingress-class=nginx
tls:
- secretName: somedomain-cr-prod
hosts:
- cr.somedomain.com
storage: s3
secrets:
htpasswd: |-
username:someBcryptPassword
s3:
accessKey: "someaccesskey"
secretKey: "someaccesssecret"
s3:
region: region
regionEndpoint: region.digitaloceanspaces.com
secure: true
bucket: somebucketname
image:
repository: somerepo
tag: latest
Maybe someone can answer where did I go wrong?
I have tried different formats to enter htpasswd, but it did produce the same error.
Related
Assume there is a backend application with a private key stored in a .env file.
For the project file structure:
|-App files
|-Dockerfile
|-.env
If I run the docker image locally, the application can be reached normally by using a valid public key during the API request. However, if I deploy the container into AKS cluster by using same docker image, the application failed.
I am wondering how the container in a AKS cluster handle the .env file. What should I do to solve this problem?
Moving this out of comments for better visibility.
First and most important is docker is not the same as kubernetes. What works on docker, won't work directly on kubernetes. Docker is a container runtime, while kubernetes is a container orchestration tool which sits on top of docker (not always docker now, containerd is used as well).
There are many resources on the internet which describe the key difference. For example this one is from microsoft docs
First configmaps and secrets should be created:
Creating and managing configmaps and creating and managing secrets
There are different types of secrets which can be created.
Use configmaps/secrets as environment variables.
Further referring to configMaps and secrets as environment variables looks like (configmaps and secrets have the same structure):
apiVersion: v1
kind: Pod
metadata:
name: pod-example
spec:
containers:
- ...
env:
-
name: ADMIN_PASS
valueFrom:
secretKeyRef: # here secretref is used for sensitive data
key: admin
name: admin-password
-
name: MYSQL_DB_STRING
valueFrom:
configMapKeyRef: # this is not sensitive data so can be used configmap
key: db_config
name: connection_string
...
Use configmaps/secrets as volumes (it will be presented as file).
Below the example of using secrets as files mounted in a specific directory:
apiVersion: apps/v1
kind: Deployment
metadata:
...
spec:
containers:
- ...
volumeMounts:
- name: secrets-files
mountPath: "/mnt/secret.file1" # "secret.file1" file will be created in "/mnt" directory
subPath: secret.file1
volumes:
- name: secrets-files
secret:
secretName: my-secret # name of the Secret
There's a good article which explains and shows use cases of secrets as well as its limitations e.g. size is limited to 1Mb.
I have a Python web service that collects data from frontend clients. Every few seconds, it creates a Pulsar producer on our topic and sends the collected data. I have also set up a dockerfile to build an image and am working on deploying it to our organization's Kubernetes cluster.
The Pulsar code relies on certificate and key .pem files for TLS authentication, which are loaded over file paths in the test code. However, if the .pem files are included in the built Docker image, it will result in an obvious compliance violation from the Twistlock scan on our Kubernetes instance.
I am pretty inexperienced with Docker, Kubernetes, and security with certificates in general. What would be the best way to store and load the .pem files for use with this web service?
You can mount certificates in the Pod with Kubernetes secret.
First, you need to create a Kubernetes secret:
(Copy your certificate to somewhere kubectl is configured for your Kubernetes cluster. For example file mykey.pem and copy it to the /opt/certs folder.)
kubectl create secret generic mykey-pem --from-file=/opt/certs/
Confirm it was created correctly:
kubectl describe secret mykey-pem
Mount your secret in your deployment (for example nginx deployment):
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: nginx
name: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
strategy:
type: Recreate
template:
metadata:
labels:
app: nginx
spec:
containers:
- image: nginx
name: nginx
volumeMounts:
- mountPath: "/etc/nginx/ssl"
name: nginx-ssl
readOnly: true
ports:
- containerPort: 80
volumes:
- name: nginx-ssl
secret:
secretName: mykey-pem
restartPolicy: Always
After that .pem files will be available inside the container and you don't need to include them in the docker image.
I've deployed my first app on my Kubernetes prod cluster a month ago.
I could deploy my 2 services (front / back) from gitlab registry.
Now, I pushed a new docker image to gitlab registry and would like to redeploy it in prod:
Here is my deployment file:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
reloader.stakater.com/auto: "true"
labels:
app: espace-client-client
name: espace-client-client
namespace: espace-client
spec:
replicas: 1
strategy: {}
template:
metadata:
labels:
app: espace-client-client
spec:
containers:
- envFrom:
- secretRef:
name: espace-client-client-env
image: registry.gitlab.com/xxx/espace_client/client:latest
name: espace-client-client
ports:
- containerPort: 3000
resources: {}
restartPolicy: Always
imagePullSecrets:
- name: gitlab-registry
I have no clue what is inside gitlab-registry. I didn't do it myself, and the people who did it left the crew :( Nevertheless, I have all the permissions, so, I only need to know what to put in the secret, and maybe delete it and recreate it.
It seems that secret is based on my .docker/config.json
➜ espace-client git:(k8s) ✗ kubectl describe secrets gitlab-registry
Name: gitlab-registry
Namespace: default
Labels: <none>
Annotations: <none>
Type: kubernetes.io/dockerconfigjson
Data
====
.dockerconfigjson: 174 bytes
I tried to delete existing secret, logout with
docker logout registry.gitlab.com
kubectl delete secret gitlab-registry
Then login again:
docker login registry.gitlab.com -u myGitlabUser
Password:
Login Succeeded
and pull image with:
docker pull registry.gitlab.com/xxx/espace_client/client:latest
which worked.
file: ~/.docker/config.json is looking weird:
{
"auths": {
"registry.gitlab.com": {}
},
"HttpHeaders": {
"User-Agent": "Docker-Client/18.09.6 (linux)"
},
"credsStore": "secretservice"
}
It doesn't seem to contain any credential...
Then I recreate my secret
kubectl create secret generic gitlab-registry \
--from-file=.dockerconfigjson=/home/julien/.docker/config.json \
--type=kubernetes.io/dockerconfigjson
I also tried to do :
kubectl create secret docker-registry gitlab-registry --docker-server=registry.gitlab.com --docker-username=<your-name> --docker-password=<your-pword> --docker-email=<your-email>
and deploy again:
kubectl rollout restart deployment/espace-client-client -n espace-client
but I still have the same error:
Error from server (BadRequest): container "espace-client-client" in pod "espace-client-client-6c8b88f795-wcrlh" is waiting to start: trying and failing to pull image
You have to update the gitlab-registry secret because this item is used to let Kubelet to pull the protected image using credentials.
Please, delete the old secret with kubectl -n yournamespace delete secret gitlab-registry and recreate it typing credentials:
kubectl -n yournamespace create secret docker-registry my-secret --docker-server=DOCKER_REGISTRY_SERVER --docker-username=DOCKER_USER --docker-password=DOCKER_PASSWORD[ --docker-email=DOCKER_EMAIL]
where:
- DOCKER_REGISTRY_SERVER is the GitLab Docker registry instance
- DOCKER_USER is the username of the robot account to pull images
- DOCKER_PASSWORD is the password attached to the robot account
You could ignore docker-email since it's not mandatory (note the square brackets).
I've tried https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
and the base 64 encoded solution in a yaml file (which is ultimately what I need to do) doesn't authenticate. (apparently this is a common problem and if anyone's got a yaml file that has it working I'd love to see it or a method that allows secure deployment from a private repo, just so we don't get stuck in the x-y problem)
So I tried the following:
kubectl create secret generic registrykey --from-file=.dockerconfigjson=/home/dbosh/.docker/config.json --type=kubernetes.io/dockerconfigjson
with the deployment file:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my_deployment
spec:
selector:
matchLabels:
app: my_deployment
tier: backend
track: stable
replicas: 7
template:
metadata:
labels:
app: my_deployment
tier: backend
track: stable
spec:
containers:
- name: my_deployment
image: "my_private_repo:image_name"
ports:
- name: http
containerPort: 8082
imagePullSecrets:
- name: registrykey
However whenever I try to deploy, I keep getting that the "pull access denied for my_private_repo, repository does not exist or may require 'docker login".
Now to create the docker login file, I have indeed logged in and tested again with logging in immediately before regenerating the secret and then redeploying and it still doesn't authenticate.
Any help appreciated please.
UPDATE (thanks to a useful comment):
It would appear that my config.json after logging in looks likethis:
cat .docker/config.json
{
"auths": {
"https://index.docker.io/v1/": {}
},
"HttpHeaders": {
"User-Agent": "Docker-Client/18.09.2 (linux)"
},
"credsStore": "secretservice"
This would appear to not contain a token. I generated this from running docker login and providing my credentials. Any ideas anyone?
There's no token for your private repo in the config.json file, but only for docker hub.
So you need to re-authenticate within your private repository:
docker logout <my_private_repo> && docker login <my_private_repo> -u <user> -p <pass> && cat ~/.docker/config.json
Should be a bit of this:
"auths": {
"my_private_repo": {
"auth": "c3VraG92ZXJzsdfdsQXNocmV2b2h1czg4"
}
I am trying to push a docker image into a private registry in Drone 0.8.5 and it works when I hardcode username and password into the pipeline however I have tried adding both the registry details in the registry tab and as secrets.
Registry Pipeline
docker-registry-push:
image: plugins/docker
repo: registry.domain.com:5000/app
registry: registry.domain.com:5000
insecure: true
pull: true
Fails with no basic auth credentials
Finally I've tried variable substitution. (with $REGISTRY_USERNAME and $$REGISTRY_USERNAME variables. All result in a error msg="Error authenticating: exit status 1"
docker-registry-push:
image: plugins/docker
repo: registry.domain.com:5000/app
registry: registry.domain.com:5000
secrets:
- source: registry_username
target: username
- source: registry_password
target: password
insecure: true
pull: true
another attempt
docker-registry-push:
image: plugins/docker
repo: registry.domain.com:5000/app
registry: registry.domain.com:5000
username: ${REGISTRY_USERNAME}
password: ${REGISTRY_PASSWORD}
secrets: [ registry_username, registry_password ]
insecure: true
pull: true
It is really frustrating. I need to add secrets for Rancher accesskey secretkey also after this via the correct method.
I have read other topics and the drone docs and am still stumped.
Thanks in advance.
The secrets need to be injected into the docker container via the environment with the names docker_username and and docker_password.
Your .drone.yml file should look something like this:
pipeline:
docker:
image: plugins/docker
repo: username/app
registry: registry.domain.com:5000
insecure: true
pull: true
secrets:
- source: registry_username
target: docker_username
- source: registry_password
target: docker_password
See the drone plugin docs for more configuration options.
here is to manage drone secret key http://docs.drone.io/manage-secrets/#pull-requests
also, you might wanna consider using .netrc inside Dockerfile on your build, so your credential is embeded inside of your docker images