I deploy my Rails 6 app in a Kubernetes cluster and I think about how to implement my ENV vars.
Regularly in Rails apps I use dotenv with regular ENV vars on the host. But it seems, that I can omit them now and make the use Rails credentials. But just because features exists doesn't mean it has to be used or must be better, right?
So I am not sure how to solve this env/security puzzle:
Approach ConfigMap
create a ConfigMap on the cluster to provide ENV vars
put all ENV vars in the ConfigMap
omit Rails credentials
Approach Credentials
provide a Kubernetes Secret or ConfigMap with the RAILS_MASTER_KEY
use Rails credentials for all vars I need
(Do some ENV-vars have to stay in a ConfigMap like RAILS_ENV?)
The downside I am worry about is, that when I wanna change a ENV var (fix typos, scale workers, switch db, credentials...), I have to pass a lot of steps: make a git push, build and tag the container and wait for a deploy.
With a ConfigMap I simply kubectl apply the change.
I like the Rails way "convention over configuration", so scattering vars to two or three different kinds seems not so practical to mem, but I am afraid, I have to.
Which approach is more secure?
Which one is more "productive" or "developer friendly"?
When to use credentials then?
What's best practice in 2021?
You wouldn't use (only) a ConfigMap since that's not safe but you might use a Secret to hold all the env vars in the same way as you describe. Really its up to which workflow you prefer. No matter how you do it, you have some Kubernetes Secrets object somewhere, just a question of scope for it. So you 100% need to have a workflow for that side of things. But if you prefer the day-to-day secrets edits to be via the Rails workflow so you don't need to touch the Kubernetes side as much, that's cool.
Personally I think having fewer systems touching secrets data is better even if it means the Rails devs need to learn a new tool.
Related
I'm currently migrating my docker deployment to k8s manifests and I was wondering about the handling of secretes. Currently my docker container fetches /run/secrets/app_secret_key to get the sensitive information inside the container as env var. but does that have any benefit in comparison to k8s secrets handling as on the other side I can also do something like this in my manifest.yaml:
env:
- name: MYSQL_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-password
key: password
which than directly brings the secret as a env-variable inside the container ...
The only difference I was able to notice is that if I fetch /run/secrets/app_secret_key inside the container like so (docker-entrypoint.sh):
export APP_SECRET_KEY="$(cat /run/secrets/app_secret_key)"
the env var is not visible when I access the container after deployment, it seems that the env var is only available at the "session" where docker-entrypoint.sh gets initially triggered (at container/pod startup).
So my question now is what does make more sense here: simply go with the env: statement shown above or stay with manual fetching /run/secrets/app_secret_key inside the container ...
Thanks in advance
To be frank both are different implementation of same thing you can choose either one but I will prefer kubernetes approch as mounting secret than container reading at run time simply because of visibility.
Won't matter if you look for one container but when we have 30-40+ microservice running accross 4-5+ environment and have like 100 or even 200 secret. In this case one deployment go wrong we can look at deployments manifest and can figure out entire application. We don't have to search for docker file to understand what happening.
Exposing secret as env var or file is just a flavor to use the secret the k8s way.
Some secret like password is just a one line long string, so it’s convenient to use it as env var. Other secret like ssh private key or TLS certificate can be multiple line, that’s why you can mount the secret as volume instead.
Still, it’s recommended to declare your secret as k8s secret resources. That way you can fetch the value needed via kubectl without having to go inside the container. You can also make a template like helm chart that generate the secret manifests at deployment. With RBAC, you can also control who can read the secret manifests.
As per your comments, yes any user that can go inside the container will have access to the resource available to the shell user.
I have a front-end (React) application. I want to build it and deploy to 3 environments - dev, test and production. As every front-end app it needs to call some APIs. API addresses will vary between the environments. So they should be stored as environment variables.
I utilize S2I Openshift build strategy to create the image. The image should be built and kind of sealed for changes, then before deployment to each particular environment the variables should be injected.
So I believe the proper solution is to have chained, two-stage build. First one S2I which compiles sources and puts it into Nginx/Apache/other container, and second which takes the result of the first, adds environment variables and produces final images, which is going to be deployed to dev, test and production.
Is it correct approach or maybe simpler solution exists?
I would not bake your environmental information into your runtime container image. One of the major benefits of containerization is to use the same runtime image in all of your environments. Generating a different image for each environment would increase the chance that your production deployments behave differently that those you tested in your lower environments.
For non-sensitive information the typical approach for parameterizing your runtime image is to use one or more of:
ConfigMaps
Secrets
Pod Environment Variables
For sensitive information the typical approach is to use:
Secrets (which are not very secret as anyone with root accesss to the hosts or cluster-admin in the cluster rbac can read them)
A vault solution like Hashicorp Vault or Cyberark
A custom solution you develop in-house that meets your security needs
I use an Kubernetes Init container to provision the application's database. After this is done I want to provide the DB's credentials to the main container via environment variables.
How can this be achieved?
I don't want to create a Kubernetes Secret inside the Init container, since I don't want to save the credentials there!
I see several ways to achieve what you want:
From my perspective, the best way is to use Kubernetes Secret. #Nebril has already provided that idea in the comments. You can generate it by Init Container and remove it by PreStop hook, for example. But, you don't want to go that way.
You can use a shared volume which will be used by InitConatainer and your main pod. InitContainer will generate the environment variables file db_cred.env in the volume which you can mount, for example, to /env path. After that, you can load it by modifying a command of your container in the Pod spec and add the command source /env/db_cred.env before the main script which will start your application. #user2612030 already gave you that idea.
Another alternative way can be Vault by Hashicorp, you can use it as storage of all your credentials.
You can use some custom solution to write and read directly to Etcd from Kubernetes apps. Here is a library example - k8s-kv.
But anyway, the best and the most proper way to store credentials in Kubernetes is Secrets. It is more secure and easier than almost any other way.
My open-source app uses AWS's Parameter Store feature to keep secure copies of app secrets (database passwords, etc). When my app is deployed to EC2 a script fetches these secrets and makes them available to the code, and I run this same script locally too.
Some of my tests need database access to run and therefore I need my Travis build script to have access.
Is it safe for me to run that script on my (public) Travis build? As far as I can tell, Travis doesn't expose the build artefacts anywhere (beyond what's on GitHub, which doesn't have my secrets). I know I can encrypt config items in my .travis.yml file but ideally there'd be a single place where this data lives, then I can rotate config keys without updating them in multiple places.
Is there a safe/better way I can do this?
Not really.
If you're accepting pull requests, it's trivially easy to create a pull request that dumps the publicly dumps the keys to the Travis console. Since there's no restrictions on what PRs can modify, edit, etc., wherever the keys are, someone could easily modify the code & print them.
Travis built it secure environment variables to prevent this type of attack, i.e. by not exposing the variables to PRs. That means that tests requiring secure environment variables can't be run with encrypted variables, but that's a trade off that one has to make.
As StephenG has mentioned it is trivial to expose a secret as it is simply an environment variable that the CI systems try to mask.
Example with Bash:
$ SECRET=mysecret
$ rev <<< $SECRET
tercesym
Since the secret is now longer a perfect string match TravisCI, Jenkins, GitHub Actions will not be able to mask you secret anymore and let it be displayed on the console. There are other ways such as uploading a secret to an external server, and so on. For example one could just do env >> debug.log and if that debug.log file is archived, then the secret would be in that file.
Rather than connecting to your real backends, which is not a good idea for CI pipelines anyways, I would much rather recommend that you use a service container.
We do not use TravisCI so I don't how practical it is with Travis, but with Jenkins on Kubernetes and GitHub Actions you can add a service container that runs parallel to your tests.
For example if your integration tests need DB access to mysql or PSQL just run the containers for them. For dynamoDB Amazon provides a container implementation for the explicit purpose of testing your code against the DynamoDB API. If you need more AWS services, LocalStack offers fake AWS core services such as S3.
Now if you actually need to write data to your DBs in AWS ... you should probably expose that data as build artifacts and trigger a notification event instead so that a custom backend on your side can fetch the data on the trigger. Something like a small Lambda function for example.
For the last few months I've managed passwords for my docker containers by putting them in the ENV variables.
Example:
web:
environment:
- PASSWORD=123456
Then I bumped into Docker Secrets. So my questions are:
Which are the reasons why I should use them?
Are they more secure? How?
Can you provide a simple example to show their functionalities?
It depends on a use case.
If you're running one application on your own machine for development that accesses just one secret, you don't need docker secrets.
If you're running dozens of machines in production with a dozen of clustered services all requiring secrets for each other, you do need the secret management.
Apart from security concern, it's just plain easier to have a standardized way of accessing, creating and removing your secrets.
a basic docker inspect (among other things) will show all your environment variables, so this is not secure at all.
You can also have a look at
keywhiz
square.github.io/keywhiz
or vault
hashicorp.com/blog/vault.html
From: https://github.com/moby/moby/issues/13490
Environment Variables. Environment variables are discouraged, because they are:
Accessible by any proces in the container, thus easily "leaked"
Preserved in intermediate layers of an image, and visible in docker inspect
Shared with any container linked to the container