I've built myself a web api that uses a sql database. I've used Visual Studio to create this project, and have the ability to right click and "manage user secrets" on my project file.
Its in user secrets that I've stored my connection string and I dont want to add it to my github (private) repo.
The user secret is a json file.
How do I noe include these secrets? Do I include them in the project, making them a part of the image? Or do I do something fancy with the running instance?
There's many ways to go about doing this, but typically you either:
Extract your secrets from your codebase (GIT repo), inject them through environment variables at container startup and then access them from your application code like any other environment var. This is not the most secure option, but at least your secrets won't be in your VCS anymore.
Pull the secrets from some type of secrets manager (e.g. AWS Secrets Manager) straight from your application code. This is more secure than the first option, but requires more code changes and creates a dependency between your application and your secrets manager.
Related
I am using a GCP service account file as a GitHub Codespaces secret, and I am able to access it from the Codespace container, as explained here.
Now, I want to also support developing locally without GitHub Codespaces but still use VSCode devcontainers.
I also hold the service account file on my local filesystem, but outside of the git repo (for obvious reasons). How should I reference it?
You can use the mounts property in devcontainer.json. Codespaces ignores bind mounts (more info can be found in the documentation) so you should be able to mount the file from your local filesystem without affecting how your Codespaces are built/ run.
Update
I have release an extension on the marketplace to solve this usecase: https://marketplace.visualstudio.com/items?itemName=pomdtr.secrets
It stores the secrets in the user keychain. Since it is a web extension, it runs on the client and also works with devcontainers.
Previous Answer
You can use the terminal.integrated.env.linux to pass the secret in your settings.json file.
You can disable settings sync using the settingsSync.ignoredSettings array:
{
"terminal.integrated.env.linux": {
"GITHUB_TOKEN": "<your-token>"
},
"settingsSync.ignoredSettings": [
"terminal.integrated.env.linux"
]
}
I want to know if there is a suggested approach on how to configure Docker machines using configuration files. I have a service that I configure for several users, it is basically a Django app.
Until now I had a shared base image and a bunch of scripts. When I need to create a new machine for a new user, I create it in Google Cloud Engine using the base image. Then I :
SSH into it
Launch a script that download everything via git and launch all services
Copy required credential files using scp
Is there a way to optimize some steps with Docker (using secrets or some external config management tool)?
Thanks!
I'm new to docker so I have a very simple question: Where do you put your config files?
Say you want to install mongodb. You install it but then you need to create/edit a file. I don't think they fit on github since they're used for deployment though it's not a bad place to store the files.
I was just wondering if docker had any support for storing such config files so you can add them as part of running an image.
Do you have to use swarms?
Typically you'll store the configuration files on the Docker host and then use volumes to bind mount your configuration files in the container. This allows you to separately manage the configuration file from the running containers. When you make a change to the configuration, you can just restart the container.
You can then use a configuration management tool like Salt, Puppet, or Chef to manage copying/storing the configuration file onto the Docker host. Things like passwords can be managed by the secrets capabilities of the tool. When set up this way, changing a configuration file just means you need to restart your container and not build a new image.
Yes, in most cases you definitely want to keep your Dockerfiles in version control. If your org (or you personally) use GitHub for this, that's fine, but stick them wherever your other repos are. One of the main ideas in DevOps is to treat infrastructure as code. In fact, one of the main benefits of something like a Dockerfile (or a chef cookbook, or a puppet file, etc) is that it is "used for deployment" but can also be version-controlled, meaningfully diffed, etc.
I use an Kubernetes Init container to provision the application's database. After this is done I want to provide the DB's credentials to the main container via environment variables.
How can this be achieved?
I don't want to create a Kubernetes Secret inside the Init container, since I don't want to save the credentials there!
I see several ways to achieve what you want:
From my perspective, the best way is to use Kubernetes Secret. #Nebril has already provided that idea in the comments. You can generate it by Init Container and remove it by PreStop hook, for example. But, you don't want to go that way.
You can use a shared volume which will be used by InitConatainer and your main pod. InitContainer will generate the environment variables file db_cred.env in the volume which you can mount, for example, to /env path. After that, you can load it by modifying a command of your container in the Pod spec and add the command source /env/db_cred.env before the main script which will start your application. #user2612030 already gave you that idea.
Another alternative way can be Vault by Hashicorp, you can use it as storage of all your credentials.
You can use some custom solution to write and read directly to Etcd from Kubernetes apps. Here is a library example - k8s-kv.
But anyway, the best and the most proper way to store credentials in Kubernetes is Secrets. It is more secure and easier than almost any other way.
My open-source app uses AWS's Parameter Store feature to keep secure copies of app secrets (database passwords, etc). When my app is deployed to EC2 a script fetches these secrets and makes them available to the code, and I run this same script locally too.
Some of my tests need database access to run and therefore I need my Travis build script to have access.
Is it safe for me to run that script on my (public) Travis build? As far as I can tell, Travis doesn't expose the build artefacts anywhere (beyond what's on GitHub, which doesn't have my secrets). I know I can encrypt config items in my .travis.yml file but ideally there'd be a single place where this data lives, then I can rotate config keys without updating them in multiple places.
Is there a safe/better way I can do this?
Not really.
If you're accepting pull requests, it's trivially easy to create a pull request that dumps the publicly dumps the keys to the Travis console. Since there's no restrictions on what PRs can modify, edit, etc., wherever the keys are, someone could easily modify the code & print them.
Travis built it secure environment variables to prevent this type of attack, i.e. by not exposing the variables to PRs. That means that tests requiring secure environment variables can't be run with encrypted variables, but that's a trade off that one has to make.
As StephenG has mentioned it is trivial to expose a secret as it is simply an environment variable that the CI systems try to mask.
Example with Bash:
$ SECRET=mysecret
$ rev <<< $SECRET
tercesym
Since the secret is now longer a perfect string match TravisCI, Jenkins, GitHub Actions will not be able to mask you secret anymore and let it be displayed on the console. There are other ways such as uploading a secret to an external server, and so on. For example one could just do env >> debug.log and if that debug.log file is archived, then the secret would be in that file.
Rather than connecting to your real backends, which is not a good idea for CI pipelines anyways, I would much rather recommend that you use a service container.
We do not use TravisCI so I don't how practical it is with Travis, but with Jenkins on Kubernetes and GitHub Actions you can add a service container that runs parallel to your tests.
For example if your integration tests need DB access to mysql or PSQL just run the containers for them. For dynamoDB Amazon provides a container implementation for the explicit purpose of testing your code against the DynamoDB API. If you need more AWS services, LocalStack offers fake AWS core services such as S3.
Now if you actually need to write data to your DBs in AWS ... you should probably expose that data as build artifacts and trigger a notification event instead so that a custom backend on your side can fetch the data on the trigger. Something like a small Lambda function for example.