My open-source app uses AWS's Parameter Store feature to keep secure copies of app secrets (database passwords, etc). When my app is deployed to EC2 a script fetches these secrets and makes them available to the code, and I run this same script locally too.
Some of my tests need database access to run and therefore I need my Travis build script to have access.
Is it safe for me to run that script on my (public) Travis build? As far as I can tell, Travis doesn't expose the build artefacts anywhere (beyond what's on GitHub, which doesn't have my secrets). I know I can encrypt config items in my .travis.yml file but ideally there'd be a single place where this data lives, then I can rotate config keys without updating them in multiple places.
Is there a safe/better way I can do this?
Not really.
If you're accepting pull requests, it's trivially easy to create a pull request that dumps the publicly dumps the keys to the Travis console. Since there's no restrictions on what PRs can modify, edit, etc., wherever the keys are, someone could easily modify the code & print them.
Travis built it secure environment variables to prevent this type of attack, i.e. by not exposing the variables to PRs. That means that tests requiring secure environment variables can't be run with encrypted variables, but that's a trade off that one has to make.
As StephenG has mentioned it is trivial to expose a secret as it is simply an environment variable that the CI systems try to mask.
Example with Bash:
$ SECRET=mysecret
$ rev <<< $SECRET
tercesym
Since the secret is now longer a perfect string match TravisCI, Jenkins, GitHub Actions will not be able to mask you secret anymore and let it be displayed on the console. There are other ways such as uploading a secret to an external server, and so on. For example one could just do env >> debug.log and if that debug.log file is archived, then the secret would be in that file.
Rather than connecting to your real backends, which is not a good idea for CI pipelines anyways, I would much rather recommend that you use a service container.
We do not use TravisCI so I don't how practical it is with Travis, but with Jenkins on Kubernetes and GitHub Actions you can add a service container that runs parallel to your tests.
For example if your integration tests need DB access to mysql or PSQL just run the containers for them. For dynamoDB Amazon provides a container implementation for the explicit purpose of testing your code against the DynamoDB API. If you need more AWS services, LocalStack offers fake AWS core services such as S3.
Now if you actually need to write data to your DBs in AWS ... you should probably expose that data as build artifacts and trigger a notification event instead so that a custom backend on your side can fetch the data on the trigger. Something like a small Lambda function for example.
Related
I'm exploring how best to use Github Codespaces for my organization. Our dev environment consists of a Docker dev environment that we run on local machines. It relies on pulling other private repos we maintain via the local machine's ssh-agent. I'd ideally like to keep things as consistent as possible and have our Codespaces solution use the same Docker dev environment from within the codespace.
There's a naive solution of just building a new codespace with no devcontainer.json and going through all the setup for a dev environment each time you create a new one... but I'd like to avoid this. Ideally, I keep the same dev experience and am able to get the codespace to prebuild by building the docker image and somehow getting access to our other private repos.
An extremely hacky-feeling solution that works for automated building is creating an ssh key and storing it as a user codespace secret, then setting up the ssh-agent with that ssh-key as part of the postCreateCommand. My understanding is that this would not work with the onCreateCommand because "it will not typically have access to user-scoped assets or secrets.". To reiterate, this works for automated building, but not pre-building.
From this Github issue it looks like cloning via ssh is a complete no-go with prebuilds because ssh will need a user-defined ssh key, which isn't available from the onCreateCommand. The only potential workaround I can see for this is having an organization-wide read-only ssh-key... which seems potentially even sketchier than having user-created ssh keys as user secrets.
The other possibility I can think of is switching to https for the git clones. This would require adding access to the other repos, which is no big deal. BUT I can't quite see how to get access from within the docker image. When I tried this, I was getting errors because I was asked for a username and password when I ran a git clone from within docker... even though git clone worked fine in the base codespace. Is there a way to forward whatever tokens Github uses for access to other repos into the docker build process? Is there a way to have user-generated tokens get passed into the docker build process and use that for access instead?
Thoughts and roasts welcome.
I've built myself a web api that uses a sql database. I've used Visual Studio to create this project, and have the ability to right click and "manage user secrets" on my project file.
Its in user secrets that I've stored my connection string and I dont want to add it to my github (private) repo.
The user secret is a json file.
How do I noe include these secrets? Do I include them in the project, making them a part of the image? Or do I do something fancy with the running instance?
There's many ways to go about doing this, but typically you either:
Extract your secrets from your codebase (GIT repo), inject them through environment variables at container startup and then access them from your application code like any other environment var. This is not the most secure option, but at least your secrets won't be in your VCS anymore.
Pull the secrets from some type of secrets manager (e.g. AWS Secrets Manager) straight from your application code. This is more secure than the first option, but requires more code changes and creates a dependency between your application and your secrets manager.
I am working on a migration task from an on-premise system to a cloud composer, the thing is that Cloud composer is a fully managed version of airflow which restrict access to file systems behind, actually on my on-premise system I have a lot of environment variables for some paths we're saving them like /opt/application/folder_1/subfolder_2/....
When looking at the Cloud composer documentation, they said that you can access and save your data on the data folder which is mapped by /home/airflow/gcs/data/ which implies that in case I move forward that mapping, I will be supposed to change my env variables values to something like : /home/airflow/gcs/data/application/folder_1/folder_2 things that could be a bit painful, knowing that I'm running many bash scripts that rely on those values.
Is there any approach to solve such problem ?
You can specify your env variables during Composer creation/update process [1]. These vars are then stored in the YAML files that create the GKE cluster where Composer is hosted. If you SSH into a VM running the Composer GKE cluster, then enter one of the worker containers and run env, you can see the env variables you specified.
[1] https://cloud.google.com/composer/docs/how-to/managing/environment-variables
I have a front-end (React) application. I want to build it and deploy to 3 environments - dev, test and production. As every front-end app it needs to call some APIs. API addresses will vary between the environments. So they should be stored as environment variables.
I utilize S2I Openshift build strategy to create the image. The image should be built and kind of sealed for changes, then before deployment to each particular environment the variables should be injected.
So I believe the proper solution is to have chained, two-stage build. First one S2I which compiles sources and puts it into Nginx/Apache/other container, and second which takes the result of the first, adds environment variables and produces final images, which is going to be deployed to dev, test and production.
Is it correct approach or maybe simpler solution exists?
I would not bake your environmental information into your runtime container image. One of the major benefits of containerization is to use the same runtime image in all of your environments. Generating a different image for each environment would increase the chance that your production deployments behave differently that those you tested in your lower environments.
For non-sensitive information the typical approach for parameterizing your runtime image is to use one or more of:
ConfigMaps
Secrets
Pod Environment Variables
For sensitive information the typical approach is to use:
Secrets (which are not very secret as anyone with root accesss to the hosts or cluster-admin in the cluster rbac can read them)
A vault solution like Hashicorp Vault or Cyberark
A custom solution you develop in-house that meets your security needs
I have a Powershell script that uploads audit logs to an S3 repository. The script works fine when I run it while logged in but I need to define a scheduled task and the task needs to be run as SYSTEM user. Can someone recommend a way that I can provide the SYSTEM user with the AWS credentials so that they are not stored on the machine in clear text?
I just found what I think is the answer: if I run the script thru the task scheduler once with the 'Set-AWSCredentials' command it creates the encrypted key info in C:\Windows\System32\config\systemprofile\AppData\Local\AWSToolkit\RegisteredAccounts.json. Then I was able to remove the 'Set-AWSCredentials' command and the script seems to run successfully.
I don't believe this is possible. Your authentication info will have to be stored in the clear on the client machine one way or another, and windows doesn't provide any convenient methods for protecting that information.
You might find it more convenient to manage access if you use ssh + encryption certificates as credentials (i believe there's an ssh client for powershell, though i haven't tried using it for aws work)