I have the django application, which I developing locally and deploying to pythonanywhere.
Local Setting.py contain the sqlite-db, local static paths etc. Setting.py on pythonanywhere contain mysql-db, cdn links for static etc.
I want to have one settings file in my git, with some checks like this
if host_type=='pythonanywhere':
DATABASES = { "default": { "ENGINE": "django.db.backends.mysql",
Where is a best practice for settings.py for this?
And which os.environ's are provided by pythonanywhere?
PythonAnywhere doesn't give you a UI for setting environment variables, although you could edit your /home/yourusername/.bashrc file and set one in there using
export MY_ENV_VAR="something"
From your question, I'm guessing you want to avoid storing your database settings in version control. So, as an alternative to environment variables, one technique I've used is to have a file, outside of version control, which contains custom settings:
echo "production_settings.py" >> .gitignore
in settings.py:
#... default dev settings
DATABASES = { "default": {"ENGINE": "django.db.backends.sqlite3"
#...
try:
from production_settings import DATABASES
except ImportError:
pass
So on PythonAnywhere, you can keep your production settings in a file called production_settings.py, and on your dev machine, you do nothing at all.
You could also use this method to set a custom SECRET_KEY on the server, which would be good security practice...
I have been using python-dotenv (from pip) and a ~/.env file (which is excluded from git) on my pythonanywhere account.
Then in my settings.py:
from dotenv import load_dotenv
load_dotenv()
SECRET_KEY = os.getenv('DJANGO_SECRET_KEY')
This means the only file unique to pythonanywhere is the .env file, and the same approach works locally, and (reading env vars directly not with dotenv) on Heroku where a settings.py file excluded from git isn't possible.
Related
Is database.yml the right place to read the AWS keys from bashrc? database.yml sounds like a place only for database configs. Is there a more appropriate place where the AWS configs from bashrc could be read inside my Rails app?
Rails 5.2 onwards
Rails 5.2 has introduced the concept of encrypted credentials. Basically, from Rails 5.2 onwards, there is an encrypted credentials file that is generated on initializing the app in config/credentials.yml.enc. This file is encrypted, and hence, can be pushed to your source control tool. There is also a master.key file which is generated while initializing the app, which can be used to decrypt the credentials file, and make changes to it.
So, credentials for AWS could be added to it as:
aws:
access_key_id: 123
secret_access_key: 345
These keys could be accessed in your app as Rails.application.credentials.aws[:secret_access_key]. Other sensitive config, like credentials to other external services that are being used, can also be added to this config. Check out this blog by Marcelo Casiraghi for more details.
Pre Rails 5.2
There was no concept of a credentials system prior to Rails 5.2. There are a couple of ways in which you could try to come up with a solution to store your configuration.
A. You could create a YAML file for defining your config from scratch.
Create a file called my_config.yml and place it in the config folder. Path: config/my_config.yml
Add whatever configuration is required to the file, in YAML format (s described for AWS above)
Make changes in application.rb to load this file during initialization as follows:
APP_CONFIG = YAML.load(ERB.new(File.new(File.expand_path('../my_config.yml', __FILE__)).read).result)[Rails.env] rescue {}
Using this approach, you will then be able to use APP_CONFIG['aws']['access_key_id'] for the AWS configuration defined above. For this use case, it is strongly recommended to have separate configuration files for development and production environments. The production file should probably not be checked in to version control for security.
B. Another approach would be to use some gems for managing configurations like railsconfig/config
NOTE: To answer the bit about storing this configuration in database.yml, it is strongly recommended to not do so. database.yml is a configuration file for storing configuration related to databases. Separation of concerns really helps while scaling any application, and hence, it is recommended to place such configurations in a separate file, which can be independently maintained, without any reliance on the database config.
Absolutely. The standard place to configure things like AWS would be inside config/initializers. You can create a file in there called aws.rb.
app/
bin/
config/
|__ initializers/
|__ aws.rb
and inside this file you can configure your AWS setup using the environment variables from your bashr
Aws.config.update({
credentials: Aws::Credentials.new('your_access_key_id', 'your_secret_access_key')
})
Files inside this directory are executed on app start, so this configuration will be executed right when your app starts, before it starts handling requests.
It may also be useful to note that the AWS SDK for Ruby will automatically search for specific environment variables to configure itself with. If that's what you're using, and if you have the following environment variables set up in your bashrc
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
then you won't need any additional code in your Rails app to configure AWS. Check out more details here.
I have read about the Twelve Factor App's Config - Section III and searched for a way to do it in NodeJS. It seems that most people recommend dotenv to store configs in environment variables.
However, it seems that dotenv contradicts the Twelve-Factor App, as states:
Another approach to config is the use of config files which are not checked into revision control, such as config/database.yml in Rails. This is a huge improvement over using constants which are checked into the code repo, but still has weaknesses: it’s easy to mistakenly check in a config file to the repo; there is a tendency for config files to be scattered about in different places and different formats, making it hard to see and manage all the config in one place. Further, these formats tend to be language- or framework-specific.
The twelve-factor app stores config in environment variables (often shortened to env vars or env). Env vars are easy to change between deploys without changing any code; unlike config files, there is little chance of them being checked into the code repo accidentally; and unlike custom config files, or other config mechanisms such as Java System Properties, they are a language- and OS-agnostic standard.
Making sense of this statement, it seems that using dotenv, you will create a config file .env which will then export them as environment variables.
Wouldn't this violate the Twelve-Factor App, since the .env file can get accidentally checked into the code repo?
12factor is not violated until somebody actually commits and pushes the .env ;)
The .env file can also be stored outside the repo itself, since a library or app is still has to read the .env file and push the variables into the environment. Depending on your implementation, this can be as simple as changing the path from ".env" to "../.env".
Using .env files can be a good compromise to allow developers to manage environments easily, but still be compatible with better environment practices during deployment. I might have 30-40 12factor-flavored apps running in a virtual machine, and having to manage each environment separately is daunting without a "shim" like .env.
The OP asked a well-thought-out question.
I would say the dotenv contradict the 12-factor, section 3, for 2 reasons.
By definition, i.e. this paragraph: "Another approach to config is the use of config files which are not checked into revision control, ... still has weaknesses: it’s easy to mistakenly check in a config file to the repo; ... (therefore the 12-factor app uses a different approach as) stores config in environment variables", now you see, just because an .env file could/should be declared inside a .gitignore, does not make dotenv exempt from that "easy to mistakenly check in a config file to the repo" scrutiny.
An app could otherwise be fully complied with 12-factor, section 3, if it read config from and only from env var. But a dotenv feature is to allow the app to automatically pickup ./.env if it is available. In that sense, the .env file - despite its deceptive name - IS a config file, through and through. Again, this falls into the config approach category that the 12-factor explicitly avoids.
That being said, dotenv is still one of the viable option. Other options include: managing env vars in docker layer; or in a unix user's .*rc file; or in web server config; or in /etc/profile (quoted from this another SO post). dotenv offers one of the most universal file format (fwiw, docker env_file is equally straightforward although their specs are different), however dotenv is the only solution that "polutes" the target app's code base with one more dependency.
Bottom line, dotenv is fine, it is also funny that many dotenv implementation inherit the claim that it started from the 12-factor app tenet, but only few admits that its .env approach deviates from 12-factor app.
Most version control systems have ways of ignoring certain files.
Git has the .gitignore
SVN has "Special Ignores"
As a side note, these techniques are similarly used to ignore the node_modules directory to avoid needlessly copying files.
I'm working in a team in a Rails project.
Each of us have a local database for development purpose. We have a problem: Everyone have different configuration for the local database. When someone make a commit without reset the /config/database.yml the other members of the team can't use their database because the access is not configured.
Can I have a local configuration not commited? To each one can works without problem and without the need of re-set the file every time? Sometime like the local_settings.py in Django
You can configure the database in Rails via both the config/database.yml file and the DATABASE_URL env var. They are merged but settings from DATABASE_URL take precedence.
A good solution is to check in the config/database.yml with a basic configuration thats just enough to support postgres.app or whatever is the most common solution.
Developers that need other settings can set the DATABASE_URL env var to customize the database configuration. Gems like figaro and dotenv make this easier.
If you have ever wondered this how Heroku manages to use the correct DB no matter what you throw into database.yml and the DATABASE_URL ENV var is how you should be configuring your production DB.
Why should I use ENV vars and not more database_*.yaml files?
The twelve-factor app stores config in environment variables (often
shortened to env vars or env). Env vars are easy to change between
deploys without changing any code; unlike config files, there is
little chance of them being checked into the code repo accidentally;
and unlike custom config files, or other config mechanisms such as
Java System Properties, they are a language- and OS-agnostic standard.
https://12factor.net/config
Add config/database.yml in to the .gitignore file at root path of your rails-app.
Copy config/database.yml with the values you need for production into config/database_example.yml.
Now you can modify your local database and in production you copy config/database_expample.yml to config/database.yml
If the config file is ignored by git, everyone can change it locally without getting tracked by git.
EDIT:
HERE YOU SEE HOW YOU CAN REMOVE FILE FROM TRACKING!!!
Ignore files that have already been committed to a Git repository
I am just learning Phoenix and Elixir I am confused, what is the best way to handle environment variables for multiple machines and environments? I keep running into different approaches, from using System.get_env, .env files and mentions of Mix env's. I also keep reading about problems compiling env variables at deployment.
Does anyone have an explanation of how Mix variables, system environment variables and possible .env files or .secret files should be used for local development, stage and production servers?
I have been working mostly in Rails and Python recently so that maybe a helpful contextual piece.
Thanks for the help,
Cory
I personally stick with what Phoenix is giving you by default, e.g. using config files for different environments. Since config files are meant for application configuration, e.g. configuring database adapters, they are checked into source control. By default, these are controlled by the MIX_ENV environment variable. If you look at the bottom of your main config/config.exs file, you will notice this:
import_config "#{Mix.env}.exs"
From what I understand from the documentation, Mix.env is just a shorthand for getting the MIX_ENV value.
Phoenix comes with config files for "develop", "prod" and "test" (all in the config directory) which you can modify for your own use. You can also easily add more configurations — if you want to have a "staging"-specific configuration, for example, just set MIX_ENV=staging on the relevant server, and create config/staging.exs.
For sensitive information, like API keys, environment variables are better, since they're not checked into source control and can be easily changed. You can access those env variables from within your config files or from anywhere within your application.
Hope that helps!
I've enjoyed using Rails on Heroku, and like that I can adjust the configuration property of a Heroku app without having to commit a change to xyz.yml and redeploy.
It would be nice to completely do away with the Yaml config files in my Rails app, and rely as much as possible on storing configuration in ENV. This goes along with the 12-factor config principle.
However, there are some trade-offs in switching from a Yaml-based configuration management to a Heroku/12-factor-based one.
While it's true that a proliferation of deployments (qa, stage, prod, dev, demo, labs) can lead to a proliferation of Yaml files, it's very easy to copy-paste to create a new configuration profile. I don't see a way to 'copy' configuration profiles from one deployment to another in Heroku.
Storing configuration data in the repo means that, in the case of Heroku, deploying and configuring and application are accomplished in a single operation. If I were to move my configuration out of Yaml files and into ENV variables, I would have to configure my application in a separate step after deployment.
Would like to hear from people who have used 12-factor style configuration in their private applications, and how they have managed lots of configuration variables across lots of deployments.
How do you quickly configure a new deployment?
Where do you keep your authoritative source of configuration variables, if not the repo? How do you distribute it among developers?
Thanks!
What I typically use is Yaml using the ENV and provide defaults. For instance, YAML can be ERB'ed happily to include your ENV vars:
foo:
var: ENV["MY_CONFIG"] || "default_value"
You just need to make sure that you load the Yaml with ERB when you read it:
YAML.load(ERB.new(File.read("#{Rails.root}/config/app_config.yml")).result)
By doing this your code works fine in dev, but also allows you to set config vars in the environment as well.
You can accomplish this relatively easy with some simple shell scripts, iterate existing variables via heroku config or heroku release:info v99, and then set heroku config:set k=v --app
But if its a problem/pain/friction perhaps you have too much inside your env var configuration.
A bit of a late answer, but I believe this is what you are looking for.
I developed a gem called settei, allowing you to use YAML files to configure the app. However the gem will serialize the YAML file as one environment variable during deploy. This way one get the best of both worlds: YAML for ease of management/creating derivative environment, and ENV for 12-factor compliance.