I'm pretty new to this and I was curious as to why this information may have been given to me in a separate .yml file to be used on a RoR app.
I assumed that it was info to the put into my bash profile as it has corresponding environmental variables in the app itself.
BASE_URL: 'http://localhost.com:5000'
development:
MAX_THREADS: '1'
PORT: '5000'
WEB_CONCURRENCY: '1'
test:
I'm also curious as to why you would want to set your url differently as the information states.
Thanks a bunch.
I'd think changing the default port a matter of preference unless there's another part of the stack the development team likes to leave running at 3000 by default, for example, a Node.JS server or other projects.
The .yml file you've been given should be picked up when running bundle exec <command>, but not as part of your bash environment variables.
Related
I am having a very difficult time trying to launch a sample Rails 6 application to Elastic Beanstalk. For context, I am following these instructions
ADD RDS to Ruby Application
ADD an RDS to Beanstalk
I have followed these instructions to a tee and am still unable to connect to the rds database that I have provisioned. I keep receiving the following error:
PG::ConnectionBad: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
Whenever I try to run RAILS_ENV=production rails db:migrate or any other rake task, I keep getting that error.
On my AWS console, under Configuration and Software, I have the following environment variables:
Also in my database.yml file I have the rds configured variables listed as such.
production:
adapter: postgresql
database: <%= ENV['RDS_DB_NAME'] %>
username: <%= ENV['RDS_USERNAME'] %>
password: <%= ENV['RDS_PASSWORD'] %>
host: <%= ENV['RDS_HOSTNAME'] %>
port: <%= ENV['RDS_PORT'] %>
I have mapped my values as instructed in the documentation and am certain that they are correct.
Finally, I have sshed into my beanstalk provisioned ec2 instance and have executed the following command:
psql -U username -p 5432 -h examplehost.rds.amazonaws.com -d ebdb
provided the password and am able to connect. I am really at my wits end, I've spent too much time trying to diagnose this and am running out of ideas. I don't know where to look too next for ideas on how to trouble shoot this. I've read so many stack overflow questions and blogs that my head is spinning. If anyone has any ideas on how to resolve this, I would greatly appreciate it.
---Update----
I have created a new environment variable on the elastic beanstalk console.
ENV['DATABASE_URL'] = postgres://YourUserName:YourPassword#YourHostname:5432/YourDatabaseName
I made the necessary configurations, uploaded my .zip file and the connection to the database failed.
---- UPDATE-----
printenv does not show the varialbes provided by beanstalk, however this command does sudo /opt/elasticbeanstalk/bin/get-config environment.
My first advice is that, in my opinion, it is a much better option to create an Amazon RDS on their own, and not tied to Beanstalk.
As the AWS documentation indicates (emphasis mine):
AWS Elastic Beanstalk provides support for running Amazon Relational Database Service (Amazon RDS) instances in your Elastic Beanstalk environment. To learn about that, see Adding a database to your Elastic Beanstalk environment. This works great for development and testing environments. However, it isn't ideal for a production environment because it ties the lifecycle of the database instance to the lifecycle of your application's environment.
And:
To decouple your database instance from your environment, you can run a database instance in Amazon RDS and configure your application to connect to it on launch. This enables you to connect multiple environments to a database, terminate an environment without affecting the database, and perform seamless updates with blue-green deployments.
In my opinion, even for testing or development, it is always advisable to configure a small database instance and give your application the ability of define the most appropriate mechanism for connecting to your database.
The only downside is that you will probably need to configure a VPC, although it should not be actually a problem and, in ay case, it is worth value.
If for any reason you need to use the Beanstalk provisioned RDS database perhaps you have some workarounds to your problem (it should be a workaround because your configuration looks fine - please, only, verify that the database configuration is defined for the right Beanstalk environment).
For instance, one thing you can try is to store the database connection configuration in a S3 bucket, as also suggested in the AWS documentation. The idea is basically create some configuration file with the necessary connectivity information, store it in S3, and read that configuration in your application, i.e., process that file, in order to initialize your database.
But maybe you can try another approach.
Please, consider this SO question, and the answer from Jon McAuliffe and others. As indicated, Beanstalk will provide your application with environment variables, but maybe this variables will not be exposed as shell variables, they will be exposed to your application in different ways depending on the runtime the application needs to be executed on.
In the case of Ruby, you are accessing these variables in the correct way but, for any reason, your program is not having access to that information.
This probably also explains why printenv does not print any if your variables but the get-config script does.
But maybe you can take advantage of the fact that get-config provides you the right information and, either define this variables in your ENV by executing the get-config script for every RDS* key, perhaps in your environment.rb - please, be aware that I programmed in Ruby when I was a student but there is a long time since that, do the task in the file you consider appropriate - or using .ebextensions and a custom configuration file. You can find several examples here.
For instance, consider the following (copy and paste, with minor modifications of this example configuration):
commands:
01_update_env:
command: "/tmp/update_environment_variables.sh"
files:
"/tmp/update_environment_variables.sh":
mode: "000755"
content : |
#!/bin/bash
RDS_HOSTNAME=$(/opt/elasticbeanstalk/bin/get-config environment -k RDS_HOSTNAME)
if [ -z "$RDS_HOSTNAME" ]; then
echo "Could not determine RDS hostname"
exit 1
fi
echo "RDS hostname $RDS_HOSTNAME..."
# Just export the variable at OS level, or make it visible to
# the rails env in some other way
export RDS_HOSTNAME=$RDS_HOSTNAME
# Process the rest of the variables...
# Probably we should create a list and iterate through it
A similar approach could be the one exposed in this stackoverrun question, but restricted to the container that Beanstalk will use to encapsulate your app. AFAIK, the container should receive as env variables the different RDS* ones corresponding to the database configuration.
Dan, be aware that I have not tested these solutions, they are only ideas: please, be careful with that, I do not want to cause any damage to your system.
I found an answer for this problem with a mysql server that might still help you. Basically, even though I followed all your steps, could see my envars using sudo /opt/elasticbeanstalk/bin/get-config environment and could connect directly to my database with the mysql command, I was still getting the following error:
Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) (Mysql2::Error::ConnectionError)
The solution turned out to be the fact that Elastic Beanstalk was not connecting my envars to my bundle exec rails console command in the eb ssh instance access. I solved the issue by prepending all of the required envars explicitly to any rails commands I ran from within the eb ssh instance access. So for example, in order to run rails console, I had to run the following:
RAILS_MASTER_KEY=xxxxxxx RAILS_ENV=production RDS_HOSTNAME=xxxxxxx RDS_PASSWORD=xxxxxxx RDS_USERNAME=xxxxxxx RDS_DB_NAME=xxxxxxx AWS_REGION=xxxxxxx AWS_BUCKET=xxxxxxx bundle exec rails c
Replace the xxxxxxxs above with the values from the corresponding variables in your EB > Configuration > Software tab, and you should be able to connect to the remote database and run migrations, rake tasks and other database-reliant functions.
For Linux2 instances I was having the same issue and just noticed that the env variables I set in the config just didn't exist for su that I had set myself to -- if I remain the default login after eb ssh env prints everything I expected
edit: sorry -- env printing of variables on linux 2 instance enabled by
https://aws.amazon.com/premiumsupport/knowledge-center/elastic-beanstalk-env-variables-shell/
so what I did was find where those env variables were being exported for default user shell, which was /etc/profile.d/sh.local as noted in the above aws knowledge center link and just source that file when I needed to start the rails console as su
My personal rails project uses a few API's for which I store the API keys/secrets in config/environments/production.yml and development.yml as global variables. I now want to push this project to github for others to use, but I don't want them to have those bits of sensitive data. I also don't want this file in .gitignore because it's required for the app to run. I've considered putting them in the DB somewhere, but am hoping to find a better solution.
TLDR: Use environment variables!
I think #Bryce's comment offers an answer, which I'll just flush out. It seems one approach Heroku recommends is to use environment variables to store sensitive information (API key strings, database passwords). So survey your code and see in which you have sensitive data. Then create environment variables (in your .bashrc file for example) that store the sensivite data values. For example for your database:
export MYAPP_DEV_DB_DATABASE=myapp_dev
export MYAPP_DEV_DB_USER=username
export MYAPP_DEV_DB_PW=secret
Now, in your local box, you just refer to the environment variables whenever you need the sensitive data. For example in database.yml :
development:
adapter: mysql2
encoding: utf8
reconnect: false
database: <%= ENV["MYAPP_DEV_DB_DATABASE"] %>
pool: 5
username: <%= ENV["MYAPP_DEV_DB_USER"] %>
password: <%= ENV["MYAPP_DEV_DB_PW"] %>
socket: /var/run/mysqld/mysqld.sock
I think database.yml gets parsed just at the app's initialization or restart so this shouldn't impact performance. So this would solve it for your local development and for making your repository public. Stripped of sensitive data, you can now use the same repository for the public as you do privately. It also solves the problem if you are on a VPS. Just ssh to it and set up the environment variables on your production host as you did in your development box.
Meanwhile, if your production setup involves a hands off deployment where you can't ssh to the production server, like Heroku's does, you need to look at how to remotely set up environment variables. For Heroku this is done with heroku config:add. So, per the same article, if you had S3 integrated into your app and you had the sensitive data coming in from the environment variables:
AWS::S3::Base.establish_connection!(
:access_key_id => ENV['S3_KEY'],
:secret_access_key => ENV['S3_SECRET']
)
Just have Heroku create environment variables for it:
heroku config:add S3_KEY=8N022N81 S3_SECRET=9s83159d3+583493190
Another pro of this solution is that it's language neutral, not just Rails. Works for any app since they can all acquire the environment variables.
How about this...
Create a new project and check it into GitHub with placeholder values in the production.yml and development.yml files.
Update .gitignore to include production.yml and development.yml.
Replace the placeholder values with your secrets.
Now you can check your code into GitHub without compromising your secrets.
And anyone can clone your repo without any extra steps to create missing files (they'll just replace the placeholder values as you did).
Does that meet your goals?
They're probably best put in initializers (config/initializers/api.yaml) though I think what you've got cooked up is fine. Add the actual keys to your .gitignore file and run git rm config/environments/production.yml to remove that sensitive data from your repo. Fair warning, it will remove that file too so back it up first.
Then, just create a config/environments/production.yml.example file next to your actual file with the pertinent details but with the sensitive data left out. When you pull it out to production, just copy the file without the .example and substitute the appropriate data.
Use environment variables.
In Ruby, they're accessible like so:
ENV['S3_SECRET']
Two reasons:
The values will not make it into source control.
"sensitive data" aka passwords tend to change on a per-environment basis anyways. e.g. you should be using different S3 credentials for development vs production.
Is this a best practice?
Yes: http://12factor.net/config
How do I use them locally?
foreman and dotenv are both easy. Or, edit your shell.
How do I use them in production?
Largely, it depends. But for Rails, dotenv is an easy win.
What about platform-as-a-service?
Any PaaS should give you a way to set them. Heroku for example: https://devcenter.heroku.com/articles/config-vars
Doesn't this make it more complicated to set up a new developer for the project?
Perhaps, but it's worth it. You can always check a .env.sample file into source control with some example data in it. Add a note about it to your project's readme.
Rails 4.1 has now a convention for it. You store this stuff in secrets.yml. So you don't end up with some global ENV calls scattered across Your app.
This yaml file is like database.yml erb parsed, so you are still able to use ENV calls here. In that case you can put it under version control, it would then serve just as a documentation which ENV vars has to be used. But you also can exlcude it from version control and store the actual secrets there. In that case you would put some secrets.yml.default or the like into the public repo for documentation purposes.
development:
s3_secret: 'foo'
production:
s3_secret: <%= ENV['S3_SECRET']%>
Than you can access this stuff under
Rails.application.secrets.s3_secret
Its discussed in detail at the beginning of this Episode
I work with Rails 4 and Ruby 2.1 and sorry but I am working on Windows
I have read a lot about this topic "Missing secret token, secret key base" but actually I do not understang anything.
I do not use Heroku, Git, Puma, Passenger or everything else I've read. I just thought I could instead of running rails s as usual run rails s -e production and see what is the version of my web application in production.
But I have the error "Missing secret_token and secret_key_base for production environment, set these values in config/secrets.yml"
I read about solutions using openSSL, export SECRET_KEY_BASE=<the long string> but I do understand the solutions.
I thought it was a problem related to the system of connection by password I settled thanks to Rails tutorial of Micheal Hartl. So disabled SSL connection. But nothing change.
This is my config/secrets.yml :
production:
secret_key_base: <%= ENV["SECRET_KEY_BASE"] %>
Can someone explain how to concretely solve this issue ?
GDMN I am sorry that everyone gave you such poor explanations and instructions. Ok onto it then, shall we......
First everyone is right you no longer need "secret_token", you do however need "secret_key_base". What this does is it secures your visitors connection and keeps your system and app more secure. That is a simple explanation but all you need to worry about at the beginners level.
Second the ENV stands for "Environment Variable" they are used in all operating systems and they refer to variables on the OS level that hold information that you do not want to be accessible t someone trying to gain access to your site. For instance in Ruby On Rails if you HARD CODE the secrety_token_base string/hash then I hacker can gain access by using your security_token against you. I have seen this happen and it is not pretty, if the individual is skilled enough then they can gain access to even your root/admin account.
Now on to setting it all up. I only know the linux way and I know you are looking for the windows method but this should at least give you an understanding to seek out the information relevant to your operating system.
First thing you would need to do is generate your secret_token_base by running
bundle exec rake secret
To my knowlege this is the way you do it in all Operating Systems. After you run the above command the console will return a string and you would need to copy it. Once copied you would run the following command:
export SECRET_KEY_BASE=WhatYouJustCopied
Then we would check to make sure the Environment Variable SECRET_KEY_BASE is set by running:
env | grep -E "SECRET_TOKEN|SECRET_KEY_BASE"
If you do not have SECRET_TOKEN set you will only get the KEY_BASE.
If you want to learn more in depth information please visit this link it may be a little dated but most of it is still relevant and conceptually it is the same.
I wish you luck on your new found ROR Adventure! It is fun once you get the hang of it!
From your command prompt, run:
bundle exec rake secret
It will generate a long string of characters. Copy this string and paste it into config/secrets.yml as follows:
production:
secret_key_base: <paste the string here>
Note: only do this if you are not using a public repository. This key should not be accessible to anyone else. An alternate, and more secure way of doing this is using an environment variable. See this: http://daniel.fone.net.nz/blog/2013/05/20/a-better-way-to-manage-the-rails-secret-token/
So, if you look inside the secrets.yml file, you will see where your secret_key_base is set for each of your environments. When you look at the production setting, it wants an env variable to initialize your secret_key_base. Normally, in production, you would want your app server to fetch the value from a general place in case you need to spin multiple servers up, you wouldn't have to hard-code your secret_key_base everywhere since that is not a secure way of setting that variable.
Basically, you will have to have that env variable set up on the machine that will run your rails app in production. There are so many different ways to set this.
What I have set up to initialize my ENV variables for production is have a separate yml file that is constructed like this
# config/env_provider.yml
production:
SECRET_KEY_BASE: "KEY GOES HERE"
other_production_variables: #...etc
Then my separate servers will be told where to find this file before initializing the variables (This is not checked into version control). After the file is in place it will know to initialize the variables from the following code in environment.rb before your app initializes
#config/environment.rb
YAML.load_file("#{::Rails.root}/config/env_provider.yml")[::Rails.env].each {|k,v| ENV[k] = v }
# This is before Rails.application.initialize!
The thing with this set up is to make sure that you do not have this file accessible to everyone to see, only allow your application servers to use it. Anyway, this is how I deal with ENV variables and deploy them to production. I hope this helps you.
I want to run the same rails app several times with a few configuration differences on the same server. Each app must have its own:
database
ports
cookie_store key(not sure if needed)
secret_key_base
Let's say I want to run the same code multiple times to service different cities:
newyork.myapp.com and boston.myapp.com. I wonder what would be the best way to store and use the different configurations.
Use environments:
Add a file to config/environments, one for each site you want to host. Name it something along the lines of 'production_[city]', replacing [city] with the city name. Copy the production.rb file contents into each.
in config/database.yml find the 'production' block of yml and duplicate it once for each site you want to host. Rename the root node of each block to production_[city], matching the filenames above. For example:
production_ny:
adapter: mysql2
username: my_user
password: my_pa$$w0rd%&*#
database: production_ny
This takes care of the database settings per app.
Assuming Rails 4, your secret key base will be in config/secrets.yml under an environment node, as per config/database.yml so just add an entry per site:
production_ny:
secret_key_base: xxxxxxxx1111111122222223333333344444444...
All sites will need an end-point. Using a different domain for each would give you separate cookies and sessions for free. Or you could go the subdomain route:
http://guides.rubyonrails.org/configuring.html#deploy-to-a-subdirectory-relative-url-root
tldr:
In your config/environments/production_ny.rb:
config.relative_url_root = "/ny"
for example. Then set up your webserver to handle subdirectories. You may need to add a path to cookies in order to scope them to the virtual directory. Just use:
Rails.configuration.relative_url_root
Ports, again, will have to be set up at the webserver level. (Apache, nginx, etc.)
To see if it all works, try this in the command-line:
RAILS_ENV=production_ny bundle exec rails s
This should start a development style webserver for you to access, but use the production_ny environment.
You will need to create and set up your database as normal - create, migrate, seed.
The final step is setting the RAILS_ENV environment variable to production_[city] per app using your web server. The steps to do this will depend on your technology choice.
In my app I have some code that changes when running locally versus in development on the remote server, such as app_domain in application.yml, etc. Besides having to manually change them each time I run locally versus deploy to remote machine, is there a better way of managing this?
EDIT - for yml
It depends a bit on how you are loading the application.yml but generally when you load yml files ruby parses it all into a hash for you. Thus you can set something up like the database.yml and structure all the yml under your different environments:
development:
domain: localhost
production:
domain: mydomain.com
test:
domain: foo
and then when you load your yml file you want to grab the settings for the specific environment you are in, like this:
MY_CONFIGS = YAML.load_file("[path to my yaml file]/application.yml")[RAILS_ENV]
I like Matthew's answer unless you have some secrets in those files (passwords).
If you have passwords you don't want in source, what I tell my UrbanDeploy customers when they have this type of scenario is to use a script (or we have a built-in script) that effectively does a token replace based on environment. Where the replacements are coming from the deployment system, or a magic file somewhere on the deployment target.
Either way, treat an environment as a first order thing that has data about it that needs to be managed.
Using kwateeSDCM you can customize any file at deployment time on a server by server basis. This way you only need one templatized application.xml and parametrize it depending on the target server.