I've got a local .csv file with updated user data that I want to run a simple one-time script on in production (on Heroku, Rails 4 app). It's open source, so I can't just include the file in the repo without exposing the data. I'd like to be able to do it from the CLI but can't seem to figure it out.
There's probably a better solution, but I've been trying unsuccessfully to set the contents of the file (which isn't huge) to an environment variable on Heroku. My Bash skills are weak...
$ heroku config:set MY_CSV=<./my_local_csv_path.csv
didn't work -- it will not set anything. (running $ heroku config shows the env variable blank -- MY_CSV:).
Is there a better way to make this .csv file accessible to a script on Heroku? I suspect it's a similar problem to just accessing the local filesystem from the Heroku console.
I just did something similar to what you are doing where the user uploads a CSV for processing in my app. I am also hosting on heroku. Check out...
http://railscasts.com/episodes/396-importing-csv-and-excel
Basically let your model doing your processing for you that way you don't have to store any files on heroku.
I was able to do this with the heroku cli -e flag. It has to be a relatively small file. (all arguments combined are limited to 32kb)
It goes something like this:
# gzip and base64 encode the file
# (without the encoding, the gzip characters break this argument)
FILE_CONTENTS=$(cat /path/to/file.csv | gzip -fc | base64)
# Pass the variable from above to the dyno above.
heroku run -a APP_NAME -e "FILE_CONTENTS=\"$FILE_CONTENTS\"" rails c
Then in your console, you can get the csv by running:
csv_contents = ActiveSupport::Gzip.decompress(
Base64.decode64(
ENV['FILE_CONTENTS']
)
)
Related
So I am trying to get my rails app to deploy in production mode, but I get the error: Missing secret_token and secret_key_base for 'production' environment, set these values in config/secrets.yml
My secrets.yml file is as expected:
development:
secret_key_base: xxxxxxx
test:
secret_key_base: xxxxxxx
production:
secret_key_base: <%= ENV["SECRET_KEY_BASE"] %>
But even after google and research, I have no idea what to do with the production secret key base. Most of the info out there assumes I have certain background knowledge, but the reality is that I'm a noob.
Can anyone explain to me how to set my secret key and get this to work in production mode?
You can generate the key by using following commands
$ irb
>> require 'securerandom'
=> true
>> SecureRandom.hex(64)
=> "3fe397575565365108556c3e5549f139e8078a8ec8fd2675a83de96289b30550a266ac04488d7086322efbe573738e7b3ae005b2e3d9afd718aa337fa5e329cf"
>> exit
The errors you are getting just indicate that the environment variable for secret_key_base are not properly set on the server.
You can use various scripts like capistrano that automate the process of setting these before the application is run.
As for a quick fix try this:
export SECRET_KEY_BASE=YOUR SECRET BASE
Validate the environment variables and check if these have been set.
Command:
env | grep -E "SECRET_TOKEN|SECRET_KEY_BASE"
If your values pop up then these are set on the production server.
Also it is best practice to use ENV.fetch(SECRET_KEY) as this will raise an exception before the app even tries to start.
This answer helped me a lot. He indicates you how to config the secrets.yml file in production and how to read it from the environment:
original link:
https://stackoverflow.com/a/26172408/4962760
I had the same problem and I solved it by creating an environment
variable to be loaded every time that I logged in to the production
server and made a mini guide of the steps to configure it:
https://gist.github.com/pablosalgadom/4d75f30517edc6230a67
I was using Rails 4.1 with Unicorn v4.8.2, when I tried to deploy my
app it didn't start properly and in the unicorn.log file I found this
error message:
"app error: Missing secret_key_base for 'production' environment, set
this value in config/secrets.yml (RuntimeError)"
After some research I found out that Rails 4.1 changed the way to
manage the secret_key, so if you read the secrets.yml file located at
[exampleRailsProject]/config/secrets.yml you'll find something like
this:
Do not keep production secrets in the repository,
instead read values from the environment. production: secret_key_base: <%= ENV["SECRET_KEY_BASE"] %> This means that rails
recommends you to use an environment variable for the secret_key_base
in your production server, in order to solve this error you should
follow this steps to create an environment variable for Linux (in my
case Ubuntu) in your production server:
1.- In the terminal of your production server execute the next command:
$ RAILS_ENV=production rake secret This returns a large string with
letters and numbers, copy that (we will refer to that code as
GENERATED_CODE).
2.1- Login as root user to your server, find this file and edit it: $ vi /etc/profile
Go to the bottom of the file ("SHIFT + G" for capital G in VI)
Write your environment variable with the GENERATED_CODE (Press "i" key
to write in VI), be sure to be in a new line at the end of the file:
export SECRET_KEY_BASE=GENERATED_CODE Save the changes and close the
file (we push "ESC" key and then write ":x" and "ENTER" key for save
and exit in VI)
2.2 But if you login as normal user, lets call it example_user for this gist, you will need to find one of this other files:
$ vi ~/.bash_profile $ vi ~/.bash_login $ vi ~/.profile These files
are in order of importance, that means that if you have the first
file, then you wouldn't need to write in the others. So if you found
this 2 files in your directory "~/.bash_profile" and "~/.profile" you
only will have to write in the first one "~/.bash_profile", because
Linux will read only this one and the other will be ignored.
Then we go to the bottom of the file ("SHIFT + G" for capital G in VI)
And we will write our environment variable with our GENERATED_CODE
(Press "i" key to write in VI), be sure to be in a new line at the end
of the file:
export SECRET_KEY_BASE=GENERATED_CODE Having written the code, save
the changes and close the file (we push "ESC" key and then write ":x"
and "ENTER" key for save and exit in VI)
3.- You can verify that our environment variable is properly set in Linux with this command:
$ printenv | grep SECRET_KEY_BASE or with:
$ echo $SECRET_KEY_BASE When you execute this command, if everything
went ok, it will show you the GENERATED_CODE from before. Finally with
all the configuration done you should be able to deploy without
problems your Rails app with Unicorn or other.
When you close your shell terminal and login again to the production
server you will have this environment variable set and ready to use
it.
And thats it!! I hope this mini guide help you to solve this error.
Disclaimer: I'm not a Linux or Rails guru, so if you find something
wrong or any error I will be glad to fix it!
nowadays (rails 6) rails generate a secret key base in tmp/development_secret.txt for you.
and in production environment the best is having SECRET_KEY_BASE as en env variable, it will get picked up by rails.
you can check with Rails.application.secret_key_base.
should give you a long string of numbers and characters from 'a' to 'f' (a 128 chars long hexadecimal encoded string)
As you can see, there is a hardcoded value for the development and test environments, but the one for production comes from a variable. First of all, why this way? It is a security feature. This way, if you check this file into version control such as git or svn, the development and test values get shared, which is fine, but the production one (the one that would be used on a real website) isn't, so no one can look at the source to get that secret.
As for the variable used, ENV["SECRET_KEY_BASE"], this is an environment variable from the environment Rails is run in (not to be confused with the Rails "environment", such as development, test, and production). These environment variables come from the shell. As mentioned in JensD's post, you can set this environment variable temporarily with:
export SECRET_TOKEN=YOUR SECRET TOKEN
export SECRET_KEY_TOKEN=YOUR SECRET BASE
To generate a new secret token, use the rake secret command in the command line.
That is temporary, however, and not a good final solution. For a final solution, check out this article which has a quick bit near the end on implementing dotenv to load configuration secrets. Remember, if you use version control, be sure to exclude your .env file from being checked in!
Setting dotenv up takes a little bit of work, but I highly recommend it over trying to manually configure these environment variables.
I'm just getting a sitemap going with the rails gem and am having trouble generating a sitemap in production.
Running the rake command: rake sitemap:refresh in development creates the sitemap.xml.gz file in the public folder. I navigate to localhost:3000/sitemap.xml.gz and get it downloads the zipped file.
When I run it in production (Heroku-like command line with Dokku on a Digital Ocean VM) I get:
+ sitemap.xml.gz 6 links / 450 Bytes
Sitemap stats: 6 links / 1 sitemaps / 0m00s
Pinging with URL 'https://www.myapp.com/sitemap.xml.gz':
Successful ping of Google
Successful ping of Bing
It appears the file has been created, so I navigate to www.myapp.com/sitemap.xml.gz and get a 404 response.
Server say:
ActionController::RoutingError (No route matches [GET] "/sitemap.xml.gz"):
It appears that this request is hitting the Rails stack when it should be served by Nginx. I just checked to see if the file exists:
FileTest.exists?("public/sitemap.xml.gz")
It returns false so it seems like the sitemap is not actually saved on file. Is there a possibility my file system is read-only right now? How could I test that?
With the new dokku docker-options plugin, you could append persistent storage/volume from your host machine into you container.
First create a local directory in your host machine.
mkdir <path/to/dir>
Then add the following docker-options in dokku
dokku docker-options:add <app> deploy,run -v <path/to/host/dir>:<path/to/container/public/sub/dir>:rw
On your config/sitemap.rb file, add the following lines
SitemapGenerator::Sitemap.public_path = 'public/sitemap/'
SitemapGenerator::Sitemap.sitemaps_path = 'sitemap/'
The sitemap:refresh rake task should write into the sitemap sub folder within the public folder.
This would also allow sitemap_generator to ping the search engine with the right address to your sitemap.xml.gz file.
Feel free to give this a try.
I believe this is a dokku related "issue". Dokku uses Heroku buildpacks, and this yields a read-only file system like on Heroku.
I'd be curious to know if there's a way to modify that behavior in Dokku (seems unlikely if using Heroku buildpacks), but that's a bit out of my league.
I think the best solution to this problem is the same as on Heroku - using Amazon S3.
The sitemap_generator gem has docs on how to set this up.
I have a Ruby on Rails application and I want to get the data from a Heroku server to a local machine.
I have tried the steps mentioned in
https://devcenter.heroku.com/articles/heroku-postgres-import-export, but it just copies the Data Definitions not the actual data on the server.
Is there any way to get the data from Heroku to local databases?
Try using the steps detailed here.
Basically, grab the information from your Heroku database using heroku config:get. Look for your Heroku Postgres url. It will look something like this: HEROKU_POSTGRESQL_RED_URL: postgres://user3123:passkja83kd8#ec2-117-21-174-214.compute-1.amazonaws.com:6212/db982398. Note that the string is configured as database://username:password#host:port/databasename. Thus, in this example, the username is user3123, the database name is db982398. You'll need this for the next part.
Then, you'll use the information in the above postgres string to make a local copy of your Heroku database using pg_dump. Enter the following into your terminal:
pg_dump --host=<host_name> --port=<port> --username=<username> --password --dbname=<dbname> > output.sql
In each place where <....> is in the above code, enter your specific information. After typing this, the terminal will ask you for your password. Enter it.
Finally, you'll load that dumpfile into your local database using psql.
psql -d <local_database_name> -f output.sql
Be sure to put the name of your local database.
I have a Rails 4 application hosted by Heroku.
Seeing as I'm working on an open-source project, there are several unversioned files containing sensitive information listed in .gitignore. For example, I have a .secret file at the root of the app that contains the key with which cookies are encrypted. I created this file by running the rake secret command.
My problem is that I cannot send this file to my heroku app since it is not versioned, it is not included in the deployment. Furthermore, I am using Github and cannot risk having my key disclosed publicly in the commit history.
I have attempted to use the heroku run command to create the file (heroku run 'rake secret > .secret' to no avail). I have attempted to connect with the terminal using heroku run bash but as the filesystem is ephemeral, my changes are not preserved when I exit the terminal.
Do you have any idea how I could achieve having unversioned files on a Heroku application?
Secret data (keys, passwords, etc) should be stored as config vars on Heroku. They are then accessed via the ENV hash in your code.
If you use something like figaro, you can place these vars in an application.yml (don't commit the file)
application.yml:
SECRET_KEY: my_secret_key
Figaro then has a rake task to push these to heroku:
rake figaro:heroku
Or, you can manually set them:
heroku config:set SECRET_KEY=my_secret_key
Finally, access them in your app as:
ENV['SECRET_KEY']
So, I have an app deployed on Heroku, and I'm trying to populate the database on the app through a script I wrote. I have the database in a text file, and the script runs through the file and populates the database. I don't want to push the database file to the heroku server, since it's a very large file.
Is there any way to do this on Heroku? It works fine locally, but I can't get it to work on the Heroku server.
I've tried
heroku run rails runner PATH/TO/SCRIPT LOCAL/PATH/TO/DATABASE --app my_app
to no avail.
To run a local script on Heroku:
irbify.rb script.rb | heroku run rails console --app=my_app
irbify.rb is a silly tiny script I wrote to convert a script to a single eval statement.
Regarding passing data: you can serialize it in some form and put it inside script.
Hope it helps someone.
UPDATE: this does not work well anything beyond trivial datasets.
You can also upload your script to a gist and then do:
binding.eval(open("your gist raw url").read)
I had to use global variables ($dollar_prefixed) since the context would not get filled with the variables (using Pry), otherwise it went well.