I have an S3 bucket containing a bunch of data in the format of a Ruby Hash. What I'd like to do upon running rails s is to have the data retrieved from the S3 bucket and use that to seed the database. The data in the S3 bucket will always be changing and the Rails app is running inside of a container so I'm going to need to seed the DB before every run. How do I seed a database from an S3 bucket?
Related
I am having a hard time getting s3 images to show in deployment. I have 2 buckets on aws s3, one for dev and for production.
only the production bucket get filled up with images when I run the code below for my images. (You can see this on the images I attached) I also dont get any errors on chrome console.
Restaurant.last.main_photo.attach(io: File.open("/home/vault/Desktop/halal-table-photos/profile-photos/blur-breakfast-chef-cooking-262978.jpg"), filename: "23.jpg")
I've tried the Open-uri gem
require 'open-uri'
Restaurant.last.main_photo.attach(io: open("/home/vault/Desktop/halal-table-photos/profile-photos/blur-breakfast-chef-cooking-262978.jpg"), filename: "23.jpg")
Here are my s3 setups
https://github.com/MahmudAhmed/HalalTable <- git repo
https://halal-table.herokuapp.com/#/restaurants
https://halal-table.herokuapp.com/
I examine your project from Github. There is no visible problem on the code, but only thing I didn't see here is your Heroku config. You need to either say Heroku to use specific bucket for production via heroku config vars or initialize config/initializers/aws.rb and say heroku to use different bucket in production. In here you can get really got information about s3 buckets in Rails with different environments in action.
The only tool I could find, I forked and tried to update to include the S3_REGION because I was getting
$ The bucket you are attempting to access must be addressed using the specified endpoint
These are all the variables I am passing to access the bucket.
opts[:s3_key] =======> AKIAJHXXG*********YA
opts[:s3_secret] =======> uYXxuA*******************pCcXuT61DI7po2
opts[:s3_bucket] =======> *****
opts[:output_path] =======> /Users/myname/Desktop/projects/my_project/public/system
opts[:s3_region] =======> s3-us-west-2.amazonaws.com
https://github.com/rounders/heroku-s3assets has not been update in a while so Im assuming I just can't find where the actual error is breaking either in Heroku tools, or the older aws-s3 gem.
Anyone have any method to pull down production assets to Heroku server from AmazonS3?
I think I mis-understood you, so editing now...maybe experiment with something simpler:
http://priyankapathak.wordpress.com/2012/12/28/download-assets-from-amazon-s3-via-ruby/
My search returned this info:
Bucket is in a different region
The Amazon S3 bucket specified in the COPY command must be in the same
region as the cluster. If your Amazon S3 bucket and your cluster are
in different regions, you will receive an error similar to the
following:
ERROR: S3ServiceException:The bucket you are attempting to access must be addressed using the specified endpoint.
You can create an Amazon S3 bucket in a specific region either by
selecting the region when you create the bucket by using the Amazon S3
Management Console, or by specifying an endpoint when you create the
bucket using the Amazon S3 API or CLI. For more information, see
Uploading files to Amazon S3.
For more information about Amazon S3 regions, see Buckets and Regions
in the Amazon Simple Storage Service Developer Guide.
Alternatively, you can specify the region using the REGION option with
the COPY command.
http://docs.aws.amazon.com/redshift/latest/dg/s3serviceexception-error.html
So it turns out that gem was all but useless. I've gotten further to my goal of downloading all my s3 assets to public/system - but still can not figure out how to download them to my correct local rails directory using the aws s3 docs - http://docs.aws.amazon.com/AWSRubySDK/latest/AWS/S3/S3Object.html
s3 = AWS::S3.new(access_key_id: 'AKIAJH*********PFYA', secret_access_key: 'uYXxuAMcnKODn***************uT61DI7po2', s3_endpoint: 's3-us-west-2.amazonaws.com')
s3.buckets['advlo'].objects.each do |obj|
puts obj.inspect
end
I probably just need to read more unix commands and scp them over individually or something. Any ideas?
I am having issues with my Rails App on Heroku. code-dojo.herokuapp.com
After every push to heroku any images I uploaded with Carrierwave Gem return a 404 error message.
Do I need to precompile this folder or point to it ?
Does Heroku replace this folder with a blank one?
Should I create my app with all the images on locathost and then push the database?
Heroku is Read-only Filesystem
The following types of behaviors are not supported:
Caching pages in the public directory
Saving uploaded assets to local disk (e.g. with attachment_fu or paperclip)
Writing full-text indexes with Ferret
Writing to a filesystem database like SQLite or GDBM
Accessing a git repo for an app like git-wiki
You need to use an external storage solution. You can accomplish this for example by using the gem carrierwave-aws instead of the gem carrierwave, with which you can configure an Amazon S3 bucket to store your images on...
Our server ran into a file limit issue with carrierwave. Over 36000 files. We are now going to move to S3.
Is there a way to migrate the files over to S3? When we launched the code on production none of the images showed up and there was a duh moment. It's trying to grab the files from s3 when they are locally stored on the server still.
How do we migrate the files over?
You can upload the files to s3 via the s3 console in the s3 file manager. Or by using a plugin such as S3Fox for FireFox. You'll just need to make sure the pathing and the s3 bucket are such that Carrierwave will know how to point to the image via the right set of subfolders, etc.
I'm trying to upload a file using paperclip in a production environment in Heroku and the log files show:
Errno::EACCES (Permission denied - /app/public/system/photos/1/small/081811-2012-honda-cbr1000rr-leaked-003.jpg):
Will I have to use s3 or similar to handle file uploads, or can I configure path permissions to store the files on Heroku?
Yes Heroku does not allows you to add files dynamically to its server.
Though if you need upload feature on a app on heroku you need to configure s3 or other similar services
Refer this for details
http://devcenter.heroku.com/articles/read-only-filesystem
Yes, you must use S3 or another persistent store like Rackspace cloudfiles, Dropbox etc.
Whilst you can write to the tmp on all the stacks, the Cedar stack does let you write to the file system but it's not shared across dynos or dyno stop/restarts.
See http://devcenter.heroku.com/articles/dyno-isolation#ephemeral_filesystem
Yeah, it is true that Heroku does not allow you to upload files directly onto their servers. However, you can shove your files into your database. I used a gem created by Pat Shaughnessy:
http://patshaughnessy.net/2009/2/19/database-storage-for-paperclip
It worked well.