I have used Azure Active Storage for all the user files that are uploaded on my application developed using Rails v6. Works perfectly fine.
Now I wanted to make a copy of the files that are getting uploaded on my application to Another Azure storage account. How can I go about this approach. Any help would be much appreciated.
There are a couple of options available. One is using mirrors in storage.yml file. There you can configure multiple storage accounts even with different cloud service providers like Amazon and Azure. This is done at configuration level
s3_mirror:
service: S3
access_key_id: ""
secret_access_key: ""
region: ""
bucket: ""
development:
service: AzureStorage
root: <%= Rails.root.join("storage") %>
container: development
storage_account_name: my_account_name
storage_access_key: my_access_key
mirror:
service: Mirror
primary: development
mirrors:
- s3_mirror
Another way which was more usable for me was this piece of code suggested by a famous AI app
# First, install the gem
gem 'azure-storage-ruby'
# Then, in your code, you can use the Azure::Storage::Client class to connect
to a storage account
# Replace ACCOUNT_NAME and ACCOUNT_KEY with your storage account name and key
client1 = Azure::Storage::Client.create(storage_account_name: 'ACCOUNT_NAME',
storage_access_key: 'ACCOUNT_KEY')
client2 = Azure::Storage::Client.create(storage_account_name: 'ACCOUNT_NAME',
storage_access_key: 'ACCOUNT_KEY')
# You can then use the client objects to perform operations on the storage
accounts
blob_client1 = client1.blob_client
blob_client2 = client2.blob_client
# For example, you can list the blobs in a container like this:
blob_client1.list_blobs('my-container').each do |blob|
puts blob.name
end
blob_client2.list_blobs('my-container').each do |blob|
puts blob.name
end
Related
I'm using Rails 6 Active Storage to upload images to a Digital Ocean Spaces bucket.
My Storage.yml:
digitalocean:
service: S3
access_key_id: <%= Credential.digitalocean_access_key_id %>
secret_access_key: <%= Credential.digitalocean_secret_access_key %>
endpoint: https://sfo2.digitaloceanspaces.com
region: sfo2
bucket: mybucket
upload:
acl: "public-read"
As you can see I do specify a "public-read" upload ACL.
Images do upload fine using Active Storage direct upload, yet in the bucket, the file permission is Private, not Public.
Any help is appreciated, thank you.
I guess it's an issue with Digital Ocean S3 implementation! Try this command to force ACL to public:
s3cmd --access_key=YOUR_ACCESS_KEY--secret_key=YOUR_SECRET_KEY --host=YOUR_BUCKET_REGION.digitaloceanspaces.com --host-bucket=YOUR_BUCKET_NAME.YOUR_BUCKET_REGION.digitaloceanspaces.com --region=YOUR_BUCKET_REGION setacl s3://YOUR_BUCKET_NAME --acl-public
I have an app running on a ubuntu server. I have a production mode and a staging mode.
Problem is that actions being done on the production site relative to uploading and retrieving images from an S3 bucket are being done to the same bucket as my staging. When I have my configurations set up differently.
production.rb
config.s3_bucket = 'bucket-production'
config.s3_path = 'https://bucket-production.s3.us-east-2.amazonaws.com/'
staging.rb && development.rb
config.s3_bucket = 'bucket-staging'
config.s3_path = 'https://bucket-staging.s3.us-east-2.amazonaws.com/'
storage.yml
amazon:
service: S3
access_key_id: <%= Rails.application.credentials.dig(:aws, :access_key_id) %>
secret_access_key: <%= Rails.application.credentials.dig(:aws, :secret_access_key) %>
region: us-east-2
bucket: bucket-staging
endpoint: http://bucket-staging.us-east-2.amazonaws.com
I'm thinking it could be something with storage.yml but I deleted this entire file and restarted the localhost server and it didn't change anything. Is storage.yml production only?
Also, my logs are logging to staging from production.
I would like to ask is prod server/staging server(Ubuntu) running in AWS.if yes, you must have IAM ROLE attached to the server which must be fine-grained to which env application should access which bucket of S3. Also storing access_key_id and secret access key id should not be used as best practice. IAM role can take care of that. Also, I would like to add if the server is in the private subnet you need to explore either NAT gateway or use VPC S3 Endpoint to access the bucket.
Also try printing logs of S3 connection in prod mode how it is acquiring cred to access bucket. You might be granting access using some ENV variables or IAM role. Best way to see is use
printenv
before and after of S3 connection to see the variables and which bucket access is provided.
Thanks
Ashish
I am trying to set caching for the images stored in Active Storage with google cloud. I have the below configuration file
test:
service: Disk
root: <%= Rails.root.join("tmp/storage") %>
local:
service: Disk
root: <%= Rails.root.join("storage") %>
production:
service: GCS
credentials: <%= Rails.root.join("cert/gcs-production.json") %>
project: 'my-app-121212'
bucket: 'my-prod-bucket'
staging:
service: GCS
credentials: <%= Rails.root.join("cert/gcs-staging.json") %>
project: 'my-app-121212'
bucket: 'my-stage-bucket'
Can any one please suggests as to where i can specify the caching settings, I want the images to be cached.
You can set public: true on the service. This will instruct rails to use public url from GCS which by default will have caching header added. Please refer to official guide
Make sure your GCS bucket has public permission added (allUsers with viewer role)
If you are using App Engine Flexible and not Compute Engine, caching files that are publicly readable object is allowed by default when using Google Cloud Storage, indicated by the service: GCS flag you’re using, meaning that you do not have to specify any special configuration.
Keep in mind that the objects you can cache need to be static files, so images, stylesheets, JavaScript, etc.
If you wanna read more on serving static files from App Engine with Rails, you can read this article from Google Cloud.
I'm trying to configure ActiveStorage to use S3 bucket as a storage backend however I don't want to pass any of access_key_id, secret_access_key, region. Instead, I'd like to use previously defined IAM role. Such configuration is mentioned here. It reads (I've added bold):
If you want to use environment variables, standard SDK configuration files, profiles, IAM instance profiles or task roles, you can omit the access_key_id, secret_access_key, and region keys in the example above. The Amazon S3 Service supports all of the authentication options described in the AWS SDK documentation.
However I cannot get it working. My storage.yml looks similar to this:
amazon:
service: S3
bucket: bucket_name
credentials:
role_arn: "linked::account::arn"
role_session_name: "session-name"
I've run rails active_storage:install, applied generated migrations and set config.active_storage.service = :amazon in my app's config.
The issue is that when I'm trying to save a file, I'm getting an unexpected error:
u = User.first
s = StringIO.new
s << 'hello,world'
s.seek 0
u.csv.attach(io: s, filename: 'filename.csv')
Traceback (most recent call last):
2: from (irb):3
1: from (irb):3:in `rescue in irb_binding'
LoadError (Unable to autoload constant ActiveStorage::Blob::Analyzable, expected /usr/local/bundle/gems/activestorage-5.2.2/app/models/active_storage/blob/analyzable.rb to define it)
I'm using Rails 5.2.2.
Are you trying this code inside an AWS EC2 instance or locally in your machine?
If you check the authentication methods in AWS: https://docs.aws.amazon.com/sdk-for-ruby/v3/developer-guide/setup-config.html#aws-ruby-sdk-credentials-iam
You'll see the following section:
Setting Credentials Using IAM
For an Amazon Elastic Compute Cloud instance, create an AWS Identity and Access Management role, and then
give your Amazon EC2 instance access to that role. For more
information, see IAM Roles for Amazon EC2 in the Amazon EC2 User Guide
for Linux Instances or IAM Roles for Amazon EC2 in the Amazon EC2 User
Guide for Windows Instances.
This means that for this authentication method to work, you must:
Create an EC2 instance on AWS
Create an EC2 IAM Role with permissions to write to an S3 Bucket
Configure your EC2 instance attaching the new IAM Role to it
With the role attached to the instance, your config/storage.yml file will look like this:
amazon:
service: S3
bucket: test-stackoverflow-bucket-app
region: "us-west-1"
Note that region is a required parameter, you'll get an error if you skip it: https://github.com/aws/aws-sdk-ruby/issues/1240#issuecomment-231866239
I'm afraid this won't work locally, to use active_storage locally you must set the access_key_id, secret_access_key values.
Today I tried rails 5.2.0.
It generated following file template when rails newed.
# Use rails credentials:edit to set the AWS secrets (as aws:access_key_id|secret_access_key)
# amazon:
# service: S3
# access_key_id: <%= Rails.application.credentials.dig(:aws, :access_key_id) %>
# secret_access_key: <%= Rails.application.credentials.dig(:aws, :secret_access_key) %>
# region: us-east-1
# bucket: your_own_bucket
Question:
As far as I know, there is no region in S3 service. If so, what does this parameter field region mean in the realm of Active Storage?
"there is no region in S3 service", that's incorrect.
Extracted from Working with Amazon S3 Buckets, official docs.
Amazon S3 creates buckets in a region you specify. You can choose any
AWS Region that is geographically close to you to optimize latency,
minimize costs, or address regulatory requirements. For example, if
you reside in Europe, you might find it advantageous to create buckets
in the EU (Ireland) or EU (Frankfurt) regions. For a list of Amazon S3
regions, see Regions and Endpoints in the AWS General Reference.