I have a Rails app which uploads files to a server and stores them so that they can be processed by a Java application.
The current configuration is in AWS which have two instances and is balanced by ELB.
Path to upload dir in AWS-instance1 : /var/lib/rails_files
Path to upload dir in AWS-instance2 : /var/lib/rails_files
Is there a way so that I can sync the directory rails_files from both instances?
At a specific time, the Java application looks for the file name from db and picks the file from /var/lib/rails_files.
Or is it possible to attach/add a shared drive to both instances so that it can be accessed from both instances?
I don't want to go with s3 file upload.
Any help would be appreciated.
Rather than synchronizing between instances (and all the inherent security and configuration required), instead sync with Amazon S3.
For example, the AWS Command-Line Interface has a command that can sync between local directories and an Amazon S3 bucket:
aws s3 sync dir s3://BUCKET/dir
This also works in the reverse direction (from S3 to local) and between S3 buckets. By default, sync will not delete files from the destination.
Related
we're having a rails 6 app holding files via ActiveStorage. So far, requests dealing with uploading files etc. last some time because we're currently working non-asynchronous. Now we're switching to background processing. Everything related to files specifically is causing headache, because the files reside in one docker container in the realm of our rails service but need to be accessed from a separate sidekiq container where unfortunately ActiveJob tasks are not able to find the ActiveStorage objects (ActiveStorage::FileNotFoundError).
So - if any - what would be the best approach to make ActiveStorage ressources accessible to sidekiq in a multi-container docker setup?
Thanks in advance!
I guess you use disk service on active storage(I should point out that if you do not store the rails app or the storage directory on the host, then you will lose your files when the container is restarted.). If your files are in disk service, if you want to use them in the background processor, then you can link both the rails app and the app directory in the background processor from your host, or you can link the storage directory to both of them from the host. As an example, you can look at the following article;
https://nickjanetakis.com/blog/dockerize-a-rails-5-postgres-redis-sidekiq-action-cable-app-with-docker-compose
I have a production system using an Amazon S3 bucket for storing ActiveStorage blobs (Rails 6.0.3).
For development, I use the Local service because I don't want to inadvertently mess with the production data.
What is the best way to replicate/mirror/sync the production bucket locally so I can use it just a as scratch for development and testing?
I have tried replicating the production database locally and copying the bucket directly to the Local service directory but ActiveStorage doesn't see the blobs with the right keys. (Not sure why?)
I have also seen some examples of the Mirror service being used between cloud services or from Local to cloud services, but not from cloud services to Local.
Local storage is organized in subdirectories
$ tree storage
storage
├── ZD # first 2 char
│ └── ou # third and fourth car
│ └── ZDouA9nEzJjWAx71Cmb3WVeY
while S3 stores file in flat format (no under sub-directories). The above file will be store as a direct file under the bucket
ZDouA9nEzJjWAx71Cmb3WVeY
I don't know it applies to Rails (and which version). But I had succeeded in transferring development storage files to S3 using the above concept (Spree 4 - which uses rails framework 6).
I am looking for information but I can't find it. Is it possible to attach Amazon S3 storage to the Docker container so that it is via the application (Spring Boot) as a disk? If so, how to do it? If it is important, the docker image is managed by kubernetes
I'd recommend to take a look at the product Min.io.
Its client has nice features that allow you to use simple FS commands to operate S3 storage from your container:
ls list buckets and objects
mb make a bucket
rb remove a bucket
cat display object contents
head display first 'n' lines of an object
pipe stream STDIN to an object
share generate URL for temporary access to an object
cp copy objects
mirror synchronize objects to a remote site
find search for objects
sql run sql queries on objects
stat stat contents of objects
lock set and get object lock configuration
retention set object retention for objects with a given prefix
legalhold set object legal hold for objects
diff list differences in object name, size, and date between buckets
rm remove objects
event manage object notifications
watch watch for object events
policy manage anonymous access to objects
admin manage MinIO servers
session manage saved sessions for cp command
config manage mc configuration file
update check for a new software update
version print version info
For convenience, you may add shell aliases to override your common Unix tools.
alias ls='mc ls'
alias cp='mc cp'
alias cat='mc cat'
alias mkdir='mc mb'
alias pipe='mc pipe'
alias find='mc find'
It also has native Java (Pyton, Golang, .NET, Haskel) client:
MinIO Java SDK for Amazon S3 Compatible Cloud Storage
Amazon S3 is an object storage and not a block storage or file storage and you should not attach it to a container. For any practical purpose you should be able to use the s3 storage from the spring boot application using AWS SDK or API.
With kubernetes you can use block storage such as AWSElasticBlockStore out of the box.
If you want to use object storage like S3, this is also possible however far more complicated. If you are still interested in implementing specifically S3 storage on your kubernetes cluster, this article describes very well the whole procedure.
I'm working in an application to obtain some data from a web service, create a text file in the local filesystem send a command to a command line application, obtain the result and then send the results back via the web service.
I need to be able to write to the local file system, read from it and then delete the temporary file. I was reading about bind mounts and volumes but this folder can be delete if a new version of the image is uploaded is just a staging area.
Any ideas how this can be done, thanks.
When using containers in App Service, I believe you will have to link a storage account and mount file shares accordingly. Depending on the OS (windows / linux), the steps vary a bit.
If you are not using containers, then you should be able to access the temporary file locations for file-based requirements. Do note that the storage available this way is limited and not shared across site instances.
I've been following the twitter tutorial for bluemix and watson and I've read that notebooks are short-lived but what about the .txt or .json files generated by running a notebook and saved under 'static' folder?
Whenever I re-run my application, those .txt and .json files are no longer there. Is there any way to make them persistent?
Thanks.
You have to save the files to your git repository if you are using the IBM DevOps as described in the tutorial. Using this approach a brand new container for your application will be created every time you run Build and Deploy in your devops environment.
Or you can save a local copy to your root project directory and then it will be copied to your application container when you use cf push to deploy your application.
Cloud foundry applications are ephemeral so you should avoid writing to the local disk. As I mentioned in the first paragraph a brand new container is created every time your application is redeployed.
You can find more details here.