I have a serverless function that handles file uploads to my S3 bucket. What I want is that everytime a file uploads I want to see logs on the handler. Is there anyway I could skip deploying everytime and see logs for my offline application state, or I can invoke local functions on file upload handlers to my S3.
Related
I'm trying to upload a file to my rails app by sending the user's local location of the file via a parameter, so the input URL would look like this:
http://rails-app.herokuapp.com/element?file=C:\temp\data.txt
It's easy if I'm working on my local machine, since I can just use File.read(filename), however this doesn't work on heroku. Thanks for any help!
First of all Heroku has read-only file system. So you can't upload anything directly to heroku.
Use something like Amazon S3 to keep files.
The second issue is your approach.
When running app locally - it has access to your C:/ drive.
But app that is located at remote server does not have access to your computers C:/ drive, so it can't fetch file.
You should upload either through browser file field or through passing accessible to anyone http link.
I am trying to create a dashboard using CSV files, Highcharts.js, and HTML5. In a local development environment I can render the charts using CSVs both on my file system and hosted on the web. The current goal is to deploy the dashboard live on Heroku.
The CSVs will be updated manually - for now - once per day in a consistent format as required by Highcharts. The web application should be able to render the charts with these new, "standardized" CSVs whenever the dashboard page is requested. My question is: where do I host these CSVs? Do I use S3? Do I keep them on my local file system and manually push the updates to heroku daily? If the CSVs are hosted on another machine, is there a way for my application (and only my application) to access them securely?
Thanks!
Use the gem carrierwave direct to upload the file directly from the client to an Amazon S3 bucket.
https://github.com/dwilkie/carrierwave_direct
You basically give the trusted logged in client a temporary key to upload the file, and nothing else, and then the client returns information about the uploaded file to your web app. Make sure you have set the upload to be private to prevent any third parties from trying to brut force find the CSV. You will then need to create a background worker to do the actually work on the CVS file. The gem has some good docs on how to do this.
https://github.com/dwilkie/carrierwave_direct#processing-and-referencing-files-in-a-background-process
In short in the background process you will download the file temporarily to heroku, parse it out, get the data you need and then discard the copy on heroku, and if you want the copy on S3. This way you get around the heroku issue of permanent file storage, and the issue of tied up dynos with direct uploads, because there is nothing like NGINX for file uploads on heroku.
Also make sure that the file size does not exceed the available memory of your worker dyno, otherwise you will crash. Sense you don't seem to need to worry about concurrency I would suggest https://github.com/resque/resque.
I have paperclip working just fine where I can upload files to my site, but whenever I make updates and push a new version of the site all of the files I uploaded via paperclip seem to disappear (All the information that was entered into the database remains though).
I assume the problem is that I haven't pulled the files from the live version of the site, but whenever I do a git pull it tells me everything is up to date. Is there anyway for me to download the files I've uploaded. (I would prefer to not use amazon S3 to store the files currently)
The files you have uploaded are stored at public folder. And public folder is not deployed with code, so your files are assuming to be disappeared.
If, you use amazon S3, then images will be stored at s3 and it will provide a dynamic url to access images. Then, you will be able to access images properly.
You can also save images at dropbox. In this application images are stored at dropbox and running on heroku. You may take the referance:
https://github.com/aman199002/Album-App # open source app to store albums(at dropbox).
http://album-app.herokuapp.com #Url of the application running on heroku.
When you deploy your application to Heroku, your pushed code is compiled into what is called a slug - this is essentially an archive of everything needed to run your application. When you restart or scale your application your original slug is then copied to be run. However, it's a readonly slug so when you upload files they exist on the dyno that received them so if you had multiple dynos your files wouldn't exist across them and they then do not persist when your application is restarted or you push new code nor is there any way to retrieve them.
Your only way to perist files on Heroku is to use an external cloud storage solution like Amazon S3, Rackspace files etc - fortunately it's very simple to have Paperclip use S3 for it's storage mechanism - there's a tutorial at https://devcenter.heroku.com/articles/paperclip-s3
In it's simplest form...
I'm using Carrierwave in an app to auto-upload images that exist on external sites so as to put as little load on them as possible. The first time I want to display an image I trigger Carrierwave to do the uploading.
The db is seeded with an image url written in a background Java process. Then in Ruby I assign the seed url to "remote_image_url" and save! to get the upload and versioning to happen. This all works fine.
However I have to store a field in the db that tells whether the upload has been triggered so that I don't trigger it every time the image is displayed. How can I tell if the "image" has been uploaded? The "image" field is actually an Uploader, is there a method that will tell me if it has been triggered?
Carrierwave downloads the remote file as soon as you assign it via model.remote_image_url=, and it moves the file to its final location (local storage, S3, etc) as soon as you call model.save. This happens synchronously, so if model.save has completed, and there hasn't been an error, your file has been stored.
To put it another way: If you just loaded a model from the database, and model.image? is true, then model.image.url should be a working URL.
I am using cloudfoundry. I upload a file and save the file..my routine returns the path and filename
/var/vcap/data/dea/apps/Dwarfquery-0-99065f0be8880d91916257931ed91162/app/tmp/region1-legends10-11-2012-20:53.xml
However the scheduled resque routine which tries to read it using File.Open returns the following error
Errno::ENOENT
Error
No such file or directory - /var/vcap/data/dea/apps/Dwarfquery-0-99065f0be8880d91916257931ed91162/app/tmp/region1-legends10-11-2012-20:53.xml
This is the path returned by the Upload Server...I have added require 'open-uri' at the top of my Job Class
The line that is failing is
File.open(fpath, 'r+') do |f|
where fpath the the file/directory returning the error
I'm not proficient with ruby at all, but just to clarify:
Are the bit that uploads and the Resque routine part of the same "app" (in Cloud Foundry sense?)
Are you trying to read the file soon after it has been uploaded, or long after (in particular, after your app has/could have been restarted?)
This is important because:
Each "app" has its own temporary folder and obviously one app can't access another app's filesystem. This also holds if you deployed your app with multiple "instances". Each instance is a separate process that has its own filesystem.
local filesystem storage is ephemeral and is wiped clean every time the app restarts
If you need to access binary data between apps, you will want to use some kind of storage (e.g. Mongo's GridFS) to have it persisted and visible by both apps.