I'm building a rails app that interacts with a 3rd party API
When a user uploads a file to rails, it should be forwarded on to the 3rd party site via an HTTP POST.
In some cases, the upload can be several hundred MBs.
At the moment, I've just been re-posting to the API using Net::HTTP and accessing the multipart form object like so
#tempfile = params[:video][:file_upload].tempfile
This is hella slow though and feels kinda dirty.
Is there a better way to do this?
Is it possible to have the user post directly to the 3rd party service or do you have to handle the API through your Rails stack? Ideally you would be able to do this and would not have to load the file into your stack and then re-post it to the API. If you can't post directly, I would recommend seeing if the API has a streaming service so that you can send parts of the file instead of the entire thing at once. Either way I think you'll start running into Timeout errors on your side and on the API side with large files, so you'll have to increase your own timeouts or create a different type of streaming file uploader.
Spin up a background job using DelayedJob. In the delayed job, you could try rails redirect_to.
https://github.com/tobi/delayed_job
http://apidock.com/rails/ActionController/Base/redirect_to
Related
I need to send to the customers a raw emails via my rails app.
When they click a link, a new page must open and they need to be able to see the source code of an email. I have a lot of cases where there are emails really big (even 40/50 mb), and it takes a lot of time to server to send it.
E.G.
I have an email with 3 attachments, the total size is 30mb. My
controller method it takes 700 ms to process it and to retrieve the
raw source from imap server, but in the broswer, it takes up to 5
seconds. (2.5 to the first byte, 2.5 to download it).
Right now I just send the string with the render method. Is there a better way? where I am losing all that time?
To be more clear:
With the word 'send' I mean when the server has to 'send' the source code to the browser so the suer can visualize it
What about storing attachments on your server and include in email only links to those attachments? This way your emails will be blazingly fast and clients may download attachments separately. If you don't want to store files on your server, you may use Amazon S3 or some other cloud storage (there are many in these days).
To ease file uploading, I'd recommend you carrierwave library, Amazon S3 integration goes from the box.
I would suggest you to use Delayed Job: https://github.com/collectiveidea/delayed_job or Sidkiq: https://github.com/mperham/sidekiq
When the job is in processing state, you can mark that email as Sending.... and once the background job is completed you can mark it as Sent
Hope this helps
I think you could benefit from looking into solutions for HTTP streaming, which would allow you to start sending data while still processing the request.
This is difficult with Rack based servers, so Rails might have some issues with this approach.
Another approach is to try and split the raw source into chunks and request each chunk using Ajax.
This will allow your app to be more responsive and offer a better user experience. This is also known as a perceived performance approach, since the user experiences the app as more responsive even if it takes the same amount of time to load.
If I were trying to resolve the issue, I would look into an Ajax solution that would allow me to leverage IMAP's partial fetch feature, referenced in it's RFC.
It's possible to write a simple server side API that fetches a part of an email or returns a signal when there is no more to download and than use Javascript to request the data from the server until the 'no more data' return value is received.
This would allow you to display the downloaded data as it's being received.
I am working in a Rails application and below is the scenario requiring a solution.
I'm doing some time consuming processes in the background using Sidekiq and saves the related information in the database. Now when each of the process gets completed, we would like to show notifications in a separate area saying that the process has been completed.
So, the notifications area really need to pull things from the back-end (This notification area will be available in every page) and show it dynamically. So, I thought Ajax must be an option. But, I don't know how to trigger it for a particular area only. Or is there any other option by which Client can fetch dynamic content from the server efficiently without creating much traffic.
I know it would be a broad topic to say about. But any relevant info would be greatly appreciated. Thanks :)
You're looking at a perpetual connection (either using SSE's or Websockets), something Rails has started to look at with ActionController::Live
Live
You're looking for "live" connectivity:
"Live" functionality works by keeping a connection open
between your app and the server. Rails is an HTTP request-based
framework, meaning it only sends responses to requests. The way to
send live data is to keep the response open (using a perpetual connection), which allows you to send updated data to your page on its
own timescale
The way to do this is to use a front-end method to keep the connection "live", and a back-end stack to serve the updates. The front-end will need either SSE's or a websocket, which you'll connect with use of JS
The SEE's and websockets basically give you access to the server out of the scope of "normal" requests (they use text/event-stream content / mime type)
Recommendation
We use a service called pusher
This basically creates a third-party websocket service, to which you can push updates. Once the service receives the updates, it will send it to any channels which are connected to it. You can split the channels it broadcasts to using the pub/sub pattern
I'd recommend using this service directly (they have a Rails gem) (I'm not affiliated with them), as well as providing a super simple API
Other than that, you should look at the ActionController::Live functionality of Rails
The answer suggested in the comment by #h0lyalg0rithm is an option to go.
However, primitive options are.
Use setinterval in javascript to perform a task every x seconds. Say polling.
Use jQuery or native ajax to poll for information to a controller/action via route and have the controller push data as JSON.
Use document.getElementById or jQuery to update data on the page.
I'm using pusher gem to manipulate my front-end from an external API. It works fine, no problem with that.
But the thing I wonder is if there is a possibility to use push notifications at the back-end of my application? I spent a serious amount of time investigating this but couldn't find something useful.
Let me summarize:
I have an application and another API application which is tightly interacting with other. Sometimes I want to use my API to send notification to my main application and I want to be able to manipulate data at the back-end of my main application regarding the data received from API side. These are things like 'an action was completed/started/succeed' etc...
I understand that 'pusher' receives push notifications by JavaScript at the front-end. But I believe that there must be a way to use those notifications at the back-end as well.
If there is another way (maybe Faye? Websocket) to do that I'd love to learn what it is. Any clue would be appreciated.
Is it something doable?
Thank you
Pusher is a backend system too (to "push" updates to channels)
Endpoints
I think you may be interested in endpoints
From what I can gather, it seems you're looking to trigger the transfer of data to an endpoint once an action occurs in your API? For example:
User signs up on "API" app
API app sends "notification" to main app
Main app increases user count by 1
The way I can see this working is by either using ajax, or sending a curl request to your main app's endpoint (set in routes), triggering the action:
#main_app/config/routes.rb
post "endpoint", to: "application#endpoint"
#main_app/controllers/application_controller.rb
def endpoint
#count = Option.increment!(:user_count)
end
This will allow you to manipulate your data in the backend of your "main" app
API
The tricky, non-conventional part comes when you want to send the data from your API app to your Main app (this is where you got the "pusher" idea from)
I would personally look at sending a standard HTTP request to the Main app endpoint, probably with Curl (if from the backend):
Curl on Ruby on Rails
Rails curl syntax
You may want to install curb (CUrl RuBy) here: https://github.com/taf2/curb
I could write some code if you wanted?
I had asked the same question to the Pusher's support team and I got the exact answer I was looking for.
You can install a client library on your server
(http://pusher.com/docs/client_libraries) if there is one for your
server. You can then subscribe to a client channel this way.
In my case, I use Ruby gem which can be reached from https://github.com/pusher/pusher-ruby-client .
I am trying to implement basic progress bar for file uploads to work across multiple browsers without additional plugins like Flash or Silverlight. There are multiple ways to approach this problem on a client-side, however I can't find anything to work on a server.
Anywhere on MVC controller (before/on authorization and before/on executing action) file is already uploaded to server as HttpPostedFileBase. If I use basic HTTP handler for form submission, I have access to Context.Request.InputStream as well as to Context.Request.Files, but as soon I access the properties the stream/files would load silently.
I did extensive research, but could not find anything what allows me to Cache or store file upload process in Session. That would at least allowed me to use periodic AJAX requests from a client to monitor the progress.
What am I missing?
You should try a Javascript alternative, instead of handling it server side. The controller should only come in action when the request is done sending.
Try this JavaScript/jQuery alternative to Flash/SL
http://blueimp.github.io/jQuery-File-Upload/
I'm wondering what the best way to go about developing a rails application with the following features:
All of the data comes from a SOAP request to a 3rd party
A background task will make this soap request every ~10s
The background task will parse the response and then update an ActiveRecord model accordingly
The data isn't written to a database at all, if the app fails, when we start it back up the data will come from the soap request again
Users will make a request to the app which will simply show data in the model (i.e. from the soap request).
The idea is to avoid making the SOAP request for every single user as the data won't change that frequently. Not using a database avoids reading and writing of data that only ever comes from the request anyway.
I imagine that all of this can be completely quite simply with a few gems but I've had a bit of trouble sorting through what meets my requirements and what doesn't.
Thanks
I'm not sure what benefit you're getting from using ActiveRecord in this case.
Perhaps consider some other type of persistance for the SOAP calls?
If the results form the WebService are really not changing, I would recommend the Rails caching mechanism. Wherever in your Rails app, you can do:
Rails.cache.fetch "a_unique_cache_key" do
... do your SOAP request and return the result
end
This will do the work within the block just once and fetch its result from the rails cache store in the future.
The cache store be of various types (one of which is the memcache store). I usually go with the file store for medium traffic sites, but you may choose another:
http://guides.rubyonrails.org/caching_with_rails.html