I am designing a web service under Rails 3.2.8 which will execute external program for the users, the expected use case is:
1) User fill a form of parameters and submit the request
2) Rails execute a Matlab program based on user request. It would last a few minutes (less than 3mins) and generate a result file shows that the program is finished
3) During this process, redirecting user to a page shows "loading" status, and monitoring if the result file has been generated
4) Once the result file is generated, reload the page via AJAX and display the results.
Is there a "rails" way to do so? I did this before in JAVA SSH framework but pretty painful. What will be the tools I need? For example, do I need gems like backgroundjob
to manage the task queue? Or are there any "one-stand" gems can handle this? Thanks!
Use one of several job queue providers. Examples include DelayedJob (simplest one ever, no additional pieces required), Resque and Sidekiq (much faster, but you need a small Redis server to use them).
A "job" will be (depending on the provider) a class or instance, which implements the execution. If it implies running something in the shell, you can do it like
%x[matlab command --options > output.ext]
(it's just an example, you can use string interpolation too: %x[#{executable_name}])
I kind figured out what to use during the whole process:
1) User fill a form of parameters and submit the request
Using the method from this rails cast: http://railscasts.com/episodes/219-active-model for server side validation and gem 'client-side-validations' to enable client side check.
2) Rails execute a Matlab program based on user request. It would last a few minutes (less than 3mins) and generate a result file shows that the program is finished
Using gem 'DelayedJob' for task management and 'ChildProcess' to start the Matlab program.
3) During this process, redirecting user to a page shows "loading" status, and monitoring if the result file has been generated
Monitoring child process as well as the existence of the result file to check whether the task is finished.
4) Once the result file is generated, reload the page via AJAX and display the results.
Using polling method from this rails cast: http://railscasts.com/episodes/229-polling-for-changes to update the result page. I am not using pushing although it would be more secure - as polling seems to be a more light-weighted solution and can meet my requirements.
Obviously this is not the best practice, but satisfies my needs and easy to implement. Thanks for all the comments and answers.
Related
I have a logging query (a simple INSERT) that happens on every single request.
For this request only (the one that happens on every page load), I want to set the limit to 500ms in case the database is locked/slow/down it won't affect the site, where the site hangs while it waits to connect/write.
Is there a way I can specify a timeout somehow on a per-query basis that I can abort the LoggedRequest.create! if it's taking too long?
I don't want to set it in my config because I have many other queries that shouldn't have timeouts that low.
I'm using Postgres 11.7
I also don't know how I feel about setting a timeout for the entire session because I don't want that connection to be shared from the pool with other queries that can't have that timeout.
Rails 6 introduces event based triggers for notifications, logging etc that comes in very handy, provided you are using/can afford to migrate to Rails 6. Here'a useful post that demonstrates creating event based triggers for notifications/logging: https://pramodbshinde.wordpress.com/2020/03/20/custom-events-tracking-with-activesupportnotifications-and-audited/
If, for some reason, you cannot use Rails 6, perhaps this article might help you find some answers: https://evilmartians.com/chronicles/the-silence-of-the-ruby-exceptions-a-rails-postgresql-database-transaction-thriller
If I were you, I could also contemplate using AJAX with a fire-and-forget API request to server for logging/whatever that is not critical to normal functioning of the application.
I am building a website, and I have an administrator page. The admin will have to run a reporting task, meaning that, the task will iterate all the records fetch information and generate a pdf file. Now this will be heavy on the app and the database.
What is the usual approach for it ? Should I have a button that calls a method of a class or should I have a rake task? I heard that HTTP GET requests have a limit and if the report generation takes more than that then it kills the request.
I would like to use send_data(....) so the user is given a nice download pop up box when the report is done. Will it be better to use a mailer and email it?
Thanks
We have similar functionality in our Rails apps at my job.
We have one URL/action that initiates the request to generate the PDF file, and returns right away saying the request was started successfully.
Then we have another action that we can poll with AJAX that returns whether or not the report is complete, and when it is complete, it gives the user the PDF.
The actual generation is done by a Sidekiq worker which is not subject to the webserver timeout.
I have an analytics engine which periodically packages a bunch of stats in JSON format. I want to send these packages to a Rails server. Upon a package arriving, the Rails server should examine it, generate a model instance out of it (for historical purposes), and then display the contents to the user. I've thought of two approaches.
1) Have a little app residing on the same host as the Rails server to be listening for these packages (using ZeroMQ). Upon receiving a package, the app would invoke a Rails action through CURL, passing on the package as a parameter. My concern with this approach is that my Rails server checks that only signed-in users can access actions which affect models. By creating an action accessible to this listening app (and therefore other entities), am I exposing myself to a major security flaw?
2) The second approach is to simply have the listening app dump the package into a special database table. The Rails server will then periodically check this table for new packages. Upon detecting one or more, it will process them and remove them from the table.
This is the first time I'm doing something like this, so if you have techniques or experiences you can share for better solutions, I'd love to learn.
Thank you.
you can restrict access to a certain call by limiting the host name that is allowed for the request in routes.rb
post "/analytics" => "analytics#create", :constraints => {:ip => /127.0.0.1/}
If you want the users to see updates, you can use polling to refresh the page every minute orso.
1) Yes you are exposing a major security breach unless :
Your zeroMQ app provides the needed data to do authentification and authorization on the rails side
Your rails app is configured to listen only on the 127.0.0.1 interface and is thus not accessible from the outside
Like Benjamin suggests, you restrict specific routes to certain IP
2) This approach looks a lot like what delayed_job does. You might wanna take a look there : https://github.com/collectiveidea/delayed_job and use a rake task to add a new job.
In short, your listening app will call a rake task that will add a custom delayed_job when receiving a packet. Then let delayed_job handle the load. You benefit from delayed_job goodness (different queues, scaling, ...). The hard part is getting the result.
One idea would be to associated a unique ID with each job, and have the delayed_job task output the result in a data store wich associated the job ID with the result. This data store can be a simple relational table
+----+--------+
| ID | Result |
+----+--------+
or a memecache/redis/whatever instance. You just need to poll that data store looking for the result associated with the job ID. And delete everything when you are done displaying that to the user.
3) Why don't you directly POST the data to the rails server ?
Following Benjamin's lead, I implemented a filter for this particular action.
def verify_ip
#ips = ['127.0.0.1']
if not #ips.include? request.remote_ip
redirect_to root_url
end
end
The listening app on the localhost now invokes the action, passing the JSON package received from the analytics engine as a param. Thank you.
I am building an application which will send status requests to users (via email & sms) on a regular basis. I want to execute the service each hour which will:
Query the database for all requests that need to be sent (based on some logic)
Send the requests through Amazon's Simple Email Service (this is already working)
Write a record of the status request notification back to the data store
I am considering wrapping up this series of operations into a single controller with an end point that can be called remotely to kick off the process within the rails app.
Longer term, I will break this process out into an app that can be run independently of my rails app, but for now I'm just trying to keep it simple.
My first inclination is to build the following:
Controller with the following elements:
A method which will orchestrate the steps outlined above (and can be called externally)
A call to the status_request model which will bring back a collection of request needing to be sent
A loop to iterate through the pending requests, which will:
Make a call to my AWS Simple Email Service module to actually send the email, and
Make a call to the status_request model to log the request back to the database
Model:
A method on my status_request model which will bring back a collection of requests that need to be sent
A method in my status_request model which will log that a notification was sent
Since this will behave as a service that gets called periodically from an outside scheduler I don't think I'll need a view for this operation. (Will, of course, need views to show users and admins what requests have been sent, but that's later...).
As someone new to Rails, I'm asking for review of this approach and any suggestions you may have.
Thanks!
Instead of a controller which Jeff pointed out exposes a security risk, you may just want to expose a rake task and use cron to invoke it on an hourly basis.
If you are still interested in building a controller, look at devise gem and its single access token, token_authenticatable, for securing the methods you are exposing.
You may also want to look at delayed_job or resque to offload the call to status_request and the loop to AWS simple service to a background worker process.
You may want a seperate controller and view for the log file so you can review progress on demand.
And if you want to get real fancy use Amazon SNS to send you alerts when the service reaches some unacceptable level of failures, backlog, etc.
Since you are trying to invoke this from an outside process, your approach should work. You could also have a worker process that processes task when they are there.
You will need routes to expose your service, and you may want to also make security decisions. How will the service that invokes your application authenticate so all others can't hit it at will?
Another consideration should be how many emails are you sending. If there are enough, we may want to look into the fact that writing this sort of loop is going to be extremely top heavy; and may affect users on the current system if it's a web application.
In the end, there are many ways to do this. I would focus on the performance/usage you expect as well as security. There's never one perfect way to solve a problem like this, and your way should just be aware of the variables it will need to be operating within.
Resque and Redis might be helpful to you in scheduling and performing operatio n .They are simple and superfast, [here](http://railscasts.com/episodes/271-resque] is a simple tut on same.
Does Rails provide a way to execute code on the server after the view is rendered and after the response is sent to the browser?
I have an action in my application that performs a lot of database transactions, which results in a slow response time for the user. What I'd like is to (1) perform some computations, (2) send the results of those computations to the browser, and then (3) save the results to the database.
It sounds like you want to implement a background job processor. This allows you to put the job into a queue to be processed asynchronously and for your users to not notice a long page load.
There are many options available. I have used and had no issues with delayed_job. Another popular one lately, which I have not used is resque.