I have a logging query (a simple INSERT) that happens on every single request.
For this request only (the one that happens on every page load), I want to set the limit to 500ms in case the database is locked/slow/down it won't affect the site, where the site hangs while it waits to connect/write.
Is there a way I can specify a timeout somehow on a per-query basis that I can abort the LoggedRequest.create! if it's taking too long?
I don't want to set it in my config because I have many other queries that shouldn't have timeouts that low.
I'm using Postgres 11.7
I also don't know how I feel about setting a timeout for the entire session because I don't want that connection to be shared from the pool with other queries that can't have that timeout.
Rails 6 introduces event based triggers for notifications, logging etc that comes in very handy, provided you are using/can afford to migrate to Rails 6. Here'a useful post that demonstrates creating event based triggers for notifications/logging: https://pramodbshinde.wordpress.com/2020/03/20/custom-events-tracking-with-activesupportnotifications-and-audited/
If, for some reason, you cannot use Rails 6, perhaps this article might help you find some answers: https://evilmartians.com/chronicles/the-silence-of-the-ruby-exceptions-a-rails-postgresql-database-transaction-thriller
If I were you, I could also contemplate using AJAX with a fire-and-forget API request to server for logging/whatever that is not critical to normal functioning of the application.
Related
I have a very weird situation: I have a system where a client app (Client) makes an HTTP GET call to my Rails server, and that controller does some handling and then needs to make a separate call to the Client via a different pathway (i.e. it actually goes via Rabbit to a proxy and the proxy calls the Client). I can't change the pathway for that different call and I can't change the Client at all (it's a 3rd party system).
However: the issue is: the call via the different pathway fails UNLESS the HTTP GET from the client is completed.
So I'm trying to figure out: is there a way to have Rails finish the HTTP GET response and then make this additional call?
I've tried:
1) after_filter: this doesn't work because the after filter is apparently still within the Request/Response cycle so the TCP/HTTP response back to the Client hasn't completed.
2) enqueuing a worker: this works, but it is not ideal because if the workers are backed up, this call back to the client may not happen right away and it really does need to happen right after the Client calls the Rails app
3) starting a separate thread: this may work, but it makes me nervous: adding threading explicitly in Rails could be fraught with peril.
I welcome any ideas/suggestions.
Again, in short, the goal is: process the HTTP GET call to the Rails app and return a 200 OK back to the Client, completely finishing the HTTP request/response cycle and then call some extra code
I can provide any further details if that would help. I've found both #1 and #2 as recommended options but neither of them are quite what I need.
Ideally, there would be some "after_response" callback in Rails that allows some code to run but after the full request/response cycle is done.
Possibly use an around filter? Around filters allow us to define methods that wrap around every action that rails calls. So if I had an around filter for the above controller, I could control the execution of every action, execute code before calling the action, and after calling it, and also completely skip calling the action under certain circumstances if I wanted to.
So what I ended up doing was using a gem that I had long ago helped with: Spawnling
It turns out that this works well, although it required a tweak to get it working with Rails 3.2. It allows me to spawn a thread to do the extra, out-of-band callback to the Client, but let the normal, controller process complete. And I don't have to worry about thread management, or AR connection management. Spawnling handles that.
It's still not ideal, but pretty close. And it's slightly better than enqueuing a Resque/Sidekiq worker as there's no risk of worker backlog causing an unexpected delay.
I still wish there was an "after_response_sent" callback or something, but I guess this is too unusual a request.
(APEX 4.1.1.00.23)
I have two applications A and B that share the same session (because they use the same session cookie), and each has Maximum Session Idle Time set to the same value N. Having established a session and visited both applications, if I then spend more than N seconds working in application A (doing lots of page loads so not timing out), if I then navigate to application B it immediately times out and sends me to its login page.
I tried also calling APEX_UTIL.SET_SESSION_MAX_IDLE_SECONDS(N) in both applications, with p_scopr defaulting to 'SESSION', noting that the API docs say
This would be the most common use case when multiple Application
Express applications use a common authentication scheme and are
designed to operate as a suite in a common session.
However the same thing happens.
I want the timeout to apply to the session as a whole, not to each application independently. Is this not what the above is supposed to achieve, or am I doing something wrong?
I got the answer to this from Christian Neumueller on the Oracle APEX forum:
... it's no issue anymore in 4.2. Looking at the 4.1.1
code, it seems that the problem is how we stored the last access time.
While the APEX_UTIL call with SESSION scope would set the idle timeout
for both apps, we maintained a timer (FSP_LAST_REQUEST_TIME) for each
app. Working in TIMTEST1 only updated the timer for TIMTEST1, not for
TIMTEST2. After working with one app and switching back to the other
app, Apex sees the stale timer and decides that the session expired.
This is clearly a bug. The bad news is that a backport is not
feasible, because so much has changed in session state management.
I noticed that in a standard grails environment, a request is always executed to the end, even when the client connection is lost and the result can't be delivered anymore.
Is there a way to configure the environment in such a way that execution of a request is canceled as soon as the client connection is lost?
Update: Thanx fo the answers. Yes - most of the problems I am trying to avoid can be avoided by better coding:
caching can make nearly every page fast
a token can help to avoid submitting something twice
but there are some requests which still could consume some time. Let's take a map service as example. Calculating a route will take some time. One solution to avoid resubmitting the request could be a "calculationInProgress" flag together with a message to the user. But then it is still possible to create a lot of sessions and thus a lot of requests in order to do a DOS attack...
I am still curious: is there no way to configure the server to cancel the request? I used to develop on a system where the server behaved this way and it was great :-)
Probably there is no such way. And I'm sure grails (and your webcontainer) is designed to
accept incoming request
process it on server side
send response
if something happened during phase 2, i'll know about it only on send response phase. Actually you can send data to HttpSerlvetRespone by yourself, handle IOException, etc - but it will be too much low-level way, I think. And it will not help you with canceling your DB operations, while you're preparing data to send.
Btw, it's common pattern to use an web frontend, like nginx, that accepts incomming request and and handle all this problems with cancelled requests, slow requests (i guess it's the real problem?), etc.
According to your comment it is reload and multiple clicks that you are trying to avoid. The proper technique should be to use Grails support for handling multiple form submissions:
http://grails.org/doc/2.0.x/guide/theWebLayer.html#formtokens
I'm aware of the model that involves a scheduled task runninng in the back ground which runs jobs registered with a web request but how about this for an idea that keeps everything within ASP.net...
User uploads a CSV file with, perhaps, several thousand rows. The rows are persisted to the database. I think this would take maybe a minute or so which would be an acceptable wait.
Request returns to the browser and then an automatic Ajax request would go back to the server and request, say, ten rows at a time and process them. (Each row requires a number of web service requests.)
Ajax call returns, display is updated and then another automatic Ajax request goes back for more rows. This repeats until all rows are completed.
If user leaves the web page, then they could return and restart the job.
Any thoughts?
Cheers, Ian.
If i get you right, you actually dont need any "interaction" between background jobs and the long-running request, you just want to "lauch" background jobs with incoming requests? Not such a good idea. Take a look at the Quartz.NET project, it is scheduler embeddable into ASP.NET application, it will handle this stuff for you without need of requests. Of course, if there is app pool shutdown, also your scheduler goes down, but this you cant guarantee not to happen even with your long-running requests solution, dependent on browser waiting on other side.
Also take a look on this interesting article from phil haack on this topic, with his own little scheduler library specific for ASP.NET :
http://haacked.com/archive/2011/10/16/the-dangers-of-implementing-recurring-background-tasks-in-asp-net.aspx
A server side program (or ideally service) could still be quick and dirty and would be more reliable. You could still do step 1 as you have proposed, upload the file and insert the data ( don't forget to increase the maxRequestLength time out value in the web.config ). Then have a program running on the server that checks for new records and processes them.
If the user needs status you could store an entry in the database for each file and update the database record when the import is complete.
Maybe I'm reading the question and interpreting it in a weird way, but why couldn't you read the file into a database and store in a table the current line of the file that you've completed through. You could then track your progress via the db and just send small json objects telling the user how far along you are. That way if their connection drops you can keep processing their request, and if they return later you can notify them of how far along the job is. Also, if multiple clients are connecting you can use the db to queue and throttle (by serializing) the workload. Or if the user connects mid-job with another file, then their new request will be queued up after their current job.
I am trying to use jQuery's .ajax functionality to make a progress bar.
A request is submited via .ajax, which starts a long running process. Once submited another .ajax request is called on an interval which checks the progress of this process. Then a progress meter is updated using this information.
However, the progress .ajax call only returns once the long running process has completed. Its like its being blocked by the initial request.
The weird thing is this process works fine on dev but is failing on the deployment server. I am running on IIS using ASP.Net MVC.
Update: Apparently, it is browser related because it is working fine on IE 7 but is not working on IE 8. This is strange because IE 8 allows up to 6 connections on broadband where IE 7 only allows 2 requests per domain.
Update2: I think it's a local issue because it appears to be working fine on another IE 8 machine.
The server will only run one page at a time from each user. When you send the requests to get the progress status, they will be queued.
The solution is to make the page that returns the status sessionless, using EnableSessionState="false" in the #Page directive. That way it's not associated with any user, so the request isn't queued.
This of course means that you can't use session state to communicate the progress state from the thread running the process to the thread getting the status. You have to use a different way of keeping track of running processes and send some identifier along with the requests that gets the status so that you know which user it came from.
Some browsers (in particular, IE) only allows two requests to the same domain at the same time. If there are any other requests happening at the same time, then you might run into this limitation. One way around it is to have a few different aliases for the domain (some sites use "www1.example.com" and "www2.example.com", etc)
You should be able to use Firebug or Fiddler to determine how many requests are in progress, etc.
Create an Asynchronus handler (IHttpAsyncHandler) for your second ajax request.
use any parameter required via the .ashx querystring in order to process what you want because the HttpContext won't have what you'll need. You barely will have access to the Application object.
Behind the scenes ASP.NET will create for you a thread from the CLR pool, not the application pool, so You'll have an extra performance gain with IHttpAsyncHandler