We are using fetchmail to retrieve mails from different mailboxes and create tickets in queues configured in the request tracker. Fetching mail from one of the mail-ids is taking long. Anybody had similar experience? What can be the possible reasons for delay or any means to debug it?
One approach I tried was to spawn another fetchmail process for the mail causing delay, but request tracker mandates being run by root user and a single user cannot run multiple fetchmail process at a time
A little bit of digging into apache logs, I was able to figure it out that the size of the mails being downloaded from the one particular mailbox was quite large, which in turn cause the timeout.
Changing the FcgidIOTimeout configuration from 90 to 180 in the following configuration file helped.
/etc/httpd/conf.d/fcgid.conf.
Related
I had a python application, which, with the use of shell tricks, was able to send me emails when new error messages appeared in the log during an execution session, scheduled with cron.
Now I am packing it in docker and was able to reproduce most of its functionality with docker-compose.
But when it comes to emails on failures, I am not sure of what is the best way to implement it.
What are your suggestions? Are there any best practices?
Update:
The app runs couple of times a day. Previously, all prints to stderr was duplicated to stdout to preserve chronological order in the main log file. Then, the wrapper script would accumulate all stderr from a single session in another, temporary file. And if that file was not empty after the session, its content was send in a single email from me to myself through SMTP with proper authentication. I was happy to receive and able to handle them for the last few months.
Right now I see three possible solutions:
Duplicating everything worth sending to a temporary file right in the app, this way docker logs would persist. Then sending it after the session from the entrypoint, provided, there is a way to setup in container all the requirements.
Grepping docker log from the outside. But that's somewhat missing the point of docker.
Relaying reports via local net to another container, with something like https://hub.docker.com/r/juanluisbaptiste/postfix/ which then will send it in a email
I was not able to properly setup postfix or use mail utility inside the container, but python seems to work just fine https://realpython.com/python-send-email/
TL;DR look at NewRelic's free tier.
There is a lot to unpack here. First, it would be helpful to know more about what you had been doing previously with the backticks. Including some more information about what commands might change how I'd respond to this. Without that I might make some assumptions that are incorrect/not applicable.
I would consider errors via email a bad idea for a few reasons:
If many errors occur quickly, you inbox will get flooded with emails and/or flood a mail server with messages or flood the network with traffic. It's your mailbox and your network so you can do what you want, but it tends to fail dramatically when it happens, especially in production.
To send an email you need to send those messages through a SMTP server/gateway/relay of some sort and often times automated scripts like this get blocked as they trigger spam detections. When that happens and there is an error, the messages get silently dropped and production issues go unreported. Again, it's your data/errors and you can do that if you want, but I wouldn't consider it best practices as far as reliability goes. I've seen it fail to alert many times in my past experience for this very reason.
It's been my experiences (over 20 years in the field) that any alerting sent via email quickly gets routed to a sub-folder via a mail rule and start getting ignored as they are noisy. As soon as people start to ignore the messages, serious errors get lost in the inbox along with them and go unnoticed.
De-duplication of error messages isn't built-in and you could get hundreds or thousands of emails in a few seconds, often with the one meaningful error like finding a needle in a haystack of emails.
So I'd recommend not using email for those reasons. If you were really set on using email you could have a script running inside the container that tails the error log and sends the emails off using some smtp client (that could be installed in the docker container). You'd likely have to setup credentials depending on your mail server but it could be done. Again, I'd suggest against it.
Instead I'd recommend having the logs be sent to something like AWS SQS or AWS CloudWatch where you can setup rules to alert (via SNS which supports email alerting) if there are N messages in N minutes. You can configure those thresholds as you see fit and it can also handle de-duplication.
If you didn't want to use AWS (or some other Cloud provider) you could perhaps use something like Elasticache to store the events with a script to check for recent events/perform de-duplication.
There are plenty of 3rd party companies that will take this data and give you an all-in-one solution for storing the logs and a nice dashboard with custom notifications (via email/SMS/etc.), some of which are free. NewRelic comes to mind and is free assuming you don't need log retention.
I don't work for any of these companies, just some tools I've worked with that I'd consider using before I tried to roll a cronjob to send SMTP messages (although I've done that several times in my career which I needed something quick and dirty).
The problem I have is to do with generating large reports. I am doing so using the prawn gem to got results however I was wondering if this could be migrated to a background process.
Since I am using faye for push notifications and sidekiq for background tasks, a potential solution would be to generate the report in a sidekiq worker and use faye to notify the client of the completion of the worker. The issue with this is I don't see a way of cleaning up the generated file elegantly. I don't think generating the file within the controller action is feasible as it leads to unreasonable loading times and blocks other requests.
Is this system possible? Or am I thinking about this in the wrong way?
You are right, this is a perfectly valid to do things.
I am not sure what you means about "cleaning up the generated file". If you mean deleting from the file system, you could do it in the controller who download it, and eventually add a daily cron job who remove all remaining files.
We used such system in various projects.
Another option if the generation is really long is to send an email (if you got it) once the report is generated with the report embedded or with a link to it.
Its my understanding that when I upload a file to my heroku instance its a synchronous request and I will get a 200 back when the request is done, which means my upload has been processed and stored by paperclip.
I am using plupload which does a serial upload (one file at a time). On Heroku I have 3 dynos and my app becomes unresponsive and I get timeouts trying to use the app. My upload should really only tie up at most a single dyno while all the files are being uploaded since its done serially and file 2 doesnt start until a response is returned from file 1.
As a test I bumped my dynos to 15 and ran the upload. Again I see the posts come into the logs and then I start seeing output of paperclip commands (cant remember if it was identify or convert) and I start getting timeouts.
I'm really lost as to why this is happening. I do know I 'can' upload directly to s3 but my current approach should be just fine. Its an admin interface that is only used by a single person and again at most it should tie up a single dyno since all the uploaded files are sent serially.
Any ideas?
I've been working on the same problem for a couple of days. The problem, so far as I understand, is that when uploading files through heroku, your requests are still governed by the 30 second timeout limit. On top of this, it seems that subsequent requests issued to the same dyno (application instance) can cause it to accrue the response times and terminate. For example, if you issue two subsequent requests to your web app that each take 15 seconds to upload, you could recieve a timeout, which will force the dyno to terminate the request. This is most likely why you are receiving timeout errors. If this continues on multiple dynos, you could end up with an application crash, or just generally poor performance.
What I ended up doing was using jquery-file-upload. However, if you are uploading large files (multiple MBs), then you will still experience errors as heroku is still processing the uploads. In particular I used this technique to bypass heroku entirely and upload directly from the client's browser to s3. I use this to upload to a temp directory, and then use carrierwave to 're-download' the file and process medium and thumbnail versions in the background by pushing the job to Qu. Now, there are no timeouts, but the user has to wait for the jobs to get processed in the background.
Also important to note is that heroku dynos operate independently of each other, so by increasing the number of web dynos, you are creating more instances of your application for other users, but each one is still subject to 30 second timeouts and 512Mb of memory. Regardless of how many dynos you have, you will still have the same issues. More dynos != better performance.
You can use something like Dropzonejs in order to divide your files in queue and send them separately. That way the request wont timeout.
I have a web app which has a single long running task - generating a PDF report. Various graphs are generated, and it takes about 15 sec to process in all. The report is generated by a user.
Processing the report at the time of request currently causes the process to be tied up, and more importantly (given that use of this website is not heavy) sometimes the request times out.
I am therefore redesigning the architecture of this section of the app (Rails 2.3.8). To put this in context, it's unlikely that more than a couple of these reports will be generated per day, and this is an extremely niche application, so significant further scaling isn't a major concern. I do intend to hand off the project in the future though, so stability is.
The most obvious solution I think is to use Spawn to generate a report, and fire a download link to the user in an email once it's complete. Another solution I've looked into is DelayedJob.
Can anyone who's done something similar recommend one approach over another?
delayed_job, or some other queueing mechanism is going to be the easiest thing to set up. With delayed_job you would just enqueue your worker instead of creating the PDF, and a background process on the server would be working from the queue doing whatever work was available. Using spawn to fork your whole process seems a little heavy-handed, and doesn't seem to lend itself well to other minor, but still longer running tasks (like sending emails).
Should verification emails be processed / sent in the background using some kind of background job (Resque, Delayed_Job, etc)?
The app does have a tendency to hang until a verification email is sent. However, it only hangs for a split second or two. Don't know if its worth sending over as a background task.
What are your thoughts?
The main thing to keep in mind is that this delay isn't affecting just the current user, it's also affecting other users because it's holding up the Rails process (the exact effects of this will depend on the web server you're using and your setup).
If this application doesn't have a lot of users (and won't in the future) and you don't have any background job processing at the moment, then it may not be worth adding it. Otherwise it's probably a good idea.
If it's a few split seconds then I think that you can live with it. But if it takes longer then you may move it to delayed_job unless your delayed_job is always clogged and you are too much into background jobs. In that case it may take a bit longer to send the email to user and may ruin user experience. Even in that case you could set a higher priority for sending the verification emails.
I have no doubts about the capability of delayed_job and have been personally using it for quite some time now. Even Github recommends it while releasing resque:
https://github.com/blog/542-introducing-resque
We need a background job system as serious as our web framework. I highly recommend DelayedJob to anyone whose site is not 50% background work.