I am new to Sidekiq and currently using a worker to send push notifications asynchronously. Push notifications are sent when a message is sent by a user to another.
Since Sidekiq best practices are to make worker params small and simple, is it fine to pass the message text (<140 characters) directly in the params ?
perform(message_text, user_id)
Or is it better to fetch it with the message_id?
Yeah, that's totally fine if the text is all you need. Even better might be to include the message_id as another parameter so you can lookup the message if you need to in the future, i.e.
perform(message_text, user_id, message_id)
Related
I've read most of the other answers on this topic, but a lot of them related to either third-party services like MailChimp (which I'm not necessarily opposed to) or how not to upset the host's email server.
I believe this case is unique so that it'll contribute...
I have my own DigitalOcean droplet running a rails app. I need to send out 100-1000 emails every so often, each with a unique message (a link I'm using for tracking clicks originating from the email).
I'm also operating my own iRedMail server.
Can someone recommend how to best-handle this task? I was going to simply cycle through the list of emails and use the template.html.erb to drop in my link, but what types of problems might I run into?
Thank you!
You should decouple your Rails App from the mail sending so that you don't have to wait in your view for the mails to be sent (assuming that you click on something that triggers the start of your mail sending). Use something like delayed_job or another queueing mechanism that Rails offers and only queue up the sending job of the e-mails. Then when the queue comes to execute the particular job you can customize the message with an HTML part and a text part or whatever else you need and pass them on individually to your MTA.
Sorry for the basic question about Sidekiq's delaying ActionMailer. As per this article, Sidekiq can delay sending out emails by just saying UserMailer.delay_for(1.hour).....
Does this mean this is handled in the background now, or does it mean that it simply just delays sending the email out for an hour but once that hour comes, then the email is basically being sent like a regular ActionMailer, which slows down response time?
Or is it that if I truly want to do this in the background then I would have to do the other sidekiq stuff like putting it in a specific Worker and then firing it up that way?
Also, separately, if I do just do it via UserMailer.delay..., I presume I won't need a worker dyno on Heroku to save some money, correct?
Thanks for the help!
Yes, for emails you don't need to do anything else. It's like calling the Mailer 1 hour later. You just need to make sure you don't pass any complex objects into the mailer, for example a user object, you should only pass the user_id, because it will be stored in redis. On the Mailer fetch the user object with the given id.
I am building an application which will send status requests to users (via email & sms) on a regular basis. I want to execute the service each hour which will:
Query the database for all requests that need to be sent (based on some logic)
Send the requests through Amazon's Simple Email Service (this is already working)
Write a record of the status request notification back to the data store
I am considering wrapping up this series of operations into a single controller with an end point that can be called remotely to kick off the process within the rails app.
Longer term, I will break this process out into an app that can be run independently of my rails app, but for now I'm just trying to keep it simple.
My first inclination is to build the following:
Controller with the following elements:
A method which will orchestrate the steps outlined above (and can be called externally)
A call to the status_request model which will bring back a collection of request needing to be sent
A loop to iterate through the pending requests, which will:
Make a call to my AWS Simple Email Service module to actually send the email, and
Make a call to the status_request model to log the request back to the database
Model:
A method on my status_request model which will bring back a collection of requests that need to be sent
A method in my status_request model which will log that a notification was sent
Since this will behave as a service that gets called periodically from an outside scheduler I don't think I'll need a view for this operation. (Will, of course, need views to show users and admins what requests have been sent, but that's later...).
As someone new to Rails, I'm asking for review of this approach and any suggestions you may have.
Thanks!
Instead of a controller which Jeff pointed out exposes a security risk, you may just want to expose a rake task and use cron to invoke it on an hourly basis.
If you are still interested in building a controller, look at devise gem and its single access token, token_authenticatable, for securing the methods you are exposing.
You may also want to look at delayed_job or resque to offload the call to status_request and the loop to AWS simple service to a background worker process.
You may want a seperate controller and view for the log file so you can review progress on demand.
And if you want to get real fancy use Amazon SNS to send you alerts when the service reaches some unacceptable level of failures, backlog, etc.
Since you are trying to invoke this from an outside process, your approach should work. You could also have a worker process that processes task when they are there.
You will need routes to expose your service, and you may want to also make security decisions. How will the service that invokes your application authenticate so all others can't hit it at will?
Another consideration should be how many emails are you sending. If there are enough, we may want to look into the fact that writing this sort of loop is going to be extremely top heavy; and may affect users on the current system if it's a web application.
In the end, there are many ways to do this. I would focus on the performance/usage you expect as well as security. There's never one perfect way to solve a problem like this, and your way should just be aware of the variables it will need to be operating within.
Resque and Redis might be helpful to you in scheduling and performing operatio n .They are simple and superfast, [here](http://railscasts.com/episodes/271-resque] is a simple tut on same.
In my rails app, I'm using the SendGrid parse API which posts mail to my server. Every now and then SendGrid's Parse API submits the same email twice.
When I get a posted mail I place it in the IncomingMail model. so in order to prevent this double submitting issue, I look at each IncomingMail when processing to see if there is a duplicate in the table within the last minute. That tested great on development, it caught all the double submissions.
Now I pushed that live to heroku, where I have 2+ dynos and it didn't work. My guess being that it has something to do with replication. So that being the case, how can scalable sites with multiple server deal with something like this?
Thanks
You should look at using a background job queue. Heroku has "Workers" (which was Delayed Job). Rather than sending the email immediately, you push it onto the queue. Then one or more Heroku 'workers' need to be added to your account, and each one will pull jobs in sequence. This means there can be a short delay (depending on load) before the email is sent, but this delay is not presented to the user, and should there be a lot of email to send you just add more workers.
Waiting for an external service like an email provider on each user action is dangerous because any network problem will take down your site as several users have to 'wait' for their HTTP requests to be responded to while Heroku is blocked with these third party calls.
In this situation with workers each job would fail but would be retried and eventually succeed.
This sounds like it could be a transaction issue. If you have multiple workers running simultaneously their operation may be 'interleaved'. For instance this sequence of events would result in 2 mails being sent.
Worker A : Checks for an existing record and doesn't find one
Worker B : Checks for an existing record and doesn't find one
Worker A : Post to Sendgrid
Worker B : Post to Sendgrid
You could wrap everything in a transaction to keep this from happening. Something like this should do it.
class IncomingMail < ActiveRecord::Base
def check_and_send(email_address)
transaction do
# your existing code for preventing duplicates and sending
end
end
end
Following a specific action the user takes on my website, a number of messages must be sent to different emails. Is it possible to have a separate thread or worker take care of sending multiple emails so as to avoid having the response from the server take a while to return if there are a lot of emails to send?
I would like to avoid using system process or scheduled tasks, email queues.
You can definitely spawn off a background thread in your controller to handle the emails asynchronously.
I know you want to avoid queues, but another thing i have done in the past is written a windows service that pulls email from a DB queue and processes it at certain intervals. This way you can separate the 2 applications if there is a lot of email to be sent.
This can be done in many different ways, depending on how large your application is and what kind of reliability you want. Any of these ways should help you achieve what you want (in ascending order based on complexity):
If you're using IIS SMTP Server or another mail server that supports a pickup directory option, you can go with that. With this option, instead of sending the emails directly, they are saved first in the pickup directory. Your call will immediately return after the email is saved in the pickup directory, so the user won't have to wait until the email is sent. On the other hand, the server will try to send the email as soon as it's saved in the pickup directory so it's almost immediate (just without blocking the call).
You can use a background thread like described in other answers. You'll need to be careful with this option as the thread can end unexpectedly before it finishes its job. You'll need to add some code to make sure this works reliably (personally, I'd prefer not to use this option).
Using a messaging queue server like MSMQ. This is more work and you probably should only look into this if you have a large scale application or have good reasons not to use the first option with the pickup directory.
There are a few ways you could do this.
You could store enough details about the message in the database, and write a windows service to loop through them and send the email. When the user submits the form it just inserts the required data about the message and trusts the service will pick it up. Almost an email queue which you said you didn't want, but you're going to end up in a queue situation with almost any solution.
Another option would be to drop in NServiceBus. Use that for these kinds of tasks.
I typically compile the message body and store that in a table in the db along with the from and to addresses, a subject, and a timestamp indicating when the email was sent. Then I have a background task check the table periodically and pull any that haven't been sent. This task attempts to send each email and updates the timestamp accordingly. One advantage of storing the compiled message body up front is that the background task doesn't have to do any processing of context-specific data, and therefore can be pretty darn simple.
Whenever an operation like is hingent upon an event, there is always the possibility something will go wrong.
In ASP.NET you can spawn multiple threads and have those threads do the action. Make sure you tell the thread it's a background thread, otherwise ASP.NET might way for the thread to finish before rendering your page:
myThread.IsBackground = true;
I know you said you didn't want to use system process or scheduled tasks, but a windows service would be a viable approach to this as well. The approach would be to use MS Queue, or save the actions needing to be done in a DataBase table. Then have a windows service check every minute or so and do those actions.
This way, if something fails (Email server down) those emails / actions can still be done.
They will also be recorded for audit's (which is very nice to have).
This method allows you're web site to function as a website while offloading these tasks to another service. The last thing you need is for multiple ASP.NET processes to be used up waiting for emails to send. let something else handle that.