I am working on a Rails application where customer refunds are handed to a Sweatshop worker. If a refund fails (because we cannot reach the payment processor at that time) I want to requeue the job.
class RefundWorker < Sweatshop::Worker
def process_refund(job)
if refund
Transaction.find(job[:transaction]).update_attributes(:status => 'completed')
else
sleep 3
RefundWorker.async_process_refund(job) # requeue the job
end
end
Is there any better way to do this than above? I haven't found any "delay" feature in RabbitMQ, and this is the best solutions I've come up with so far. I want to avoid a busy loop while requeueing.
Have you looked at things like Ruote and Minion?
Some links here: http://delicious.com/alexisrichardson/rabbitmq+work+ruby
You could also try Celery which does not speak native Ruby but does speak HTTP+JSON.
All of the above work with RabbitMQ, so may help you.
Cheers
alexis
Have a timed-delivery service? You'd send the message to deliver as payload, wrapped up with a time-to-deliver, and the service would hold onto the message until the specified time had been reached. Nothing like that exists in the RabbitMQ server or any of the AMQP client libraries, as far as I'm aware, but it'd be a useful thing to have.
It doesn't seem like AMQP (or at least RabbitMQ) supports the idea of "delay this job." So the approach to re-queue the same job from inside the worker if it fails seems to be the best solution at this time.
I have the code working in a demo environment and it meeting my needs so far.
Related
I want to send email batch at specific time like CRON.
I think whenever gem (https://github.com/javan/whenever) is not to fit in Cloud Foundry Environment. Because Cloud Foundry can't use crontab.
Please inform me what options are available to me.
There's a node.js app here that you could use to schedule a specific rake task.
I haven't worked with cloudfare so I'm not sure if it'll serve your needs, but you can also try some of the batch job processing tools rails has available: Delayed job and sidekiq. Those store data for recurring jobs either on your database (DJ) or in a separate redis database (Sidekiq) and both need keeping extra processes up and running, so review them deeply and the changes you'd need for your deployment process before using each one. There's also resque, and here's a tutorial to use it with rails for scheduling tasks.
There are multiple solutions here, but the short answer is that whatever you end up doing needs to implement its own scheduler. This is because there is no cron service available to your application when it runs on CF. This means there is nothing to trigger or schedule your actions. Any project or solution that depends on cron will not work when deploying to CF. Any project that implements it's own scheduler should work fine.
Some specific things I've seen people do successfully:
Use a web service that sends HTTP requests to your app on predefined intervals. The requests trigger your action. It's the services responsibility to let you define when to trigger and to send the HTTP requests. I'm intentionally avoiding mentioning any specific services, but you can find them by searching for "cron http service" or something like that.
Importing a library that has cron like functionality. I'm not familiar with Ruby, so I don't know the landscape there. #mlabarca has mentioned a couple that you might try out. Again, look to see that they implement the scheduling functionality and do not depend on cron. I'm more familiar with Java where you have Quartz and Spring, which has some scheduling functionality too.
Implement a "clock" process or scheduler. This would generally be a second app that you deploy on CF. It would be lightweight and probably not have a web interface. It could be as simple as do something, sleep, loop for ever repeating those two steps. It really depends on your needs. You could even get fancy and implement something like the first option above where you're sending some sort of request to your other apps to trigger the actual events.
There are probably other solutions as well, those are just some examples to get you started.
Probably also worth mentioning that the Cloud Controller v3 API will have first class features to run tasks. In this case, the "task" is some job that runs in a finite amount of time and exits (like a batch job). This is opposed to the standard "app" that when run on CF should continue executing forever (i.e. if it exits, it's cause of a crash). That said, I do not believe it will include a scheduler so you'd still need something to trigger the task.
I have 4 Rails apps: App1, App2, App3 and App4.
App2, App3 and App4 needs to update the same information (an email address for example, something as simple as that) when it has been updated on App1.
For now, and since the data I need to sync are not often changed, I went by using my already-in-place Sidekiq implementation to send a post request to the other Apps within a background job. But that definitly does not sounds as an ideal solution. This change is still in it's own branch and has not been yet merge on production. That's why I'm here for.
I've been tackling RabbitMQ as I read it definitely has this capabilities. However, RMQ sound a bit overkill for my use case, IMHO. So I would like to know how would you guys go for? Is there any other lightweight alternative ?
If you want to take the RMQ route you have to do clustering and federation which is very bloated for this kind of a simple problem. You can implement your own worker queue using Redis which Sidekiq is already using it as a message queue. Use RabbitMQ only for critical real time messaging queue systems.
RPOPLPUSH - Redis
I'd like to run a small amount of code asynchronously in my rails application. The code logs some known information. I'd like this task to not block the rest of the response from my app. The operation is way too lightweight and frequent for it to be done as a delayed job tasks.
I'm thinking to just use:
Thread.new do
# my logging code
end
and call it a day. Is this achieving what I want it to achieve? Are there any drawbacks?
It may be overkill for your particular usage, but have you considered using some form of Message Queuing Middleware such as STOMP, AMQP, OpenWire or even Jabber?
The basic outline (pseudo-code!) would be:
s = client.create_connection(user,pass,server_ip,port)
s.message_send("Log Message Goes Here")
You would then have at least one "consumer" at the other end of the message queue which would write the log message to a file/database/chatroom/IRC Channel/whatever-you-want-really-it's-all-code... :)
It would also mean that if in future you wanted to hand-off high-intensity processing jobs (invoice generation for example) you would already have the infrastructure in place to do so.
Also if you're looking for a really easy Messaging server, I recommend RabbitMQ - it's written in Erlang (but don't let that put you off!) and is very easy to setup.
This is my first post so I can't post any more than two links, but I've published links to all the technologies mentioned above in a gist # https://gist.github.com/1090372
Aside from handling some baseline resource contention things, this should be sufficient. The database resource comes to mind if you're logging there...a file if you're logging there. But the basic approach is fine I would think. Simpler is better...
I have a rails app using delayed_job. I need my jobs to communicate with each other for things like "task 5 is done" or "this is the list of things that need to be processed for task 5".
Right now I have a special table just for this, and I always access the table inside a transaction. It's working fine. I want to build out a cleaner api/dsl for it, but first wanted to check if there were existing solutions for this already. Weirdly I haven't found a single things, I'm either googling completely wrong, or the task is so simple (set and get values inside a transaction) that no one has abstracted it out yet.
Am I missing something?
clarification: I'm not looking for a new queueing system, I'm looking for a way for background tasks to communicate with one another. Basically just safely shared variables. Do the below frameworks offer this facility? It's a shame that delayed job does not.
use case: "do these 5 tasks in parallel, and then when they are all done, do this 1 final task." So, each of the 5 tasks checks to see if it's the last one, and if it is, it fires off the final task.
I use resque. Also there are lots of plugins, which should make inter-process comms easier.
Using redis has another advantage: you can use the pub-sub channels for communication between workers/services.
Another approach (but untested by me): http://www.zeromq.org/, which also has ruby bindings. If you like to test new stuff, then try zeromq.
Update
To clarify/explain/extend my comments above:
Why I should switch from DelayedJobs to Resque is the mentioned advantage that I have queue and messages in one system because Redis offers this.
Further sources:
https://github.com/blog/542-introducing-resque
https://github.com/defunkt/resque#readme
If I had to stay on DJ I would extend the worker classes with redis or zeromq/0mq (only examples here) to get the messaging in my extisting background jobs.
I would not try messaging with ActiveRecord/MySQL (not even queueing actually!) because this DB isn't the best performing system for this use case especially if the application has too many background workers and huge queues and uncountable message exchanges in short times.
If it is a small app with less workers you also could implement a simple messaging via DB, but also here I would prefer memcache instead; messages are short living data chunk which can be handled in-memory only.
Shared variables will never be a good solution. Think of multiple machines where your application and your workers can live on. How you would ensure a save variable transfer between them?
Okay, someone could mention DRb (distributed ruby) but it seems not really used anymore. (never seen a real world example so far)
If you want to play around with DRb however, read this short introduction.
My personal preference order: Messaging (real) > Database driven messaging > Variable sharing
memcached
rabbitmq
You can use Pipes:
reader, writer = IO.pipe
fork do
loop do
payload = { name: 'Kris' }
writer.puts Marshal.dump(payload)
sleep(0.5)
end
end
loop do
begin
Timeout::timeout(1) do
puts Marshal.load(reader.gets) # => { name: 'Kris' }
end
rescue Timeout::Error
# no-op, no messages to receive
end
end
One way
Read as a byte stream
Pipes are expressed as a pair, a reader and a writer. To get two way communication you need two sets of pipes.
I'm writing a simple Windows Service that sends out emails to all employees every month. My question is, how to stop itself when it's done? I'm a noobie in this field so please help me out. Really appreciated.
It will be deployed on the server to be run monthly. I did not start this thing and the code was given to me like that. It is written in VB.NET and I'm asked now to change a few things around it. I noticed that there is only 'Sub OnStart' and wondered when the service would stop? After the main sub is done, what it the status of this service? Is it stopped or just hung in there? Sorry, as I said, I am really new to this....
If you have a task that recurs monthly you may be better off writing a console app, and then using Windows Task Scheduler to set it to run monthly. A service should be used for processes that need to run for a long time or constantly, with or without a user logged on
As every other answer has noted, it sounds like this should be an executable or script that you run as a scheduled task.
However, if you are obligated for some reason to run as a Windows Service and you're working in .NET, you just have to call the Stop() method inherited from ServiceBase once your service completes its work. From the MSDN documentation for the method:
The Stop method sets the service state
to indicate a stop is pending and
calls the OnStop method. After the
application is stopped, the service
state is set to stopped. If the
application is a hosted service, the
application domain is unloaded.
There's one important caveat here: the user account under which the service is running must have permission to stop services (which is a topic for ServerFault).
Once a service's OnStart method completes, it will continue running (doing nothing) until something tells it to stop in one of the following ways:
Programatically, by calling Stop
within the service itself or from an
external process using the method
Colin Gravill describes in his
answer.
Via the command-line.
Through the windows Computer Management console's "Services" panel.
If this is a Win32 service (i.e. written in C or C++), then you simply call SetServiceStatus(SERVICE_STOPPED) and return from ServiceMain.
On the other hand, if you're just sending emails once a month, why are you using a service at all? Use the Windows Task Scheduler and run a normal application or script.
net stop [service_name] ...on the command line will do it too.
But, I agree with everyone else; it seems that Windows Task Scheduler will meet your needs better.
It might be better to write this as a scheduled task, it would certainly be easier to develop initially. Then it would naturally terminate and wouldn't be consuming resources for the rest of the month.
To answer the original question, you can get a list of the current running services in C#
services = System.ServiceProcess.ServiceController.GetServices();
Then look for the one you want and set the status to stopped
locatedService.Status == ServiceControllerStatus.Stopped
Full example on msdn
Is there a reason it has to be a Windows service? If not, then follow #Macros solution. However, if it does, then why stop the service? If you stop it, then it'll just have to be restarted when the emails need to be sent. Based on your description, it doesn't sound like it would require a lot of resources, so I'd suggest just installing it and letting it run, firing up once a month to send the emails.
here's what i did in a similar situation.
windows service runs 24/7 and processes work units. it gets work units through a database view.
table Message
ProcessingStartTime
CompletionDTE
...
the database view only pulls records marked not-complete and have a ProcessingStartTime in the past. So after the service confirms the transaction it executes a stored procedure that updates the database record. For this system, end-user upload excel files to asp.net webfrom that imports them into the database.