I am using ASP.NET MVC with NServiceBus and where as the vast majority of commands can be executed with eventual consistency in mind, there are a small minority of tasks where immediate consistency would appear to simplify things.
I have done plenty of research on the various methods used to accomplish this but few come with any kind of justification as to why that particular method is preferable. I don't have any experience with NSB in a production environment, so it would also be nice to know if any methods limit scalability in any way.
The following are broadly the methods I have come across: -
No synchronisation, fake the information back to the client. My reservations with this one are firstly, you have to deal with the case where you have faked the data and the command has failed (unlikely scenario) and more importantly, if the initialisation of any data within the command is complex, the ability to fake this data is not necessarily feasible anyway.
Reply (or publish event to be recieved by client) when the task is completed. My reservation with this one is that it means that the distributed architecture becomes more complex and I am not sure if load balanced clients would cause issues as only one of the client machines should be recieving the reply.
Poll the read store until data is present. My reservation with this one is that it puts the read store under more load than the other options.
Are there any options which are better than the above three and if so, why? If not, which of the above three are better and why?
I am assuming that the answer is not subjective and one suits using NServiceBus to implement the command infrastructure in CQRS better than the others.
Thanks.
My take on this is that the actual endpoint should not be performing the work but be handing it off to some 'Task' (Application Service / Operation Script) object. That object is performing the work immediately.
So for cases where you absolutely have to have 100% consistency rather call that same task object rather than sending a command for later processing. You may still want that command for other scenarios.
Related
I have a multitenant-Rails app with multiple delayed_job workers.
In order to avoid overlapping tenant-specific work, I would like to separate the workers from each other in such a way that each one works on only one tenant-specific task at a time.
I thought about using the (named) queue column and add "tenant_1", "tenant_2" and so on. Unfortunately the queues have to be named during configuration, so this principle is not flexible enough for many tenants.
Is there a way to customize the way delayed_job picks the next task? Is there another way to define a scope?
Your best bet is probably to spin a custom solution that implements a distributed lock - essentially, the workers all run normally and pull from the usual queues, but before performing work check with another system (Redis, RDBMS, API, whatever) to verify that no other worker is yet performing a job for that tenant. If that tenant is not being worked, then set the lock for the tenant in question and work the job. If the tenant is locked, don't perform the work. It's your call on a lot of the implementation details like whether to move on to try another job, re-enqueue the job at the back of the queue, whether to consider it a failure and bind it to your retry limits, or do something else entirely. This is pretty open-ended, so I'll leave the details to you, but here are some tips:
Inheritance will be your friend; define this behavior on a base job and inherit from it on the jobs you expect your workers to run. This also allows you to customize the behavior if you have "special" cases for certain jobs that come up without breaking everything else.
Assuming you're not running through ActiveJob (since it wasn't mentioned), read up on delayed_job hooks: https://github.com/collectiveidea/delayed_job/#hooks - they may be an appropriate and/or useful tool
Get familiar with some of the differences and tradeoffs in Pessimistic and Optimistic locking strategies - this answer is a good starting point: Optimistic vs. Pessimistic locking
Read up on general practices surrounding the concept of distributed locks so you can choose the best tools and strategies for yourself (it doesn't have to be a crazy complicated solution, a simple table in the database that stores the tenant identifier is sufficient, but you'll want to consider the failure cases - how to you manage locks that are abandoned, for example)
Seriously consider not doing this; is it really strictly required for the system to operate properly? If so, it's probably indicative in an underlying flaw in your data model or how you've structured transformations around that data. Strive for ACIDity in your application when thinking about operations on the data and you can avoid a lot of these problems. There's a reason it's not a commonly available "out of the box" feature on background job runners. If there is an underlying flaw, it won't just bite you on this problem but on something else - guaranteed!
If you are trying to avoid two different workers working on the same tenant then that's a bad design choice. something is smelling. fix that first. however, if you want the same kind of worker instances working on different tenents below is the easiest solution. These relationships are my hypotheses.
ExpiredOrderCleaner = Struct.new(:tenant_id) do
def perform
Order.where(tenant_id: tenant_id).expired.delete_all
end
end
Tenant.each do |tenant|
Delayed::Job.enqueue ExpiredOrderCleaner.new(tenant.id)
end
this will create unique jobs for each tenant. single worker instance will work on a specific tenant. however, there can be other kinds of jobs working on the same tenant. which is good as it should be. if you need to more smaller scope, just pass more arguments for the worker and use in the query and use database transactions to avoid collisions.
these best practices are true for any background worker.
Make your job idempotent and transactional means that your job can safely execute multiple times
Embrace Concurrency design your jobs so you can run lots of them in parallel
your work will be a lot easier if you use apartment gem and active job wrappers. see the examples from there documents.
Problem: I have a program that will do a lot of post-processing on a file and then sending it ANOTHER web-service for more processing. Does my design smell and is this the right way to ‘tackle a problem’
Rails Accepts file and kicks off a resque_job to do work
The resque job will sends work via REST to another web-service cluster(very slow.. does MORE work) && places the task to be monitored in a monitor_queue within Rabbit MQ for completion . The Resque job will not wait another-webservice to complete task. It exits
There are some smell issues in my design, perhaps gut reactions that could be misguided?
Is it good design to have TWO message queues. The rational is that Rabbit_MQ has a built in method for creating worker_queues (they even give sample code). Resque_Job uses redis, and seems to be the accepted method of having ‘delayed jobs’ with Rails.
What I like about RabbitMQ: It has round-robin tasking abilities (so all threats get tasked) , and guarantees work will not be removed from a QUEUE without a message acknowledgement.
Resque seems to primary suggested solution for launching delayed_jobs within Rails.
Followup: When performing the polling, I was thinking a simple worker_queue of just iterating through the entire queue with seperate 'workers' makes the most sense? Do you agree.
I don't think this is a bad design. It's a type of Service Oriented Architecture, and even though you have separate queuing systems, they're completely separate applications and would only communicate through a specific interface, which has some pros and cons. I didn't quite understand the reasoning for using RabbitMQ, though. Also, a lot of new apps seem to be using Sidekiq; IMHO it is superior in every way to Resque.
I currently have an API for one of my projects and a service that is responsible for generating export files as CSVs, archive and store them somewhere in the cloud.
Since my API is written in Rails and my service in plain Ruby, I use the Her gem in the service to interact with the API. But I find my current implementation less performant, since I do a Model.all in my service, which in turn triggers a request that may contain way too many objects in the response.
I am curious on how to improve this whole task. Here's what I've thought of:
implement pagination at API level and call Model.where(page: xxx) from my service;
generate the actual CSV at API level and send the CSV back to the service (this may be done sync or async).
If I were to use the first approach, how many objects should I retrieve per page? How big should a response be?
If I were to use the second approach, this would bring quite an overhead to the request (and I guess API requests shouldn't take that long) and I also wonder whether it's really the API's job to do this.
What approach should I follow? Or, is there something better that I'm missing?
You need to pass a lot of information through a ruby process, that's always not simple, I don't think you're missing anything here.
If you decide to generate CSVs at the API level then what do you get with maintaining the service? You could just ditch the service altogether because replacing your service with an nginx proxy would do the same thing better (if you're just streaming the response from API host)?
If you decide to paginate, there will be a performance reduction for sure, but nobody can tell you exactly how much you should paginate - bigger pages will be faster and consume more memory (reducing throughput by being able to run less workers), smaller pages will be slower and consume less memory but demand more workers because of IO wait times,
exact numbers will depend on the IO response times of your API app and the cloud and your infrastructure, I'm afraid no one can give you a simple answer you can follow without experimentation with a stress test, and once you set up a stress test, you will get a number of your own anyway - better than anybody's estimate.
A suggestion, write a bit more about your problem, constraints you are working under etc and maybe someone can help you with a bit more radical solution. For some reason I get the feeling that what you're really looking for is a background processor like sidekiq or delayed job, or maybe connect your service to the DB directly through a DB view if you are anxoius to decouple your apps, or an nginx proxy for API responses, or nothing at all... but I really can't tell without more information.
I think it really depends how you want do define 'performance' and what your goal for your API is. Do you want to make sure no request to your API takes longer than 20msec to respond, than adding pagination would be a reasonable approach. Especially if the CSV generation is just an edge case, and the API is really built for other services. The number of items per page would then be limited by the speed at which you can deliver them. Your service would not be particularly more performant (even less so), since it needs to call the service multiple times.
Creating an async call (maybe with a webhook as callback) would be worth adding to your API if you think it is a valid use case for services to dump the whole record set.
Having said that, I think strictly speaking it is the job of the API to be quick and responsive. So maybe try to figure out how caching can improve response times, so paging through all the records is reasonable. On the other hand it is the job of the service to be mindful of the amount of calls to the API, so maybe store old records locally and only poll for updates instead of dumping the whole set of records each time.
I have an ASP.NET MVC application which gathers data from multiple Databases.
The databases hold information for various sites and for every new site we have a new Database. The database for each site is connected at two points, from the site and then from HQ.
A web application updated data every minute from the site and the data is is served to the HQ (via another web application) every minute. Sometimes the application response is very slow and from what I have investigated, it may be because the connection pool starts filling up swiftly.
I want to ask what is the best approach to such application, where I can get the best performance out of it. Any guidance is welcome.
How to improve your web application performance regarding to database, really depends on your architecture. But there are some general rules which you should always follow:
Check about thread starvation:On the Web server, the .NET Framework
maintains a pool of threads that are used to service ASP.NET
requests. When a request arrives, a thread from the pool is
dispatched to process that request. If the request is processed
synchronously, the thread that processes the request is blocked
while the request is being processed, and that thread cannot service
another request.
This might not be a problem, because the thread
pool can be made large enough to accommodate many blocked threads.
However, the number of threads in the thread pool is limited. In
large applications that process multiple simultaneous long-running
requests, all available threads might be blocked. This condition is
known as thread starvation. When this condition is reached, the Web
server queues requests. If the request queue becomes full, the Web
server rejects requests with an HTTP 503 status (Server Too Busy).
for "thread starvation" the best approach is using "Asynchronous
Methods". refer here for more information.
Try to use using block for your datacontext, to dispose them immediately after finishing with them.
Huge data amount in transaction: you should check your code.
May be you using too much data without need to all of them. For
example you transfer all object which you may need just one
properties of object. In this case use "projection"(refer here for
an example).
Also you may use "lazy loading" or "eager loading" base on you
scenarios. But please be noted that none of these are magic tool for
every scenario. In some cases "lazy loading" improve performance and
on others "eager loading" makes things faster. It depends to your
deep understanding of these two terms and also your case of issue,
your code and your design.
Filter your data on server side or client side. Filtering data on server side helps to keep your server load and network traffic as less as possible. It also makes your application more responsive and with better performance. Use IQueryable Interface for server side filtering (check here for more information).
One side effect of using server side filtering is having better security
Check your architecture to see do you have any bottleneck. A
controller which gets called too much, a methods which handles lots
of objects with lots of data, a table in database which receives
requests continuously, all are candidates for bottle neck.
Ues cashing data when applicable for most requested data. But again
use cashing wisely and based on your situation. Wrong cashing makes
your server very slow.
If you think your speed issue is completely on your database, the best approach is using sql profiling tools to find out which point you have critical situation. Maybe redesign of your own tables could be an answer. Try to separate reading and writing tables as much as possible. Separation could be done by creating appropriate views. Also check this checklist for monitoring your database.
I have an ASP.NET MVC 4 app hosted as an Azure web role. I want to do something that seems like it should be pretty standard: I want to create a function that I can call that initiates a VIP swap and raises and event (or calls a callback) when the VIP Swap operation is done.
Just to add some context to the situation: My website implements a workflow that takes about an hour (or less) to complete. If I want to release a new version of the website code, it's convenient (i.e. much less "backward compatibility" code to write) to first let all of the current users complete the workflow so that the new code doesn't need to deal with data created by the previous version of the code. So a management function in my website would first poke a value into the database that disables new workflows; it would then wait until all current workflows are done; it would then call the "VIP Swap" routine; finally, when the VIP Swap routine signals its completion, it would poke the database value to re-enable new workflows.
I found the Microsoft documentation for how to programmatically initiate a VIP swap here:
http://msdn.microsoft.com/en-us/library/ee460814.aspx
The procedure involves POSTing to a magic URL and including some headers in the POST, then periodically performing a GET to a magic URL and checking the response code.
The more I think about this, the more non-trivial it seems. In addition to the basic complexities of wiring up a background timer and completion notification, I don't know what complexities, if any, I might run into trying to do this stuff in the IIS environment. Can I even perform HTTP operations on a background thread? For that matter, will I run into complications just trying to use any of the half dozen or so different "do things in the background" mechanisms baked into .NET?
Any help or guidance will be greatly appreciated. In particular, I'd be ecstatic if someone could point me at a ready-to-go implementation of this function!
I don't think you will find an easy solution to this as the fabric controller is setup to do some very fancy things without your involvement. Running hour-long workflows on a cloud computing environment, where an instance can be pulled out from underneath you, (with a maximum of 5 minutes from the OnStopping event being called to clean up) requires that you do other work anyway to make sure that all of your tasks complete.
The simple question is "What do you do if an instance goes down when workflows are still running?" Do you restart them or are they lost? If they get lost then you don't care anyway, so killing workflows for an upgrade are equally unimportant. If you re-start them then use that same mechanism to decide whether or not a node is due to be shut down, and distribute the jobs accordingly. This pattern is eerily similar to the Hadoop JobTracker. Don't just run the workflows on any 'ol instance. Submit them to a (job tracker) service that decides what to do. The (job tracker) service can then use the service management API to scale up as many instances as you need running the version that you want, run workflows on the appropriate node, and shut them down when they are no longer needed or are outdated.
Unfortunately this may not be the simple solution that you are looking for, but something in your architecture needs to change, rather than trying to force PaaS to fit with your current approach. Decompose your workloads, create loosely coupled services, design for failure, and a few other cloud/distributed computing practices need to be considered. There is a reason why Hadoop is built the way that it is — and it has a reputation for being able to get work done on a bunch of somewhat unreliable commodity hardware.