Matching service use consistent hash decide which queue is assigned to which server.
Most of time, the server will poll task from cache instead of persistent database.
If I add a new matching service, All cache in queue will be re-consistent-hash to new places, and this will cause all old cache outdated. Will it cause any problem?
Most of the time tasks are not cached but matched immediately to a waiting long poll. We call it a sync match. So adding a matching service shouldn't affect the health of the running applications.
Related
I have an orchestration that runs as a singleton by using the same instance id each time. It also runs infinitely by using ContinueAsNew at the end of each iteration to keep the history manageable. However, I have noticed that the history of each past iteration is kept in the history table, each with a different execution id (as is expected when ContinueAsNew is called).
I also use PurgeInstanceHistoryAsync once a day to delete any completed, failed, terminated or cancelled orchestrations that are more than 14 days old. However, since the infinite singleton orchestration is never in any of these states will PurgeInstanceHistoryAsync ever clean up the old execution histories?
The same question can be asked for a periodic singleton orchestration (i.e. an orchestration that runs periodically but uses the same instance Id each time). If the purge process happens whilst the orchestration is running, will any old histories be removed, or would it be a matter of luck that the orchestration is not actually running at the time the purge executes?
If you look in your history table in the azure storage account and query for your instance you should see that using ContinueAsNew will actually purge history automatically. (In my test it seemed to be at most 1 execution behind.)
From Docs: https://learn.microsoft.com/sv-se/azure/azure-functions/durable/durable-functions-eternal-orchestrations?tabs=csharp#resetting-and-restarting
When ContinueAsNew is called, the instance enqueues a message to itself before it exits. The message restarts the instance with the new input value. The same instance ID is kept, but the orchestrator function's history is effectively truncated.
I have some resources that I want to be cached and served at top speed to my app.
When I used appcache I got great serving speeds, but i was stuck with an appcache.
So I've replaced it with a service worker.
Then I tried the simplest strategy, just cache the static assets on install and serve them from the cache whenever fetched.
It worked, when I checked chrome's network panel I was happy to see my service worker in action, BUT - the load times were horrible, each resource load time doubled.
So I started thinking about other strategies, here you can find plenty of them, the cache and network race sounded interesting but i was deterred by the data usage.
So I've tried something different, I tried to aggressively cache the resources in the service worker's memory. Whenever my service worker is up and running it pools the relevant resources from the cache and save the response objects in memory for later use. When it gets a matching fetch it just responds with a clone of the in memory response.
This strategy proved to be fastest, here's a comparison I made:
So my question is pretty vague as my understanding in service workers is still pretty vague...
Does this all makes sense, can I keep static resources cache in memory?
What about the bloated memory usage, are there any negative implications in that? for instance - maybe the browser shuts down more frequently service workers with high memory consumption.
You can't rely on keeping Response objects in memory inside of a service worker and then responding directly with them, for (at least) two reasons:
Service workers have a short lifetime, and everything in the global scope of the service worker is cleared each time the service worker starts up again.
You can only read the body of a Response object once. Responding to a fetch request with a Response object will cause its body to be read. So if you have two requests for the same URL that are both made before the service worker's global scope is cleared, using the Response for the second time will fail. (You can work around this by calling clone() on the Response and using the clone to respond to the fetch event, but then you're introducing additional overhead.)
If you're seeing a significant slowdown in getting your responses from the service worker back to your page, I'd take some time to dig into what your service worker code actually looks like—and also what the code on your client pages look like. If your client pages have their main JavaScript thread locked up (due to heavyweight JavaScript operations that take a while to complete and never yield, for instance) that could introduce a delay in getting the response from the service worker to the client page.
Sharing some more details about how you've implemented your cache-based service worker would be a good first step.
I have a PendingEmail table which I push many records to for emails I want to send.
I then have multiple Que workers which process my app's jobs. One of said jobs is my SendEmailJob.
The purpose of this job is to check PendingEmail, pull the latest 500 ordered by priority, make a batch request to my 3rd party email provider, wait for array response of all 500 responses, then delete the successful items and mark the failed records' error column. The single job will continue in this fashion until the records returned from the DB are 0, and the job will exit/destroy.
The issues are:
It's critical only one SendEmailJob processes email at one time.
I need to check the database every second if a current SendEmailJob isn't running. If it is running, then there's no issue as that job will get to it in ~3 seconds.
If a table is locked (however that may be), my app/other workers MUST still be able to INSERT, as other parts of my app need to add emails to the table. I mainly just need to restrict SELECT I think.
All this needs to be FAST. Part of the reason I did it this way is for performance as I'm sending millions of email in a short timespan.
Currently my jobs are initiated with a clock process (Clockwork), so it would add this job every 1 second.
What I'm thinking...
Que already uses advisory locks and other PG mechanisms. I'd rather not attempt to mess with that table trying to prevent adding more than one job in the first place. Instead, I think it's ok that potentially many SendEmailJob could be running at once, as long as they abort early if there is a lock in place.
Apparently there are some Rails ways to do this but I assume I will need to execute code directly to PG to initiate some sort of lock in each job, but before doing that it checks if there already is one lock, and if there is it aborts)
I just don't know which type of lock to choose, whether to do it in Rails or in the database directly. There are so many of them with such subtle differences (I'm using PG). Any insight would be greatly appreciated!
Answer: I needed an advisory lock.
I am using in my app a background job system (Sidekiq) to manage some heavy job that should not block the UI.
I would like to transmit data from the background job to the main thread when the job is finished, e.g. the status of the job or the data done by the job.
At this moment I use Redis as middleware between the main thread and the background jobs. It store data, status,... of the background jobs so the main thread can read what it happens behind.
My question is: is this a good practice to manage data between the scheduled job and the main thread (using Redis or a key-value cache)? There are others procedures? Which is best and why?
Redis pub/sub are thing you are looking for.
You just subscribe main thread using subscribe command on channel, in which worker will announce job status using publish command.
As you already have Redis inside your environment, you don't need anything else to start.
Here are two other options that I have used in the past:
Unix sockets. This was extremely fiddly, creating and closing connections was a nuisance, but it does work. Also dealing with cleaning up sockets and interacting with the file system is a bit involved. Would not recommend.
Standard RDBMS. This is very easy to implement, and made sense for my use case, since the heavy job was associated with a specific model, so the status of the process could be stored in columns on that table. It also means that you only have one store to worry about in terms of consistency.
I have used memcached aswell, which does the same thing as Redis, here's a discussion comparing their features if you're interested. I found this to work well.
If Redis is working for you then I would stick with it. As far as I can see it is a reasonable solution to this problem. The only things that might cause issues are generating unique keys (probably not that hard), and also making sure that unused cache entries are cleaned up.
I'm going to use rabbitMQ as a message broker and switch most of the scripts to sending data to queue instead of performing direct writes/reads. Consumer will get those messages and perform corresponding operations. In my dreams this will give me more flexibility choosing DB engine, app level sharding and so on. But is it a good idea generally? Or am I missing something? Current write load is ~15k inserts/deletes for mysql and 30-50k sets for redis instances. Read load is the same ~15-20k selects, and 50-70k gets for redis.
The biggest issue you'll face will be the fact that your DB writes will be asynchronously processed. If a client writes data to the DB and then instantly reads it back, the value might not be what it originally inserted because the Rabbit queue might have been very busy or slow, delaying the update operation. Or an admin might accidentally purge your queue and then you'll have all these clients thinking their transactions had been committed but nothing will have been stored.
This sounds like a classic case of premature optimization. It's a solution in search of a problem, and you should probably avoid doing it.
With amqp you can run a none asynchronous operations using a RPC way, with this kind of architecture you should figure out all problems related with asynchronous operations.