I am going through Netflix Conductor documentation. It keeps using task polling and loop polling but I can't seem to get my head around it. Please help. TIA.
Conductor server has a queue, when a task is scheduled, it mean conductor put a record into that queue (image like insert into queue_message values ...) ~ Image the manager creating a issue on Jira on some project board
Conductor worker/client will keep polling the server, using REST that will result in querying the above queue_message for a specific taskType (depend on the worker) ~ Image the developer keeps checking the board for thing to do and pick up the thing to work on once he sees it.
Related
When the Spring Cloud Data Flow server uses local deployer to handle the task lifecycle management(launch, stop, etc.,), the corresponding task execution log can be obtained only when the task execution status is RUNNING.
This is by design because the local task launcher prunes the task instance history every time a new task instance is launched and hence the access to the log is not available, which is explored by the code here.
The reason was not to grow the number of Task process IDs in the local deployer' in process Map. You can see the issue related to it here.
But, this causes some side effects as discussed in another thread as not being able to show the previous instances' task execution log in local deployer mode.
I think it would be ok to consider having some X number of task executions in history and that way at least we can avoid these side effects for a few executions in the history. Created a GH issue to track this.
I am using Dask behind a Django server and the basic setup I have is summarised here: https://github.com/MoonVision/django-dask-demo/ where the Dask client can be found here: https://github.com/MoonVision/django-dask-demo/blob/master/demo/daskmanager/daskmanager.py
I want to be able to separate the saving of a task from the server that submitted it for robustness and scalability. I also would like more detailed information as to the processing status of the task, right now the future status is always pending even if the task is processing. Having a rough estimate of percent complete would also be great.
Right now, if the web server were to die, the client would get deleted and the task would stop as no client is still holding the future. I can get around this by using fire_and_forget but I then have no way to save the task status and result when it completes.
Ways I see to track the status and save the result after a fire_and_forget:
I could have a scheduler plugin that sends all transfers to AMPQ server (RabbitMQ). I like the robustness and being able to subscribe to certain messages that are output by the scheduler and knowing every message will be processed. I'm not sure how I could get the result it self with this method. I could manually adding a node to the end of every graph to save the result but would rather have it be behind the scenes.
get_task_stream on separate server or use it in some way. With this, it seems I could miss some messages if the server were to go down so seems like a worse option 1.
Other option?
What would be the best way to accomplish this?
Edit: Just tested and it seems when the client that submitted a task shuts down, all futures it created are moved from processing to forgotten, even if calling fire_and_forget.
You probably want to look at Dask's coordination primitivies like Queues and Pub/Sub. My guess is that putting your futures into a queue would solve your problem.
https://docs.dask.org/en/latest/futures.html#coordination-primitives
If you have a recurring task that runs once per day, you use a Scheduled Task.
If you have a recurring task that runs every 10 seconds, you use a Service.
At what point do you switch between the two? Is there official guidance on this somewhere?
i`m not sure the interval is the main issue here.
here are a few thing to consider:
how much state this task needs in memory - do you load stuff from a file of DB ?
does the system that needs this task to run, have a need to communicate with the task
other that when its running ?
do you need more control over the process lifecycle when the task is up?
you can see where i`m going with this , that a service is a resident entity, and a sched task isn't.
i think it depends on the point if your programm is made for only one task or for more. if it's just doin' one "stupid" thing (like running a stored procedure in a database every 20 seconds) i would concidering a sheduled task, but if it does more than that and maybe got some dependencies (maybe what time it is running or some file-operations) I would concider a service.
I would also concider a service if the intervals when the operation is made are different. Let's say your programm runs a single stored procedure in a database and depending on the fact that it made "real" changes to the db. If it did something the next run is in 5 seconds and if not the next run is in 20 seconds. That's one of the perfect examples for a service.
I am developing my first web application using ASP.Net MVC, and I am in a situation where I would like a background service to process status notifications outside of the application, not unlike the reputation/badge system on stackoverflow.
What is the best way to handle something like this? Is it even possible in a shared-hosting environment like Godaddy, which I am using.
I don't need to communicate with the background worker directly, since I will be adding notification records to a database table with a column set to an "unprocessed" state. Then the worker will just scan the table on a regular schedule and processes what is ready.
Thanks for your advice.
Have you tried with quartz.net? I think it may fit your needs.
also take a look at this Simulate a Windows Service using ASP.NET to run scheduled jobs article.
it explains a nice way to schedule operations with no outer dependence.
The idea is to use Cache timeout to control the schedule. I've implemented it successfully on a project which required regular temp file cleaning. This cleaning is a bit heavy so we move this clean operation in a scheduled job (using the asp.net cache) to avoid having to deploy scheduled task or custom program.
To answer whether GoDaddy will support a seperate service you need to ask them.
However there are a number of creative ways that you can "get around" this issue on shared hosting.
Have a secure page that's purpose is to execute your background work. You could have scheduled task on a machine under your control that calls to this web page at set intervals.
Use a variation of the Background Worker Thread answer from #safi. Your background worker thread could check to see if another is already processing and stop, so that only one instance is running at a time.
If only one background task is enough for you then use the WebBackgrounder
And this is the article with detailed explanation.
I'm writing a simple Windows Service that sends out emails to all employees every month. My question is, how to stop itself when it's done? I'm a noobie in this field so please help me out. Really appreciated.
It will be deployed on the server to be run monthly. I did not start this thing and the code was given to me like that. It is written in VB.NET and I'm asked now to change a few things around it. I noticed that there is only 'Sub OnStart' and wondered when the service would stop? After the main sub is done, what it the status of this service? Is it stopped or just hung in there? Sorry, as I said, I am really new to this....
If you have a task that recurs monthly you may be better off writing a console app, and then using Windows Task Scheduler to set it to run monthly. A service should be used for processes that need to run for a long time or constantly, with or without a user logged on
As every other answer has noted, it sounds like this should be an executable or script that you run as a scheduled task.
However, if you are obligated for some reason to run as a Windows Service and you're working in .NET, you just have to call the Stop() method inherited from ServiceBase once your service completes its work. From the MSDN documentation for the method:
The Stop method sets the service state
to indicate a stop is pending and
calls the OnStop method. After the
application is stopped, the service
state is set to stopped. If the
application is a hosted service, the
application domain is unloaded.
There's one important caveat here: the user account under which the service is running must have permission to stop services (which is a topic for ServerFault).
Once a service's OnStart method completes, it will continue running (doing nothing) until something tells it to stop in one of the following ways:
Programatically, by calling Stop
within the service itself or from an
external process using the method
Colin Gravill describes in his
answer.
Via the command-line.
Through the windows Computer Management console's "Services" panel.
If this is a Win32 service (i.e. written in C or C++), then you simply call SetServiceStatus(SERVICE_STOPPED) and return from ServiceMain.
On the other hand, if you're just sending emails once a month, why are you using a service at all? Use the Windows Task Scheduler and run a normal application or script.
net stop [service_name] ...on the command line will do it too.
But, I agree with everyone else; it seems that Windows Task Scheduler will meet your needs better.
It might be better to write this as a scheduled task, it would certainly be easier to develop initially. Then it would naturally terminate and wouldn't be consuming resources for the rest of the month.
To answer the original question, you can get a list of the current running services in C#
services = System.ServiceProcess.ServiceController.GetServices();
Then look for the one you want and set the status to stopped
locatedService.Status == ServiceControllerStatus.Stopped
Full example on msdn
Is there a reason it has to be a Windows service? If not, then follow #Macros solution. However, if it does, then why stop the service? If you stop it, then it'll just have to be restarted when the emails need to be sent. Based on your description, it doesn't sound like it would require a lot of resources, so I'd suggest just installing it and letting it run, firing up once a month to send the emails.
here's what i did in a similar situation.
windows service runs 24/7 and processes work units. it gets work units through a database view.
table Message
ProcessingStartTime
CompletionDTE
...
the database view only pulls records marked not-complete and have a ProcessingStartTime in the past. So after the service confirms the transaction it executes a stored procedure that updates the database record. For this system, end-user upload excel files to asp.net webfrom that imports them into the database.