Background:
In our typical working model of state machine , we use state machine to monitor spring-batch processing done by spring cloud stream micro-services .
More precisely saying we create stream ,say, named ItemStream , having source , process , sink micro services deployed that read , process and write records in bulk ; respectively .
While doing these activities , we fire REST calls each call consisting of one event in between the operations of streams. These REST calls we consume in State Machine micro-service and fire those events within accordingly , on machine instance . Here at state machine level each time once processing is done , we persist state machine context to the persistent store (database/Redis cache) using JPA/Redis State Machine Persister .
And when new REST call is received from streams , we restore back the machine context from the persistent store and populate one fresh state machine object with it ; so as to prepare that fresh machine object to have a state of the machine as existed previously .
Note: Here 'populate the fresh state machine object' simply means we create each time new instance using machine factory calls , as we are yet to plug the machine object pool implementation to push and pop used objects of StateMachine as and when needed rather than creating fresh new instance every time .
Problem :
However , now problem we are facing is , we have configured the deferred event (say for e.g.CompleteChunk_1 ) for particular state(e.g. state PROCESSEDCHUNK ) in uml file.
Now after firing CompleteChunk_1 event when machine is at PROCESSEDCHUNK state , this event does get deferred i.e. parked aside correctly . Now it is expected that machine should fire this parked event on its own whenever that particular future state(i.e. WRITECHUNK ) is reached for which that event was mapped in a transition .
But once machine reaches at that state i.e. at WRITECHUNK state , the machine is not firing that CompleteChunk_1 event on its own which was parked previously in machines deferred list .
Observations
**1.**After trouble shooting the issue what I could notice is State machine's member property 'stateMachineExecutor' holds the DefaultStateMachineExecutor instance who maintains the deferred list in a data structure 'ConcurrentLinkedList' within itself . This executor is every time instantiated to the new one in an onInit() method of state machine in AbstractStateMachine class . So I believe each time whenever fresh instance is created from factory and machine object is revived with the previous persisted context , we are losing the deferred events that were parked in that executor's "deferredList" . Because OOB persistence/restoring does not persist/restore the executor's state thus deferred list present on that executor will be lost .
So in our case by the time when machine comes to the correct state i.e. WRITECHUNK , the executor has already lost the deferred event that was parked in past event calls . Thus they can not be now fired by machine automatically because parked events now don't exist in executor.
**2.**In simple state machine examples given on spring documentation I tried to configured deferred events and it worked correctly and after valid state was reached parked events did get fired automatically . In these examples , state machine was launched and running constantly since then from first event to the last event. Thus same instance object of machine holding that same DefaultStateMachineExecutor instance till end worked .
3.Maven dependency lib versions are as below :
a.spring-statemachine-core : 2.0.0
b.spring-statemachine-uml : 2.0.0
Requesting for a Solution ??
So is there any design change needed or a way to deal with this situation of DefaultStateMachineExecutor when looking at the fact that we persist the context and restore it on each REST call that is received ?? Or I am miss understanding something regarding above mentioned conclusions which I have arrived at while troubleshooting ??
Thanking you in advance.
Yes this going to need some changes in a machine itself. It's a little complex issue and generally speaking we need to overhaul other event related things with persistence. These are tracked in gh-550
Related
I want to create and host a file on a remote server and allow cytoscape to poll it every time it starts up to see if there are any changes. I'm not sure where to even start with that. Was hoping I might get some information here.
Actually, it's pretty easy. As part of your app initialization in your CyActivator, you could easily poll your remote site. The CyActivator start method gets called when Cytoscape starts up (or you app changes). I would strongly suggest that you do your poll in a separate thread since Cytoscape won't actually start until your start method returns...
-- scooter
I have an orchestration that runs as a singleton by using the same instance id each time. It also runs infinitely by using ContinueAsNew at the end of each iteration to keep the history manageable. However, I have noticed that the history of each past iteration is kept in the history table, each with a different execution id (as is expected when ContinueAsNew is called).
I also use PurgeInstanceHistoryAsync once a day to delete any completed, failed, terminated or cancelled orchestrations that are more than 14 days old. However, since the infinite singleton orchestration is never in any of these states will PurgeInstanceHistoryAsync ever clean up the old execution histories?
The same question can be asked for a periodic singleton orchestration (i.e. an orchestration that runs periodically but uses the same instance Id each time). If the purge process happens whilst the orchestration is running, will any old histories be removed, or would it be a matter of luck that the orchestration is not actually running at the time the purge executes?
If you look in your history table in the azure storage account and query for your instance you should see that using ContinueAsNew will actually purge history automatically. (In my test it seemed to be at most 1 execution behind.)
From Docs: https://learn.microsoft.com/sv-se/azure/azure-functions/durable/durable-functions-eternal-orchestrations?tabs=csharp#resetting-and-restarting
When ContinueAsNew is called, the instance enqueues a message to itself before it exits. The message restarts the instance with the new input value. The same instance ID is kept, but the orchestrator function's history is effectively truncated.
I'm developing a beam pipeline for dataflow runner. I need the below functionality in my use case.
Read input events from Kafka topic(s). Each Kafka message value derives [userID, Event] pair.
For each userID, I need to maintain a profile and based on the current Event, a possible update to the profile is possible. If the profile is updated:
Updated profile is written to output stream.
The next Event for that userID in the pipeline should refer to the updated profile.
I was thinking of using the provided state functionality in Beam, without depending on an external key-value store for maintaining the user profile. Is this feasible with the current version of beam (2.1.0) and dataflow runner? If I understand correctly the state is scoped to the elements in a single window firing (i.e even for a GlobalWindow, the state will be scoped to the elements in a single firing of the window caused by a trigger). Am I missing something here?
State would be perfectly appropriate for your use case.
The only correction is that state is scoped to a single window, but trigger firings do not affect it. So, if your state is small you can store it in a global window. When a new element arrives, you can use use the state, output elements as needed, and make changes to the state.
The only thing to consider would be if you have an unbounded number of user IDs, how big the state may become. For instance, you may want an inactivity timer to clear old user state after some period of time.
If you haven't read them, the blog posts Stateful Processing with Apache Beam and Timely (and Stateful) Processing with Apache Beam provide a good introduction to these concepts and APIs.
I'm trying to chain a few jobs in Quartz.NET through JobChainingJobListener. I first create a couple of durable jobs (while using ADO JobStore with SQL Server) and this part works well - the jobs are visible across Quartz.NET restarts.
When I chain my jobs with Scheduler.ListenerManager.AddJobListener(listener, matchers)the listener fires correctly, but its definition cannot be made durable in the database. After every server restart, I have to define all listeners again.
Looking at the DB tables, there are no tables for listeners, nor does the code for ListenerManagerImpl contain any hints of listener persistence.
I'm planning to add listener durability and reload the global listener dictionary on server restart. Before I do that, I'm wondering if there are any reasons why the project does not already do so? Considering how mature Quartz.NET is, someone would have already ran into this, so it seems I'm missing something.
Can anyone please point to any pitfalls in implementing listener durability?
From Quartz's perspective listeners are just a configuration issue. Just like you configure job store type or other settings for the library. Commonly listeners are stateless and thus need no persistence services, unlike triggers and jobs that hold state that need to be persisted between invocations and possible job processing nodes.
If you have sound configuration management plan this shouldn't be an issue. Just handle the listener configuration like you would other aspects of the setup. If you have state management in your listeners that would need storage between restarts, that's a different story. Then you'd naturally need custom persistence.
I have been researching Mobile Agents, and was wondering if it is possible to send a running process to another node in erlang. I know it is possible to send a process on another node a message. I know it is possible to load a module on all nodes in a cluster. Is it possible to move a process that might be in some state on a particular node to another node and resume it's state. That is, does erlang provide strong mobility? Or is it possible to provide strong mobility in erlang?
Yes, it is possible, but there is no "Move process to node" call. However, if the process is built with a feature for migration, you can certainly do it by sending the function of the process and its state to another node and arrange for a spawn there. To get the identity of the process right, you will need to use either the global process registry or gproc, as the process will change pid.
There are other considerations as well: The process might be using an ETS table whose data are not present on the other node, or it may have stored stuff in the process dictionary (state from the random module comes to mind).
The general consensus in Erlang is that processes are not mobilized to move between machines. Rather, one either arranges for a takeover of applications between nodes should a node die. Or for distribution of the system so data are already distributed to another machine. In any case, the main problem of making state persistent in the event of errors still hold, mobility or not - and distribution is a nice tool to solve the persistence problem.