Unable to acquire Singleton lock in Azure WebJob after removing SingletonAttribute - azure-webjobssdk

I have got a WebJob containing functions having a TimeTriggerAttribute. They also have a SingletonAttribute on them, so they don't get executed in parallel.
Deployed in the App Service everything works as expected.
When running it locally it did work as expected for a while and then the output of the Job host reports:
Development settings applied
Found the following functions:
MyNamespace.WebJobs.Functions.MyFunctionAsync
Unable to acquire Singleton lock (bd99766f202e4ac4ba230557b7180bd2/MyNamespace.WebJobs.Functions.MyFuncitonAsync.Listener).
I removed the singleton attribute, but the message keeps showing. I also re-built the project and restarted the machine. Nothing helped.
I renamed the function and it was again scheduled as expected. Now I could also re-add the SingletonAttribute and everything worked as expected. When I rename it back to the original, the error comes back.
What can I do in order to be able to acquire the lock again from the Job host?

By reading this article I figured out that on top of the SingletonAttribute I did set, the TimeTriggerAttribute uses another one behind the scenes. Thank you Janley Zhang.
The ID of the host is, by default, constant across deployments. So when multiple developers that use the same storage account execute the code locally, they would try to acquire the same blob lease and only the first would be successful.
I ended up using a distinct host ID for each developer, like
if (jobHostConfiguration.IsDevelopment)
{
jobHostConfiguration.UseDevelopmentSettings();
jobHostConfiguration.HostId = Environment.UserName.ToLowerInvariant();
}

Related

Running Camunda in clustered environment locks further deployment

I have a problem with my clustered Camunda environment. What I am trying is to run multiple Camunda instances on my Openshift Cluster. All of them are connected to a single oracle db instance.
My problem is, that the deployment of the first instance is working as expected. However as soon as I'm trying to scale the pods to e.x. 3 instances, at least one of them failes and remains stuck on the following output:
{"timestamp":"2020-07-15 14:04:39.503","level":"DEBUG","thread":"main","logger":"org.camunda.bpm.engine.cmd","message":"ENGINE-13009 opening new command context","context":"default"}
14:01:00.741","level":"DEBUG","thread":"main","logger":"org.camunda.bpm.engine.impl.persistence.entity.PropertyEntity.lockDeploymentLockProperty","message":"==> Preparing: SELECT VALUE_ FROM ACT_GE_PROPERTY WHERE NAME_ = 'deployment.lock' for update ","context":"default"}
{"timestamp":"2020-07-15 14:01:00.748","level":"DEBUG","thread":"main","logger":"org.camunda.bpm.engine.impl.persistence.entity.PropertyEntity.lockDeploymentLockProperty","message":"==> Parameters: ","context":"default"}
As the logs tell, it hast something to do with locking of process deployment. After further investigation I came across this article on the offical Camunda page:
https://docs.camunda.org/manual/7.13/user-guide/process-engine/deployments/
And have seen these entries in the database:
Problem: I do understand why the deployments are locked but the main problem is that the lock remains there forever and never gets released. I would appreciate any help!
Are you using autodeployment?! The mentioned article describes a weired situation where multiple nodes try to deploy the same resources. In my opinion this only should happen, when each node trys to autodeploy resources.
Using an explicit deployment (after nodes are started) should be executed on a single node.
KR, Joachim

How to deploy client-side service WSDLs

I have code using the SysOperationFrameworkService, and after a model-deploy or some other set of circumstances, many users get Function SysOperationDataContractInfo::newParameterInfo has been incorrectly called..
It's unable to be resolved with full-compile, sync, full CIL, deleting XPPIL files, deleting AUC/KTI, usage data, security (they're admin), refreshing caches (server &client), etc. and I have a ticket open with Microsoft and they're struggling.
I've narrowed the issue down to when the service group AxClient located in the AOT at \Service Groups\AxClient WSDL files get deployed to C:\Users\[User]\AppData\Local. Either those files existing there, or something happening when those files get deployed is needed.
Any idea how/what causes these files (AxClient.wsdl, etc) to be created? With a user who happens to be "working", I can close the client, delete all the files, open the client, and no files will exist...then I perform a SysOp process and those files get created during that process. With the non-working user, with same security on the same machine does it, the files don't get created.
I have found that the
"Function SysOperationDataContractInfo::newParameterInfo has been incorrectly called" has been throwed because of a possible bug in the kernel when calling the "dictMethod = new DictMethod(UtilElementType::ClassInstanceMethod, dictClass.id(), methodName);" in the SysOperationServiceController.getServiceOperation() method. In my case the new DictMethod was sometimes unable to get the method parameter from a superclass, causing the _parentMethodParameterName check fail in the SysOperationDataContractInfo.newParameterInfo.
So, I solved the issue just by doing an override of the super method where the parameter could not be retrieved and then just calling super(_parameter), then everything went well.
Have you tried axclicfg.exe -> connections -> refresh

Deadlock in TAO when colocated

I have a CORBA server that makes a twoway call to another server from a notification thread that is created in the servant. This works fine normally. I have just modified the server to be co-located with its client (both are DLLs in a host app), and now it deadlocks waiting for the reply. The offending call is to outside the colocated servers (to the naming service in fact). I am using TAO 1.6 with default policies.
The problem appears to be related to the leader-follower implementation; it is blocking waiting for the existing leader, whereas when it works there is no leader.
Thanks
Pete.
For the record, I got round it by using thread-per-connection configuration, following the advice here http://www.theaceorb.com/faq/index.html#configure_tpc and putting the optoins in a service configuration file as described here http://www.dre.vanderbilt.edu/~schmidt/DOC_ROOT/TAO/docs/Options.html#SVC (what that doesn't make entirely clear is that all you need is a file called svc.conf in the working directory). Not surprisingly, all those options are necessary to fix the problem I had.

WSS caches old Workflow version

I'm currently developing three workflows that are supposed to handle the status of items in different lists.
Each Workflow is attached to a separate list.
When I'm deploying and debugging in my development Environment, everything works fine.
Except for the case, when an item is created via an incoming mail.
I already figured out, that I have to restart some services and then it'll work, but I'm still not sure wich of the services is caching the workflow.
Afterwards I build a .wsp file which I deploy on a server.
Each time I deploy the solution, I do a retract and delete solution first.
After deployment I'll recreate the workflows on the lists
It seems to me that this has no effect. An older version of the workflow is still triggered, if I create a new instance in the list.
I already restarted the whole server and still no result.
Has anyone an idea what else I could try in order to get this working?
Thanks in advance.
If Timer Service is the one that calls your code, then restart Windows SharePoint Services Timer (OWSTIMER.EXE).
When workflow waits on something, it gets serialized (hydrated). When event happens, OWSTIMER.EXE deserializes (dehydrates) and continues workflow execution.
So timer is the one that wakes workflow up.
So this problem kind of resolved itself.
I was reading an article on Kirk Evanns Blog on an issue with the development of workflows in VS2008 for WSS.
I had not realized that I still had an illeagle reference in my Project properties.
I removed the reference. The second thing I tried was deploying with -upgradesolution rather than doing a retract-delete-add-deploy...
I don't know which of both did the trick, but I can finally see the new workflows kicking in.
Thanks for your help.

Starting a windows service fails with error 1053

I have a windows service that is failing to start, giving an error "Error 1053: The service did not respond to the start or control request in a timely fashion".
Running the service in my debugger works fine, and if I double click on the the service .exe on the remote machine a console window pops up and continues to run without problem - I can even see log messages showing me that the program is processing everything the way it should be.
The service had been running fine previously, though this is my first time, personally, trying to deploy it with the most recent changes made to the program. I've evaluated those changes and cant figure out how they might cause this problem, particuarly since everything runs fine when not started as a service.
The StartRoutine() method of the service impelmentation is empty, so should be returning in a "timely fashion".
I've checked the event logs on the computer, and it doesn't give any additional information other than it didn't hear back from the service in the 30 second requisite time frame.
Since it works on my machine, and as a double-clicked executable, how would I go about figuring out why it fails as a service?
Oh, and it's .NET 2.0, so it shouldn't be affected by the 1.1 framework bug that exhibited this symptom (http://support.microsoft.com/kb/839174)
The box is a windows server 2003 R2 machine running SP2.
This is a misleading error. It's probably an unhandled exception.
Empty your OnStart() handler then try this in your constructor...
public MainService()
{
InitializeComponent();
try
{
// All your initialization code goes here.
// For instance, my exception was caused by the lack of registry permissions
;
}
catch (Exception ex)
{
EventLog.WriteEntry("Application", ex.ToString(), EventLogEntryType.Error);
}
}
Now check the EventLog on your system for your Application Error.
Could be a number of things and it might help to get a stack trace on the machine exhibiting the problem. There are a number of ways to do this but the point is that you have to see where this is failing in the code.
You can do this with remote debugging, but a simple thing might be to just log to the event logger, or file log if you have that. Literally, putting "WriteLine("At class::function()") throughout portions of the code to see if you've made it there.
This will at least get you looking in the right direction (which ultimately is the code).
Update:
See Microsoft's How to Debug Windows Services article for details in troubleshooting startup problems using WinDbg.
This related question details nice ways to debug services that are written in .NET.
I agree with Scott, the easiest way to find out what's happening is to put some traces in the start-up code (maybe it doesn't even come to your start-up code).
If this doesn't help, you can post your code here so others can take a look.
perhaps lacking some dependence, try this :
- deregister your service
- register again
If fail at register means that lack an module.
If the StartRoutine is empty, you are probably starting it somewhere else.
IIRC you need to fire off a worker thread, and then return from StartRoutine.
One of the problems which may lead to this error is if windows service which needs to be deployed consists of some error i.e it may be simple authorization error or anything as in my case I have referenced some folders and files for logging which were not existing, but when provided the right path of those file and folders it solved my problem.
I ran through every post on this particular subject and none of the responses solved the problem, so I'm adding this response in case this helps someone else. Admittedly this only applies to a new service, not this specific case.
I was writing a File listening service. As a console app, it worked perfectly. When I ran it as a service, I got the same error as above. What I didn't know (and many of the MSDN articles about services conveniently leave out) is that you need to have your class executed from within ServiceBase.Run( YourClassName());. Otherwise, your app executes and immediately terminates and because it terminated, you get the error above even if no error or exception occurred. Here is a link to an article about this. It actually discusses setting up your app for dual use - Console app and service: Create a combo command line / Windows service app
I had that issue and the source of my problem was config file. I edited it in notepad and notepad added one special character which cause service not to run properly because config file was ruined. I saw that special character in notepadd++ and after delete it, service started to run successfully as previous did.
In my case, the correct .NET framework was not installed on the server that I was installing the Windows service on.
One other reason is If you copy the DLL in 'debug' mode to installation folder this issue will come.What you need to do is Run the project in 'Release' mode copy the DLL or directly form Release folder rather than Debug folder,,and copy that DLL in to installation folder,it will work.You can see the reduction in size of DLL ,it will not contain any debug symbols and like that

Resources