In my application I have a static class (singleton) that needs to be initialized with some environmental variables that's used through out my layers, I'm calling it my applicationContext. That in turn has customer and user contexts.
As each job runs it modifies these customer and user contexts depending on the situation.
The problem I have is that when 2 jobs fires concurrently they might overwrite each others contexts, therefor I need to keep multiple user and customer contexts alive for each running job and I need to be able to pick the right context by somehow being able to see what the current job is.
Is it possible to somehow get information about the current executing quartz.net job?
I'm envisioning something like this where "currentQuartzJob.Name" is made up and is the part I'm missing:
public class MyContextImpl : IApplicationContext {
private Dictionary<string,subContexts> allCustomerContexts;
public string CurrentContext
{
get { return allCustomerContexts[currentQuartzJob.Name] };
}
}
edit:
I don't think it's possible to do what I wanted, that is to be able to get the executing job's name in a class that doesn't know about Quartz.Net.
What I really needed was a way to keep a different context for each job. I managed to do that by looking at the executing thread's ID as they seem to differ for each running job.
Try this:
public void Execute(IJobExecutionContext context)
{
var yourJobName = context.JobDetail.Key.Name;
}
Given your statement above: "The problem I have is that when 2 jobs fires concurrently they might overwrite each others contexts", you may want to reduce the complexity of your application by making sure that jobs do not fire concurrently. This can be achieved by implementing the IStatefulJob interface instead of the usual IJob interface: http://quartznet.sourceforge.net/tutorial/lesson_3.html
Alternatively if that is not an option you can query the Scheduler object for the currently executing jobs via the Ischeduler.GetCurrentlyExecutingJobs() method. This method returns an IList of those jobs but note the following remarks when using that method (from version 1.0.3 API):
This method is not cluster aware. That is, it will only return Jobs currently executing in this Scheduler instance, not across the entire cluster.
Note that the list returned is an 'instantaneous' snap-shot, and that as soon as it's returned, the true list of executing jobs may be different. Also please read the doc associated with JobExecutionContext- especially if you're using remoting.
I'm not entirely sure of what you're doing with your application, but I will say that the name of the currently executing job is definitely available during job execution.
Inside the IJob Execute() method:
// implementation in IJob
public void Execute(JobExecutionContext context)
{
// get name here
string jobName = context.JobDetail.Name;
}
Is that what you're looking for?
Related
We are using apache beam and would like to setup the logback MDC. logback MDC is a great GREAT resource when you have a request come in and you store let's say a userId (in our case, it's custId, fileId, requestId), then anytime a developer logs, it magically stamps that information on to the developers log. the developer no longer forgets to add it every log statement he adds.
I am starting in an end to end integration type test with apache beam direct runner embedded in our microservice for testing (in production, the microservice calls dataflow). currently, I am see that the MDC is good up until after the expand() methods are called. Once the processElement methods are called, the context is of course gone since I am in another thread.
So, trying to fix this piece first. Where should I put this context such that I can restore it at the beginning of this thread.
As an example, if I have an Executor.execute(runnable), then I simply transfer context using that runnable like so
public class MDCContextRunnable implements Runnable {
private final Map<String, String> mdcSnapshot;
private Runnable runnable;
public MDCContextRunnable(Runnable runnable) {
this.runnable = runnable;
mdcSnapshot = MDC.getCopyOfContextMap();
}
#Override
public void run() {
try {
MDC.setContextMap(mdcSnapshot);
runnable.run();
} Catch {
//Must log errors before mdc is cleared
log.error("message", e);. /// Logs error and MDC
} finally {
MDC.clear();
}
}
}
so I need to do the same with apache beam basically. I need to
Have a point to capture the MDC
Have a point to restore the MDC
Have a point to clear out the MDC to prevent it leaking to another request(really in case I missed something which seems to happen now and then)
Any ideas on how to do this?
oh, bonus points if it the MDC can be there when any exceptions are logged by the framework!!!! (ie. ideally, frameworks are supposed to do this for you but apache beam seems like it is not doing this. Most web frameworks have this built in).
thanks,
Dean
Based on the context and examples you gave, it sounds like you want to use MDC to automatically capture more information for your own DoFns. Your best bet for this is, depending on the lifetime you need your context available for, to use either the StartBundle/FinishBundle or Setup/Teardown methods on your DoFns to create your MDC context (see this answer for an explanation of the differences between the two). The important thing is that these methods are executed for each instance of a DoFn, meaning they will be called on the new threads created to execute these DoFns.
Under the Hood
I should explain what's happening here and how this approach differs from your original goal. The way Apache Beam executes is that your written pipeline executes on your own machine and performs pipeline construction (which is where all the expand calls are occurring). However, once a pipeline is constructed, it is sent to a runner which is often executing on a separate application unless it's the Direct Runner, and then the runner either directly executes your user code or runs it in a docker environment.
In your original approach it makes sense that you would successfully apply MDC to all logs until execution begins, because execution might not only be occurring in a different thread, but potentially also a different application or machine. However, the methods described above are executed as part of your user code, so setting up your MDC there will allow it to function on whatever thread/application/machine is executing transforms.
Just keep in mind that those methods get called for every DoFn and you will often have mutiple DoFns per thread, which is something you may need to be wary of depending on how MDC works.
As an example, I remember in hadoop I could make classes serializable or give a path with the jars needed for my job. I had either option. I am wondering if this is true in a dataflow job such that I can have all the clients we have in jar files to be packaged for all the workers.
In our case, we have MicroserviceApi and a generated client, etc. and would prefer to output to that downstream microservice without having to make it serializable.
Is there a method to do this?
First, let me clarify about serialization
When you add implements Serializable to a class in Java, you make it such that object instances of that class can be serialized (not the class itself). The destination JVM needs to have access to the class to be able to understand the serialized instances that you send to it. So, in fact you always need to provide the JAR for it.
Beam has code to automatically find all JARs in your classpath and upload them to Dataflow or whatever runner you're using, so if the JAR is in your classpath, then you don't need to worry about it (if you're using Maven/Gradle and specifying it as a dependency, then you're most likely fine).
Now, how can I use a class in Beam if it's not serializable?
In Beam, the more important part of it is to figure out where and when the different parts of the pipeline code will execute. Some things execute at pipeline construction time and some things execute at pipeline running time.
Things that run at construction time
Constructors for all your classes (DoFns, PTransforms, etc)
The expand method of your PTransforms
Things that run at executuion time
For your DoFns: ProcessElement, StartBundle, FinishBundle, Setup, TearDown methods.
If your class does not implement serializable, but you want to access it at execution time, then you need to create it at execution time. So, suppose that you have a DoFn:
class MyDoFnImplementation extends DoFn<String, String> {
// All members of the object need to be serializable. String is easily serializable.
String config;
// Your MicroserviceApi is *not* serializable, so you can mark it as transient.
// The transient keyword ensures that Java will ignore the member when serializing.
transient MicroserviceApi client;
public MyDoFnImplementation(String configuration) {
// This code runs at *construction time*.
// Anything you create here needs to be serialized and sent to the runner.
this.config = configuration;
}
#ProcessElement
public void process(ProcessContext c) {
// This code runs at execution time. You can create your object here.
// Add a null check to ensure it's only created once.
// You can also create it at #Setup or #StartBundle.
if (client == null) client = new MicroserviceApi(this.config);
}
}
By ensuring that objects are created at execution time, you can avoid the need to make them serializable - but your configuration needs to be serializable.
I need to execute some tasks after grails transaction ends. I'm not looking for afterCommit of Hibernate, instead of that, I need a method to be called after the transaction of the service ends.
Is there any way? I've read the documentation but I couldn't find anything, just normal events like afterUpdate or afterCommit.
Thanks!
You can add an event listener to the current session
sessionFactory.currentSession.addEventListener(...)
class MyListener extends BaseSessionEventListener {
#Override
void transactionCompletion(boolean successful) {
}
}
You just need to verify the transaction was successful via the parameter. Also understand that if there are multiple transactions in a single session the listener will be invoked for each one.
That really depends on what you're trying to do. The best bet is probably to call a #Transactional method on a service, and then when that returns, run the code that you need to happen after the transaction.
An alternative, if you just want to spin off some task to do something simple, you can use:
new java.util.Timer().runAfter(1000) { //time in milliseconds
//code to run here
}
This is just a shortcut to spin off a new thread and runs the closure body after (in the above example) 1 second... I believe the closure still has access to injected grails objects, but does not have access to the session. I'd double-check that though. I've used this in several places for sending notifications that needed to wait until after some processing inside a transaction ended.
I have some integration tests written for MsTest. The integration tests have the following structure:
[TestClass]
public class When_Doing_Some_Stuff
{
[TestInitialize]
protected override void TestInitialize()
{
// create the Integration Test Context
EstablishContext();
// trigger the Integration Test
When();
}
protected void EstablishContext()
{
// call services to set up context
}
protected override void When()
{
// call service method to be tested
}
[TestMethod]
public void Then_Result_Is_Correct()
{
// assert against the result
}
}
I need to filter the code coverage results of a function by who is calling it. Namely, I want the coverage to be considered only if the function is called from a function named "When" or which has a certain attribute applied to it.
Now, even if a certain method in the system is called in the EstablishContext part of some tests, the method is marked as visited.
I believe there is no filter for this and I would like to make the changes myself, as OpenCover is... well.. open. But I really have no idea where to start. Can anyone point me in the right direction?
You might be better addressing this with the OpenCover developers; hmmm... that would be me then, if you look on the wiki you will see that coverage by test is one of the eventual aims of OpenCover.
If you look at the forking you will see a branch from mancau - he initially indicated that he was going to try to implement this feature but I do not know how far he has progressed or if he has abandoned his attempt (what he has submitted is just a small re-introduction of code to allow the tracing of calls).
OpenCover tracks by emitting a visit identifier and updating the next element in an array that resides in shared memory (shared between the profiler (C++/native/32-64bit) and the console (C#/managed/any-cpu)). What I suggested to him was (and this will be my approach when I get round to it, if no one else does and is why I emit the visit data in this way) that he may want to add markers into the sequence to indicate that he has entered/left a particular test method (filtered on [TestMethod] attribute perhaps) and then when processing the results in the console this can then be added to the model in some way. Threading may also be a concern as this could cause the interleaving of visit points for tests run in parallel.
Perhaps you will think of a different approach and I look forward to hearing your ideas.
Let's say, i have a specific information in the database that needs to be sent for a specific user by email in a specific time of the day.
a) How can i create a routine in Grails, which is basically an action that is always running - without being associated with any event? Let's say, every hour that action is runned.
I was thinking about something like this:
while(true){
...
myCodeHere
...
wait 30minutes
}
Will this actually work? Without too much processing? And how can i have an action permanently running no matter what. I there is a specific way of doing this?
Thanks in advanced,
RR
The usual way to do this in a grails app is with the Quartz scheduler plugin. The plugin provides a simple cron-like DSL for scheduling jobs. For example, to run a job every 30 minutes, you could configure it like this:
class MyJob {
static cronExpression = "0 0/30 * * * ?"
def execute(){ /* do something useful */ }
}
If you want to run a background thread all the time, take a look at the executor plugin which provides an ExecutorService wrapped up properly to get a hibernate session.
Avoiding quartz and plugins you may use pure Spring Framework
1) add to container
<task:annotation-driven executor="executor" scheduler="scheduler"/>
<task:executor id="executor" pool-size="5"/>
<task:scheduler id="scheduler" pool-size="10"/>
(do not forget to define task and tx namespaces)
2) Create some bean and add method
#Scheduled(fixedDelay=4000)
public void method() {
// do something every 4 seconds
}
finish!
For more info see spring framework