I have a time consuming script, and I want to periodically log it execution time. How do I find out the current execution time?
The symfonian way to log execution times is using the timer manager that comes with symfony.
//Get timer instance called 'myTimer'.
$timer = sfTimerManager::getTimer('myTimer');
//Start timer.
$timer->startTimer();
// Do things
...
// Stop the timer and add the elapsed time
$timer->addTime();
This timer will be saved into any logger you have configured with your symfony.
By default symfony has the sfWebDebugLogger for the DEV environment but you can create your own and configure it in the factories.yml file.
The nice thing about this logger is that it logs also the number of calls to the timer.
Why not use date('Y-m-d H:i:s') and log that... and/or calculate difference from a start time obtained with date() as well?
Think about what you are asking, each action is logged in symfony log (root/log/app_name_[env].log). This logging is done once the operation has ended, (there is no easy way to figure out the execution time of a thread executing a php from php). You could try messing up with code and add code in order to log at certain points of the code, the current execution time, something like:
$init = microtime();
log("Process started at:". $init);
foreach($this->getLotsOfRecords() as $index=>$record)
{
$start = microtime();
log ($index." record started at".microtime());
[do stuff]
$end = microtime();
log ($index." record ended at". $end . " time elapsed: ". ($start - $end));
}
$last = microtime();
log("Total elapsed time is: ". ($init - $last));
This is pseudo code but i believe you can figure out the rest, hope this helps!
I ended up writing my own static class:
class Timer
{
protected static $start = null;
public static function getDuration($format = '%H:%I:%S')
{
$now = new DateTime();
if (!self::$start)
{
self::$start = new DateTime();
}
$period = $now->diff(self::$start);
return $period->format($format);
}
}
And logging it in partial (that is looped):
<?php $logger->log(sprintf("Duration: %s", $duration = Timer::getDuration())) ?>
Related
This is the overall flow of my setup:
void ScriptPlayer::new_lua_state() {
lua = {};
lua.open_libraries(sol::lib::base, sol::lib::package, sol::lib::coroutine, sol::lib::math);
[...]
// Proceeds to initialize the state with usertype definitions and values
}
void ScriptPlayer::play(std::string path) {
main_coroutine = sol::nil;
script_env = sol::environment(lua, sol::create, lua.globals());
auto result = lua.load_file(path);
main_coroutine = sol::coroutine(result);
script_env.set_on(main_coroutine);
}
void ScriptPlayer::update() {
if (main_coroutine) {
main_coroutine();
}
}
"new_lua_state" is called once at the beginning of everything, then "play" is called anytime I want to execute a new lua script (that yields). "update" is executed every frame, and progresses the coroutine until it's finished, at which point it stops.
The problem:
If I call "play" while the previous script coroutine has yielded but hasn't yet finished, I expect lua to discard the whole environment and create a new one, discard the old coroutine, parse the script again, create a brand new coroutine and start its execution from the beginning.
What I get instead is that the coroutine will STILL be running from the state of the previous script's coroutine (which should be completely discarded) and not from the very beginning.
How is this possible? Where exactly is the state of the coroutine stored?
I tried wrapping the state with a thread, I tried calling lua.clear_stack but nothing made any difference in the fact that the new coroutine never starts from the beginning of the function when I re-parse the script and re-create the sol::coroutine object.
Any clarification is hugely appreciated.
Here was the solution:
https://github.com/ThePhD/sol2/issues/1061
Apparently my attempt of wrapping the state with a thread was faulty, because that was exactly the thing to do.
So to solve this, here's what I did:
void ScriptPlayer::play(std::string path) {
script_env = sol::nil;
main_coroutine = sol::nil;
script_thread = sol::thread::create(lua);
script_env = sol::environment(script_thread.state(), sol::create,
script_thread.state().globals());
auto result = script_thread.state().load_file(path);
main_coroutine = sol::coroutine(result);
script_env.set_on(main_coroutine);
}
One this that still blows my mind is that if I remove the second line (that reset the C-held reference to the lua coroutine), the result goes back to wrongly resuming the previous coroutine, despite that very same variable being set to a different value shortly after..
This baffles me deeply.
I am trying to get build duration for our report but it always returns 0.
From reading docs, going through Slack plugin source and reading other resources I should be able to do one of the following:
def duration = currentBuild.duration
def duration = currentBuild.durationString
def duration = currentBuild.durationString()
def duration = currentBuild.getDurationString()
none of which works. From my understanding this might be because I am calling this before the build actually finished and so th duration is not available yet.
Structure of the pipeline looks something like this:
node {
try {
stage("Stage 1"){}
stage("Stage 2"){}
} catch (e) {
currentBuild.result = "FAILED"
throw e
} finally {
notifyBuild(currentBuild.result)
}
}
def notifyBuild(String buildStatus = 'STARTED') {
def duration = currentBuild.duration;
}
My question is:
Why don't I get the duration
Is there a way in pipeline to specify "after build" step? From what I read try-catching should work that way
My temporary solution is to use:
int jobDuration = (System.currentTimeMillis() - currentBuild.startTimeInMillis)/1000;
Which works fine, but always gives time in seconds and I think the currentBuild.duration should be smart enough to give different units (?)
Update 2018-02-19, this was fixed with 2.14 release of Pipeline Support API Plugin, see this issue
In unable to find any documentation about when duration is expected to be valid. But judging from the implementation it seems like it's set directly after the run/build completes. I'm guessing that it is available on the currentBuild object since it's the same object that is used for representing currentBuild.previousBuild, which may have completed.
So to answer your questions:
The duration field is only valid after the build has completed.
No, there is no way of specifying an "after build" step.
With that said, I think your workaround is a good solution (may be wrap it in a function and put it in an GPL (Global Public Library).
As for your final bonus question I think the currentBuild.duration should be smart enough to give different units (?). If you are referring to the nicely formatted string, like Took 10min 5sec, currentBuild.duration won't give you any nice formatting since it simply returns a long value with the number of seconds that has elapsed. Instead, what you can do is call hudson.Util#getTimeSpanString(long duration). Like this:
import hudson.Util;
...
echo "Took ${Util.getTimeSpanString(System.currentTimeMillis() - currentBuild.startTimeInMillis)}"
This will return a nicely formatted string with the current build duration.
I know how to get the Execution Id and Instance Id of a job using the Job Context. But if i restart a job, is there way to know if the job execution is the first execution or a restart within the job, for instance inside the reader?
No, but there is an open issue asking for that:
https://java.net/bugzilla/show_bug.cgi?id=7473
This is a bit overly complicated (as the other answer noted, there's an issue opened to consider enhancement for the future Batch 1.1).
You could do this:
//
// Assumes JobContext injected into 'jobCtx' field
//
private boolean isRestart() {
JobOperator jo = BatchRuntime.getJobOperator();
JobInstance jobInstance = jo.getJobInstance(jobCtx.getExecutionId());
int numExecutions = jo.getJobExecutions(jobInstance).size();
return numExecutions > 1;
}
I want to stop/sleep executing to simulate long time process, unfortunately I can't find information about it. I've read the following topic (How can I "sleep" a Dart program), but it isn't what I look for.
For example sleep() function from dart:io packages isn't applicable, because this package is not available in a browser.
For example:
import 'dart:html';
main() {
// I want to "sleep"/hang executing during several seconds
// and only then run the rest of function's body
querySelect('#loading').remove();
...other functions and actions...
}
I know that there is Timer class to make callbacks after some time, but still it doesn't prevent the execution of program as a whole.
There is no way to stop execution. You can either use a Timer, Future.delayed, or just use an endless loop which only ends after certain time has passed.
If you want a stop the world sleeping function, you could do it entirely yourself. I will mention that I don't recommend you do this, it's a very bad idea to stop the world, but if you really want it:
void sleep(Duration duration) {
var ms = duration.inMilliseconds;
var start = new DateTime.now().millisecondsSinceEpoch;
while (true) {
var current = new DateTime.now().millisecondsSinceEpoch;
if (current - start >= ms) {
break;
}
}
}
void main() {
print("Begin.");
sleep(new Duration(seconds: 2));
print("End.");
}
We have a Grails project that runs behind a load balancer. There are three instances of the Grails application running on the server (using separate Tomcat instances). Each instance has its own searchable index. Because the indexes are separate, the automatic update is not enough keeping the index consistent between the application instances. Because of this we have disabled the searchable index mirroring and updates to the index are done manually in a scheduled quartz job. According to our understanding no other part of the application should modify the index.
The quartz job runs once a minute and it checks from the database which rows have been updated by the application, and re-indexes those objects. The job also checks if the same job is already running so it doesn’t do any concurrent indexing. The application runs fine for few hours after the startup and then suddenly when the job is starting, LockObtainFailedException is thrown:
22.10.2012 11:20:40 [xxxx.ReindexJob] ERROR Could not update searchable index, class org.compass.core.engine.SearchEngineException:
Failed to open writer for sub index [product]; nested exception is
org.apache.lucene.store.LockObtainFailedException: Lock obtain timed
out:
SimpleFSLock#/home/xxx/tomcat/searchable-index/index/product/lucene-a7bbc72a49512284f5ac54f5d7d32849-write.lock
According to the log the last time the job was executed, re-indexing was done without any errors and the job finished successfully. Still, this time the re-index operation throws the locking exception, as if the previous operation was unfinished and the lock had not been released. The lock will not be released until the application is restarted.
We tried to solve the problem by manually opening the locked index, which causes the following error to be printed to the log:
22.10.2012 11:21:30 [manager.IndexWritersManager ] ERROR Illegal state, marking an index writer as open, while another is marked as
open for sub index [product]
After this the job seems to be working correctly and doesn’t become stuck in a locked state again. However this causes the application to constantly use 100 % of the CPU resource. Below is a shortened version of the quartz job code.
Any help would be appreciated to solve the problem, thanks in advance.
class ReindexJob {
def compass
...
static Calendar lastIndexed
static triggers = {
// Every day every minute (at xx:xx:30), start delay 2 min
// cronExpression: "s m h D M W [Y]"
cron name: "ReindexTrigger", cronExpression: "30 * * * * ?", startDelay: 120000
}
def execute() {
if (ConcurrencyHelper.isLocked(ConcurrencyHelper.Locks.LUCENE_INDEX)) {
log.error("Search index has been locked, not doing anything.")
return
}
try {
boolean acquiredLock = ConcurrencyHelper.lock(ConcurrencyHelper.Locks.LUCENE_INDEX, "ReindexJob")
if (!acquiredLock) {
log.warn("Could not lock search index, not doing anything.")
return
}
Calendar reindexDate = lastIndexed
Calendar newReindexDate = Calendar.instance
if (!reindexDate) {
reindexDate = Calendar.instance
reindexDate.add(Calendar.MINUTE, -3)
lastIndexed = reindexDate
}
log.debug("+++ Starting ReindexJob, last indexed ${TextHelper.formatDate("yyyy-MM-dd HH:mm:ss", reindexDate.time)} +++")
Long start = System.currentTimeMillis()
String reindexMessage = ""
// Retrieve the ids of products that have been modified since the job last ran
String productQuery = "select p.id from Product ..."
List<Long> productIds = Product.executeQuery(productQuery, ["lastIndexedDate": reindexDate.time, "lastIndexedCalendar": reindexDate])
if (productIds) {
reindexMessage += "Found ${productIds.size()} product(s) to reindex. "
final int BATCH_SIZE = 10
Long time = TimeHelper.timer {
for (int inserted = 0; inserted < productIds.size(); inserted += BATCH_SIZE) {
log.debug("Indexing from ${inserted + 1} to ${Math.min(inserted + BATCH_SIZE, productIds.size())}: ${productIds.subList(inserted, Math.min(inserted + BATCH_SIZE, productIds.size()))}")
Product.reindex(productIds.subList(inserted, Math.min(inserted + BATCH_SIZE, productIds.size())))
Thread.sleep(250)
}
}
reindexMessage += " (${time / 1000} s). "
} else {
reindexMessage += "No products to reindex. "
}
log.debug(reindexMessage)
// Re-index brands
Brand.reindex()
lastIndexed = newReindexDate
log.debug("+++ Finished ReindexJob (${(System.currentTimeMillis() - start) / 1000} s) +++")
} catch (Exception e) {
log.error("Could not update searchable index, ${e.class}: ${e.message}")
if (e instanceof org.apache.lucene.store.LockObtainFailedException || e instanceof org.compass.core.engine.SearchEngineException) {
log.info("This is a Lucene index locking exception.")
for (String subIndex in compass.searchEngineIndexManager.getSubIndexes()) {
if (compass.searchEngineIndexManager.isLocked(subIndex)) {
log.info("Releasing Lucene index lock for sub index ${subIndex}")
compass.searchEngineIndexManager.releaseLock(subIndex)
}
}
}
} finally {
ConcurrencyHelper.unlock(ConcurrencyHelper.Locks.LUCENE_INDEX, "ReindexJob")
}
}
}
Based on JMX CPU samples, it seems that Compass is doing some scheduling behind the scenes. From 1 minute CPU samples it seems like there are few things different when normal and 100% CPU instances are compared:
org.apache.lucene.index.IndexWriter.doWait() is using most of the CPU time.
Compass Scheduled Executor Thread is shown in the thread list, this was not seen in a normal situation.
One Compass Executor Thread is doing commitMerge, in a normal situation none of these threads was doing commitMerge.
You can try increasing the 'compass.transaction.lockTimeout' setting. The default is 10 (seconds).
Another option is to disable concurrency in Compass and make it synchronous. This is controlled with the 'compass.transaction.processor.read_committed.concurrentOperations': 'false' setting. You might also have to set 'compass.transaction.processor' to 'read_committed'
These are the compass settings we are currently using:
compassSettings = [
'compass.engine.optimizer.schedule.period': '300',
'compass.engine.mergeFactor':'1000',
'compass.engine.maxBufferedDocs':'1000',
'compass.engine.ramBufferSize': '128',
'compass.engine.useCompoundFile': 'false',
'compass.transaction.processor': 'read_committed',
'compass.transaction.processor.read_committed.concurrentOperations': 'false',
'compass.transaction.lockTimeout': '30',
'compass.transaction.lockPollInterval': '500',
'compass.transaction.readCommitted.translog.connection': 'ram://'
]
This has concurrency switched off. You can make it asynchronous by changing the 'compass.transaction.processor.read_committed.concurrentOperations' setting to 'true'. (or removing the entry).
Compass configuration reference:
http://static.compassframework.org/docs/latest/core-configuration.html
Documentation for the concurrency of read_committed processor:
http://www.compass-project.org/docs/latest/reference/html/core-searchengine.html#core-searchengine-transaction-read_committed
If you want to keep async operations, you can also control the number of threads it uses. Using compass.transaction.processor.read_committed.concurrencyLevel=1 setting would allow asynchronous operations but just use one thread (the default is 5 threads). There are also the compass.transaction.processor.read_committed.backlog and compass.transaction.processor.read_committed.addTimeout settings.
I hope this helps.