I've made a workflow that uses a loop and is designed to be called from an application. The problem I'm having is that sometimes I'll want the loop to run 10 times, and sometimes I want it to run 25 times. I have a variables set to the number of times the loop needs to run, but I can't put that in the text field for the number of times the loop should run. Can this be done, or is there an alternate way of accomplishing this?
Related
Mistakenly, many months ago, I used the following logic to essentially implement the same functionality as the PCollectionView's asList() method:
I assigned a dummy key of “000” to every element in my collection
I then did a groupBy on this dummy key so that I would essentially get a list of all my elements in a single array list
The above logic worked fine when I only had about 200 elements in my collection. When I ran it on 5000 elements, it ran for hours until I finally killed it. Then, I created a custom “CombineFn” in which I essentially put all the elements into my own local hash table. That worked, and even on my 5000 element situation, it ran within about a minute or less. Similarly, I later learned that I could use the asList() method, and that too ran in less than a minute. However, what concerns me – and what I don't understand – is why the group by took so long to run (even with only 200 elements it would take more than a few seconds) and with 5000 it ran for hours without seeming to accomplish anything.
I took a look at the group by code, and it seems to be doing a lot off steps I don't quite understand… Is it somehow related to the fact that the group by statement is attempting to run things across a cluster? Or is it may be related to using an efficient coder? Or am I missing something else? The reason I ask is that there are certain situations in which I'm forced to use a group by statement, because the data set is way too large to fit in any single computer's RAM. However, I am concerned that I'm not understanding how to properly use a group by statement, since it seems to be so slow…
There are a couple of things that could be contributing to the slow performance. First, if you are using SerializableCoder, that is quite slow, and AvroCoder will likely work better for you. Second, if you are iterating through the Iterable in your ParDo after the GBK, if you have enough elements you will exceed the cache and end up fetching the same data many times. If you need to do that, explicitly putting the data in a container will help. Both the CombineFn and asList approaches do this for you.
I want to run my delphi program, but I need to see how many times a specific loop is executed. I remember in school when using Delphi 7 we used to set it so that when running it would only run 1 line at a time instead of run all of them at once. I can unfortunately not, for the life of me, remember how to do this.
How do you set Delphi to only run one line at a time and require you to step it forward before running the next one?
Run the program in the debugger. Typically you would start this by pressing F9, but that will set the program running. Instead you would use the Step Over action, with shortcut F8. Then each time you want to step over a line, press F8 again. As the name implies, if the line has a function call you will "step" over that call. To step into a function use Step Into, F7. When you do that the debugger will move to the first line of the function.
If you start the program in stepping mode, it might take you a while to reach the point of interest. Get there more quickly by setting a breakpoint with the F5 key, and by the Run to Cursor action, shortcut F4.
All of these actions can be found on the menu or the toolbar, but it is crushingly slow to use the mouse to step through code. That's why all half-way proficient developers use the keyboard shortcuts when debugging. It is way more productive and that really matters when debugging.
For more information, please refer to the documentation: http://docwiki.embarcadero.com/RADStudio/en/Overview_of_Debugging
F7 will execute one line (and will step into a subroutine).
F8 will execute one line (but will not step into a subroutine so actually many lines can be executed).
Shortcut keys are documented here.
You say that you need to know how many times a certain loop executes.
One way to do this is by placing a breakpoint on one of the lines inside the loop that is executed each time (not inside some conditional block or another nested loop) running your application in Debug Mode (F9 key shortcut) which would cause your program to stop once it reaches that breakpoint.
Then you use F9 key to step over that breakpoint to the next one and count how many times did your program stop at that specific breakpoint.
While the above approach is easy to setup it is not very friendly if you have loop with lots of iterations (a few dozens) as it can take you quite some time to count each iteration manually. Not to mention that you can easily miscount yourself.
So in order to avoid the above mentioned problems you can ask the debugger to help you with counting. How do you do this?
You do this in two steps.
In first step you place one breakpoint inside your loop like in the first example. But this time after you place it you right click on it and chose Breakpoint Properties from the Popup Menu. This will open Breakpoint Properties Dialog from where you can define different properties which affects what happens when the breakpoint is reached.
The property that you are interested here is Pass count.
Usually this property is used to break your program after specific breakpoint is passed for certain times.
But in our case we are not interested in stopping of our program but instead in counting the number of passes so you should set its value to be higher than the expected number of loop iterations so that this breakpoint won't cause your program to stop at all.
Then the next step is to add additional breakpoint on the first line which will be executed after the loop.
So your setup should now stop your program just after the loop and you can evaluate how many times was the first breakpoint (the one inside your loop) passed by moving your mouse cursor over it.
As you will notice the number of passes is shown as Pass count: X of Y where X is the number of times that breakpoint has already been passed and Y is the amount of passes that you specified in the breakpoint properties.
This second approach makes counting of loop iteration much easier especially when you have a few dozens or even a few hundreds of loop iterations.
But bear in mind that this approach can only be used to count the number of your loop iterations the first time the loop is used. Why?
Because the debugger is counting the number of breakpoint passes since the program start. So entering a loop again later would mean that the debugger would just continue counting from where it stopped last time.
You can however force the debugger to reset the counter by setting it to 0 and leaving your program to stop on it. After that you can again set the breakpoint Pass count property to some larger number.
NOTE: Resetting the breakpoint pass counter like explained above does needed you to step into one loop iteration so the number of total loop iteration returned will be lowered by one (the loop iteration that your program stopped in in order to reset the counter).
With all the talk about adding breakpoints and stepping through the code I want to offer another option: use a profiler.
With a profiler (like AQtime) you can easily find out how often which function or line in your program is executed without having to add additional variables or stepping through the code in a debugger.
If the only information you're interested in is the amount of times that a loop is executed, I think this is the cleanest option.
I want to determine necessary time to execute a procedure, time value must be returned in the first instruction in this procedure, like this:
procedure myProcedure;
begin
// determine necessary time here
//other instructions
end;
First, of all, is this possible? if yes, how can I do this?
Thanks for your replies
In general you cannot expect to do this. Unless you have carefully profiled the code on the machine on which you run it, under the exact conditions in which it runs. Without that information there's no way to reliably predict run time up front.
You actually want to display progress. Typically you do that by keeping track of how the overall task is progressing. If the task is made up of 50 similarly sized sub tasks, then you can measure proportion complete simply by knowing how many sub tasks are complete.
That's the normal way to measure progress for a progress bar. If you wish to estimate time to completion then make a note of when the task starts. Then, as worl proceeds, you can use simple maths to work out an estimate of the time remaining. If the first 10 sub tasks took 30 seconds, then estimate that the next 10 will take another 30 seconds.
I've tried every possible fields but can not find the number of times functions are called.
Besides, I don't get Self and # Self. What do these two numbers mean?
There are several other ways to accomplish this. One is obviously to create a static hit counter and an NSLog that emits and increments a counter. This is intrusive though and I found a way to do this with lldb.
Set a breakpoint
Execute the program until you hit the breakpoint the first time and note the breakpoint number on the right hand side of the line you hit (e.g. "Thread 1: breakpoint 7.1", note the 7.1)
Context click on the breakpoint and choose "Edit Breakpoint"
Leave condition blank and choose "Add Action"
Choose "Debugger Command"
In the command box, enter "breakpoint list 7.1" (using the breakpoint number for your breakpoint from step 2). I believe you can use "info break " if you are using gdb.
Check Options "Automatically Continue after evaluating"
Continue
Now, instead of stopping, llvm will emit info about the breakpoint including the number of times it has been passed.
As for the discussion between Glenn and Mike on the previous answer, I'll describe a performance problem where function execution count was useful: I had a particular action in my app where performance degraded considerably with each execution of the action. The Instruments time profiler showed that each time the action was executed, a particular function was taking twice as long as the time before until quickly the app would hang if the action was performed repeatedly. With the count, I was able to determine that with each execution, the function was called twice as many times as it was during the previous execution. It was then pretty easy to look for the reason, which turned out to be that someone was re-registering for a notification in NotificationCenter on each event execution. This had the effect of doubling the number of response handler calls on each execution and thus doubling the "cost" of the function each time. Knowing that it was doubling because it was called twice as many times and not because the performance was just getting worse caused me to look at the calling sequence rather than for reasons the function itself could be degrading over time.
While it's interesting, knowing the number of times called doesn't have anything to do with how much time is spent in them. Which is what Time Profiler is all about. In fact, since it does sampling, it cannot answer how many times.
It seems you cannot use Time Profiler for counting function calls. This question seems to address potential methods for counting.
W/ respect to self and #self:
Self is "The number of times the symbol calls itself." according to the Apple Docs on the Time Profiler.
From the way the numbers look though, it seems self is the summed duration of samples that had this symbol at the bottom of its stack trace. That would make:
# self: the number of samples where this symbol was at the bottom of the stack trace
% self: the percent of self samples relative to total samples of currently displayed call tree
(eg - #self / total samples).
So this wouldn't tell you how many times a method was called. But it would give you an idea how much time is spent in a method or lower in the call tree.
NOTE: I too am unsure about the various 'self' meanings though. Would love to see someone answer this authoritatively. Arrived here searching for that...
IF your objective is to find out what you need to fix to make the program as fast as possible,
Number of calls and self time may be interesting but are irrelevant.
Look at my answer to this question, in particular points 6 and 8.
EDIT: To clarify the point further, suppose the following is the timeline of execution of the program. Some of that time (in this case about 50%) is spent in an activity that can be removed, if you know what it is, such as needless buried I/O, excessive calls to new, runaway notifications, or "insignificant" data validation. If a random-time sample is taken, it has a 50% chance of occurring in that activity, and an examination of the call stack and/or program variables shows that it is doing something that can be removed. Then, if 10 such samples are taken, the activity will be seen on roughly 5 of them, regardless of whether the activity occurs in a few large chunks of time, or many small ones. The activity may be a few lines of code in a function doing something unnecessary, or it may be something much more generalized. Regardless, you recognize it, fix it, and get roughly a factor of 2 speedup. Call counts and self time contribute nothing to this process.
I've been writing some scripts for a game, the scripts are written in Lua. One of the requirements the game has is that the Update method in your lua script (which is called every frame) may take no longer than about 2-3 milliseconds to run, if it does the game just hangs.
I solved this problem with coroutines, all I have to do is call Multitasking.RunTask(SomeFunction) and then the task runs as a coroutine, I then have to scatter Multitasking.Yield() throughout my code, which checks how long the task has been running for, and if it's over 2 ms it pauses the task and resumes it next frame. This is ok, except that I have to scatter Multitasking.Yield() everywhere throughout my code, and it's a real mess.
Ideally, my code would automatically yield when it's been running too long. So, Is it possible to take a Lua function as an argument, and then execute it line by line (maybe interpreting Lua inside Lua, which I know is possible, but I doubt it's possible if all you have is a function pointer)? In this way I could automatically check the runtime and yield if necessary between every single line.
EDIT:: To be clear, I'm modding a game, that means I only have access to Lua. No C++ tricks allowed.
check lua_sethook in the Debug Interface.
I haven't actually tried this solution myself yet, so I don't know for sure how well it will work.
debug.sethook(coroutine.yield,"",10000);
I picked the number arbitrarily; it will have to be tweaked until it's roughly the time limit you need. Keep in mind that time spent in C functions etc will not increase the instruction count value, so a loop will reach this limit far faster than calls to long-running C functions. It may be viable to set a far lower value and instead provide a function that sees how much os.clock() or similar has increased.