How to get dxl scripts to run faster - ibm-doors

I have created a DXL script that goes through every row of a couple modules. I am printing out certain rows and its information. I am doing this by having a for loop that goes through the rows and if it hits a row that I am interested in, I save the elements in the columns of this row to different string variables and print those string variables. The script does not take too long to run if the module does not have a lot of rows I am interested in but if I want to run multiple modules at the same time or if a module has a lot of rows I am interested in, the script can take hours. I can show the code that I have if this is not enough to come up with solutions. Any help would be appreciated!
I have tried using a skip list to store the print statements in that and then tried going through the skip list to print each value but that did not make the script run any faster.
string sep=","
for o in m do
{
string ver1= o."column1"
if (checkIf(o) && (!(isDeleted(o))))
{
string ver2= o."column2"
string onum=number(o)
""
string otext = o."Object Text"
print ver1 sep ver2 sep onum
}
}

Initial optimization:
for o in m do
{
if (checkIf(o) && (!(isDeleted(o)))) {
//This doesn't appear to be used?
//string otext = o."Object Text"
print o."column1" "," o."column2" "," number(o) "\n"
}
}
Reasoning: DOORS has a system in place called the string table that holds declared strings in memory- and doesn't necessarily do the best at clearing it out when appropriate. By constantly declaring strings in your loop, you might be bumping into the memory limits of that system.
Problem with this is that the results all end up in that 'DXL editor' little window, and then have to be copy and pasted somewhere else to actually use it.
Secondary optimization:
// Turn off runlimit for timing
pragma runLim , 0
// Set file location - CHANGE FOR YOUR COMPUTER
string csv_location = "C:/Users/Username/Desktop/Info_Collection.csv"
// Open stream
Stream out = append csv_location
// Set headers
out << "Module,Column 1,Column 2,Object Number" "\n"
// Define your loop constraints
Module m = current
Object o
// Run your loop
for o in m do
{
if (checkIf(o) && (!(isDeleted(o)))) {
//This doesn't appear to be used?
//string otext = o."Object Text"
out << fullName(m)","o."column1" "," o."column2" "," number(o) "\n"
}
}
close out
This will let you run the same script in different modules, all outputting to the same CSV file, which you can then load into Excel or your data manipulation engine of choice.
This keeps the data collection happening outside of DOORS, so if something goes awry, you can track down where it occurred.
My third optimization would be to use a list of modules in, say, excel as an input and do this analysis, but that might be going too far.
If this doesn't help, then we can start examining other issues.
Note- I still would like to know what 'checkIf' is/does.

If your objective is to speed up the execution of the script, since most of the objects are of no interest to you, the most effective way I know of is to filter out most of the objects that are not interesting, e.g., a filter which is obj."Object Text" != "" would filter out Headings, if you are just interested in requirements, obj."Object Text" contains "[Ss]hall" etc. Save as a view for later use.
for o in m do { respects the display set, so if you don't touch most of the objects it will speed it up a lot!
Hope this helps.
Don

Related

DXL: Need to extract data from previous object baselines

Basically, I have to go through all baselines of an object until I get the author who modified "_ReqStatus" attribute and copy that value in "_Ownr" attribute. Everything is working fine for the current baseline, but I cannot get through the older baselines of the module. I have to mention that I run the script for 2000 objects each one having at least 20 baselines.
My code looks like this:
//scriptul recunoaste obiectele cu "ReqStatus modifica"
pragma runLim, 0
Module m = current
History h
HistoryType ht
Object o
string attributName
string attributNameBaseline
string authorName
string newOwner
Baseline lBaseLine
noError()
for o in entire m do {
**for lBaseLine in module(o) do{ //These 2 code lines were my try to load all baselines
Module lBaseMod = load(module(o), false) //but with no results**
for h in o do
{
string owner = ""
attributName=""
attributName = h.attrName
authorName=""
owner = o."_Owner"
if isDeleted(o) then continue
if(attributName=="_ReqStatus")
{
authorName=h.author
//print authorName
//print "\n"
if(null owner)
{
print identifier(o)
print "\n"
newOwner = authorName
print newOwner"\n"
o."_Ownr" = newOwner
//print newOwner
break
}
}
}
}
}
ErrMess = lastError()
Thanks
Tony Goodman provides a script "Smart History Viewer" at http://www.smartdxl.com/content/?p=418, Michael Sutherland has one at https://www.ibm.com/mysupport/s/forumsquestion?id=0D50z00006HIGSUCA5, also, if you do a search on https://www.ibm.com/mysupport/s/forumshome for e.g. "dxl history" you might find some more code examples which should help you with this topic.
Your problem is the variable o. It is set only in the loop for o in entire m, it always refers to the o in the current version of m, never to the corresponding Object in a baseline. When you open the Module lBaseMod, o is not automatically reassigned.
So, use a new variable of type Object, set it to the Object in the baseline corresponding to o and browse through this new Object's history, see the linked scripts for reference.
Also, you should rework the flow of your script. With your approach, you open all the Baselines of m for each Object. In your case this means 2000 * 20 load and close. It will be much faster (though not necessarily less memory consuming) to open all the baselines of m in an outer loop, probably using a Skip list to collect the necessary information.
Also note that when you test your script, "print" becomes very slow after some time. It will be faster if you remember all the values that you want to print e.g. in a Buffer and print it at the end of your script, or write the output to a file.

How to identify text within an object's string

I want to find specified text within a string in a given column, and count how many times that string is repeated throughout the entire column.
For example, Find "XX" within a string in a column and print to dialogue box the number of times that text was found.
Module m = current
Object o
string s
string x
int offset = null
int len = null
int c
for o in m do
{
string s = probeAttr_(o, "AttributeA")
x = o."Object Text" ""
if(findPlainText(s, "XX", offset, len, false)){
print "Success "
} else {
print "Failed to match"
}
}
I have tried to use command findPlainText but I am inadvertently passing every object as true.
As well I placed the output to print 'success' or 'Failed to match' so I can at least get a number count of what is being passed. Unfortunately it seems like everything is being passed!
My understanding is that 'probeAttr_(o, "AttributeA")' allows me to specify and enter what column to search. As well o."Object Text" "" now allows me to look within any object and search for any text contained. I also realize that variable x is not being used but assume it has some way of being used to solve this issue.
I only use DOORS at a surface level but having this ability will save other staff tons of time. I realize this may be accomplished using the DOORS advanced filtering capability but I'd be able to compound this code with other simple commands to save time.
Thank you in advance for your help!!
If you want to count every occurence of a specified string in a text in an attribute for all objects, I think Mike's proposal is the correct answer. If you are only interested, if the specified string occurs once in that object's attribute, I suggest using Regexp, as I find it very fast, quite powerful and nevertheless easy to use, e.g.:
Regexp reSearch = regexp2 "XX"
int iCounter = 0
string strOT = ""
for o in m do {
strOT = o."Object Text" ""
if (reSearch strOT) {
iCounter++
}
}
print "Counted: '" iCounter "'\n"
Most of this has been answered in (DXL/Doors) How to find and count a text in a string?
You can easily exchange the "print" with a counter.

find Variables in a module

I'm new to DXL and I want to extract the variables containing
_I_,_O_ and _IO_
from a module and export then to csv file. Please help me with this
EG:
ADCD_WE_I_DFGJDGFJ_12_QWE and CVFDFH_AWEQ_I_EHH[X] is set to some value
This question has two parts.
You want to find variables that contain those parts in their name
You want to export to a .csv file
Another person may be able to expand on a better way, but the only way coming to mind right now for 1. is this:
Loop over the attributes in the module (for ad in m do {}) and get the string of the attribute names.
I am assuming that your attributes are valued at _I_, _O_ or _OI_? Like alpha = "_I_"? Are these enumerate values?
If they are, then you only need to check the value of each object's attribute. If one of those are the attribute values, then add it to something like a skip list. Having a counter here would be useful, maybe one for each attribute, like countI, countO, countOI, you can then use the counters as keys for the put() function for the skip list.
Once you have found all the attributes then you can move on to writing to csv
Stream out = write("filepathname/filename.csv") // to declare the stream
out << "_I_, _O_, _OI_\n"
Then you could loop over your Skip lists at the same time
int ijk = 0; bool finito = false
while(!finito) do {
if(ijk<countI) {
at = get(skipListI, ijk)
out << at.name ","
}
else out << ","
if(ijk<countO) {
at = get(skipListO, ijk)
out << at.name ","
}
else out << ","
if(ijk<countOI) {
at = get(skipListOI, ijk)
out << at.name "\n"
}
else out << "\n"
ijk++
// check if the next iteration would be outside of bounds on all lists
if(ijk >= countI && ijk >= countO && ijk >= countIO) finito = true
}
Or instead of at.name, you could print out whatever part of the attribute you wanted. The name, the value, "name:value" or whatever.
I have not run this, so you will be left to do any troubleshooting.
--
I hope this idea gets you started, write out what you want on paper first and then follow that plan. The key things I have mentioned that would be useful here are Skip lists, and Stream write (or append, if you want to keep adding).
In the future, please consider making your question more clear. Are you looking for those search terms in the name of the attribute, or in the value of the attribute. Are you looking to print out the names or the values, or the what? What kind of format for the .csv are you going to have? Any information will help your question be answered.

how to keep track of object and object history information in a loop

I'm writing a DXL script to extract history information from all objects and write some of the history parameters into other attributes (columns) in the DOORS module. I started out with the example script in the DXL Reference Manual (rev 9.6, near page 333), which just prints the information into the DXL editor window. I tried to add some code to write to the attribute _Reviewer -- see below. The code as written looks at the currently selected object rather than the one to which the current h history belongs to. What's the safest variable to pass into the function print so I can access the desired object and write to its _Reviewer attribute?
// history DXL Example
/* from doors manual
Example history DXL program.
Generate a report of the current Module's
history.
*/
// print a brief report of the history record
// hs is a variable of type HistorySession
// module is a variable of type Module
void print(History h) {
HistoryType ht = h.type
print h.author "\t" h.date "\t" ht "\t"
//next 3 lines are the code I added to the manual's example
Buffer authortmp = create;
authortmp = h.author "\t" h.date "\t" ht "\t"
(current Object)."_Reviewer" = authortmp;
// other code from original deleted
}
// Main program
History h
print "All history\n\n"
for h in current Module do print h
print "\nHistory for current Object\n\n"
for h in current Object do print h
print "\nNon object history\n\n"
for h in top current Module do print h
I imagine that you want to set the _Reviewer attribute not only for one object but rather for all objects of the module. So you will have a loop over all objects and for each object you will have a loop over each of its history entries.
So, the main loop would be like
Module m = current
string sHistoryAttributeName = "_Reviewer"
if (null m) then {infoBox "Open this script from a module";halt)
// […]add more code to check whether the attribute "_Reviewer" already exists in the current module and whether the module is open in edit mode
Object o
for o in entire m do {
if isDeleted(o) then continue // deleted objects are not of interest
// perhaps there are more objects that are not of interest. add relevant code here
if (!canModify o.sHistoryAttributeName) then {warn "cannot modify history entry for object " (identifier o) "\n"; continue}
Buffer bContentOfReview = create
History h
for h in o do {
bContentOfReview += getHistoryContent(h) "\n"
}
o.sHistoryAttributeName = sContentOfReview
delete bContentOfReview
}
save m
and your function getHistoryContent would be similar to your function void print (History h), only that you will return a string instead of printing the history entry. Something like
string getHistoryContent (History h) {
HistoryType ht = h.type
string sReturnValue = h.author "\t" h.date "\t" ht ""
return sReturnValue
}
One additional hint: you wrote "into other attributes (columns)". The above solution is for persistent attributes. Instead of this, you might want to show the information in a view as a DXL Layout column or or as a DXL attribute -- both possibilities have the advantage that the information is more or less always up to date, but with a persistent attribute the information will only be current after you run the script. Also note that this approach will only give you the changes since the last baseline. If you need more, the problem will be more complex. See the Rational DXL forum or google for more complex solutions of showing history entries
//Edit: removed typo in string concatenation, use Buffer insted

Boost spirit can handle Postscript/PDF like languages?

I noticed that Boost spirit offers some limits, in a question here on SO there is an user asking for help about boost spirit and the other user who gave the answer specified that boost spirit works well with statements and not with "generic text" ( I'm sorry if I don't recall it correctly ).
Now I would like to think about Postscript and PDF in terms of tokens and simplify my approach to this formats this way, the problem is that the PDF is kind of a mix between a markup language and a programming language with jumps and tables in it, and I can't think about something similar when considering the most popular file formats like XML, C++ code and others languages and formats.
There is also another fact: I can't really find people that had some kind of experience with boost::spirit wiriting a pdf parser or writer, so I'm asking, boost::spirit it's capable of parsing a PDF file and output the elements as tokens ?
Although this has nothing to do with Boost, let me assure you that the parsing of PDF (and PostScript) are about as trivial as you could want. Let's say that you have a scanner object that returns a series of tokens. The token types you will get from the scanner are:
String
Dict begin (<<)
Dict End (>>)
Name (/whatever)
Number
Hex array
Left Angle (<)
Right Angle (>)
Array begin ([)
Array end (])
Procedure begin ({)
Procedure end (})
Comment (%foo)
Word
My scanner is a finite-state automata with states for Start, Comment, String, HexArray, Token, DictEnd, and Done.
The way you parse PDF is not by parsing it, but by executing it. Given these tokens, my "parser" looks like this (in C#):
while (true) {
MLPdfToken = scanner.GetToken();
if (token == null)
return MachineExit.EndOfFile;
PdfObject obj = PdfObject.FromToken(token);
PdfProcedure proc = obj as PdfProcedure;
if (proc != null)
{
if (IsExecuting())
{
if (token.Type == PdfTokenType.RBrace)
proc.Execute(this);
else
Push(obj);
}
else {
proc.Execute(this);
}
if (proc.IsTerminal)
return Machine.ParseComplete;
}
else {
Push(obj);
}
}
I'll also add that if you give every PdfObject an Execute() method such that the base class implementation is machine.Push(this) and IsTerminal that returns false, the REPL gets easier:
while (true) {
MLPdfToken = scanner.GetToken();
if (token == null)
return MachineExit.EndOfFile;
PdfObject obj = PdfObject.FromToken(token);
if (IsExecuting())
{
if (token.Type == PdfTokenType.RBrace)
obj.Execute(this);
else
Push(obj);
}
else {
obj.Execute(this);
if (obj.IsTerminal)
return Machine.ParseComplete;
}
}
There's more support in Machine - Machine has a Stack of PdfObject and a few methods for accessing it (Push, Pop, Mark, CountToMark, Index, Dup, Swap), as well as ExecProcBegin and ExecProcEnd.
Beyond that, it's very light. The only thing that is slightly odd is that PdfObject.FromToken takes a token and if it is a primitive type (number, string, name, hex, bool) returns a corresponding PdfObject. Otherwise, it takes the given token and looks in a "proc set" dictionary of procedure names associated with PdfProcedure objects. So when you encounter the token << that gets looked up in a the proc set and comes up with this code:
void DictBegin(PdfMachine machine)
{
machine.Push(new PdfMark(PdfMarkType.Dictionary));
}
So << really means "mark the stack as the start of a dictionary. >> gets more interesting:
void DictEnd(PdfMachine machine)
{
PdfDict dict = new PdfDict();
// PopThroughMark pops the entire stack up to the first matching mark,
// throws an exception if it fails.
PdfObject[] arr = machine.PopThroughMark(PdfMarkType.Dictionary);
if ((arr.Length & 1) != 0)
throw new PdfException("dictionaries need an even number of objects.");
for (int i=0; i < arr.Length; i += 2)
{
PdfObject key = arr[i], val = arr[i + 1];
if (key.Type != PdfObjectType.Name)
throw new PdfException("dictionaries need a /name for the key.");
dict.put((PdfName)key, val);
}
machine.Push(dict);
}
So >> Pops up to the nearest dictionary mark into an array then puts each pair into the dictionary. Now, I could have done this without allocating the array. I could just pop pairs, putting them into the dictionary until I either hit the mark, fail to get a name or underflow the stack.
The important takeaway is that there really isn't any syntax in PDF, nor is there any in PostScript. At least not so much as you'd notice. The only real Syntax (and the read-eval-(push) loop shows it) is '}'.
So when you this is a PDF 14 0 obj << /Type /Annot /SubType /Square >> endobj what your really seeing is a series of procedures:
Push 14
Push 0
Execute obj (Pop two numbers and push a "definition" object).
Execute dictionary begin
Push /Type
Push /Annot
Push /SubType
Push /Square
Execute dictionary end
Execute endobj (pop the top object and then get (not pop) the next one. If the second is a definition, set its "value" to the first object, else throw).
Since "endobj" is terminal, parsing ends and the top of the stack is the result.
So when you are asked to look up object 14 in the PDF, the cross-reference table tells you where to seek to, you make a new Machine with the stream pointer at that location and run it. If the top of the stack is a "definition" object, you've succeeded.
About now you should be nodding but not trusting me, since you're thinking about PDF streams, which look like this:
<< [/key value]* >> stream ...raw data... endstream endobj
Again, there is no syntax. The proc stream looks at the top of the stack, which should be a PdfDict. If it is, it consumes characters until the next newline (scanner does this), stores the current file position in the stream as data start, reads the stream length from the dict (which may cause another Machine to get newed up), and skips past the end of stream and pushes the new stream object on the stack. endstream is a no-op. The only difference between a PdfDict and a PdfStream is that a PdfStream has a start position and a bool saying that it's a stream, otherwise I dual-purpose the object.
PostScript is almost identical except that the execution environment is a little more complex. For example, you need several stacks in your machine: a parameter stack, a dictionary stack, and an execution stack. From there, you more or less just bind your tokenizer into the set of primitive procedures as well as the word exec, and then most of your interpreter is written in PS itself.
If you're talking about boost, you're looking at C++, which means that you can't be as fast and loose with memory as I am, so you'll want to either use smart pointers or figure out where you scope is and be careful to dispose objects instead of blithely throwing them away, but that's just the normal C++ stuff.
Currently, I make PDF tools for my company in .NET, but in a former life I worked on Acrobat versions 1-4, and most of what I described is exactly what Acrobat did under the hood (well, more or less - it was C, not C++, but it's the same approach).
With respect to the xref table (or xref stream), you read that first - the spec tells you that if you jump to EOF and scan back, you find the start of the xref table. You parse that (which is a CS 101 assignment), parse the trailer, seek to the /Prev if any and repeat until no more /Prev entries. That gives you a complete xref for looking up objects.
As for writing - there are a number of approaches that you can take. The most obvious one is that when an object is meant to be referenced, you create a new reference object by assigning the newest available xref entry to it. Whenever objects refer to other objects for writing, they ask if these objects are referenced. If they are, they write the reference (ie, 14 0 R). When it comes time to write a referenced object, you get the current stream pointer and store it in the xref, then write <objnum> <generation> obj <object contents> endobj. For example, my code to write a dictionary looks like this:
public override ToStream(PdfStreamingContext context)
{
if (context.HasReference(this)) // is object referenced in xref
{
PdfUtils.WriteObjectDefinitionBegin(this, context);
}
context.Writer.Indent();
context.Writer.WriteLine("<<");
WriteContents(context);
context.Writer.Exdent();
context.Writer.Writeline(">>");
if (context.HasReference(this))
{
PdfUtils.WriteObjectDefinitionEnd(this, context);
}
}
I've chopped out some chaff so you can see the wheat underneath. The context is an object that holds a new xref table as well as an object for writing to streams that automagically handles appropriate newline discipline, indentation, line wrapping, and so on.
What you should see is that the basics here are straight forward, if not trivial. And now's when you should be asking yourself the question, "if it's trivial, how come there isn't more (serious) competition for Acrobat in the market? The answer is that even though it's trivial, it's still easy to write PDFs that aren't spec compliant and Acrobat handles most of those. The real challenge is to be able to honor the spec and make sure that you include all required values in a dictionary and that they are in range and semantically correct. Hell, even the date time format--which is pretty well-specified--is a mound of special case code in my library to manage where other people have screwed it up royally. Being able to generate consistently correct PDF is hard and consuming the garbage in the sea of PDFs in the world is harder.
I could (and probably should) write a book about how to do this. While a lot of the fringe code is grubby, the overall structure can be very pretty.
tl;dr - If you're thinking of a recursive descent parser for PDF, you're thinking too hard. All you need is a tokenizer and a simple REPL.

Resources