Error reported always until I added a manual check point to save the data frame after cube and before pivot.
Only manual save and reload the table works, other ways such as persist or check point all no use.
My question is how to avoid this. The error message:
Conflicting attributes: groupby_keys#90179
;;
'Join UsingJoin(Inner,List(groupby_keys))
My code is a little mess for to test in spark-shell but anyways:
case class FlagsStr(groupby_keys: String, profit_flag_str: String, cmd_str: String, open_followed_str: String, close_followed_str: String, close_or_open_flag_str: String, point: Int)
import org.apache.spark.sql.functions._
import org.apache.spark.sql.SparkSession
import org.apache.spark.storage.StorageLevel
def merge_flags(flag1: String, flag2: String): String = {
var ret: String = null
if( flag1 != null & flag2 != null)
ret = flag1 + "_" + flag2
else if ( flag1 != null || flag2 != null) {
if( flag1 != null)
ret = flag1
else
ret = flag2
} else ret = null
ret
}
def concat_str_flags = udf((profit_flag_str: String, cmd_str: String, open_followed_str: String, close_followed_str: String, close_or_open_flag_str: String) => {
//val retArr: Array[String] = Array(str_flags:_*)
val retArr: Array[String] = Array(profit_flag_str, cmd_str, open_followed_str, close_followed_str, close_or_open_flag_str)
val ret = retArr.reduce(merge_flags)
ret
})
def add_target_2_flags(flag: String, prefix: String): String = {
if( flag != null)
prefix + "_" + flag
else
prefix
}
def add_target_agg_2_flags(flag: String, prefix: String, suffix: String): String = {
if( flag != null)
prefix + "_" + flag + "_" + suffix
else
prefix + "_" + suffix
}
def add_order_2_flags = udf((flag: String) => {
add_target_2_flags(flag, "order")
})
def add_point_2_flags = udf((flag: String) => {
add_target_2_flags(flag, "point")
})
def add_point_max_2_flags = udf((flag: String) => {
add_target_agg_2_flags(flag, "point", "max")
})
val ss = SparkSession.builder().appName("TestCubePivot").enableHiveSupport().getOrCreate()
import ss.implicits._
val sqlContext = ss.sqlContext
val flagsStrDataFrame = sqlContext.createDataFrame(List(
FlagsStr("3_223", "profit", "long", "os", "cs", "close", 5),
FlagsStr("4_423", null, "long", "os", "cs", "close", 10),
FlagsStr("5_523", "profit", null, "os", "cs", "close", 15)
))
val h_mt4trades_with_target_agg_4_pivot_in_memory=flagsStrDataFrame .cube($"groupby_keys", $"profit_flag_str", $"cmd_str", $"open_followed_str", $"close_followed_str", $"close_or_open_flag_str") .agg(
count($"*").as("count_1"),
sum($"point").as("sum_point"),
max($"point").as("max_point")
) .filter($"groupby_keys".isNotNull) .withColumn("flags",concat_str_flags($"profit_flag_str", $"cmd_str", $"open_followed_str", $"close_followed_str", $"close_or_open_flag_str"))
h_mt4trades_with_target_agg_4_pivot_in_memory.write.format("orc").saveAsTable("h_mt4trades_with_target_agg_4_pivot_in_memory")
val h_mt4trades_with_target_agg_4_pivot = sqlContext.read.table("h_mt4trades_with_target_agg_4_pivot_in_memory")
//h_mt4trades_with_target_agg_4_pivot.checkpoint
h_mt4trades_with_target_agg_4_pivot.persist(StorageLevel.MEMORY_AND_DISK)
ss.sparkContext.broadcast(h_mt4trades_with_target_agg_4_pivot)
val h_mt4trades_stat_cube_order = h_mt4trades_with_target_agg_4_pivot.withColumn("flags_order", add_order_2_flags($"flags")) .groupBy($"groupby_keys") .pivot("flags_order") .max("count_1")
//h_mt4trades_stat_cube_order.checkpoint
val h_mt4trades_stat_cube_point = h_mt4trades_with_target_agg_4_pivot.withColumn("flags_point", add_point_2_flags($"flags")) .groupBy($"groupby_keys") .pivot("flags_point") .max("sum_point")
//h_mt4trades_stat_cube_point.checkpoint
val h_mt4trades_stat_cube_point_max = h_mt4trades_with_target_agg_4_pivot.withColumn("flags_point_max", add_point_max_2_flags($"flags")) .groupBy($"groupby_keys") .pivot("flags_point_max") .max("max_point")
val test = h_mt4trades_stat_cube_order.join(h_mt4trades_stat_cube_point, "groupby_keys").join(h_mt4trades_stat_cube_point_max, "groupby_keys")
The background to the question is, that I would like to create software change requests from Doors requirement changes.
For this I have to get the differences of requirements between two user selectable baselines of a module in a human readable format.
In the GUI I use the "Baseline Compare" function.
How can I access these results from a script (inside or outside of Doors) in a structured format?
A good starting point might the Doors DXL library Tools -> DXL Library -> baseline comparator example, but it needs heavy modifications for your overall use-case I guess. Take a look.
Found a modified version of the baseline compare script at this forum . I haven't tried it yet, but here's the code, which does use a GUI but you should be able to redirect the input & output pretty easily:
// Your Company Baseline Compare with Attributes
/*
Date Posted: 16-Jan-2008 00:03
Posted By: Juergen Albrecht
*/
/*
// (Your Company Baseline Compare) *
//---------------------------------------------------------------------------*
// *
// Project: Your Company Ottobrunn *
// *
//---------------------------------------------------------------------------*
// Module: all modules for baseline compare *
//---------------------------------------------------------------------------*
// Description: ... *
// *
// Functions: compare any two baselines of the current module *
// (or a baseline against the current version). *
// Sets a filter on objects which differ with request *
// to Object Heading, Object Text, Object Type and .... *
// *
// *
// Copyright (C) Your Company Your Town 2005-2007 Confidential. All rights reserved. *
//---------------------------------------------------------------------------*
// *
// $Header: /doors/lib/dxl/addins/user/BaselineCompare.dxl Ver. 1.0 15.03.06 18:00 UID-USER $
// *
//---------------------------------------------------------------------------*
// $History: BaselineCompare.dxl $
//
//********************************* Version 1.0 ****************************
// User: Date: Time: *
// Jürgen Albrecht (uid-al28423) 15.01.08 18:00
// Created in $/doors/lib/dxl/addins/user/BaselineCompare.dxl
//---------------------------------------------------------------------------*
// *
// *
// *
//---------------------------------------------------------------------------*
*/
pragma runLim, 0
////////////////////////////////////////////////////// constants for Attributes
const string DUMMY_LIST[] = {}
const string strAffectsArr[] = {"Ch.Bars", "Ch.Date", "History"}
const string strCharacterArr[] = {"System ", "Hidden ", "Customised"}
const string strValidityArr[] = {"Module ", "Object"}
const string strPropertyArr[] = {"Inherited", "Multi value"}
////////////////////////////////////////////////////// variables for Attributes
DB dbBLCompare = null
DBE dbeAttrDefNameList = null
DBE dbeAttrCharacter = null
DBE dbeAttrAffects = null
DBE dbeAttrValidity = null
DBE dbeAttrProperty = null
DBE dbeAttrType = null
DBE dbeDefaultValue = null
DBE dbeAttrTypeList = null
DBE dbeAttrTypeSelect = null
////////////////////////////////////////////////////// functions for Attributes
int chopInsert2(DBE inlist, string s)
{ int no_inlist = noElems inlist
int low = 0
int high = no_inlist-1
int hw = 0
string c
if (no_inlist == 0)
{ insert(inlist,0,s)
setCheck(inlist,0,false)
return 0
}
while (high-low > 1)
{ hw = low + ((high-low) / 2)
c = get(inlist,hw)
if (s < c)
high = hw
else
low = hw
}
c = get(inlist,high)
if (c < s)
{ insert(inlist,high+1,s)
setCheck(inlist,high+1,false)
return high + 1
}
c = get(inlist,low)
if (c > s)
{ insert(inlist,low,s)
setCheck(inlist,low,false)
return low
}
insert(inlist,low+1,s)
setCheck(inlist,low+1,false)
return low+1
} // chopInsert2
void populateLstAttrDef(bool bIncludeSystemAttr)
{ AttrDef ad
for ad in current Module do
{ if (ad.module)
continue
if (!bIncludeSystemAttr)
{ if (ad.system || ad.hidden)
continue
}
chopInsert2(dbeAttrDefNameList, ad.name "")
}
set(dbeAttrDefNameList, 0, true)
} // populateLstAttrDef
void doNothing(DBE dbeKlick, int iKlick)
{
} // doNothing
void cbSystemSelect(DBE dbeKlick)
{ bool bSelect = get(dbeAttrTypeSelect)
empty (dbeAttrDefNameList)
populateLstAttrDef (bSelect)
} // cbSystemSelect
void doSetAttrTypes(void)
{
AttrDef ad = find(current Module, get(dbeAttrDefNameList, get(dbeAttrDefNameList)))
AttrType at = find(current Module, ad.typeName "")
string strDefVal = "", strAttrType = "", s = ""
int iAttrChar = 0, iAttrValid = 0, iProperty = 0, iAffects = 0
if (ad.system)
iAttrChar = 1
else
iAttrChar = 4
if (ad.hidden)
iAttrChar = iAttrChar | 2
set(dbeAttrCharacter, iAttrChar)
if (ad.object)
iAttrValid = 2
if (ad.module)
iAttrValid = iAttrValid | 1
set(dbeAttrValidity, iAttrValid)
if (ad.inherit)
iProperty = 1
if (ad.multi)
iProperty = iProperty | 2
set(dbeAttrProperty, iProperty)
if (!ad.nobars)
iAffects = 1
if (!ad.nochanges)
iAffects = iAffects | 2
if (!ad.nohistory)
iAffects = iAffects | 4
set(dbeAttrAffects, iAffects)
if ( null at )
{ strAttrType = "Unable to determine type of attribute"
set(dbeAttrType, strAttrType)
return
}
strAttrType = at.type ""
if ( at.type == attrInteger )
{ int i
if ( at.minValue )
{ i = at.minValue
s = s "Min > " i "\n"
}
if ( at.maxValue )
{ i = at.maxValue
s = s "Max < " i "\n"
}
}
else if ( at.type == attrReal )
{ real r
if ( at.minValue )
{ r = at.minValue
s = s "Min > " r "\n"
}
if ( at.maxValue )
{ r = at.maxValue
s = s "Max < " r "\n"
}
}
else if ( at.type == attrDate )
{ Date d
if ( at.minValue )
{ d = at.minValue
s = s "Min > " d "\n"
}
if ( at.maxValue )
{ d = at.maxValue
s = s "Max < " d "\n"
}
}
else if ( at.type == attrEnumeration )
{ string strAttrTypeName = at.name
strAttrType = strAttrType ": " strAttrTypeName
int i
for ( i = 0; i < at.size; ++i )
s = s at.strings[i] "\n"
}
set(dbeAttrType, strAttrType)
set(dbeAttrTypeList, s)
if (ad.defval)
{ if (strDefVal == "")
strDefVal = ad.defval
set (dbeDefaultValue, strDefVal)
}
else
set (dbeDefaultValue, "")
// Ausgabe in File
// output << ad.name"\n"
// output << s"\n"
} // doSetAttrTypes
void cbDoSetAttrTypes(DBE dbeKlick, int iKlick)
{ doSetAttrTypes
} // cbDoSetAttrTypes
void showAttributes()
{ if ( null current Module )
{ ack "This tool must be run within a module..."
halt
}
// Ausgabe in File
// output << "\t\t"(current Module)."Name""\n"
populateLstAttrDef(true)
doSetAttrTypes()
// close output
} // showAttrInvestigatorDB
/////////////////////////////////////////// Definition for Modules
DBE lblOldList, lblNewList
DBE dbeListCurrent, dbeListBaseline // two global lists containing the baseline selected (or current version)
DBE dbeOutFileBrowse, dbeOutFileLabel
DBE dbeChangeBarToggle
DBE dbeShowDiffToggle
Stream outFile
Skip skBaselines = create // cache current baselines
bool bShowDiff
string strCellBorders = "\\clbrdrt\\brdrs\\brdrw15\\brdrcf11 \\clbrdrl\\brdrs\\brdrw15\\brdrcf11 \\clbrdrb\\brdrs\\brdrw15\\brdrcf11 \\clbrdrr\\brdrs\\brdrw15\\brdrcf11 "
string lastHeadingNumber(Object o)
{ Regexp reg = regexp "\\.0-[0-9]+$"
string onum = number o
if (onum[0:0] == "0")
{ return "0"
}
else if (reg onum)
{ int i = start 0
string hd = onum[0:i-1]
while (reg hd)
{ int j = start 0
hd = hd[0:j-1]
}
return hd
}
else
{ return onum
}
} // lastHeadingNumber
bool oleInsDel(Object o1,o2)
{ bool oleRec = false
string modType = ""
int oldOle = oleCount(o2."Object Text")
int newOle = oleCount(o1."Object Text")
if (newOle > oldOle)
{ oleRec = true
modType = "Figure/Table inserted"
}
if (newOle < oldOle)
{ oleRec = true
modType = "Figure/Table deleted"
}
if (oleRec)
{ outFile << "\\trowd \\trgaph108\\trleft-108\\trkeep"
outFile << strCellBorders "\\cellx1026 " strCellBorders "\\cellx2727 " strCellBorders "\\cellx6129 " strCellBorders "\\cellx9531\n"
outFile << "\\intbl " identifier(o1) " section " lastHeadingNumber(o1) "\\cell OLE\\cell " modType "\\cell \\cell\\row\n"
}
return oleRec
} // oleInsDel
bool oleChange(Object o1,o2)
{ int i = oleCount o1."Object Text"
if (i > 0)
{ Buffer b1 = create
Buffer b2 = create
b1 = richTextWithOle o1."Object Text"
b2 = richTextWithOle o2."Object Text"
int l1 = length b1
int l2 = length b2
delete b1
delete b2
if (l1 != l2)
{ outFile << "\\trowd \\trgaph108\\trleft-108\\trkeep"
outFile << strCellBorders "\\cellx1026 " strCellBorders "\\cellx2727 " strCellBorders "\\cellx6129 " strCellBorders "\\cellx9531\n"
outFile << "\\intbl " identifier(o1) " section " lastHeadingNumber(o1) "\\cell OLE\\cell Figure/Table modified\\cell \\cell\\row\n"
accept o1 // set filter
accept o2 // on both objects
return true
}
}
return false
} // oleChange
bool compareObjects(int absno, Object o1, o2, string attr)
{ // function to compare an attribute of two objects with same absolute number
Buffer s1 = create
Buffer s2 = create
Buffer s3 = create
s1 = o1.attr
s2 = o2.attr
if (s1!=s2)
{ outFile << "\\trowd \\trgaph108\\trleft-108\\trkeep"
// if ((attr=="Object Text") && bShowDiff)
if (bShowDiff)
// {
// diff(s3, s2, s1,"\\cf1\\strike ","\\cf3\\ul ")
// outFile << strCellBorders "\\cellx1026 " strCellBorders "\\cellx2727 " strCellBorders "\\cellx9531\n"
// outFile << "\\intbl \\fs16 " identifier(o1) " section " lastHeadingNumber(o1) "\\cell " attr "\\cell " s3 "\\cell " s2 "\\cell\\row\n"
// }
{ diff(s3, s2, s1)
Regexp ct = regexp "colortbl[^}]*}"
string frag
if (ct s3)
{ frag = s3[end 0 + 1:(length s3) - 3]
}
else
{ frag = richTextFragment stringOf(s3)
}
outFile << strCellBorders "\\cellx1026 " strCellBorders "\\cellx2727 " strCellBorders "\\cellx9531\n"
outFile << "\\intbl \\fs16 " identifier(o1) " section " lastHeadingNumber(o1) "\\cell " attr "\\cell " frag "\\cell\\row\n"
}
else
{ outFile << strCellBorders "\\cellx1026 " strCellBorders "\\cellx2727 " strCellBorders "\\cellx6129 " strCellBorders "\\cellx9531\n"
outFile << "\\intbl \\fs16 " identifier(o1) " section " lastHeadingNumber(o1) "\\cell " attr "\\cell " s1 "\\cell " s2 "\\cell\\row\n"
}
accept o1 // set filter
accept o2 // on both objects
delete s1
delete s2
delete s3
return false
}
delete s1
delete s2
delete s3
return true
} // compareObjects
Skip getAbsnos(Module m)
{ // Build a skip list which maps absnos onto their corresponding objects. Also initialize the DXL filter to "reject"
Skip res = create
Object o
for o in m do
{ int a = o."Absolute Number"
reject o // filter those mentioned in report
put(res, a, o)
}
return res
} // getAbsnos
void applyCompareFunction (DB dbKlick)
{ // Main comparison routine: find out which modules to compare
// compare objects present in both, report on
// those present in only one.
string name1, name2, outf
int idx1, idx2
bool updateChangeBars
idx1 = get dbeListCurrent // position in list
idx2 = get dbeListBaseline
name1 = get dbeListCurrent // baseline name
name2 = get dbeListBaseline
outf = get dbeOutFileBrowse
updateChangeBars = get dbeChangeBarToggle
bShowDiff = get dbeShowDiffToggle
if (idx1 < 0 || idx2 < 0) // error checking
{ ack "two selections are needed"
return
}
else if (idx1 == idx2)
{ ack "same selection on both sides"
return
}
Regexp slash = regexp "[\\\\]"
if (!slash outf)
{ outf = currentDirectory "\\" outf
}
Baseline sel1, sel2
outFile = write outf
outFile << "{\\rtf1\\deff0{\\fonttbl{\\f0\\fswiss\\fcharset177 Times New Roman;}}{\\colortbl ;\\red255\\green0\\blue0;\\red0\\green255\\blue0;\\red0\\green0\\blue255;}\n"
outFile << "\\paperw11906\\paperh16838\\margl1134\\margr567\\margt1134\\margb851\\headery567\\footery567\n"
string where = (current Module)."Name"
outFile << "\\ul \\fs24 Modified Objects in Module: " where "\\ul0 \\par \\par\n"
outFile << "\\trowd \\trgaph108\\trleft-108\\trhdr"
outFile << strCellBorders "\\cellx1026 " strCellBorders "\\cellx2727 " strCellBorders "\\cellx6129 " strCellBorders "\\cellx9531\n"
outFile << "\\intbl \\fs20 Identifier\\cell Attribute\\cell Current Baseline\\cell Old Baseline\\cell\\row\n"
string str
for str in skBaselines do // find each baseline
{ Baseline b = key skBaselines // the baseline is the key
string str = (major b) "." (minor b) (suffix b)
if (name1==str) sel1 = b
if (name2==str) sel2 = b
}
progressStart(dbBLCompare, "Baseline Compare", "", 1)
Module old = current
Module b1, b2
if (idx1==0)
b1 = old // i.e. the current Module
else
{ progressMessage("Load First Baseline Module")
b1 = load(sel1, true) // load the baselines on the screen
}
progressStop()
progressStart(dbBLCompare, "Baseline Compare", "", 1)
if (idx2==0)
b2 = old // i.e.e the current Module
else
{ progressMessage("Load Second Baseline Module")
b2 = load(sel2, true)
}
progressStop()
if (updateChangeBars and (idx1 == 0))
{ AttrDef atcb
atcb = find(b1,"ChangeBar")
if (null atcb)
{ updateChangeBars = false
ack "No ChangeBar attribute in current module"
}
else
{ atcb = find(b1,"WordDocChangeLog")
if (!null atcb)
{ b1."WordDocChangeLog" = outf
}
}
}
else if (updateChangeBars)
{ updateChangeBars = false
ack "Can only update ChangeBar in current module"
}
Skip skCompAttributes = create
string aName
AttrDef atb1,atb2
int i, iSkip = 0, iElems = noElems (dbeAttrDefNameList)
bool bCheck
for (i=0;i<iElems;i++)
{ bCheck = getCheck (dbeAttrDefNameList, i)
if (bCheck)
{ aName = get (dbeAttrDefNameList, i)
atb1 = find(b1,aName)
atb2 = find(b2,aName)
if ((null atb1) or (null atb2))
{ if (null atb1)
infoBox aName " not in later baseline"
else
infoBox aName " not in earlier baseline"
}
else
put(skCompAttributes, iSkip++, aName)
}
}
current = b1 // make sure filtering is off
filtering off // on both sides.
current = b2
filtering off
current = old
Skip absno1 = getAbsnos b1 // build caches of absnos -> objects
Skip absno2 = getAbsnos b2
Object o1, o2
int iObj=0, iMaxObj=0, diffs=0
bool StartInsert = true
for o1 in absno1 do // loop through side 1
iMaxObj++
progressStart(dbBLCompare, "Baseline Compare", "", iMaxObj)
progressMessage("Compare selected attributes of all objects by Identifier")
for o1 in absno1 do // loop through side 1
{ progressStep(iObj++)
Object o2
int i = (int key absno1)
if (find(absno2, i, o2)) // absno exists in other baseline
{ bool ChangeFound = false
for aName in skCompAttributes do
{ ChangeFound = !compareObjects(i, o1, o2, aName) || ChangeFound
}
ChangeFound = oleInsDel(o1,o2) || ChangeFound
if (!ChangeFound)
{ ChangeFound = oleChange(o1,o2)
}
if (ChangeFound)
{ diffs++ // found a difference
if (updateChangeBars)
{ o1."ChangeBar" = true
}
}
else if (updateChangeBars)
{ o1."ChangeBar" = false
}
delete(absno2, i) // remove from dbeListBaseline
}
else
{ if (StartInsert)
{ outFile << "\\pard\\par\\ul \\fs24 Inserted Objects\\ul0\\par \\par "
StartInsert = false
}
outFile << "\\fs20" identifier(o1) " section " lastHeadingNumber(o1) "\\par "
accept o1
if (updateChangeBars)
{ o1."ChangeBar" = true
}
diffs++
}
}
progressStop()
raise (dbBLCompare)
infoBox "Baseline Compare has finished"
if (StartInsert)
{ outFile << "\\pard\\par\\ul \\fs24 Inserted Objects\\ul0\\par \\par "
}
outFile << "\n\\par\\ul \\fs24 Deleted Objects\\ul0\\par \\par "
for o2 in absno2 do // now we can check for objects not in dbeListCurrent
{ int i = (int key absno2)
outFile << "\\fs20" identifier(o2) " section " lastHeadingNumber(o2) "\\par "
accept o2
diffs++
}
delete absno1 // delete caches
delete absno2
delete skCompAttributes
bool doFilter // set to true if differences
if (diffs==0)
{ outFile << "\n\\par \\fs24 no differences found\\par "
doFilter=false
}
else // set filtering on in baselines
{ if (diffs==1)
outFile << "\n\\par \\fs24 one difference found\\par "
else
outFile << "\n\\par \\fs24 " diffs " differences found\\par "
doFilter=true
}
outFile << "\\par }"
close(outFile)
current = b1 // set filters
if (b1 != old)
close b1
// filtering doFilter
// refresh current
current = b2
if (b2 != old)
close b2
// filtering doFilter
// refresh current
current = old // return to former current module
} // applyCompareFunction
//// MAIN PROGRAM ////////////////////////
Module m = current // check calling context
if (null m)
{ ack "program requires current Module"
halt
}
Baseline b
int i=0
for b in m do // count number of baselines
{ i++
}
if (i==0)
{ ack "no baselines to compare"
halt
}
// Now make a dialog for selecting two baselines for comparison
string where = (current Module)."Name"
dbBLCompare = create "Your Company Baseline Compare V. 1.0 startet at Module: \"" where "\""
string strEmptyArr[] = {}
dbeListCurrent = list(dbBLCompare, "Current/Baseline:", 300, i+1 <? 5, strEmptyArr) // make maximum size of 5 elements
dbeListCurrent->"right"->"unattached" // make lists side by side
dbeListBaseline = list(dbBLCompare, "Baseline to compare with:", 300, i+1 <? 5, strEmptyArr)
dbeListBaseline->"left"->"flush"->dbeListCurrent
dbeListBaseline->"top"->"aligned"->dbeListCurrent
dbeListBaseline->"right"->"unattached"
separator (dbBLCompare)
dbeAttrDefNameList = listView(dbBLCompare, listViewOptionCheckboxes , 250, 14, DUMMY_LIST)
dbeAttrDefNameList->"left"->"form"
dbeAttrDefNameList->"right"->"unattached"
dbeAttrAffects = checkBox(dbBLCompare, "", strAffectsArr, 0)
dbeAttrAffects->"top"->"aligned"->dbeAttrDefNameList
dbeAttrAffects->"left"->"flush"->dbeAttrDefNameList
dbeAttrAffects->"right"->"unattached"
inactive (dbeAttrAffects)
dbeAttrCharacter = checkBox(dbBLCompare, "", strCharacterArr, 0)
dbeAttrCharacter->"top"->"flush"->dbeAttrAffects
dbeAttrCharacter->"left"->"flush"->dbeAttrDefNameList
dbeAttrCharacter->"right"->"unattached"
inactive (dbeAttrCharacter)
dbeAttrValidity = checkBox(dbBLCompare, "valid for: ", strValidityArr, 0)
dbeAttrValidity->"top"->"flush"->dbeAttrCharacter
dbeAttrValidity->"left"->"flush"->dbeAttrDefNameList
dbeAttrValidity->"right"->"unattached"
inactive (dbeAttrValidity)
dbeAttrProperty = checkBox(dbBLCompare, "Properties: ", strPropertyArr, 0)
dbeAttrProperty->"top"->"flush"->dbeAttrValidity
dbeAttrProperty->"left"->"flush"->dbeAttrDefNameList
dbeAttrProperty->"right"->"unattached"
inactive (dbeAttrProperty)
dbeAttrType = field (dbBLCompare, "Type: ", "", 27, true)
dbeAttrType->"top"->"spaced"->dbeAttrProperty
dbeAttrType->"left"->"flush"->dbeAttrDefNameList
dbeAttrType->"right"->"unattached"
dbeDefaultValue = field (dbBLCompare, "Default value:", "", 21, true)
dbeDefaultValue->"top"->"flush"->dbeAttrType
dbeDefaultValue->"left"->"flush"->dbeAttrDefNameList
dbeDefaultValue->"right"->"unattached"
dbeAttrTypeList = text(dbBLCompare, "Possible values or defined borders:", "", 215, 142, true)
dbeAttrTypeList->"top"->"spaced"->dbeDefaultValue
dbeAttrTypeList->"left"->"flush"->dbeAttrDefNameList
dbeAttrTypeList->"right"->"unattached"
//dbeAttrTypeList->"bottom"->"unattached"
dbeAttrTypeSelect = toggle(dbBLCompare,"show System and Hidden Attributes", true)
dbeAttrTypeSelect->"top"->"flush"->dbeAttrDefNameList
dbeAttrTypeSelect->"left"->"form"
dbeAttrTypeSelect->"right"->"unattached"
dbeAttrTypeSelect->"bottom"->"unattached"
separator (dbBLCompare)
dbeOutFileLabel = label(dbBLCompare, "Output to:")
dbeOutFileBrowse = fileName(dbBLCompare, "D:\\Project - DOORS Demonstration\\BL_Comp_Result.rtf", "*.rtf", "Rich Text files", false)
dbeChangeBarToggle = toggle(dbBLCompare,"Update ChangeBar attribute", false)
beside (dbBLCompare)
dbeShowDiffToggle = toggle(dbBLCompare,"Show Object Text changes as markup", true)
// dummy
DBE dbeDummy = label(dbBLCompare, "")
dbeDummy->"top"->"spaced"->dbeShowDiffToggle
dbeDummy->"left"->"form"
// Copyright
DBE dbeCopyRight = label(dbBLCompare, "Copyright (C) Your Company Your Town 2005-2007 Confidential. All rights reserved.")
dbeCopyRight->"bottom"->"spaced"->dbeDummy
dbeCopyRight->"left"->"form"
apply(dbBLCompare, "Compare Now", applyCompareFunction)
set(dbeAttrDefNameList, cbDoSetAttrTypes, doNothing, doNothing)
set(dbeAttrTypeSelect, cbSystemSelect)
realize (dbBLCompare, 0, 0) // we realize so that the lists can be populated using insert
insertColumn(dbeAttrDefNameList, 0, "Attribute Name", 200, iconNone)
for b in m do // fill up the baselines skip list with current baselines
{ string str = (major b) "." (minor b) (suffix b)
put(skBaselines, b, str)
insert(dbeListCurrent, 0, str)
insert(dbeListBaseline, 0, str)
}
insert(dbeListCurrent, 0, "current") // put current at head of lists
set(dbeListCurrent, 0)
insert(dbeListBaseline, 0, "current")
set(dbeListBaseline, 1)
showAttributes()
show dbBLCompare // off we go.......
// end of BaselineCompare.dxl
I would like to get data from a URL using the GO 3rd party library from http://godoc.org/code.google.com/p/go.net/html . But I came across a problem, that is I couldn't get the content of an html.Node.
There's an example code in the reference document, and here's the code.
s := `<p>Links:</p><ul><li>Foo<li>BarBaz</ul>`
doc, err := html.Parse(strings.NewReader(s))
if err != nil {
log.Fatal(err)
}
var f func(*html.Node)
f = func(n *html.Node) {
if n.Type == html.ElementNode && n.Data == "a" {
for _, a := range n.Attr {
if a.Key == "href" {
fmt.Println(a.Val)
break
}
}
}
for c := n.FirstChild; c != nil; c = c.NextSibling {
f(c)
}
}
f(doc)
The output is:
foo
/bar/baz
If I want to get
Foo
BarBaz
What should I do?
The tree of <strong>Foo</strong>Bar looks basically like this:
ElementNode "a" (this node also includes a list off attributes)
ElementNode "strong"
TextNode "Foo"
TextNode "Bar"
So, assuming that you want to get the plain text of the link (e.g. FooBar) you would have to walk trough the tree and collect all text nodes. For example:
func collectText(n *html.Node, buf *bytes.Buffer) {
if n.Type == html.TextNode {
buf.WriteString(n.Data)
}
for c := n.FirstChild; c != nil; c = c.NextSibling {
collectText(c, buf)
}
}
And the changes in your function:
var f func(*html.Node)
f = func(n *html.Node) {
if n.Type == html.ElementNode && n.Data == "a" {
text := &bytes.Buffer{}
collectText(n, text)
fmt.Println(text)
}
for c := n.FirstChild; c != nil; c = c.NextSibling {
f(c)
}
}
I'm reading email headers (in Node.js, for those keeping score) and they are VARY varied. E-mail addresses in the to field look like:
"Jake Smart" <jake#smart.com>, jack#smart.com, "Development, Business" <bizdev#smart.com>
and a variety of other formats. Is there any way to parse all of this out?
Here's my first stab:
Run a split() on - to break up the different people into an array
For each item, see if there's a < or ".
If there's a <, then parse out the email
If there's a ", then parse out the name
For the name, if there's a ,, then split to get Last, First names.
If I first do a split on the ,, then the Development, Business will cause a split error. Spaces are also inconsistent. Plus, there may be more e-mail address formats that come through in headers that I haven't seen before. Is there any way (or maybe an awesome Node.js library) that will do all of this for me?
There's a npm module for this - mimelib (or mimelib-noiconv if you are on windows or don't want to compile node-iconv)
npm install mimelib-noiconv
And the usage would be:
var mimelib = require("mimelib-noiconv");
var addressStr = 'jack#smart.com, "Development, Business" <bizdev#smart.com>';
var addresses = mimelib.parseAddresses(addressStr);
console.log(addresses);
// [{ address: 'jack#smart.com', name: '' },
// { address: 'bizdev#smart.com', name: 'Development, Business' }]
The actual formatting for that is pretty complicated, but here is a regex that works. I can't promise it always will work though. https://www.rfc-editor.org/rfc/rfc2822#page-15
const str = "...";
const pat = /(?:"([^"]+)")? ?<?(.*?#[^>,]+)>?,? ?/g;
let m;
while (m = pat.exec(str)) {
const name = m[1];
const mail = m[2];
// Do whatever you need.
}
I'd try and do it all in one iteration (performance). Just threw it together (limited testing):
var header = "\"Jake Smart\" <jake#smart.com>, jack#smart.com, \"Development, Business\" <bizdev#smart.com>";
alert (header);
var info = [];
var current = [];
var state = -1;
var temp = "";
for (var i = 0; i < header.length + 1; i++) {
var c = header[i];
if (state == 0) {
if (c == "\"") {
current.push(temp);
temp = "";
state = -1;
} else {
temp += c;
}
} else if (state == 1) {
if (c == ">") {
current.push(temp);
info.push (current);
current = [];
temp = "";
state = -1;
} else {
temp += c;
}
} else {
if (c == "<"){
state = 1;
} else if (c == "\"") {
state = 0;
}
}
}
alert ("INFO: \n" + info);
For something complete, you should port this to JS: http://cpansearch.perl.org/src/RJBS/Email-Address-1.895/lib/Email/Address.pm
It gives you all the parts you need. The tricky bit is just the set of regexps at the start.