Related
Suppose there is a table like below:
datatable(str:string) [
"a,b,2,10,d,e;a,b,c,14,d,e;a,b,c,10,d,e",
"a,b,c,11,d,e;a,b,c,12,d,e;a,b,c,13,d,e;a,b,c,10,d,e",
"a,b,c,20,d,e;a,b,c,25,d,e",
]
I need to add 4th value in each string separated by semicolon
e.g. Answer for above table is
10+14+10=34
11+12+13+10=46
20+25=45
I tried below which works for single row:
let calculateCostForARow = (str:string) {
print row = split(str,";")
| mv-expand row
| parse row with * "," * "," * "," cost:long "," *
| summarize sum(cost)
};
calculateCostForARow("a,b,c,11,d,e;a,b,c,12,d,e;a,b,c,13,d,e;a,b,c,10,d,e")
but doesn't work for table as to_scalar has issues with table
let calculateCostForARow = (str:string) {
toscalar(print row = split(str,";")
| mv-expand row
| parse row with * "," * "," * "," cost:long "," *
| summarize sum(cost))
};
datatable(str:string) [
"a,b,c,10,d,e;a,b,c,10,d,e;a,b,c,10,d,e",
"a,b,c,10,d,e;a,b,c,10,d,e;a,b,c,10,d,e;a,b,c,10,d,e",
"a,b,c,10,d,e;a,b,c,10,d,e",
]
| project calculateCostForARow(str)
Let me know if there are other ways to do this?
you could try this, using mv-apply:
datatable(str:string) [
"a,b,2,10,d,e;a,b,c,14,d,e;a,b,c,10,d,e",
"a,b,c,11,d,e;a,b,c,12,d,e;a,b,c,13,d,e;a,b,c,10,d,e",
"a,b,c,20,d,e;a,b,c,25,d,e",
]
| mv-apply s = split(str, ";") on (
summarize result = sum(tolong(split(s, ",", 3)[0]))
)
str
result
a,b,2,10,d,e;a,b,c,14,d,e;a,b,c,10,d,e
34
a,b,c,11,d,e;a,b,c,12,d,e;a,b,c,13,d,e;a,b,c,10,d,e
46
a,b,c,20,d,e;a,b,c,25,d,e
45
I'm discovering the pyparsing module which is really cool. I'm trying to parse a set of simple boolean expressions in which some identifiers (foo/bar/bla and zoo) are compared to numeric values. The parser is used to check that user expression is correct but I would like also to get the name of the identifiers used in the expression (i.e which combination of foo/bar/bla and zoo was used). I can't get a simple way to do it.
In the example below foo and bar are used in the expression. But how can I get this information ?
Best
from pyparsing import oneOf
from pyparsing import Group
from pyparsing import Regex
from pyparsing import operatorPrecedence
from pyparsing import opAssoc
from pyparsing import Literal
from pyparsing import Word
from pyparsing import nums
from pyparsing import Combine
from pyparsing import Optional
from pyparsing import CaselessLiteral
from pyparsing import alphanums
from pyparsing import quotedString
from pyparsing import Forward
lparen = Literal("(")
rparen = Literal(")")
and_operator = CaselessLiteral("and")
or_operator = CaselessLiteral("or")
comparison_operator = oneOf(['==','!=','>','>=','<', '<='])
point = Literal('.')
e = CaselessLiteral('E')
plusorminus = Literal('+') | Literal('-')
number = Word(nums)
integer = Combine( Optional(plusorminus) + number )
float_nb = Combine( integer +
Optional( point + Optional(number) ) +
Optional( e + integer ))
value = float_nb
value.resultsName = 'value'
identifier = oneOf(['foo','bar', 'bla', 'zoo'], caseless=False)
identifier.resultsName = 'key'
group_1 = Group(identifier + comparison_operator + value)
group_2 = Group(value + comparison_operator + identifier)
comparison = group_1 | group_2
boolean_expr = operatorPrecedence(
comparison,
[(and_operator, 2, opAssoc.LEFT),
(or_operator, 2, opAssoc.LEFT)])
boolean_expr_par = "(" + boolean_expr + ")"
expression = Forward()
expression << boolean_expr | boolean_expr_par
exp = expression.parseString('2.5 > foo and (3 < bar or (foo > 10 and bar < 3)) ' , parseAll=True)
# Now how can I get the 'identifiers used in exp' ?
I had a similar problem.
I used the 'setParseAction' method in your 'identifier' parser to set a function recording each occurrence of the token matching 'identifier'. Then I print the recorded items:
I declare :
idSet = set()
def recordID(tokens):
idSet.add(tokens[0])
return
I modify your 'identifier' parser as follows:
identifier = oneOf(['foo','bar', 'bla', 'zoo'], caseless=False).setParseAction(recordID)
At the end of you script I print 'idSet':
exp = expression.parseString('2.5 > foo and (3 < bar or (foo > 10 and bar < 3)) ' , parseAll=True)
print(idSet)
It gives the following outcome:
{'foo', 'bar'}
I'm playing with FsCheck so I have this implementation:
let add a b =
if a > 100
then failwith "nasty bug"
else a + b
...and this FsCheck based test:
fun (a:int) -> (add a 0) = a
|> Check.QuickThrowOnFailure
and the test never fails. My guess is that the 100 values produced by the random generator are never bigger than 100.
Shouldn't the values be more "random"?
When you use Check.QuickThrowOnFailure, it uses the configuration Config.QuickThrowOnFailure, which has these values:
> Config.QuickThrowOnFailure;;
val it : Config =
{MaxTest = 100;
MaxFail = 1000;
Replay = null;
Name = "";
StartSize = 1;
EndSize = 100;
QuietOnSuccess = false;
Every = <fun:get_Quick#342>;
EveryShrink = <fun:get_Quick#343-1>;
Arbitrary = [];
Runner = <StartupCode$FsCheck>.$Runner+get_throwingRunner#355;}
The important values to consider here are StartSize, but particularly EndSize. Some of the generators in FsCheck uses the size context to determine the size or range of values it generates.
If you change the EndSize to e.g. 1,000 you can make your test fail:
> Check.One({Config.QuickThrowOnFailure with EndSize = 1000}, fun (a:int) -> (add a 0) = a);;
System.Exception: Falsifiable, after 15 tests (0 shrinks) (StdGen (1912816373,296229213)):
Original:
101
with exception:
> System.Exception: nasty bug
at FSI_0040.add(Int32 a, Int32 b)
at FSI_0055.it#69-6.Invoke(Int32 a)
at FsCheck.Testable.evaluate[a,b](FSharpFunc`2 body, a a) in C:\Users\Kurt\Projects\FsCheck\FsCheck\src\FsCheck\Testable.fs:line 161
at <StartupCode$FsCheck>.$Runner.get_throwingRunner#365-1.Invoke(String message) in C:\Users\Kurt\Projects\FsCheck\FsCheck\src\FsCheck\Runner.fs:line 365
at <StartupCode$FsCheck>.$Runner.get_throwingRunner#355.FsCheck-IRunner-OnFinished(String , TestResult ) in C:\Users\Kurt\Projects\FsCheck\FsCheck\src\FsCheck\Runner.fs:line 365
at FsCheck.Runner.check[a](Config config, a p) in C:\Users\Kurt\Projects\FsCheck\FsCheck\src\FsCheck\Runner.fs:line 275
at <StartupCode$FSI_0055>.$FSI_0055.main#()
Stopped due to error
I am struggling to get my head around LPEG. I have managed to produce one grammar which does what I want, but I have been beating my head against this one and not getting far. The idea is to parse a document which is a simplified form of TeX. I want to split a document into:
Environments, which are \begin{cmd} and \end{cmd} pairs.
Commands which can either take an argument like so: \foo{bar} or can be bare: \foo.
Both environments and commands can have parameters like so: \command[color=green,background=blue]{content}.
Other stuff.
I also would like to keep track of line number information for error handling purposes. Here's what I have so far:
lpeg = require("lpeg")
lpeg.locale(lpeg)
-- Assume a lot of "X = lpeg.X" here.
-- Line number handling from http://lua-users.org/lists/lua-l/2011-05/msg00607.html
-- with additional print statements to check they are working.
local newline = P"\r"^-1 * "\n" / function (a) print("New"); end
local incrementline = Cg( Cb"linenum" )/ function ( a ) print("NL"); return a + 1 end , "linenum"
local setup = Cg ( Cc ( 1) , "linenum" )
nl = newline * incrementline
space = nl + lpeg.space
-- Taken from "Name-value lists" in http://www.inf.puc-rio.br/~roberto/lpeg/
local identifier = (R("AZ") + R("az") + P("_") + R("09"))^1
local sep = lpeg.S(",;") * space^0
local value = (1-lpeg.S(",;]"))^1
local pair = lpeg.Cg(C(identifier) * space ^0 * "=" * space ^0 * C(value)) * sep^-1
local list = lpeg.Cf(lpeg.Ct("") * pair^0, rawset)
local parameters = (P("[") * list * P("]")) ^-1
-- And the rest is mine
anything = C( (space^1 + (1-lpeg.S("\\{}")) )^1) * Cb("linenum") / function (a,b) return { text = a, line = b } end
begin_environment = P("\\begin") * Ct(parameters) * P("{") * Cg(identifier, "environment") * Cb("environment") * P("}") / function (a,b) return { params = a[1], environment = b } end
end_environment = P("\\end{") * Cg(identifier) * P("}")
texlike = lpeg.P{
"document";
document = setup * V("stuff") * -1,
stuff = Cg(V"environment" + anything + V"bracketed_stuff" + V"command_with" + V"command_without")^0,
bracketed_stuff = P"{" * V"stuff" * P"}" / function (a) return a end,
command_with =((P("\\") * Cg(identifier) * Ct(parameters) * Ct(V"bracketed_stuff"))-P("\\end{")) / function (i,p,n) return { command = i, parameters = p, nodes = n } end,
command_without = (( P("\\") * Cg(identifier) * Ct(parameters) )-P("\\end{")) / function (i,p) return { command = i, parameters = p } end,
environment = Cg(begin_environment * Ct(V("stuff")) * end_environment) / function (b,stuff, e) return { b = b, stuff = stuff, e = e} end
}
It almost works!
> texlike:match("\\foo[one=two]thing\\bar")
{
command = "foo",
parameters = {
{
one = "two",
},
},
}
{
line = 1,
text = "thing",
}
{
command = "bar",
parameters = {
},
}
But! First, I can't get the line number handling part to work at all. The function within incrementline is never fired.
I also can't quite work out how nested capture information is passed to handling functions (which is why I have scattered Cg, C and Ct semirandomly over the grammar). This means that only one item is returned from within a command_with:
> texlike:match("\\foo{text \\command moretext}")
{
command = "foo",
nodes = {
{
line = 1,
text = "text ",
},
},
parameters = {
},
}
I would also love to be able to check that the environment start and ends match up but when I tried to do so, my back references from "begin" were not in scope by the time I got to "end". I don't know where to go from here.
Late answer but hopefully it'll offer some insight if you're still looking for a solution or wondering what the problem was.
There are a couple of issues with your grammar, some of which can be tricky to spot.
Your line increment here looks incorrect:
local incrementline = Cg( Cb"linenum" ) /
function ( a ) print("NL"); return a + 1 end,
"linenum"
It looks like you meant to create a named capture group and not an anonymous group. The backcapture linenum is essentially being used like a variable. The problem is because this is inside an anonymous capture, linenum will not update properly -- function(a) will always receive 1 when called. You need to move the closing ) to the end so "linenum" is included:
local incrementline = Cg( Cb"linenum" /
function ( a ) print("NL"); return a + 1 end,
"linenum")
Relevant LPeg documentation for Cg capture.
The second problem is with your anything non-terminal rule:
anything = C( (space^1 + (1-lpeg.S("\\{}")) )^1) * Cb("linenum") ...
There are several things to be careful here. First, a named Cg capture (from incrementline rule once it's fixed) doesn't produce anything unless it's in a table or you backref it. The second major thing is that it has an adhoc scope like a variable. More precisely, its scope ends once you close it in an outer capture -- like what you're doing here:
C( (space^1 + (...) )^1)
Which means by the time you reference its backcapture with * Cb("linenum"), that's already too late -- the linenum you really want already closed its scope.
I always found LPeg's re syntax a bit easier to grok so I've rewritten the grammar with that instead:
local grammar_cb =
{
fold = pairfold,
resetlinenum = resetlinenum,
incrementlinenum = incrementlinenum, getlinenum = getlinenum,
error = error
}
local texlike_grammar = re.compile(
[[
document <- '' -> resetlinenum {| docpiece* |} !.
docpiece <- {| envcmd |} / {| cmd |} / multiline
beginslash <- cmdslash 'begin'
endslash <- cmdslash 'end'
envcmd <- beginslash paramblock? {:beginenv: envblock :} (!endslash docpiece)*
endslash openbrace {:endenv: =beginenv :} closebrace / &beginslash {} -> error .
envblock <- openbrace key closebrace
cmd <- cmdslash {:command: identifier :} (paramblock? cmdblock)?
cmdblock <- openbrace {:nodes: {| docpiece* |} :} closebrace
paramblock <- opensq ( {:parameters: {| parampairs |} -> fold :} / whitesp) closesq
parampairs <- parampair (sep parampair)*
parampair <- key assign value
key <- whitesp { identifier }
value <- whitesp { [^],;%s]+ }
multiline <- (nl? text)+
text <- {| {:text: (!cmd !closebrace !%nl [_%w%p%s])+ :} {:line: '' -> getlinenum :} |}
identifier <- [_%w]+
cmdslash <- whitesp '\'
assign <- whitesp '='
sep <- whitesp ','
openbrace <- whitesp '{'
closebrace <- whitesp '}'
opensq <- whitesp '['
closesq <- whitesp ']'
nl <- {%nl+} -> incrementlinenum
whitesp <- (nl / %s)*
]], grammar_cb)
The callback functions are straight-forwardly defined as:
local function pairfold(...)
local t, kv = {}, ...
if #kv % 2 == 1 then return ... end
for i = #kv, 2, -2 do
t[ kv[i - 1] ] = kv[i]
end
return t
end
local incrementlinenum, getlinenum, resetlinenum do
local line = 1
function incrementlinenum(nl)
assert(not nl:match "%S")
line = line + #nl
end
function getlinenum() return line end
function resetlinenum() line = 1 end
end
Testing the grammar with a non-trivial tex-like str with multiple lines:
local test1 = [[\foo{text \bar[color = red, background = black]{
moretext \baz{
even
more text} }
this time skipping multiple
lines even, such wow!}]]
Produces the follow AST in lua-table format:
{
command = "foo",
nodes = {
{
text = "text",
line = 1
},
{
parameters = {
color = "red",
background = "black"
},
command = "bar",
nodes = {
{
text = " moretext",
line = 2
},
{
command = "baz",
nodes = {
{
text = "even ",
line = 3
},
{
text = "more text",
line = 4
}
}
}
}
},
{
text = "this time skipping multiple",
line = 7
},
{
text = "lines even, such wow!",
line = 9
}
}
}
And a second test for begin/end environments:
local test2 = [[\begin[p1
=apple,
p2=blue]{scope} scope foobar
\end{scope} global foobar]]
Which seems to give approximately what you're looking for:
{
{
{
text = " scope foobar",
line = 3
},
parameters = {
p1 = "apple",
p2 = "blue"
},
beginenv = "scope",
endenv = "scope"
},
{
text = " global foobar",
line = 4
}
}
I'm struggling to make work setParseAction when inherit definitions (I don't know how express this in english, so the example):
from __future__ import division
from decimal import Decimal
from pyparsing import Word, alphas, ParseException, Literal, CaselessLiteral \
, Combine, Optional, nums, Or, Forward, ZeroOrMore, StringEnd, alphanums, Suppress \
, sglQuotedString, dblQuotedString, Group \
, restOfLine, Regex, stringEnd
class ASTNode(object):
def __init__(self, tokens):
self.tokens = tokens
self.assignFields()
def __str__(self):
return self.__class__.__name__ + ':' + str(self.__dict__)
__repr__ = __str__
class ConstantNode(ASTNode):
def assignFields(self):
#print " ", self.tokens
self.setValue(self.tokens[0])
def transform(self, value):
return value
def setValue(self, value):
self.constant = self.transform(value)
del self.tokens
class StringNode(ConstantNode):
pass
class BoolNode(ConstantNode):
def transform(self, value):
return bool(value)
class IntNode(ConstantNode):
def transform(self, value):
return int(value)
class FloatNode(ConstantNode):
def transform(self, value):
print value
return Decimal(value)
class AssignmentNode(ASTNode):
def assignFields(self):
#print self.tokens
self.lhs, self.rhs = self.tokens
del self.tokens
LPAR, RPAR, LBRACK, RBRACK, LBRACE, RBRACE, SEMI, COMMA = map(Suppress, "()[]{};,")
PLUS = Literal("+")
MINUS = Literal("-")
MULT = Literal("*")
DIV = Literal("/")
ASSIGN = Literal("=")
POINT = Literal('.')
TRUE = Literal('True')
FALSE = Literal('False')
SEP = Literal(':').suppress()
NAME = Word(alphas + '_?', alphanums + '_?')
TYPE = SEP + NAME
COMMENT = "#" + restOfLine
BOOLEANS = TRUE | FALSE
BOOLEANS.setParseAction(BoolNode)
EXPR = Forward()
ADDOP = PLUS | MINUS
MULTOP = MULT | DIV
PLUSORMINUS = PLUS | MINUS
#Strings
STR = dblQuotedString.setParseAction(ConstantNode) | sglQuotedString.setParseAction(ConstantNode)
STRINGS = STR
#Numbers
NUMBER = Word(nums)
INTEGER = Combine(Optional(PLUSORMINUS) + NUMBER)
FLOATNUMBER = Combine(INTEGER.copy() +
Optional(POINT + Optional(NUMBER)) +
Optional(INTEGER.copy())
)
MONEY = Combine(FLOATNUMBER.copy() + Word("$").suppress())
TYPED_FLOATNUMBER = Combine(FLOATNUMBER + Word(alphas))
INTEGER.setParseAction(IntNode)
FLOATNUMBER.setParseAction(FloatNode)
NUMBERS = MONEY | TYPED_FLOATNUMBER | FLOATNUMBER
TEST_GRAMMAR = """
#Single values
True
False
1 #Int32
1.0 #Float
1$ #MONEY
25.3mt #Typed number"""
Everything parse, but the Boolean and Int node are not called, only the float.
['True']
['False']
1
[FloatNode:{'constant': Decimal('1')}]
1.0
[FloatNode:{'constant': Decimal('1.0')}]
['1']
['25.3mt']
[ConstantNode:{'constant': "'hello world'"}]
[ConstantNode:{'constant': '"hello world"'}]
['2002-08-10']
['100000']
['2002-08-10-100000']
1
[AssignmentNode:{'rhs': FloatNode:{'constant': Decimal('1')}, 'lhs': 'x'}]
1.0
[AssignmentNode:{'rhs': FloatNode:{'constant': Decimal('1.0')}, 'lhs': 'x'}]
[AssignmentNode:{'rhs': '1', 'lhs': 'x'}]
[AssignmentNode:{'rhs': '12.2mt', 'lhs': 'x'}]
I understand that setParseAction is tied to the definition of a partial grammar. However, I find it don't "clear" if chain something like FLOATNUMBER because is based on INTEGER.
You left out the code that does the actual parsing, so I added this code:
for line in TEST_GRAMMAR.splitlines():
if not line or line[0] == '#': continue
print (BOOLEANS^INTEGER^NUMBERS).parseString(line)
Giving this output:
[BoolNode:{'constant': True}]
[BoolNode:{'constant': False}]
[IntNode:{'constant': 1}]
1.0
[FloatNode:{'constant': Decimal('1.0')}]
['1']
['25.3mt']
I also had to fix a minor bug in BoolNode:
class BoolNode(ConstantNode):
def transform(self, value):
return value.lower()=='true' #bool(value)
In transform, value is going to be one of the strings "True" or "False", but the bool value of both of these strings is True - only the empty string "" will return a bool of False.
One of the problems you have is that your definition of FLOAT will also match an INTEGER, so you might consider redefining FLOAT to require a leading, trailing, or embedded decimal point. I got around this by using '^' instead of '|' as the "or" operator. '^' will test all the given alternatives and select the longest match, '|' will short-circuit and select the first match.