Search a section of text, which occurs regularly in tcl - parsing

I have a file called log_file with following text:
....some text....
line wire (1)
mode : 2pair , annex : a
coding : abcd
rate : 1024
status : up
....some text....
line wire (2)
mode : 4pair , annex : b
coding : xyz
rate : 1024
status : down
....some text....
The values may differ but the attributes are always the same. Is there a way to find each line wire and display their attributes? The number of line wires also may differ.
EDIT: File doesn't have any blank lines. There are more attributes but only these are needed. Can I get like the first "n" lines, instead of searching for every line? i.e if there is line wire (1), copy that line plus the next 4 lines.
And I am copying the searched lines to a output file $fout, which I have used earlier in the script with the same $line.

Given your sample:
set fh [open log_file r]
while {[gets $fh line] != -1} {
switch -glob -- $line {
{line wire*} {puts $line}
{mode : *} -
{coding : *} -
{rate : *} -
{status : *} {puts " $line"}
}
}
close $fh
outputs
line wire (1)
mode : 2pair , annex : a
coding : abcd
rate : 1024
status : up
line wire (2)
mode : 4pair , annex : b
coding : xyz
rate : 1024
status : down
Edit: print the next "n" lines following the "line wire" line to a file
set in [open log_file r]
set out [open log_file_filtered w]
set n 4
while {[gets $in line] != -1} {
if {[string match {line wire*} $line]} {
puts $line
for {set i 1} {$i <= $n} {incr i} {
if {[gets $in line] != -1} {
puts $out " $line"
}
}
}
}
close $fh

Related

AWK take some data input from file and set as variable in output

I have some data in file and need to print in output some format to the data in print.
Example content to parse:
012231-33339411.sxz.ree.fg*-*
U2FsdGVkX1+1pfXeR/h4u6P/BrItX75L0wHVIka4yA6tqS9a5CFUWvLu1AB4x2m8NpmJ>fyoXdADqlWDiGWi6Pw1a8NgNDbdTOlMtGBz4FCi8n97UdVQX9f0a2u9d5l7lOCxVDDzd>wJXbi9x4O+Dmo/lm9DbWAjBGKwWu0tTQxsU2TIpqv
FhUZmGd3E6vN+puPXz4yXeVQhMfQ+K8OpSM2ZuTpKCtDgm0SdUDyFnalA4lxHaFZqh+E>3+9JgHK7/KiiZmIJshUmqrwnkX0yKihCcOXCzaFITiByxBM/7PGeJo0IBAjyKI/GflgQ>8GsIWWRkCJnz2OMiYKr8uOMOAfTHnW57Dq+orDG1p
012236-33349111.sxz.ree.fg*-*
bCRIVArOSClIWrZz6KciBFT2iPjqsS/qMRSBYinBzpDmESj8kZHoGQ46BMq+LgHJiY5P>7yygNxCkEv25GKGViKTX1X6KSSLZ+RVNEts4N7jzVLoufZ+X/TAv2Ib7pnnEj7h4rWDn>y7KP1XrTynItaas5z5fpFt2zUHFNElvNmyrjbFZVp
DUsnWWDuvemWUr5YwOLxeRCnwTvfw71gwGEVeBzIJq4TsZb2/G8j9vpb/L7KNybsyQNN>DlOTMW5CHzd5otyYaNBcYo9V/4ky63q2vZMzQDWtCwVPaTKREPUqPLRKea3VkQnnsUic>/iBe+6Sv5GYl+XPGbIjWbTJWLQmc1kv8LXPyvUmTm
cUVypKp9fDlyFUkOkEVAxW8dMxHJ0c83BPw37GkCvsR9itkzO0FpX0Zn+OvRQRkUCyzr>dgijhcH
I need some way to take in Awk the first variable from begin to "-"
Example:
variable1=012231
and
variable1=012236
Variable 2 the 4 digits after the - character
Example:
Variable2=3333
and
variable2=3334
Variable 3 the 2 digits after the 4 digits of variable2
Example:
variable3=94
and
variable3=91
Variable 4 as the text before the newline
Example:
variable4=U2FsdGVkX1+1pfXeR/h4u6P/BrItX75L0wHVIka4yA6tqS9a5CFUWvLu1AB4x2m8NpmJ>fyoXdADqlWDiGWi6Pw1a8NgNDbdTOlMtGBz4FCi8n97UdVQX9f0a2u9d5l7lOCxVDDzd>wJXbi9x4O+Dmo/lm9DbWAjBGKwWu0tTQxsU2TIpqv
FhUZmGd3E6vN+puPXz4yXeVQhMfQ+K8OpSM2ZuTpKCtDgm0SdUDyFnalA4lxHaFZqh+E>3+9JgHK7/KiiZmIJshUmqrwnkX0yKihCcOXCzaFITiByxBM/7PGeJo0IBAjyKI/GflgQ>8GsIWWRkCJnz2OMiYKr8uOMOAfTHnW57Dq+orDG1p
and
variable4=bCRIVArOSClIWrZz6KciBFT2iPjqsS/qMRSBYinBzpDmESj8kZHoGQ46BMq+LgHJiY5P>7yygNxCkEv25GKGViKTX1X6KSSLZ+RVNEts4N7jzVLoufZ+X/TAv2Ib7pnnEj7h4rWDn>y7KP1XrTynItaas5z5fpFt2zUHFNElvNmyrjbFZVp
DUsnWWDuvemWUr5YwOLxeRCnwTvfw71gwGEVeBzIJq4TsZb2/G8j9vpb/L7KNybsyQNN>DlOTMW5CHzd5otyYaNBcYo9V/4ky63q2vZMzQDWtCwVPaTKREPUqPLRKea3VkQnnsUic>/iBe+6Sv5GYl+XPGbIjWbTJWLQmc1kv8LXPyvUmTm
cUVypKp9fDlyFUkOkEVAxW8dMxHJ0c83BPw37GkCvsR9itkzO0FpX0Zn+OvRQRkUCyzr>dgijhcH
Example print expected in output:
'012231' '3333' '94' 'U2FsdGVkX1+1pfXeR/h4u6P/BrItX75L0wHVIka4yA6tqS9a5CFUWvLu1AB4x2m8NpmJ>fyoXdADqlWDiGWi6Pw1a8NgNDbdTOlMtGBz4FCi8n97UdVQX9f0a2u9d5l7lOCxVDDzd>wJXbi9x4O+Dmo/lm9DbWAjBGKwWu0tTQxsU2TIpqv
FhUZmGd3E6vN+puPXz4yXeVQhMfQ+K8OpSM2ZuTpKCtDgm0SdUDyFnalA4lxHaFZqh+E>3+9JgHK7/KiiZmIJshUmqrwnkX0yKihCcOXCzaFITiByxBM/7PGeJo0IBAjyKI/GflgQ>8GsIWWRkCJnz2OMiYKr8uOMOAfTHnW57Dq+orDG1p'
'012236' '3334' '91' 'bCRIVArOSClIWrZz6KciBFT2iPjqsS/qMRSBYinBzpDmESj8kZHoGQ46BMq+LgHJiY5P>7yygNxCkEv25GKGViKTX1X6KSSLZ+RVNEts4N7jzVLoufZ+X/TAv2Ib7pnnEj7h4rWDn>y7KP1XrTynItaas5z5fpFt2zUHFNElvNmyrjbFZVp
DUsnWWDuvemWUr5YwOLxeRCnwTvfw71gwGEVeBzIJq4TsZb2/G8j9vpb/L7KNybsyQNN>DlOTMW5CHzd5otyYaNBcYo9V/4ky63q2vZMzQDWtCwVPaTKREPUqPLRKea3VkQnnsUic>/iBe+6Sv5GYl+XPGbIjWbTJWLQmc1kv8LXPyvUmTm
cUVypKp9fDlyFUkOkEVAxW8dMxHJ0c83BPw37GkCvsR9itkzO0FpX0Zn+OvRQRkUCyzr>dgijhcH'
Haved tested the following code with result of print selecting by number of record and counting the fixed width of the field, without care the format or shape of the content.
awk -v FIELDWIDTHS="6 1 4 2 2 15" 'NR==1{print $1" "$3" "$4}NR==2{print}NR==3{print $1" "$3" "$4}NR==4{print}' file
But it`s a large file with variable lenght of number of records in the large string so the equal will not work for this case I will need catch this string to a variable to print it later in the output as field in all the sequences of show this field.
Could help me with some code to parse the input and print the output as close to the need, please explain how to take the positions in the input.
Thank in advance.
Using any awk in any shell on every Unix box:
$ cat tst.awk
split($0,f,"-") > 1 {
if ( NR > 1 ) {
prt()
delete var
}
var[1] = f[1]
var[2] = substr(f[2],1,4)
var[3] = substr(f[2],5,2)
next
}
{ var[4] = var[4] $0 }
END { prt() }
function prt( i) {
for ( i=1; i<=4; i++ ) {
printf "\047%s\047%s", var[i], (i<4 ? OFS : ORS)
}
}
$ awk -f tst.awk file
'012231' '3333' '94' 'U2FsdGVkX1+1pfXeR/h4u6P/BrItX75L0wHVIka4yA6tqS9a5CFUWvLu1AB4x2m8NpmJ>fyoXdADqlWDiGWi6Pw1a8NgNDbdTOlMtGBz4FCi8n97UdVQX9f0a2u9d5l7lOCxVDDzd>wJXbi9x4O+Dmo/lm9DbWAjBGKwWu0tTQxsU2TIpqvFhUZmGd3E6vN+puPXz4yXeVQhMfQ+K8OpSM2ZuTpKCtDgm0SdUDyFnalA4lxHaFZqh+E>3+9JgHK7/KiiZmIJshUmqrwnkX0yKihCcOXCzaFITiByxBM/7PGeJo0IBAjyKI/GflgQ>8GsIWWRkCJnz2OMiYKr8uOMOAfTHnW57Dq+orDG1p'
'012236' '3334' '91' 'bCRIVArOSClIWrZz6KciBFT2iPjqsS/qMRSBYinBzpDmESj8kZHoGQ46BMq+LgHJiY5P>7yygNxCkEv25GKGViKTX1X6KSSLZ+RVNEts4N7jzVLoufZ+X/TAv2Ib7pnnEj7h4rWDn>y7KP1XrTynItaas5z5fpFt2zUHFNElvNmyrjbFZVpDUsnWWDuvemWUr5YwOLxeRCnwTvfw71gwGEVeBzIJq4TsZb2/G8j9vpb/L7KNybsyQNN>DlOTMW5CHzd5otyYaNBcYo9V/4ky63q2vZMzQDWtCwVPaTKREPUqPLRKea3VkQnnsUic>/iBe+6Sv5GYl+XPGbIjWbTJWLQmc1kv8LXPyvUmTmcUVypKp9fDlyFUkOkEVAxW8dMxHJ0c83BPw37GkCvsR9itkzO0FpX0Zn+OvRQRkUCyzr>dgijhcH'

Jena command line and --strict

I've been going through the code in Jena to figure out what happens when passing the --strict command line. It seems like that there is no difference except for logging of warnings. In CmdLangParse.java the strict mode is set and SysRIOT.setStrictMode(true) is called and under the covers both StrictXSDLexicialForms and strictMode flags are set.
#Override
protected void exec() {
boolean oldStrictValue = SysRIOT.isStrictMode() ;
if ( modLangParse.strictMode() )
SysRIOT.setStrictMode(true) ;
try { exec$() ; }
finally { SysRIOT.setStrictMode(oldStrictValue) ; }
}
Then only in CheckerLiters validateByDatatypeand validateByDatatypeNumericuse the StrictXSDLexicalForms attribute and if it finds a space, \n or \r it just logs it. Below is the snippet from validateByDatatypeNumeric
// Do a white space check as well for numerics.
if ( lexicalForm.contains(" ") ) {
handler.warning("Whitespace in numeric XSD literal: '" + lexicalForm + "'", line, col) ;
return false ;
}
if ( lexicalForm.contains("\n") ) {
handler.warning("Newline in numeric XSD literal: '" + lexicalForm + "'", line, col) ;
return false ;
}
if ( lexicalForm.contains("\r") ) {
handler.warning("Carriage return in numeric XSD literal: '" + lexicalForm + "'", line, col) ;
return false ;
}
Am I missing something or is there no difference between to two modes (strict and non strict)?
--strict is a general flag for Jena command line tools.
For parsing, there is little difference as you note. It switches off features and provides the minimum for the standard. For parsing, there isn't much.

Weird antlr grammar rule

I have found an old file that define antlr grammar rules like that:
rule_name[ ParamType *param ] > [ReturnType *retval]:
<<
$retval = NULL;
OtherType1 *new_var1 = NULL;
OtherType2 *new_var2 = NULL;
>>
subrule1[ param ] > [ $retval ]
| subrule2 > [new_var2]
<<
if( new_var2 == SOMETHING ){
$retval = something_related_to_new_var2;
}
else{
$retval = new_var2;
}
>>
{
somethingelse > [new_var_1]
<<
/* Do something with new_var_1 */
$retval = new_var_1;
>>
}
;
I'm not an Antlr expert and It's the first time that i see this kind of semantic for a rule definition.
Does anybody know where I can find documentation/informations about this?
Even a keyword for a google search is welcome.
Edit:
It should be ANTLR Version 1.33MR33.
Ok, I found! Here is the guide:
http://www.antlr2.org/book/pcctsbk.pdf
I quote the interesting part of the pdf that answer to my question.
1) Page 47:
poly > [float r]
: <<float f;>>
term>[$r] ( "\+" term>[f] <<$r += f;>> )*
;
Rule poly is defined to have a return value called $r via the "> [float r]" notation; this is similar to the output redirection character of UNIX shells. Setting the value of $r sets the return value of poly. he first action after the ":" is an init-action (because it is the first action of a rule or subrule). The init-action defines a local variable called f that will be used in the (...)* loop to hold the return value of the term.
2) Page 85:
A rule looks like:
rule : alternative1
| alternative2
...
| alternativen
;
where each alternative production is composed of a list of elements that can be references to rules, references to tokens, actions, predicates, and subrules. Argument and return value definitions looks like the following where there are n arguments and m return values:
rule[arg1,...,argn] > [retval1,...,retvalm] : ... ;
The syntax for using a rule mirrors its definition:
a : ... rule[arg1,...,argn] > [v1,...,vm] ...
;
Here, the various vi receive the return values from the rule rule, each vi must be an l-value.
3) Page 87:
Actions are of the form <<...>> and contain user-supplied C or C++ code that must be executed during the parse.

Powershell script for data parsing

I am looking for a Powershell script that could help me for this task:
Got data like this:
"No.","time","1-1","1-2","1-3","1-4","1-5","1-6","1-7","1-8","1-9","1-10","1-11","1-12","1-13","1-14","1-15","5-1","5-2","5-3","5-4","5-5","5-6","5-7","5-8","5-9","5-11","5-13","5-15","9-1","9-3","9-5","9-7","9-8","9-9","13-1","13-2","13-3","13-4","13-5","13-6","13-7","13-8","13-9","13-10","17-1","17-2","17-3","17-4","17-5","17-6","17-7","17-8","17-9","E1-1","00:FE:FFX(X2049-1)","00:00:8DX(X2050-1)","00:00:8CX(X2051-1)","00:00:8BX(X2052-1)","00:00:8EX(X2053-1)","00:00:8FX(X2054-1)","00:00:97X(X2055-1)","00:00:96X(X2056-1)","00:00:92X(X2057-1)","00:00:99X(X2058-1)","00:00:98X(X2059-1)","00:00:94X(X2060-1)","00:00:93X(X2061-1)","00:00:90X(X2062-1)","00:00:95X(X2063-1)","00:00:91X(X2064-1)","00:00:9FX(X2065-1)","00:00:9CX(X2066-1)","00:00:A0X(X2067-1)","00:00:A1X(X2068-1)","00:00:9AX(X2069-1)","00:00:9EX(X2070-1)","00:00:A5X(X2071-1)","00:00:A3X(X2072-1)","00:00:A4X(X2073-1)","00:00:9BX(X2074-1)","00:00:A2X(X2075-1)","00:02:D2X(X2076-1)","00:00:A6X(X2077-1)","00:00:A7X(X2078-1)","00:01:0CX(X2079-1)","00:60:48X(X2080-1)","00:00:B2X(X2081-1)","00:02:B4X(X2082-1)","00:02:43X(X2083-1)","00:00:AEX(X2084-1)","00:00:ADX(X2085-1)","00:02:E4X(X2086-1)","00:02:BDX(X2087-1)","00:00:B1X(X2088-1)","00:00:DFX(X2089-1)","00:00:B3X(X2090-1)","00:60:40X(X2091-1)","00:60:41X(X2092-1)","00:00:B5X(X2093-1)","00:00:B7X(X2094-1)","00:00:C3X(X2095-1)","00:60:42X(X2096-1)","00:00:C9X(X2097-1)","00:00:C2X(X2098-1)","00:00:C1X(X2099-1)","00:00:C4X(X2100-1)","00:00:B4X(X2101-1)","00:00:2FX(X2102-1)","00:00:BAX(X2103-1)","00:00:B6X(X2104-1)","00:00:BFX(X2105-1)","00:00:C8X(X2106-1)","00:00:D3X(X2107-1)","00:00:B8X(X2108-1)","00:00:C5X(X2109-1)","00:00:CFX(X2110-1)","00:00:CAX(X2111-1)","00:00:CCX(X2112-1)","00:60:43X(X2113-1)","00:00:D9X(X2114-1)","00:00:BCX(X2115-1)","00:00:A8X(X2116-1)","00:00:C7X(X2117-1)","00:00:D0X(X2118-1)","00:00:BBX(X2119-1)","00:01:3BX(X2120-1)","00:01:3EX(X2121-1)","00:00:BEX(X2122-1)","00:00:BDX(X2123-1)"
"1","2013/11/04 15:45",0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-5,-5,-5,-5,-5,-5,-5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-1,-1,-1,-5,-5,-5,1550,0,1010,58,81,73,197,91,275,286,378,44,58,101,140,41,66,144,107,62,17,36,8,46,76,98,-5,130,217,-5,-5,0,-5,-5,0,0,-5,-5,144,0,5,-5,-5,15,281,2859,-5,1,442,724,13,12,880,97,171,130,30,0,49,15,0,82,12,-5,0,443,0,55,64,1269,-5,-5,41,172
"2","2013/11/04 15:46",0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-5,-5,-5,-5,-5,-5,-5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-1,-1,-1,-5,-5,-5,1710,0,903,57,91,42,172,95,609,281,274,34,126,384,254,39,49,315,90,46,20,197,8,71,61,89,-5,247,220,-5,-5,0,-5,-5,0,0,-5,-5,126,0,12,-5,-5,16,258,3298,-5,4,647,716,1,9,868,101,208,26,30,0,53,17,0,89,9,-5,0,448,0,36,68,1394,-5,-5,39,67
"3","2013/11/04 15:47",0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-5,-5,-5,-5,-5,-5,-5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-1,-1,-1,-5,-5,-5,1548,0,853,55,91,71,193,145,103,269,272,38,77,142,184,39,180,796,85,44,18,517,7,101,64,88,-5,549,138,-5,-5,0,-5,-5,0,0,-5,-5,156,0,3,-5,-5,22,260,2496,-5,18,448,620,15,6,789,194,239,66,96,0,31,13,0,164,8,-5,0,344,0,33,55,1121,-5,-5,72,121
"4","2013/11/04 15:48",0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-5,-5,-5,-5,-5,-5,-5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-1,-1,-1,-5,-5,-5,1558,0,874,34,76,38,201,550,113,288,158,18,64,116,458,42,51,127,90,44,16,50,6,69,66,102,-5,116,294,-5,-5,0,-5,-5,0,0,-5,-5,116,0,1,-5,-5,7,210,3038,-5,5,81,553,5,6,834,53,248,26,88,0,36,17,0,17,9,-5,0,78,0,206,55,1450,-5,-5,45,92
"5","2013/11/04 15:49",0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-5,-5,-5,-5,-5,-5,-5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-1,-1,-1,-5,-5,-5,1620,0,900,39,88,37,229,171,211,311,264,23,104,128,506,42,201,50,98,46,19,62,6,61,59,102,-5,102,306,-5,-5,0,-5,-5,0,0,-5,-5,126,0,3,-5,-5,16,241,3235,-5,11,353,740,8,8,818,68,244,24,111,0,21,14,0,19,10,-5,0,91,0,93,63,1567,-5,-5,50,103
"6","2013/11/04 15:50",0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-5,-5,-5,-5,-5,-5,-5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-1,-1,-1,-5,-5,-5,1745,0,907,44,83,37,189,213,293,265,130,47,68,514,222,42,106,142,92,62,18,338,6,49,79,88,-5,140,231,-5,-5,0,-5,-5,0,0,-5,-5,135,0,5,-5,-5,43,376,3095,-5,1,300,656,1,9,790,91,263,54,103,0,29,14,0,15,11,-5,0,91,0,81,58,1579,-5,-5,57,104
"7","2013/11/04 15:51",0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-5,-5,-5,-5,-5,-5,-5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-1,-1,-1,-5,-5,-5,1786,0,972,45,84,55,195,110,798,324,150,31,191,1406,332,1126,225,60,87,57,70,203,7,45,62,81,-5,112,235,-5,-5,0,-5,-5,0,0,-5,-5,121,0,60,-5,-5,4,354,3378,-5,2,421,629,2,136,737,81,196,128,92,0,21,16,0,18,13,-5,0,71,0,90,55,1184,-5,-5,41,170
"8","2013/11/04 15:52",0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-5,-5,-5,-5,-5,-5,-5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-1,-1,-1,-5,-5,-5,1704,0,928,31,87,38,199,111,286,341,195,24,299,1065,292,329,60,54,87,45,18,54,6,67,72,89,-5,102,204,-5,-5,0,-5,-5,0,0,-5,-5,172,0,22,-5,-5,5,494,3337,-5,9,169,792,6,15,764,159,227,45,92,0,36,16,0,16,11,-5,0,78,0,93,65,1706,-5,-5,61,81
"9","2013/11/04 15:53",0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-5,-5,-5,-5,-5,-5,-5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-1,-1,-1,-5,-5,-5,1494,0,857,28,112,47,188,649,111,318,153,21,87,445,288,34,45,52,87,44,29,94,10,61,74,98,-5,152,129,-5,-5,0,-5,-5,0,0,-5,-5,172,0,1,-5,-5,10,324,3371,-5,1,46,625,3,7,824,54,216,25,85,0,34,17,0,34,12,-5,0,85,0,104,66,1578,-5,-5,32,40
"10","2013/11/04 15:54",0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-5,-5,-5,-5,-5,-5,-5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-1,-1,-1,-5,-5,-5,1565,0,850,80,116,38,217,98,127,370,329,174,96,251,184,37,107,66,380,43,18,92,8,41,65,96,-5,104,231,-5,-5,0,-5,-5,0,0,-5,-5,162,0,2,-5,-5,6,272,3743,-5,11,314,545,7,5,962,66,5,20,28,0,13,15,0,17,11,-5,0,40,0,149,65,1419,-5,-5,31,63
"11","2013/11/04 15:55",0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-5,-5,-5,-5,-5,-5,-5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-1,-1,-1,-5,-5,-5,1650,0,841,55,77,37,168,80,133,291,286,17,64,138,152,43,57,936,97,57,16,112,8,52,72,103,-5,134,407,-5,-5,0,-5,-5,0,0,-5,-5,129,0,5,-5,-5,2,274,3401,-5,3,297,522,2,8,805,96,5,23,23,0,16,14,0,15,12,-5,0,37,0,186,74,1623,-5,-5,14,45
"12","2013/11/04 15:56",0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-5,-5,-5,-5,-5,-5,-5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-1,-1,-1,-5,-5,-5,1471,0,826,42,81,38,162,92,477,284,191,32,68,130,144,45,66,244,100,63,16,146,14,139,102,96,-5,104,302,-5,-5,0,-5,-5,0,0,-5,-5,127,0,10,-5,-5,8,298,3363,-5,2,440,582,3,18,1010,79,8,68,19,0,14,15,0,15,11,-5,0,45,0,129,68,1539,-5,-5,4,93
"13","2013/11/04 15:57",0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-5,-5,-5,-5,-5,-5,-5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-1,-1,-1,-5,-5,-5,1035,0,1002,39,308,36,226,101,104,269,185,24,91,122,137,46,140,59,87,49,18,273,7,156,75,87,-5,113,145,-5,-5,0,-5,-5,0,0,-5,-5,202,0,3,-5,-5,6,214,3794,-5,9,192,500,4,18,1095,161,90,142,84,0,15,15,0,25,17,-5,0,59,0,207,59,1563,-5,-5,29,164
"14","2013/11/04 15:58",0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-5,-5,-5,-5,-5,-5,-5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-1,-1,-1,-5,-5,-5,707,0,968,33,230,37,179,139,138,303,255,21,92,104,161,234,67,55,100,43,18,168,6,145,87,93,-5,126,294,-5,-5,0,-5,-5,0,0,-5,-5,140,0,2,-5,-5,13,305,3448,-5,1,262,648,4,30,928,58,281,51,163,0,19,18,0,40,17,-5,0,155,0,90,50,1631,-5,-5,15,60
"15","2013/11/04 15:59",0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-5,-5,-5,-5,-5,-5,-5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-1,-1,-1,-5,-5,-5,761,0,954,58,103,41,176,79,107,310,109,20,86,142,146,846,51,68,91,50,18,184,6,45,71,96,-5,109,142,-5,-5,0,-5,-5,0,0,-5,-5,254,0,3,-5,-5,6,276,3513,-5,3,171,545,4,4,958,51,91,34,60,0,27,16,0,22,12,-5,0,62,0,91,52,1651,-5,-5,12,51
"No.","time","00:01:3CX(X2124-1)","00:00:C0X(X2125-1)","00:00:C6X(X2126-1)","00:01:04X(X2127-1)","00:01:08X(X2128-1)","00:00:DBX(X2129-1)","00:01:B9X(X2130-1)","00:00:DDX(X2131-1)","00:00:DCX(X2132-1)","00:01:64X(X2133-1)","00:00:E0X(X2134-1)","00:00:E1X(X2135-1)","00:00:E2X(X2136-1)","00:00:E6X(X2137-1)","00:00:E8X(X2138-1)","00:00:E5X(X2139-1)","00:00:E4X(X2140-1)","00:00:E3X(X2141-1)","00:00:E7X(X2142-1)","00:00:E9X(X2143-1)","00:00:CEX(X2144-1)","00:00:D8X(X2145-1)","00:00:AAX(X2146-1)","00:00:EDX(X2147-1)","00:60:3FX(X2148-1)","00:00:F7X(X2149-1)","00:00:31X(X2150-1)","00:00:D6X(X2151-1)","00:00:D7X(X2152-1)","00:00:EEX(X2153-1)","00:00:EFX(X2154-1)","00:60:46X(X2155-1)","00:00:F0X(X2156-1)","00:00:F1X(X2157-1)","00:00:ECX(X2158-1)","00:00:F3X(X2159-1)","00:00:EBX(X2160-1)","00:00:F4X(X2161-1)","00:00:32X(X2162-1)","00:01:86X(X2163-1)","00:00:2BX(X2164-1)","00:02:10X(X2165-1)","00:02:11X(X2166-1)","00:00:2CX(X2167-1)","00:01:0AX(X2168-1)","00:01:0BX(X2169-1)","00:00:A9X(X2170-1)","00:60:02X(X2171-1)","00:60:01X(X2172-1)","00:60:03X(X2173-1)","00:60:04X(X2174-1)","00:60:05X(X2175-1)","00:60:06X(X2176-1)","00:60:07X(X2177-1)","00:60:08X(X2178-1)","00:60:09X(X2179-1)","00:60:0AX(X2180-1)","00:60:00X(X2181-1)","00:60:3EX(X2182-1)","00:01:06X(X2183-1)","00:01:0DX(X2184-1)","00:01:07X(X2185-1)","00:01:05X(X2186-1)","00:02:7BX(X2187-1)","00:02:7CX(X2188-1)","00:02:B5X(X2189-1)","00:02:E5X(X2190-1)","00:02:0FX(X2191-1)","00:01:0EX(X2192-1)","00:01:11X(X2193-1)","00:01:14X(X2194-1)","00:01:10X(X2195-1)","00:01:12X(X2196-1)","00:01:13X(X2197-1)","00:01:09X(X2198-1)","00:00:FBX(X2199-1)","00:00:33X(X2200-1)","00:01:0FX(X2201-1)","00:01:27X(X2202-1)","00:01:15X(X2203-1)","00:01:1DX(X2204-1)","00:01:1BX(X2205-1)","00:01:1AX(X2206-1)","00:01:1CX(X2207-1)","00:02:4CX(X2208-1)","00:01:39X(X2209-1)","00:01:16X(X2210-1)","00:01:38X(X2211-1)","00:02:E7X(X2212-1)","00:01:18X(X2213-1)","00:00:FEX(X2214-1)","00:01:19X(X2215-1)","00:00:FDX(X2216-1)","00:00:FFX(X2217-1)","00:01:29X(X2218-1)","00:01:28X(X2219-1)","00:01:17X(X2220-1)","00:01:2DX(X2221-1)","00:01:2EX(X2222-1)","00:01:2FX(X2223-1)","00:01:2BX(X2224-1)","00:01:2CX(X2225-1)","00:60:0BX(X2226-1)","00:02:07X(X2227-1)","00:60:0FX(X2228-1)","00:60:0CX(X2229-1)","00:60:0DX(X2230-1)","00:01:00X(X2231-1)","00:01:4CX(X2232-1)","00:01:56X(X2233-1)","00:01:61X(X2234-1)","00:01:4EX(X2235-1)","00:01:55X(X2236-1)","00:01:58X(X2237-1)","00:01:59X(X2238-1)","00:01:52X(X2239-1)","00:01:5DX(X2240-1)","00:01:60X(X2241-1)","00:01:4DX(X2242-1)","00:01:5AX(X2243-1)","00:01:54X(X2244-1)","00:01:46X(X2245-1)","00:01:5EX(X2246-1)","00:01:5CX(X2247-1)","00:01:49X(X2248-1)","00:01:4AX(X2249-1)","00:01:50X(X2250-1)","00:01:4BX(X2251-1)"
"1","2013/11/04 15:45",-5,9,62,-5,-5,0,-5,0,0,-5,7,0,0,40,21,55,21,79,24,203,3,0,88,51,-5,0,2,272,15,1967,51,-5,61,58,31,243,24,0,3,-5,0,-5,-5,13,-5,-5,0,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,0,0,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,0,-5,1,0,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5
"2","2013/11/04 15:46",-5,10,47,-5,-5,0,-5,0,0,-5,7,0,0,45,24,68,25,94,24,185,3,0,93,40,-5,0,3,285,116,2195,75,-5,117,70,41,216,27,0,3,-5,0,-5,-5,13,-5,-5,0,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,9,0,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,35,-5,24,0,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5
"3","2013/11/04 15:47",-5,19,111,-5,-5,0,-5,0,0,-5,2,0,0,44,30,62,24,91,32,190,1,0,93,121,-5,0,3,346,283,1534,10,-5,93,29,32,218,14,0,3,-5,0,-5,-5,12,-5,-5,0,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,34,0,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,125,-5,74,0,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5
...
... etc etc ...
Its a CSV file which seems to be split when the number of columns > 130: next columns are added to the file with new lines.
I don't know the number of columns which is dynamic, but I always have 1 Header Line + 15 Results followed by 1 Header Line + 15 Results and so on a certain number of times.
What I'm looking is reserve this split thing and have one correct CSV file that I can add into Splunk later. That means EACH LINE HAVE ONE DISTINCT TIME (And No.) so my new file much have only 16 lines. (1 Header Line + 15 Results, 1 by minute)
So I need to append :
all the (16+1)*n lines (n belongs to 1,EndOfFile) to the 1rst line
without the first 2 columns (No and Time are the same)
all the
(16+2)*n lines to the 2nd line without the first 2 columns (No and
Time are the same)
all the (16+3)*n lines to the 3nd line without the
first 2 columns (No and Time are the same)
etc etc...
If someone can help me with this script that would be awesome !
EDIT: Here's where I am but no success:
Import-Csv .\data.txt |Group-Object -Property No.,time |% {
$text = $_.name+","
$text += ($_.group | % {$i=0;$j=$_.count}{$i++ ; ($_|%{$_.toString() + ","})*($j-$i -gt 0)})
$text += "`n"
Write-Output $text
}
EDIT 2: My problem is that I got an hashtable but I don't know any Names to get all the elements. I tried with getEnumerator() without success :
Method invocation failed because [System.Management.Automation.PSCustomObject] doesn't contain a method named 'getEnumerator'.
Import-Csv .\data.txt |Group-Object -Property No.,time |% {
$text = $_.name+","
$text += ($_.group | % {$i=0;$j=$_.count}{$i++ ; ($_.GetEnumerator()|%{$_ + ","})*($j-$i -gt 0)})
$text += "`n"
Write-Output $text
}
If I put a column name like "1-1" instead of getEnumerator() its working but I can't do that for all columns since I don't know the names.
I used get-content and array treatment its a lot more easier than trying to get these ** cmd-let working...
$ofs = ',' # ! Variable Interne utilisée pour le cast de [string] venant d'un [array], définit le séparateur
#Lecture des fichiers en entrée
$txt = gc .\data.txt
if($txt -is [system.array]){
#Declaration des variables
$res = #()
$count = 0
#Traitement
foreach ($line in $txt) { #On traite toutes les lignes du fichier
$count++ #On incremente le compteur de lignes
If($count-le 16) { #Les 16 premieres lignes [1 Header + 15 Datas row]
$res += $line+"," #sont copiées telles quelles
} else { #Pour toutes les lignes suivantes [17,+inf]
$newline = $line.Split(',')[2..500] #On supprime leur deux premières colonnes
$res[($count%16)-1] += [string] $newline #On les ajoute aux 16 premières lignes avec Mod[16]
}
}
#Ecriture des fichiers de sortie
$res
}

Build CSV from parse files python

I am building a small database (for personal use), from over a 1000 files. I am looking for specific word, but the issue that I have if the word is not contained in the file how can i write a NoData line, what I would like to have is:
Africa Botswana test 51.1922546 -113.9366341
Africa Kenya Skydive Kenya -13.788388 33.78498
Africa Malawi Skydive Malawi NoData NoData
Africa Mauritius SkyDive Austral 30.5000854 -8.824510574
Africa Morocco Beni Mellal NoData NoData
for i in os.listdir(Main_Path):
if "-" in i:
for filename in os.listdir(Main_Path+i):
if ".dat" in filename and os.path.isdir(Main_Path+i):
f_split = filename.split("-")
if len(f_split) == 4:
continent.append(f_split[0])
country.append(f_split[1])
state.append(f_split[2].split(".")[0])
else:
continent.append(f_split[0])
country.append("")
state.append(f_split[1].split(".")[0])
d = open(Main_Path+i+"/" + filename, "r")
files = d.readlines()
d.close()
for k, line in enumerate(files):
if "Dropzone.com :" in line:
dzname.append(line.split(":")[1].strip())
elif 'id="lat"' in line:
lat.append(line.split("=")[3].split('"')[1].strip())
myFile = open(Main_Path+"MYFILE.csv", "wb")
wtr= csv.writer( myFile )
for a,b,c,d,e in zip(continent,country,state,dzname,lat):
wtr.writerow([a,b,c,d,e])
myFile.close()
I am stack "elif 'id="lat"' in line:" because it adds to the list "lat" only the files which contains id = lat. I do understand why but I would like the parser to return and add to the list an NoData
sorry i wrote the question from another comp.
Do you mean something like this?
That is: if no line in files contains id="lat" it will append "No Data" to lat.
snip...
d = open(Main_Path+i+"/" + filename, "r")
files = d.readlines()
d.close()
found_latitude = False
for k, line in enumerate(files):
if "Dropzone.com :" in line:
dzname.append(line.split(":")[1].strip())
elif 'id="lat"' in line:
found_latitude = True
lat.append(line.split("=")[3].split('"')[1].strip())
if not found_latitude:
lat.append("No Data")
snip...

Resources