I have the following text structure. The values below JTT JNX JNA JNO belong to previous line.
9 8 11 56507785 93
JTT JNX JNA JNO
76 98
9 8 60 3269557 58
9 8 53 7269558 150
JTT JNX JNA JNO
132 71 45-7705678
9 8 62 439559 82
I'd like to parse it in order to print the corresponding values in a single line like below:
H1 H2 H3 H4 H5 JTT JNX JNA JNO
9 8 11 56507785 93 76 98
9 8 60 3269557 58
9 8 53 7269558 150 132 71 45-7705678
9 8 62 439559 82
My issue is when I use awk with FS = space (default FS) then it takes JTT as first field and JTT has 9 spaces before, so I think should be use some technique that counts how may spaces are from left until JTT JNX JNA JNO and count number of spaces from beginning until the values below JTT JNX JNA JNO in order to positionate correctly each value. I mean, 76 and 132 below JTT header, 971 below JNX, 98 below JNA and 45-7705678 below JNO.
How can this be done in awk?
$ awk --version
GNU Awk 5.0.0, API: 2.0 (GNU MPFR 4.0.2, GNU MP 6.1.2)
Copyright (C) 1989, 1991-2019 Free Software Foundation.
$ uname -srv
CYGWIN_NT-6.1 3.0.7(0.338/5/3) 2019-04-30 18:08
Thanks in advance.
With GNU awk (which you have) for FIELDWIDTHS:
$ cat tst.awk
BEGIN {
OFS = ","
print "H1", "H2", "H3", "H4", "H5", "JTT", "JNX", "JNA", "JNO"
}
!NF || ($1 == "JTT") { next }
!/^ / {
if (NR>1) {
print rec
}
FS = " "
$0 = $0
$1 = $1
rec = $0
}
/^ / {
FIELDWIDTHS = "12 5 5 *"
$0 = $0
$1 = $1
for (i=1; i<=NF; i++) {
gsub(/^\s+|\s+$/,"",$i)
}
rec = rec OFS $0
}
END {
print rec
}
.
$ awk -f tst.awk file
H1,H2,H3,H4,H5,JTT,JNX,JNA,JNO
9,8,11,56507785,93,76,,98
9,8,60,3269557,58
9,8,53,7269558,150,132,71,,45-7705678
9,8,62,439559,82
$ awk -f tst.awk file | column -s, -t
H1 H2 H3 H4 H5 JTT JNX JNA JNO
9 8 11 56507785 93 76 98
9 8 60 3269557 58
9 8 53 7269558 150 132 71 45-7705678
9 8 62 439559 82
Replace OFS="," with OFS="\t" or otherwise massage to suit...
Related
Define a function to implement Fibonacci Sequence: 1, 1, 2, 3, 5, 8, 13, 21, 34. Please use the function output first 20 figures of Fibonacci Sequence.
Here is a python implementation
def fib(n):
a, b = 0, 1
while a < n:
print(a, end=' ')
a, b = b, a+b
print()
fib(5000)
Output
0 1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 1597 2584 4181
A recursive implementation
memo = [-1] * 21
memo[0] = 0
memo[1] = 1
print(memo[0], end=' ')
print(memo[1], end=' ')
def fibrec(n):
if(memo[n] == -1):
memo[n] = fibrec(n-2) + fibrec(n-1)
print(memo[n], end=' ')
return memo[n]
fibrec(20)
Output
0 1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 1597 2584 4181 6765
I'm relatively new to lua and programming in general (self taught), so please be gentle!
Anyway, I wrote a lua script to read a UDP message from a game. The structure of the message is:
DATAxXXXXaaaaBBBBccccDDDDeeeeFFFFggggHHHH
DATAx = 4 letter ID and x = control character
XXXX = integer shows the group of the data (groups are known)
aaaa...HHHHH = 8 single-precision floating point numbers
The last ones is those numbers I need to decode.
If I print the message as received, it's something like:
DATA*{V???A?A?...etc.
Using string.byte(), I'm getting a stream of bytes like this (I have "formatted" the bytes to reflect the structure above.
68 65 84 65/42/20 0 0 0/237 222 28 66/189 59 182 65/107 42 41 65/33 173 79 63/0 0 128 63/146 41 41 65/0 0 30 66/0 0 184 65
The first 5 bytes are of course the DATA*. The next 4 are the 20th group of data. The next bytes, the ones I need to decode, and are equal to those values:
237 222 28 66 = 39.218
189 59 182 65 = 22.779
107 42 41 65 = 10.573
33 173 79 63 = 0.8114
0 0 128 63 = 1.0000
146 41 41 65 = 10.573
0 0 30 66 = 39.500
0 0 184 65 = 23.000
I've found C# code that does the decode with BitConverter.ToSingle(), but I haven't found any like this for Lua.
Any idea?
What Lua version do you have?
This code works in Lua 5.3
local str = "DATA*\20\0\0\0\237\222\28\66\189\59\182\65..."
-- Read two float values starting from position 10 in the string
print(string.unpack("<ff", str, 10)) --> 39.217700958252 22.779169082642 18
-- 18 (third returned value) is the next position in the string
For Lua 5.1 you have to write special function (or steal it from François Perrad's git repo )
local function binary_to_float(str, pos)
local b1, b2, b3, b4 = str:byte(pos, pos+3)
local sign = b4 > 0x7F and -1 or 1
local expo = (b4 % 0x80) * 2 + math.floor(b3 / 0x80)
local mant = ((b3 % 0x80) * 0x100 + b2) * 0x100 + b1
local n
if mant + expo == 0 then
n = sign * 0.0
elseif expo == 0xFF then
n = (mant == 0 and sign or 0) / 0
else
n = sign * (1 + mant / 0x800000) * 2.0^(expo - 0x7F)
end
return n
end
local str = "DATA*\20\0\0\0\237\222\28\66\189\59\182\65..."
print(binary_to_float(str, 10)) --> 39.217700958252
print(binary_to_float(str, 14)) --> 22.779169082642
It’s little-endian byte-order of IEEE-754 single-precision binary:
E.g., 0 0 128 63 is:
00111111 10000000 00000000 00000000
(63) (128) (0) (0)
Why that equals 1 requires that you understand the very basics of IEEE-754 representation, namely its use of an exponent and mantissa. See here to start.
See #Egor‘s answer above for how to use string.unpack() in Lua 5.3 and one possible implementation you could use in earlier versions.
is it possible via ctags to get the function end line number as well
"ctags -x --c-kinds=f filename.c"
Above command lists the function definition start line-numbers. Wanted a way to get the function end line numbers.
Other Approaches:
awk 'NR > first && /^}$/ { print NR; exit }' first=$FIRST_LINE filename.c
This needs the code to be properly formatted
Example:
filename.c
1 #include<stdio.h>
2 #include<stdlib.h>
3 int main()
4 {
5 const char *name;
6
7 int a=0
8 printf("name");
9 printf("sssss: %s",name);
10
11 return 0;
12 }
13
14 void code()
15 {
16 printf("Code \n");
17 }
18
19 int code2()
20 {
21 printf("code2 \n");
22 return 1
23 }
24
Input: filename and the function start line no.
Example:
Input: filename.c 3
Output: 12
Input : filename.c 19
Output : 23
Is there any better/simple way of doing this ?
C/C++ parser of Universal-ctags(https://ctags.io) has end: field.
jet#localhost tmp]$ cat -n foo.c
1 int
2 main( void )
3 {
4
5 }
6
7 int
8 bar (void)
9 {
10
11 }
12
13 struct x {
14 int y;
15 };
16
[jet#localhost tmp]$ ~/var/ctags/ctags --fields=+ne -o - --sort=no foo.c
main foo.c /^main( void )$/;" f line:2 typeref:typename:int end:5
bar foo.c /^bar (void)$/;" f line:8 typeref:typename:int end:11
x foo.c /^struct x {$/;" s line:13 file: end:15
y foo.c /^ int y;$/;" m line:14 struct:x typeref:typename:int file:
awk to the rescue!
doesn't handle curly braces within comments but should handle blocks within functions, please give it a try...
$ awk -v s=3 'NR>=s && /{/ {c++}
NR>=s && /}/ && c && !--c {print NR; exit}' file
finds the matching brace for the first one after the specified start line number s.
I have been working on this program for hours and cannot find out how to make the numbers loop around after they hit saturday. They either go way passed it to the right or if i add and endl; they go up and down.
// This is how my output looks like (except they curve around they just go forever to the right:
Number of days: 31
Offset: 0
Su Mo Tu We Th Fr Sa
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
This what you mean?
#include <iostream>
using namespace std;
int main()
{
int i;
for (i=1; i<=31; i++) {
cout << ((i<10) ? " " : "") << i << " ";
if (i%7==0) cout << endl;
}
return 0;
}
Outputs:
1 2 3 4 5 6 7
8 9 10 11 12 13 14
15 16 17 18 19 20 21
22 23 24 25 26 27 28
29 30 31
The % sign is the modulus operator. It gives the remainder of division. So every 7th day divided by 7 is going to have a remainder of zero. That's how you check where to put the line breaks.
I tried to create a neural network to estimate y = x ^ 2. So I created a fitting neural network and gave it some samples for input and output. I tried to build this network in C++. But the result is different than I expected.
With the following inputs:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49
50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 -1
-2 -3 -4 -5 -6 -7 -8 -9 -10 -11 -12 -13 -14 -15 -16 -17 -18 -19 -20 -21 -22 -23 -24 -25 -26 -27 -28 -29 -30 -31 -32 -33 -34 -35 -36 -37 -38 -39 -40 -41 -42 -43 -44 -45 -46 -47 -48 -49 -50 -51 -52 -53 -54 -55 -56 -57 -58 -59 -60 -61 -62 -63 -64 -65 -66 -67 -68 -69 -70 -71
and the following outputs:
0 1 4 9 16 25 36 49 64 81 100 121 144 169 196 225 256 289 324 361 400
441 484 529 576 625 676 729 784 841 900 961 1024 1089 1156 1225 1296
1369 1444 1521 1600 1681 1764 1849 1936 2025 2116 2209 2304 2401 2500
2601 2704 2809 2916 3025 3136 3249 3364 3481 3600 3721 3844 3969 4096
4225 4356 4489 4624 4761 4900 5041 1 4 9 16 25 36 49 64 81 100 121 144
169 196 225 256 289 324 361 400 441 484 529 576 625 676 729 784 841
900 961 1024 1089 1156 1225 1296 1369 1444 1521 1600 1681 1764 1849
1936 2025 2116 2209 2304 2401 2500 2601 2704 2809 2916 3025 3136 3249
3364 3481 3600 3721 3844 3969 4096 4225 4356 4489 4624 4761 4900 5041
I used fitting tool network. with matrix rows. Training is 70%, validation is 15% and testing is 15% as well. The number of hidden neurons is two. Then in command lines I wrote this:
purelin(net.LW{2}*tansig(net.IW{1}*inputTest+net.b{1})+net.b{2})
Other information :
My net.b[1] is: -1.16610230053776 1.16667147712026
My net.b[2] is: 51.3266249426358
And net.IW(1) is: 0.344272596370387 0.344111217766824
net.LW(2) is: 31.7635369693519 -31.8082184881063
When my inputTest is 3, the result of this command is 16, while it should be about 9. Have I made an error somewhere?
I found the Stack Overflow post Neural network in MATLAB that contains a problem like my problem, but there is a little difference, and the differences is in that problem the ranges of input and output are same, but in my problem is no. That solution says I need to scale out the results, but how can I scale out my result?
You are right about scaling. As was mentioned in the linked answer, the neural network by default scales the input and output to the range [-1,1]. This can be seen in the network processing functions configuration:
>> net = fitnet(2);
>> net.inputs{1}.processFcns
ans =
'removeconstantrows' 'mapminmax'
>> net.outputs{2}.processFcns
ans =
'removeconstantrows' 'mapminmax'
The second preprocessing function applied to both input/output is mapminmax with the following parameters:
>> net.inputs{1}.processParams{2}
ans =
ymin: -1
ymax: 1
>> net.outputs{2}.processParams{2}
ans =
ymin: -1
ymax: 1
to map both into the range [-1,1] (prior to training).
This means that the trained network expects input values in this range, and outputs values also in the same range. If you want to manually feed input to the network, and compute the output yourself, you have to scale the data at input, and reverse the mapping at the output.
One last thing to remember is that each time you train the ANN, you will get different weights. If you want reproducible results, you need to fix the state of the random number generator (initialize it with the same seed each time). Read the documentation on functions like rng and RandStream.
You also have to pay attention that if you are dividing the data into training/testing/validation sets, you must use the same split each time (probably also affected by the randomness aspect I mentioned).
Here is an example to illustrate the idea (adapted from another post of mine):
%%# data
x = linspace(-71,71,200); %# 1D input
y_model = x.^2; %# model
y = y_model + 10*randn(size(x)).*x; %# add some noise
%%# create ANN, train, simulate
net = fitnet(2); %# one hidden layer with 2 nodes
net.divideFcn = 'dividerand';
net.trainParam.epochs = 50;
net = train(net,x,y);
y_hat = net(x);
%%# plot
plot(x, y, 'b.'), hold on
plot(x, x.^2, 'Color','g', 'LineWidth',2)
plot(x, y_hat, 'Color','r', 'LineWidth',2)
legend({'data (noisy)','model (x^2)','fitted'})
hold off, grid on
%%# manually simulate network
%# map input to [-1,1] range
[~,inMap] = mapminmax(x, -1, 1);
in = mapminmax('apply', x, inMap);
%# propagate values to get output (scaled to [-1,1])
hid = tansig( bsxfun(#plus, net.IW{1}*in, net.b{1}) ); %# hidden layer
outLayerOut = purelin( net.LW{2}*hid + net.b{2} ); %# output layer
%# reverse mapping from [-1,1] to original data scale
[~,outMap] = mapminmax(y, -1, 1);
out = mapminmax('reverse', outLayerOut, outMap);
%# compare against MATLAB output
max( abs(out - y_hat) ) %# this should be zero (or in the order of `eps`)
I opted to use the mapminmax function, but you could have done that manually as well. The formula is a pretty simply linear mapping:
y = (ymax-ymin)*(x-xmin)/(xmax-xmin) + ymin;