Debugging APL code: how to use `#`(index) and `⊢` (right tack) together? - parsing

I am attempting to read Aaron Hsu's thesis on A data parallel compiler hosted on the GPU, where I have landed at some APL code I am unable to fix. I've attached both a screenshot of the offending page (page number 74 as per the thesis numbering on the bottom):
The transcribed code is as follows:
d ← 0 1 2 3 1 2 3 3 4 1 2 3 4 5 6 5 5 6 3 4 5 6 5 5 6 3 4
This makes sense: create an array named d.
⍳≢d
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27
This too makes sense. Count the number of elements in d and create a sequence of
that length.
⍉↑d,¨⍳≢d
0 1 2 3 1 2 3 3 4 1 2 3 4 5 6 5 5 6 3 4 5 6 5 5 6 3 4
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27
This is slightly challenging, but let me break it down:
zip the sequence ⍳≢d = 1..27 with the d array using the ,¨ idiom, which zips the two arrays using a catenation.
Then, split into two rows using ↑ and transpose to get columns using ⍉
Now the biggie:
(⍳≢d)#(d,¨⍳≢d)⊢7 27⍴' '
INDEX ERROR
(⍳≢d)#(d,¨⍳≢d)⊢7 27⍴' '
Attempting to break it down:
⍳≢d counts number of elements in d
(d,¨⍳≢d) creates an array of pairs (d, index of d)
7 27⍴' ' creates a 7 x 27 grid: presumably 7 because that's the max value of d + 1, for indexing reasons.
Now I'm flummoxed about how the use of ⊢ works: as far as I know, it just ignores everything to the left! So I'm missing something about the parsing of this expression.
I presume it is parsed as:
(⍳≢d)#((d,¨⍳≢d)⊢(7 27⍴' '))
which according to me should be evaluated as:
(⍳≢d)#((d,¨⍳≢d)⊢(7 27⍴' '))
= (⍳≢d)#((7 27⍴' ')) [using a⊢b = b]
= not the right thing
As I was writing this down, I managed to fix the bug by sheer luck: if we increment d to be d + 1 so we are 1-indexed, the bug no longer manifests:
d ← d + 1
d
1 2 3 4 2 3 4 4 5 2 3 4 5 6 7 6 6 7 4 5 6 7 6 6 7 4 5
then:
(⍳≢d)#(d,¨⍳≢d)⊢7 27⍴' '
1
2 5 10
3 6 11
4 7 8 12 19 26
9 13 20 27
14 16 17 21 23 24
15 18 22 25
However, I still don't understand how this works! I presume the context will be useful
for others attempting to leave the thesis, so I'm going to leave the rest of it up.
Please explain what (⍳≢d)#(d,¨⍳≢d)⊢7 27⍴' ' does!
I've attached the raw screenshot to make sure I didn't miss something:

I'm happy to see that you found the the off-by-one error. It stems from Aaron Hsu working with index origin 0. If you set ⎕IO←0 then his code will work.
Some dyadic operators can take an array operand, giving the sequence OPERATOR operand argument, e.g. in -#(1 2 3)(4 5 6 7). This poses a problem because both the operand and the argument are arrays, and juxtaposition of arrays forms a new array with those arrays as elements by a process known as stranding. Compare:
(1 2 3)(4 5 6 7)
┌─────┬───┐
│1 2 3│4 5│
└─────┴───┘
However, in the case of the operator with its array operand, we want to "break" this strand so the left part can act as operand while the right part acts as argument. One way to break the stranding up is by applying a function to the argument, giving the sequence OPERATOR operand Function argument. Now, we don't actually need any transformation of the argument, so an identity function will do: -#(1 2 3)⊢(4 5 6 7).
As for what (⍳≢d)#(d,¨⍳≢d)⊢7 27⍴' ' actually does:
7 27⍴' ' creates a blank matrix.
(⍳≢d) are indices to insert into specified slots in the matrix.
#(d,¨⍳≢d) indicates at which locations in the matrix the above should replace the existing values
⊢ serves solely to separate (d,¨⍳≢d) from 7 27⍴' '. The code could also have been written as ((⍳≢d)#(d,¨⍳≢d))7 27⍴' ' with parentheses serving to "bind" the operand to the operator.

Related

Intercalate columns when they are in pairs

Using this table:
A
B
C
D
1
2
3
4
5
6
7
8
9
10
11
12
In Google Sheets if I do this here in column E:
={A1:B3;C1:D3}
Teremos:
E
F
1
2
5
6
9
10
3
4
7
8
11
12
But the result I want is this:
E
F
1
2
3
4
5
6
7
8
9
10
11
12
I tried multiple options with FLATTEN, but none of them returned what I wanted.
Well you can try:
=WRAPROWS(TOCOL(A1:D3),2)
You could try with MAKEARRAY
=MAKEARRAY(ROWS(A1:D3)*2,2,LAMBDA(r,c,INDEX(FLATTEN(A1:D3),c+(r-1)*2)))
GENERAL ANSWER
For you or anyone else: to do something similar but with a variable number of columns of origin or of destination, you can use this formula. Changing the range and amount of columns at the end of LAMBDA:
=LAMBDA(range,cols,MAKEARRAY(ROWS(range)*ROUNDUP(COLUMNS(range)/cols),cols,LAMBDA(r,c,IFERROR(INDEX(FLATTEN(range),c+(r-1)*cols)))))(A1:D3,2)
you can do:
={FLATTEN({A1:A3, C1:C3}), FLATTEN({B1:B3, D1:D3})}
for more columns, it could be automated with MOD

How To Skip Down by 1 Row/Cell The Formula Output and Remove The Last Sequential Output Before 1's Google Sheets?

I've got these 3 groups of data in range F2:G22 as below
(3 groups as minimal example, in reality many thousands of groups, and recurrent similar datasets expected in the future):
I need to number each group's rows sequentially, starting over at 1 at each new group.
The expected result would be like in range E1:E22.
I tried the following formula n cell C2 , then in cell D3:
=INDEX(IF(A2:A22="",COUNTIFS(B2:B22&A2:A22, B2:B22&A2:A22, ROW(B2:B22), "<="&ROW(B2:B22)),1))
In C2:
In D3:
That fixed partially the sequence issue, but there's still 2 issues I can't find remedy for.
1st remaining issue:
I'd prefer not having to manually do the C2 to D3 step each time I get new similar data (but would accomodate if there's no simple solution to this issue).
Is there a simple way to modify the formula to make it output the correct sequencing from C2 ?
2nd remaining issue:
At rows 7, 14 and 23 there still remain unecessary ending numbering for these intermediary rows in D7 , D14 , and D23:
I could only think of an extra manual step of filtering out the non-blank rows in Column A to fix this 2nd issue (i.e. Highlighting Column A > Data tab > Create Filter > Untick all > Tick Blanks > Copy All > Paste In new Sheet).
But would there be a way to do it in the same formula? I'm not seeing the way to add the proper filter or using another method in the formula.
Any help is greatly appreciated.
EDIT (Sorry for Forgotten Sample):
Formula Input A
Formula Input B
Formula Output 1
Formula Output 2
EXPECTED RESULT
rockinfreakshow
ztiaa
DATA
DATA BY GROUP
7
1
1
7
7
2
1
1
1
2
Element-1
Group-1
7
3
2
2
2
3
Element-2
Group-1
7
4
3
3
3
4
Element-3
Group-1
7
5
4
4
4
5
Element-4
Group-1
8
1
5
6
8
8
2
1
1
1
7
Element-1
Group-2
8
3
2
2
2
8
Element-2
Group-2
8
4
3
3
3
9
Element-3
Group-2
8
5
4
4
4
10
Element-4
Group-2
8
6
5
5
5
11
Element-5
Group-2
8
7
6
6
6
12
Element-6
Group-2
9
1
7
13
9
9
2
1
1
1
14
Element-1
Group-3
9
3
2
2
2
15
Element-2
Group-3
9
4
3
3
3
16
Element-3
Group-3
9
5
4
4
4
17
Element-4
Group-3
9
6
5
5
5
18
Element-5
Group-3
9
7
6
6
6
19
Element-6
Group-3
9
8
7
7
7
20
Element-7
Group-3
9
9
8
8
8
21
Element-8
Group-3
9
Can you try:
=INDEX(LAMBDA(y,z,
IF(LEN(z),COUNTIFS(y,y,ROW(z),"<="&ROW(z)),))
(LOOKUP(ROW(G2:G),FILTER(ROW(G2:G),BYROW(G2:G,LAMBDA(z,IF(z<>OFFSET(z,-1,0),row(z),0))))),G2:G))
You can simply use SCAN.
=SCAN(,G2:G,LAMBDA(a,c,IF(c="",,a+1)))
Sample sheet

YouTube Data API returning inconsistent results with duplicates

There have been numerous questions about inconsistent results from the YouTube Data API: 1, 2, 3, 4, 5, 6. Most of them have accepted answers that seem to indicate there was a problem with the API request that was fixed by the instructions in the answers. But none of those situations apply to the API request discussed here.
There have also been two questions about duplicates in the API results: 1, 2. Both of them have the same answer, which says to use the next-page token. But both questions say the token was used, so that answer is not helpful.
Yesterday, I submitted a series of API requests to get the list of most-viewed videos about 3D printing. The first request in the series was:
https://www.googleapis.com/youtube/v3/search?q=3D print&type=video&maxResults=50&part=id,snippet&order=viewCount&key=<my key>
I ran that in a VBA sub, which took the next-page token from each result and resubmitted the URL with &pageToken=<nextPageToken> inserted.
The result was a list of 649 unique video IDs. So far so good.
After making some changes in the VBA code and seeing some duplicates in the result set, I went back today and ran the original VBA sub again. The result was again a list of 649 video IDs, but this time the list included duplicates and it also included IDs that were not in yesterday's list and was missing IDs that were there yesterday. Here is a comparison from the first two pages and the last two pages of the two result sets:
Page
# on page
# overall
Run 1
Run 2
Same as
Seq
Dup
1
1
1
f2mdMcf-fJs
f2mdMcf-fJs
1
1
2
2
WSauz5KVKTU
WSauz5KVKTU
2
Seq
1
3
3
zsSCUWs7k9Q
XYIUM5TkhMo
None
1
4
4
B5Q1J5c8oNc
zsSCUWs7k9Q
3
Seq
1
5
5
cUxIb3Pt-hQ
B5Q1J5c8oNc
4
Seq
1
6
6
4yyOOn7pWnA
LDjE28szwr8
None
1
7
7
3N46jQ0Xi3c
cUxIb3Pt-hQ
5
Seq
1
8
8
08dBVz8_VzU
4yyOOn7pWnA
6
Seq
...
1
13
13
oeKIe1ik2O8
e1rQ8YwNSDs
11
Seq
1
14
14
FrG_eSECfps
RVB2JreIcoc
12
Seq
1
15
15
pPQCwz2q96o
oeKIe1ik2O8
13
Seq
1
16
16
uo3KuoEiu3I
pPQCwz2q96o
15
NOT
1
17
17
0U6aIwd5h9s
uo3KuoEiu3I
16
Seq
...
1
47
47
ShGsW68zbIo
iu9rhqsvrPs
46
Seq
1
48
48
0q0xS7W78KQ
ShGsW68zbIo
47
Seq
1
49
49
UheJQsXOAnY
0q0xS7W78KQ
48
Seq
Dup
1
50
50
H8AcqOh0wis
H8AcqOh0wis
50
NOT
Dup
2
1
51
EWq3-2VuqbQ
0q0xS7W78KQ
48
NOT
Dup
2
2
52
scuTZza4f_o
H8AcqOh0wis
50
NOT
Dup
2
3
53
bJWJW-mz4_U
UheJQsXOAnY
49
NOT
2
4
54
Ii4VYsh9OlM
EWq3-2VuqbQ
51
NOT
2
5
55
r2-OGUu57pU
scuTZza4f_o
52
Seq
2
6
56
8KTnu18Mi9Q
bJWJW-mz4_U
53
Seq
2
7
57
DconsfGsXyA
Ii4VYsh9OlM
54
Seq
2
8
58
UttEvLJP3l8
8KTnu18Mi9Q
56
NOT
2
9
59
GJOOLH9ZP2I
DconsfGsXyA
57
Seq
2
10
60
ewgmg9Q5Ab8
UttEvLJP3l8
58
Seq
...
13
35
635
qHpR_p8lA4I
FFVOzo7tSV8
639
Seq
13
36
636
DplwDDZNTRc
76IBjdM9s6g
640
Seq
13
37
637
3AObqGsimr8
qEh0uZuu7_U
None
13
38
638
88keQ4PWH18
RhfGJduOlrw
641
Seq
13
39
639
FFVOzo7tSV8
QxzH9QkirCU
643
NOT
13
40
640
76IBjdM9s6g
Qsgz4GbL8O4
None
13
41
641
RhfGJduOlrw
BSgg7mEzfqY
644
Seq
13
42
642
lVEqwV0Nlzg
VcmjbJ2q8-w
645
Seq
13
43
643
QxzH9QkirCU
gOU0BCL-TXs
None
13
44
644
BSgg7mEzfqY
IoOXQUcW24s
646
Seq
13
45
645
VcmjbJ2q8-w
o4_2_a6LzFU
647
Seq
Dup
14
1
646
IoOXQUcW24s
o4_2_a6LzFU
647
NOT
Dup
14
2
647
o4_2_a6LzFU
ijVPcGaqVjc
648
Seq
14
3
648
ijVPcGaqVjc
nk3FlgEuG-s
649
Seq
14
4
649
nk3FlgEuG-s
27ZLFn8Dejg
None
The last three columns have the following meanings:
Same as: If an ID from Run 2 is the same as an ID from Run 1, then this column has the # overall for Run 1.
Seq: Indicates whether the number in column "Same as" is one more than the previous number in that column.
Dup: Indicates whether an ID from Run 2 occurred more than once in that run.
Problems:
The videos XYIUM5TkhMo, LDjE28szwr8, qEh0uZuu7_U, Qsgz4GbL8O4, gOU0BCL-TXs, and 27ZLFn8Dejg were returned as #3, 6, 637, 640, 643, and 649 in Run 2, but were not returned at all in Run 1.
The videos FrG_eSECfps, r2-OGUu57pU, lVEqwV0Nlzg were returned as #14, 55, 642, in Run 1, but were not in Run 2.
The videos 0q0xS7W78KQ, H8AcqOh0wis, and o4_2_a6LzFU were returned as #49, 50, and 645 in Run 2, but then each appears a second time in that run (as well as appearing in Run 1 as #48, 50, and 647).
These results are troubling. They mean that no single search will return a reliable list of videos for a given value of q.
I mentioned at the beginning that previous questions about inconsistent results from the YouTube Data API had answers that seemed to resolve those inconsistencies. Is there a way to do that for this search? Is there something wrong with the way I'm composing the search that is causing the problem?
If there isn't a way to fix the search, then I suppose the only way to get a list of videos on the topic with high confidence of it being complete is to run the search multiple times and merge the results until no new IDs appear that were not in a previous result set. But even then, one would not know if there are other videos lurking undetected.

KDB combine/join different table

How can i join two different table like
all_order_ask:([]ask:();ask_qty:();exchange_name_ask:())
all_order_bid:([]bid:();bid_qty:();exchange_name_bid:())
and get =====>
final_order:ask:();ask_qty:();exchange_name_ask:();bid:();bid_qty:();exchange_name_bid:()
the two table have the same number of rows
you can use uj:
https://code.kx.com/q/ref/uj/
all_order_ask uj all_order_bid
ask ask_qty exchange_name_ask bid bid_qty exchange_name_bid
-----------------------------------------------------------
q)
If your tables look similar like this:
all_order_ask
ask ask_qty exchange_name_ask
----------------------------------
7.051033 8 bjd
1.497004 3 lln
2.400771 0 edg
1.039355 7 lij
2.353326 6 hon
6.423479 4 ncp
5.778177 6 gee
2.193148 5 ijf
1.66486 4 bbf
4.784272 2 lmi
all_order_bid
bid bid_qty exchange_name_bid
----------------------------------
15.70605 2 pjbke
10.93533 17 epjak
7.040985 11 ekaaj
14.19316 19 mpnan
9.248942 17 nogel
1.615466 18 holpj
1.073589 16 kkfpn
19.85822 13 pegin
14.45499 8 jcgnm
16.47223 0 dlhep
You can try this:
all_order_ask^all_order_bid
ask ask_qty exchange_name_ask bid bid_qty exchange_name_bid
---------------------------------------------------------------------
7.051033 8 bjd 15.70605 2 pjbke
1.497004 3 lln 10.93533 17 epjak
2.400771 0 edg 7.040985 11 ekaaj
1.039355 7 lij 14.19316 19 mpnan
2.353326 6 hon 9.248942 17 nogel
6.423479 4 ncp 1.615466 18 holpj
5.778177 6 gee 1.073589 16 kkfpn
2.193148 5 ijf 19.85822 13 pegin
1.66486 4 bbf 14.45499 8 jcgnm
4.784272 2 lmi 16.47223 0 dlhep
Since your two tables have the same number of rows, you should also be able to join your two tables horizontally using ,' as follows:
q)final_order_ask:all_order_ask,'all_order_bid
q)final_order_ask
ask ask_qty exchange_name_ask bid bid_qty exchange_name_bid
-----------------------------------------------------------

Return records in an unusual offset order

Currently I have a scope that pulls back my records in the following standard order:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
and it is converted in to a html block that's laid out as so:
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
I want a html block that's laid out like so:
1 6 11 16
2 7 12
3 8 13
4 9 14
5 10 15
So I think I need to pull the records back in the following order - (records offset by the number of rows)
1 6 11 16 2 7 12 3 8 13 4 9 14 5 10 15
Any idea what is the neatest way to do this in Rails / ActiveRecord?
You must know how many columns you want to eventually render. I think this must work for you:
columns = 5
MyModel.order(:id).in_groups_of(columns).transpose
And you get an array of lines, each one with an array of records.
If you want an unique array you can add .flatten at the end.
Can't do that with AR ordering! You will have to do that in your controller or the view.

Resources