How to list all permutations without repetition? - google-sheets

Question

The current post is a follow-up question to this linked one:
Shuffle a deck of 7 hypothetical trading cards and list out the orders they can come in a Google Sheet
Surprise! My problem child is actually for Shin Megami Tensei 3 permutations -- for a video game. I thought putting it into terms of trading cards would make more sense to the layman. Forgive me.
Okay. I messed around with a data set of 7 unique monsters (an initial set that's easy to obtain early in the video game). They can't be duplicated in my party but can be combined to make different ones. At the start, there are just 8 slots available for monsters in the video game.
This project focuses on building all the permutations of a "fusion chain" that attempts to take these monsters and arrange them into unique orders for a later combination within this chain.
It starts with A+B and then cleans that list to eliminate any B+A scenarios from the initial pairings (fusing A+B or B+A makes the same result). Then, the fusions just tack on C, D, E, F, G, and H (currently broken) to the result of the previous fusion until no more possible fusions remain (having only a single monster in my party).
The problem is this: the query or other functions within the permutation cell throw the error "The resulting array was too large" when attempting to list permutations for sorting 8 monsters at once -- even before the fusions can happen. I have isolated the issue to this formula (a bit long):
=iferror(if(counta($A$2:$A$13)>=2,arrayformula(query(query(split(flatten(flatten(flatten(flatten(flatten(flatten(
filter($F$2:$F,$F$2:$F<>"")&if(counta($A$2:$A$13)>=3,","&transpose(
filter($A$2:$A$13,$A$2:$A$13<>"")),""))&if(counta($A$2:$A$13)>=4,","&transpose(
filter($A$2:$A$13,$A$2:$A$13<>"")),""))&if(counta($A$2:$A$13)>=5,","&transpose(
filter($A$2:$A$13,$A$2:$A$13<>"")),""))&if(counta($A$2:$A$13)>=6,","&transpose(
filter($A$2:$A$13,$A$2:$A$13<>"")),""))&if(counta($A$2:$A$13)>=7,","&transpose(
filter($A$2:$A$13,$A$2:$A$13<>"")),""))&if(counta($A$2:$A$13)>=8,","&transpose(
filter($A$2:$A$13,$A$2:$A$13<>"")),"")),","),
"where Col1 <> Col2"&
if(counta($A$2:$A$13)>=3," and Col1 <> Col3 and Col2 <> Col3"&
if(counta($A$2:$A$13)>=4," and Col1 <> Col4 and Col2 <> Col4 and Col3 <> Col4"&
if(counta($A$2:$A$13)>=5," and Col1 <> Col5 and Col2 <> Col5 and Col3 <> Col5 and Col4 <> Col5"&
if(counta($A$2:$A$13)>=6," and Col1 <> Col6 and Col2 <> Col6 and Col3 <> Col6 and Col4 <> Col6 and Col5 <> Col6"&
if(counta($A$2:$A$13)>=7," and Col1 <> Col7 and Col2 <> Col7 and Col3 <> Col7 and Col4 <> Col7 and Col5 <> Col7 and Col6 <> Col7"&
if(counta($A$2:$A$13)>=8," and Col1 <> Col8 and Col2 <> Col8 and Col3 <> Col8 and Col4 <> Col8 and Col5 <> Col8 and Col6 <> Col8 and Col7 <> Col8",),),),),),),0),"where Col1 <>''",0)),"not enough data"),)
And the first range this formula was looking at is here in its previously stable form (column F):
unique init pairs
Pixie,Shikigami
Kodama,Pixie
Hua Po,Pixie
Datsue-Ba,Pixie
Angel,Pixie
Fomorian,Pixie
Kodama,Shikigami
Hua Po,Shikigami
Datsue-Ba,Shikigami
Angel,Shikigami
Fomorian,Shikigami
Hua Po,Kodama
Datsue-Ba,Kodama
Angel,Kodama
Fomorian,Kodama
Datsue-Ba,Hua Po
Angel,Hua Po
Fomorian,Hua Po
Angel,Datsue-Ba
Datsue-Ba,Fomorian
Angel,Fomorian
It was provided by a sort of "cleaner" formula I made but that isn't the problem.
The overall input I was testing is like this (in column A) and is also the input for the cleaner formulas for the initial pairs:
available
Pixie
Shikigami
Kodama
Hua Po
Datsue Ba
Angel
Fomorian
High Pixie
And the expected output... is really big. Here's a sample of the first lines to get an idea (hosted in H2 of the original sheet):
A
B
C
D
E
F
G
H
Pixie
Shikigami
Kodama
Hua Po
Datsue Ba
Angel
Fomorian
High Pixie
Pixie
Shikigami
Kodama
Hua Po
Datsue Ba
Fomorian
Angel
High Pixie
Pixie
Shikigami
Kodama
Hua Po
Angel
Datsue Ba
Fomorian
High Pixie
Pixie
Shikigami
Kodama
Hua Po
Angel
Fomorian
Datsue Ba
High Pixie
Pixie
Shikigami
Kodama
Hua Po
Fomorian
Datsue Ba
Angel
High Pixie
Pixie
Shikigami
Kodama
Hua Po
Fomorian
Angel
Datsue Ba
High Pixie
Pixie
Shikigami
Kodama
Datsue Ba
Hua Po
Angel
Fomorian
High Pixie
and so on...
I am currently at a loss for how to fix this problem. I would like to fit at least 8 starting monsters within my sheets for analysis, if not a full 12 for the end of the game.
There is probably a better, more compact way to generate these permutations than the way I have. I would probably like to boot up Excel to try this on my suped-up system and then see where it breaks offline. Yet, I want more efficient formulae to work around my "array too large" issues in Google Sheets. It's where I work best and where I have many other projects.

Answer 1

The limitations described below are because of lambda functions. The first solution can be successfully implemented without lambda:
=ARRAYFORMULA(QUERY(BASE(SEQUENCE(PERMUTATIONA(7,7)),7,7),"where not Col1 matches '.*(("&JOIN(")|(",SEQUENCE(7,1,0)&".*"&SEQUENCE(7,1,0))&")).*'",0))
The trick here is to use regex to find unique elements using query...match. The only problem with this is memory size needed will exceed 10 million for 8 items PERMUTATIONA(8,8). But that can be overcome with repeating the formula with different SEQUENCEs in a array {}.
There are different algorithms to implement this. See Permutation in computing:
The straight forward and the easiest approach is create a sequence of numbers with BASE equal to the number of items to choose from. For eg, if there are 7 items to choose from, create a sequence like this:
BASE 7(=ARRAYFORMULA(BASE(SEQUENCE(25),7,7)))
0000001
0000002
0000003
0000004
0000005
0000006
0000010
0000011
0000012
0000013
0000014
0000015
0000016
0000020
0000021
0000022
0000023
0000024
0000025
0000026
0000030
0000031
0000032
0000033
0000034
....
Notice at each position, there are 7 variables(0 to 6) and there are 7 positions. Once we get all the numbers for PERMUTATIONA(7,7), it's a simple matter of removing all the duplicates only getting numbers, where all numbers in each position are unique, i.e., COUNTUNIQUE per number = 7(eg:0124536). Here's a implementation:
=ARRAYFORMULA(LAMBDA(n,QUERY(BYROW(SPLIT(REGEXREPLACE(TO_TEXT(BASE(SEQUENCE(PERMUTATIONA(n,n)-1),n,n)),"\B","."),"."),LAMBDA(r, IF(COUNTUNIQUE(r)<>n,"👀",JOIN(,r)))),"where not Col1='👀' ",0))(5))
Unfortunately, Google arbitrarily limited execution to less than a few seconds. So, this formula is unable to get all permutations for more than n=5.
The next in the list is using factorial(Lehmer's code) to get the permutations. See permutations here. Note how there's a direct relation between a sequence of numbers and permutation.
decimal
factoradic
permutation
0
0:0:0!
(0,1,2)
1
0:1:0!
(0,2,1)
2
1:0:0!
(1,0,2)
3
1:1:0!
(1,2,0)
4
2:0:0!
(2,0,1)
5
2:1:0!
(2,1,0)
Table from https://wikipedia.org/wiki/Factorial_number_system
Licensed under CC-BY-SA 3.0
I implemented this algorithm and I've hit Google's limit again at n=5. (Code not shown here).
Up next, we have Lexicographic ordering. Algorithm is as follows:
The following algorithm generates the next permutation lexicographically after a given permutation. It changes the given permutation in-place.
Find the largest index k such that a[k] < a[k + 1]. If no such index exists, the permutation is the last permutation.
Find the largest index l greater than k such that a[k] < a[l].
Swap the value of a[k] with that of a[l].
Reverse the sequence from a[k + 1] up to and including the final element a[n].
For example, given the sequence [1, 2, 3, 4] (which is in increasing order), and given that the index is zero-based, the steps are as follows:
Index k = 2, because 3 is placed at an index that satisfies condition of being the largest index that is still less than a[k + 1] which is 4.
Index l = 3, because 4 is the only value in the sequence that is greater than 3 in order to satisfy the condition a[k] < a[l].
The values of a[2] and a[3] are swapped to form the new sequence [1, 2, 4, 3].
The sequence after k-index a[2] to the final element is reversed. Because only one value lies after this index (the 3), the sequence remains unchanged in this instance. Thus the lexicographic successor of the initial state is permuted: [1, 2, 4, 3].
Quoted from https://en.wikipedia.org/wiki/Permutation
Licensed under CC-BY-SA 3.0
Thanks to Google's latest support for recursion and named functions, I implemented this and I was able to get up to n=6(720 items) within a single formula, but I still hit the Google's recursion limit at n=7(5040 items). Having said that, it's still possible to get all the 5k permutations one by one without a array formula(and maybe even n=8(40320 items) depending on what your device can handle).
1.Pixie
2.Shikigami
3.Kodama
4.Hua Po
5.Datsue Ba
6.Angel
7.Fomorian
1.Pixie
2.Shikigami
3.Kodama
4.Hua Po
5.Datsue Ba
7.Fomorian
6.Angel
1.Pixie
2.Shikigami
3.Kodama
4.Hua Po
6.Angel
5.Datsue Ba
7.Fomorian
1.Pixie
2.Shikigami
3.Kodama
4.Hua Po
6.Angel
7.Fomorian
5.Datsue Ba
1.Pixie
2.Shikigami
3.Kodama
4.Hua Po
7.Fomorian
5.Datsue Ba
6.Angel
1.Pixie
2.Shikigami
3.Kodama
4.Hua Po
7.Fomorian
6.Angel
5.Datsue Ba
1.Pixie
2.Shikigami
3.Kodama
5.Datsue Ba
4.Hua Po
6.Angel
7.Fomorian
1.Pixie
2.Shikigami
3.Kodama
5.Datsue Ba
4.Hua Po
7.Fomorian
6.Angel
1.Pixie
2.Shikigami
3.Kodama
5.Datsue Ba
6.Angel
4.Hua Po
7.Fomorian
1.Pixie
2.Shikigami
3.Kodama
5.Datsue Ba
6.Angel
7.Fomorian
4.Hua Po
1.Pixie
2.Shikigami
3.Kodama
5.Datsue Ba
7.Fomorian
4.Hua Po
6.Angel
1.Pixie
2.Shikigami
3.Kodama
5.Datsue Ba
7.Fomorian
6.Angel
4.Hua Po
1.Pixie
2.Shikigami
3.Kodama
6.Angel
4.Hua Po
5.Datsue Ba
7.Fomorian
1.Pixie
2.Shikigami
3.Kodama
6.Angel
4.Hua Po
7.Fomorian
5.Datsue Ba
1.Pixie
2.Shikigami
3.Kodama
6.Angel
5.Datsue Ba
4.Hua Po
7.Fomorian
1.Pixie
2.Shikigami
3.Kodama
6.Angel
5.Datsue Ba
7.Fomorian
4.Hua Po
1.Pixie
2.Shikigami
3.Kodama
6.Angel
7.Fomorian
4.Hua Po
5.Datsue Ba
1.Pixie
2.Shikigami
3.Kodama
6.Angel
7.Fomorian
5.Datsue Ba
4.Hua Po
1.Pixie
2.Shikigami
3.Kodama
7.Fomorian
4.Hua Po
5.Datsue Ba
6.Angel
1.Pixie
2.Shikigami
3.Kodama
7.Fomorian
4.Hua Po
6.Angel
5.Datsue Ba
1.Pixie
2.Shikigami
3.Kodama
7.Fomorian
5.Datsue Ba
4.Hua Po
6.Angel
1.Pixie
2.Shikigami
3.Kodama
7.Fomorian
5.Datsue Ba
6.Angel
4.Hua Po
1.Pixie
2.Shikigami
3.Kodama
7.Fomorian
6.Angel
4.Hua Po
5.Datsue Ba
1.Pixie
2.Shikigami
3.Kodama
7.Fomorian
6.Angel
5.Datsue Ba
4.Hua Po
Showing the first few permutations for n=7. For the formula to work, it's important to note that there must be a inherent ascending order in the list. I added prefixes:1., 2., etc to 1.Pixie, 2.Shikigami... and so on to enforce ascending order. It is possible to the order within the formula itself, but it's not implemented.
A1:G1:
1.Pixie
2.Shikigami
3.Kodama
4.Hua Po
5.Datsue Ba
6.Angel
7.Fomorian
A2:
=GET_NEXT_LEX(A1:G1)
Drag fill or auto fill down as much as needed (40k or 5k rows). The advantage of using this method is, you can continue where you left off. If you need 2 million permutations, and Google sheets cannot handle more than 1 million. You can put the first million in one spreadsheet and continue the next million in another(all you need is the last permutation from the previous spreadsheet).
Named functions:
Create these functions
Main function:
GET_NEXT_LEX(arr):
=ARRAYFORMULA(
TRANSPOSE(
  LAMBDA(arr,    
LAMBDA(k,      
LAMBDA(sarr,k,{SPLICE(sarr,k+1,2^999);REVERSE(SPLICE(sarr,1,k+1))})  
  (SWAP(arr,k,XMATCH(TRUE,INDEX(arr,k)<SPLICE(arr,1,k+1),,-1)+k),k)
 )(XMATCH(TRUE,POP(arr)<SHIFT(arr),,-1))
  )(TRANSPOSE(arr))
)
)
Helper functions:
Functions similar to javascript or python
SPLICE(arr,i,j)
=FILTER(arr,LAMBDA(seq,(seq<i)+(seq>=j))(SEQUENCE(ROWS(arr))))
REVERSE(arr)
=POP(REDUCE(,arr,LAMBDA(a,c,{c;a})))
SWAP(arr,i,j)
=SORT(arr,LAMBDA(keys,SWITCH(keys,i,j,j,i,keys))(SEQUENCE(ROWS(arr))),1)
POP(arr)
=ARRAY_CONSTRAIN(arr,ROWS(arr)-1,1)
SHIFT(arr)
=FILTER(arr,{0;SEQUENCE(ROWS(arr)-1)})

Answer 2

instead of an algorithm, we could use just Lagrangian equation and calculate all particles needed to store all permutations... or we could also just use good old brute force...
to list all possible permutations of 12 items without repetition within one single spreadsheet is not possible due to the limitation being set to 10 million cells per spreadsheet
to put it in some perspective, to list all possible permutations of 12 items up to the 8th factorial would require 5 spreadsheets and it would use up 19958400 cells whereas the journey to calculate it would require 24723744 cells
spreadsheet 1
spreadsheet 2
spreadsheet 3
spreadsheet 4
spreadsheet 5
depends on your system but it can take some time to load the array of 8 million cells so give it some time and a few refreshes if the spreadsheet crashes. not to mention the maximum scrolling limitation of 1597829 rows so to jump to the last row we can use CTRL+END and give it some time. therefore, the only way how to select and copy everything is to click on the column label (or export the spreadsheet) because non of these works:
CTRL+DOWN ARROW
CTRL+SHIFT+DOWN ARROW
and good luck with not crashing the spreadsheet due to random memory or networking issues... just for educational purposes, the formula to populate 8 million cells looks like this:
={LAMBDA(x, y, INDEX(QUERY(FLATTEN(IF(REGEXMATCH(x, y),, x&","&y)), "where Col1 is not null", )))(QUERY({
IMPORTRANGE("1XisY71iQTLGqEQUXBb6yoGM2PSAFaqbnMccyeCtLP1M", "A1:A200000");
IMPORTRANGE("1XisY71iQTLGqEQUXBb6yoGM2PSAFaqbnMccyeCtLP1M", "A200001:A400000");
IMPORTRANGE("1XisY71iQTLGqEQUXBb6yoGM2PSAFaqbnMccyeCtLP1M", "A400001:A600000");
IMPORTRANGE("1XisY71iQTLGqEQUXBb6yoGM2PSAFaqbnMccyeCtLP1M", "A600001:A800000")}, "where Col1 is not null", ),
TRANSPOSE(QUERY(IMPORTRANGE("10fbzz3HINDs_bc2Oeihe20C-BQlq8kK_SATdevliWos", "A1:A12"), "where Col1 is not null", )));
LAMBDA(x, y, INDEX(QUERY(FLATTEN(IF(REGEXMATCH(x, y),, x&","&y)), "where Col1 is not null", )))(QUERY({
IMPORTRANGE("1XisY71iQTLGqEQUXBb6yoGM2PSAFaqbnMccyeCtLP1M", "A800001:A1000000");
IMPORTRANGE("1XisY71iQTLGqEQUXBb6yoGM2PSAFaqbnMccyeCtLP1M", "A1000001:A1200000");
IMPORTRANGE("1XisY71iQTLGqEQUXBb6yoGM2PSAFaqbnMccyeCtLP1M", "A1200001:A1400000");
IMPORTRANGE("1XisY71iQTLGqEQUXBb6yoGM2PSAFaqbnMccyeCtLP1M", "A1400001:A1600000")}, "where Col1 is not null", ),
TRANSPOSE(QUERY(IMPORTRANGE("10fbzz3HINDs_bc2Oeihe20C-BQlq8kK_SATdevliWos", "A1:A12"), "where Col1 is not null", )))}
to answer the question
list all permutations without repetition of 7 unique monsters
=INDEX(LAMBDA(a,
LAMBDA(x, y, QUERY(FLATTEN(IF(REGEXMATCH(x, y),, x&","&y)), "where Col1 is not null", ))
(LAMBDA(x, y, QUERY(FLATTEN(IF(REGEXMATCH(x, y),, x&","&y)), "where Col1 is not null", ))
(LAMBDA(x, y, QUERY(FLATTEN(IF(REGEXMATCH(x, y),, x&","&y)), "where Col1 is not null", ))
(LAMBDA(x, y, QUERY(FLATTEN(IF(REGEXMATCH(x, y),, x&","&y)), "where Col1 is not null", ))
(LAMBDA(x, y, QUERY(FLATTEN(IF(REGEXMATCH(x, y),, x&","&y)), "where Col1 is not null", ))
(LAMBDA(z, LAMBDA(x, y, QUERY(FLATTEN(IF((x<y)+(x>y), x&","&y, )), "where Col1 is not null", ))
(z, TRANSPOSE(z)))(a), TRANSPOSE(a)), TRANSPOSE(a)), TRANSPOSE(a)), TRANSPOSE(a)), TRANSPOSE(a)))
(A1:A7))
which results in 5040 unique combinations. somehow, however, the question shifted from 7 to 8 monsters so to answer the question
list all permutations without repetition of 8 unique monsters
=INDEX(LAMBDA(a,
LAMBDA(x, y, QUERY(FLATTEN(IF(REGEXMATCH(x, y),, x&","&y)), "where Col1 is not null", ))
(LAMBDA(x, y, QUERY(FLATTEN(IF(REGEXMATCH(x, y),, x&","&y)), "where Col1 is not null", ))
(LAMBDA(x, y, QUERY(FLATTEN(IF(REGEXMATCH(x, y),, x&","&y)), "where Col1 is not null", ))
(LAMBDA(x, y, QUERY(FLATTEN(IF(REGEXMATCH(x, y),, x&","&y)), "where Col1 is not null", ))
(LAMBDA(x, y, QUERY(FLATTEN(IF(REGEXMATCH(x, y),, x&","&y)), "where Col1 is not null", ))
(LAMBDA(x, y, QUERY(FLATTEN(IF(REGEXMATCH(x, y),, x&","&y)), "where Col1 is not null", ))
(LAMBDA(z, LAMBDA(x, y, QUERY(FLATTEN(IF((x<y)+(x>y), x&","&y, )), "where Col1 is not null", ))
(z, TRANSPOSE(z)))(a), TRANSPOSE(a)), TRANSPOSE(a)), TRANSPOSE(a)), TRANSPOSE(a)), TRANSPOSE(a)), TRANSPOSE(a)))
(A1:A8))
and just for fun, we can do 9 monsters (362880 combinations) like:
=INDEX(LAMBDA(a,
LAMBDA(x, y, QUERY(FLATTEN(IF(REGEXMATCH(x, y),, x&","&y)), "where Col1 is not null", ))
(LAMBDA(x, y, QUERY(FLATTEN(IF(REGEXMATCH(x, y),, x&","&y)), "where Col1 is not null", ))
(LAMBDA(x, y, QUERY(FLATTEN(IF(REGEXMATCH(x, y),, x&","&y)), "where Col1 is not null", ))
(LAMBDA(x, y, QUERY(FLATTEN(IF(REGEXMATCH(x, y),, x&","&y)), "where Col1 is not null", ))
(LAMBDA(x, y, QUERY(FLATTEN(IF(REGEXMATCH(x, y),, x&","&y)), "where Col1 is not null", ))
(LAMBDA(x, y, QUERY(FLATTEN(IF(REGEXMATCH(x, y),, x&","&y)), "where Col1 is not null", ))
(LAMBDA(x, y, QUERY(FLATTEN(IF(REGEXMATCH(x, y),, x&","&y)), "where Col1 is not null", ))
(LAMBDA(z, LAMBDA(x, y, QUERY(FLATTEN(IF((x<y)+(x>y), x&","&y, )), "where Col1 is not null", ))
(z, TRANSPOSE(z)))(a), TRANSPOSE(a)), TRANSPOSE(a)), TRANSPOSE(a)), TRANSPOSE(a)), TRANSPOSE(a)), TRANSPOSE(a)), TRANSPOSE(a)))
(A1:A9))
now let's get serious with 10 monsters... that's 3628800 unique combinations across 3 spreadsheets:
spreadsheet X
spreadsheet Y
spreadsheet Z
with formula:
=LAMBDA(x, y, INDEX(QUERY(FLATTEN(IF(REGEXMATCH(x, y),, x&","&y)), "where Col1 is not null", )))
(LAMBDA(x, y, INDEX(QUERY(FLATTEN(IF(REGEXMATCH(x, y),, x&","&y)), "where Col1 is not null", )))(QUERY({
IMPORTRANGE("1bAz3a18NGEVgF44r-RxhV_oco1AWdli1CoJ5HAhT91g", "A1:A200000");
IMPORTRANGE("1bAz3a18NGEVgF44r-RxhV_oco1AWdli1CoJ5HAhT91g", "A200001:A400000");
IMPORTRANGE("1bAz3a18NGEVgF44r-RxhV_oco1AWdli1CoJ5HAhT91g", "A400001:A600000");
IMPORTRANGE("1bAz3a18NGEVgF44r-RxhV_oco1AWdli1CoJ5HAhT91g", "A600001:A800000");
IMPORTRANGE("1bAz3a18NGEVgF44r-RxhV_oco1AWdli1CoJ5HAhT91g", "A800001:A1000000");
IMPORTRANGE("1bAz3a18NGEVgF44r-RxhV_oco1AWdli1CoJ5HAhT91g", "A1000001:A1200000");
IMPORTRANGE("1bAz3a18NGEVgF44r-RxhV_oco1AWdli1CoJ5HAhT91g", "A1200001:A1400000");
IMPORTRANGE("1bAz3a18NGEVgF44r-RxhV_oco1AWdli1CoJ5HAhT91g", "A1400001:A1600000");
IMPORTRANGE("1bAz3a18NGEVgF44r-RxhV_oco1AWdli1CoJ5HAhT91g", "A1600001:A1814400")}, "where Col1 starts with '"&A1&"'", ), TRANSPOSE(QUERY(
IMPORTRANGE("1nqIvdSihMPEtvips5zrjKqJzA8E9o1G57WEOsaK95F0", "A1:A10"), "where Col1 is not null", ))), TRANSPOSE(QUERY(
IMPORTRANGE("1nqIvdSihMPEtvips5zrjKqJzA8E9o1G57WEOsaK95F0", "A1:A10"), "where Col1 is not null", )))
where A1 is kinda dropdown/selector for string start due to the sheer size of 10 million ways (with repetitions) how to combine 10 monsters into 10 slots and the formula just takes that 10M input and renders out repetitions of the same monster within a single string so the output with no duplicate monsters is produced of size 3628800 segmented into 90 ways how to choose 10 monsters multiplied by 40320 combinations