I'm looking to generate a table of random values, but want to make sure that none of those values are repeated within the table.
So my basic table generation looks like this:
numbers = {}
for i = 1, 5 do
table.insert(numbers, math.random(20))
end
So that will work in populating a table with 5 random values between 1-20. However, it's the making sure none of those values repeat is where I'm stuck.
One approach would be to shuffle an array of numbers and then take the first n numbers. The wrong way to go about shuffling an array is to maintain a list of previously generated random numbers, checking against that with each newly generated random number before adding it to the final array. Such a solution is O(n^2) in time complexity when iterating over the array during the check; this will be painful for large arrays, or for small arrays when many must be created. Lua has constant time array access since tables are really hash tables, so you could get away with this, except: sometimes many random numbers will need to be tried before a suitable one (that has not already been used) is found. This can be a real problem near the end of an array of many random numbers, i.e., when you want 1000 random numbers and have filled all but the last slot, how many random tries (and how many iterations of the 999 numbers already selected) will it take to find the only number (42, of course) that is still available?
The right way to go about shuffling is to use a shuffling algorithm. The Fisher-Yates shuffle is a common solution to this problem. The idea is that you start at one end of an array, and swap each element with a random element that occurs later in the list until the entire array has been shuffled. This solution is O(n) in time complexity, thus much less wasteful of computational resources.
Here is an implementation in Lua:
function shuffle (arr)
for i = 1, #arr - 1 do
local j = math.random(i, #arr)
arr[i], arr[j] = arr[j], arr[i]
end
end
Testing in the REPL:
> t = { 1, 2, 3, 4, 5, 6 }
> table.inspect(t)
1 = 1
2 = 2
3 = 3
4 = 4
5 = 5
6 = 6
> shuffle(t)
> table.inspect(t)
1 = 4
2 = 5
3 = 1
4 = 6
5 = 2
6 = 3
This can easily be extended to create lists of random numbers:
function shuffled_numbers (n)
local numbers = {}
for i = 1, n do
numbers[i] = i
end
shuffle(numbers)
return numbers
end
REPL interaction:
> s = shuffled_numbers(10)
> table.inspect(s)
1 = 9
2 = 5
3 = 3
4 = 4
5 = 7
6 = 6
7 = 2
8 = 10
9 = 8
10 = 1
If you want to see what is happening during the shuffle, add a print statement in the shuffle function:
function shuffle (arr)
for i = 1, #arr - 1 do
local j = math.random(i, #arr)
print(string.format("%d (%d) <--> %d (select %d)", i, arr[i], j, arr[j]))
arr[i], arr[j] = arr[j], arr[i]
end
end
Now you can see the swaps as they occur if you recall that in the above implementation of shuffled_numbers the array { 1, 2, ..., n } is the starting point of the shuffle. Note that sometimes a number is swapped with itself, which is to say that the number in the current unselected position is a valid choice, too. Also note that the last number is automatically the correct selection, since it is the only number that has not yet been randomly selected:
> s = shuffled_numbers(10)
1 (1) <--> 5 (select 5)
2 (2) <--> 10 (select 10)
3 (3) <--> 5 (select 1)
4 (4) <--> 9 (select 9)
5 (3) <--> 8 (select 8)
6 (6) <--> 9 (select 4)
7 (7) <--> 8 (select 3)
8 (7) <--> 10 (select 2)
9 (6) <--> 9 (select 6)
> table.inspect(s)
1 = 5
2 = 10
3 = 1
4 = 9
5 = 8
6 = 4
7 = 3
8 = 2
9 = 6
10 = 7
Obtaining a selection of 5 random numbers between 1 and 20 is easy enough to accomplish using the shuffle function; one of the virtues of this approach is that the shuffling operation has been abstracted to an O(n) procedure which can shuffle any array, numeric or otherwise. The function that calls shuffle is responsible for supplying the input and returning the results.
A simple solution for more flexibility in the range of random numbers returned:
-- Take the first N numbers from a shuffled range [A, B].
function shuffled_range_take (n, a, b)
local numbers = {}
for i = a, b do
numbers[i] = i
end
shuffle(numbers)
return { table.unpack(numbers, 1, n) }
-- table.unpack won't work for very large ranges, e.g. [1, 1000000]
-- You could instead use this for arbitrarily large ranges:
-- local take = {}
-- for i= 1, n do
-- take[i] = numbers[i]
-- end
-- return take
end
REPL interaction creating a table containing 5 random values between 1 and 20:
> s = shuffled_range_take(5, 1, 20)
> table.inspect(s)
1 = 1
2 = 10
3 = 4
4 = 8
5 = 20
But, there is a disadvantage to the shuffle method in some circumstances. When the number of elements needed is small compared with the number of available elements, the above solution must shuffle a large array to obtain comparatively few random elements. The shuffle is O(n) in the number of elements available, while the memoization method is roughly O(n) in the number of elements chosen. A memoization method like that of #AlexanderMashin performs poorly when the goal is to create an array of 20 random numbers between 1 and 20, because the final numbers chosen may need to be chosen many times before suitable numbers are found. But when only 5 random numbers between 1 and 20 are needed, this problem with duplicate choices is less of an issue. This approach seems to perform better than the shuffle, up to about 10 numbers needed from 20 random numbers. When more than 10 numbers are needed from 20, the shuffle begins to perform better. This break-even point is different for larger numbers of elements to choose from; for 1000 available elements, parity is reached at about 700 chosen. When performance is critical, testing is the only way to determine the best solution.
numbers = {}
local i = 1;
while i<=5 do
n = 0
local rand = math.random(20)
for x=1,#numbers do
if numbers[x] == rand then
n = n + 1
end
end
if n == 0 then
table.insert(numbers, rand)
i = i + 1
end
n = 0
end
the method I used for this process was to use a for to scan each of the elements in the table and increase the variable n if one of them was equal to the random value given, so if x was different from 0, the value would not be inserted in the table and would not increment the variable i (I had to use the while to work with i)
if you want to print each of the elements in the table to check the values you can use this:
for i=1,#numbers do
print(numbers[i])
end
I suggest an alternative method based on the fact that it is easy to make sets in Lua: they are just tables with true values.
-- needed is how many random numbers in the table are needed,
-- maximum is the maximum value of a random non-negtive integer.
local function fill_table( needed, maximum )
math.randomseed ( os.time () ) -- reseed the random numbers generator
local numbers = {}
local used = {} -- which numbers are already used
for i = 1, needed do
local random
repeat
random = math.random( maximum )
until not used[random]
used[random] = true
numbers[i] = random
end
return numbers
end
Making a table with 20 keys (use for/do/end) and then do your desired times
rand_number=table.remove(tablename, math.random(1,#tablename))
EDIT: Corrected - See first comment
And rand_number never holds the same value. I use this as a simulation for a "Lottozahlengenerator" (german, sorry) or random video/music clips playing where duplicates are unwanted.
Suppose I have two datasets .
DS1:
a 1
b 2
c 3
d 4
e 5
DS2:
1 pass
2 fail
3 pass
4 pass
5 fail
and i want to get a output like :
a 1 pass
b 2 fail
c 3 pass
d 4 pass
e 5 fail
now my question is,what pigcommand should i use to get the desire output?
JOIN.Assuming the data in the files are tab delimited.
A = LOAD 'ds1' USING PigStorage('\t') AS (a1:charrarray,a2:int);
B = LOAD 'ds2' USING PigStorage('\t') AS (b1:int,a2:chararray);
C = JOIN A BY a2, B BY b1;
D = FOREACH C GENERATE C.$0,C.$1,B.$1;
DUMP D;
I have google the whole universe but cannot find out this.
Given data set A:
a b
1 2
3 4
1 2
I want to print this to result in this way:
a 1 3 1
b 2 4 2
Also print each variable, name first then content on one line to result.
I think you're looking for for proc transpose:
proc transpose data = A out = A_transpose;
var a b;
run;
Then you can print this with proc print:
proc print data = A_transpose;
run;
I'm wondering if there's a way for me to perform an exact match compare in SPSS. Currently, using the following will return system missing (null) in cases where one variable is sysmis:
compute var1_comparison = * Some logic here.
compute var1_check = var1 = var1_comparison.
The results look like this (hypens representing null values):
ID var1 var1_comparison var1_check
1 3 3 1
2 4 3 0
3 - 2 -
4 1 1 1
5 - - -
What I want is this:
ID var1 var1_comparison var1_check
1 3 3 1
2 4 3 0
3 - 2 0
4 1 1 1
5 - - 1
Is this possible using just plain SPSS syntax? I'm also open to using the Python extension, though I'm not as familiar with it.
Here's a slightly different approach, using temporary scratch variables (prefixed by a hash (#)):
recode var1 var1_comparison (sysmis=-99) (else=copy) into #v1 #v2.
compute Check=(#v1 = #v2).
This is to recreate your example:
data list list/ID var1 var1_comparison.
begin data
1, 3, 3
2 , 4, 3
3, , 2
4, 1, 1
5, ,
end data.
Now you have to deal separately with the situation where both values are missing, and then complete the calculation in all other situations:
do if missing(var1) or missing(var1_comparison).
compute var1_check=(missing(var1) and missing(var1_comparison)).
else.
compute var1_check = (var1 = var1_comparison).
end if.
I want to have a if condition within a loop. That is As long as id < 10,
check if Modc_initial is equal to MODC, if true then set d = 12
This is the code I tried bit not working, can anyone please help.
LOOP if (id LT 10)
IF(Modc_initial EQ MODC))
COMPUTE d = 12.
END LOOP.
EXECUTE.
You can either use a one line conditional of the form IF (condition) d = 12. or a multiple line DO IF. Below I provide an example of DO IF adapted to your syntax.
data list free / id MODC Modc_initial.
begin data
1 3 3
2 3 5
12 1 1
end data.
LOOP if (id LT 10).
DO IF (Modc_initial EQ MODC).
COMPUTE d = 12.
END IF.
END LOOP IF (d = 12).
EXECUTE.
Note you had a period missing in your original syntax on the initial LOOP. I also added an end loop condition, otherwise the code as written would just go until the maximum set number of loops per your system.