I use Linear Congruential Generators (http://en.wikipedia.org/wiki/Linear_congruential_generator) to generate IDs exposed to users.
nextID = (a * LastID + c) % m
Now I want to implement LCGs in Redis. Here's the problem:
Getting the current ID and generating the next ID outside Redis is not multi-user safe. Redis has 2 commands which can be used for simple counters: INCRBY and INCRBYFLOAT, but unfortunately Redis doesn't support modulo operation natively. At the moment the only way I see is using EVAL command and writing some lua script.
UPDATE1:
Some lua analog of
INCRBY LCG_Value ((LCG_Value*a+c)%m)-LCG_Value
seems to be a neat way to achieve this.
A server-side Lua script is probably the easiest and more efficient way to do it.
Now it can also be done using Redis primitive operations using a multi/exec block. Here it is in pseudo-code:
while not successful:
WATCH LCG_Value
$LastID = GET LCG_value
$NextID = (a*$LastID+c)%m
MULTI
SET LCG_value $NextID
EXEC
Of course, it is less efficient than the following Lua script:
# Set the initial value
SET LCG_value 1
# Execute Lua script with on LCG_value with a, c, and m parameters
EVAL "local last = tonumber(redis.call( 'GET', KEYS[1]));
local ret = (last*ARGV[1] + ARGV[2])%ARGV[3];
redis.call('SET',KEYS[1], ret);
return ret;
" 1 LCG_value 1103515245 12345 2147483648
Note: the whole script execution is atomic. See the EVAL documentation.
Related
My question is that when I run
wrk -d10s -t20 -c20 -s /mnt/c/xxxx/post.lua http://localhost:xxxx/post
the Lua script that is only executed once? It will only put one item into the database at the URL.
-- example HTTP POST script which demonstrates setting the
-- HTTP method, body, and adding a header
math.randomseed(os.time())
number = math.random()
wrk.method = "POST"
wrk.headers["Content-Type"] = "application/json"
wrk.body = '{"name": "' .. tostring(number) .. '", "title":"test","enabled":true,"defaultValue":false}'
Is there a way to make it create the 'number' variable dynamically and keep adding new items into the database until the 'wrk' command has finished its test? Or that it will keep executing the script for the duration of the test creating and inserting new 'number' variables into 'wrk.body' ?
Apologies I have literally only being looking at Lua for a few hours.
Thanks
When you do
number = math.random
you're not setting number to a random number, you're setting it equal to the function math.random. To set the variable to the value returned by the function, that line should read
number = math.random()
You may also need to set a random seed (with the math.randomseed() function and your choice of an appropriately variable argument - system time is common) to avoid math.random() giving the same result each time the script is run. This should be done before the first call to math.random.
As the script is short, system time probably isn't a good choice of seed here (the script runs far quicker than the value from os.time() changes, so running it several times immediately after one another gives the same results each time). Reading a few bytes from /dev/urandom should give better results.
You could also just use /dev/urandom to generate a number directly, rather than feeding it to math.random as a seed. Like in the code below, as taken from this answer. This isn't a secure random number generator, but for your purposes it would be fine.
urand = assert (io.open ('/dev/urandom', 'rb'))
rand = assert (io.open ('/dev/random', 'rb'))
function RNG (b, m, r)
b = b or 4
m = m or 256
r = r or urand
local n, s = 0, r:read (b)
for i = 1, s:len () do
n = m * n + s:byte (i)
end
return n
end
I'm trying to produce a Lua script that takes the members of a set (each representing a set as well) and returs the union.
This is a concrete example with these 3 sets:
smembers u:1:skt:n1
1) "s2"
2) "s3"
3) "s1"
smembers u:1:skt:n2
1) "s4"
2) "s5"
3) "s6"
smembers u:1:skts
1) "u:1:skt:n1"
2) "u:1:skt:n2"
So the set u:1:skts contains the reference of the other 2 sets and I
want to produce the union of u:1:skt:n1 and u:1:skt:n2 as follows:
1) "s1"
2) "s2"
3) "s3"
4) "s4"
5) "s5"
6) "s6"
This is what I have so far:
local indexes = redis.call("smembers", KEYS[1])
return redis.call("sunion", indexes)
But I get the following error:
(error) ERR Error running script (call to f_c4d338bdf036fbb9f77e5ea42880dc185d57ede4):
#user_script:1: #user_script: 1: Lua redis() command arguments must be strings or integers
It seems like it doesn't like the indexes variable as input of the sunion command. Any ideas?
Do not do this, or you'll have trouble moving to the cluster. This is from the documentation:
All Redis commands must be analyzed before execution to determine which keys the command will operate on. In order for this to be true for EVAL, keys must be passed explicitly. This is useful in many ways, but especially to make sure Redis Cluster can forward your request to the appropriate cluster node.
Note this rule is not enforced in order to provide the user with opportunities to abuse the Redis single instance configuration, at the cost of writing scripts not compatible with Redis Cluster.
If you still decide to go against the rules, use Lua's unpack:
local indexes = redis.call("smembers", KEYS[1])
return redis.call("sunion", unpack(indexes))
I successfully wrote an intersection of text search and other criteria using Redis. To achieve that I'm using a Lua script. The issue is that I'm not only reading, but also writing values from that script. From Redis 3.2 it's possible to achieve that by calling redis.replicate_commands(), but not before 3.2.
Below is how I'm storing the values.
Names
> HSET product:name 'Cool product' 1
> HSET product:name 'Nice product' 2
Price
> ZADD product:price 49.90 1
> ZADD product:price 54.90 2
Then, to get all products that matches 'ice', for example, I call:
> HSCAN product:name 0 MATCH *ice*
However, since HSCAN uses a cursor, I have to call it multiple times to fetch all results. This is where I'm using a Lua script:
local cursor = 0
local fields = {}
local ids = {}
local key = 'product:name'
local value = '*' .. ARGV[1] .. '*'
repeat
local result = redis.call('HSCAN', key, cursor, 'MATCH', value)
cursor = tonumber(result[1])
fields = result[2]
for i, id in ipairs(fields) do
if i % 2 == 0 then
ids[#ids + 1] = id
end
end
until cursor == 0
return ids
Since it's not possible to use the result of a script with another call, like SADD key EVAL(SHA) .... And also, it's not possible to use global variables within scripts. I've changed the part inside the fields' loop to access the list of ID's outside the script:
if i % 2 == 0 then
ids[#ids + 1] = id
redis.call('SADD', KEYS[1], id)
end
I had to add redis.replicate_commands() to the first line. With this change I can get all ID's from the key I passed when calling the script (see KEYS[1]).
And, finally, to get a list 100 product ID's priced between 40 and 50 where the name contains "ice", I do the following:
> ZUNIONSTORE tmp:price 1 product:price WEIGHTS 1
> ZREMRANGEBYSCORE tmp:price 0 40
> ZREMRANGEBYSCORE tmp:price 50 +INF
> EVALSHA b81c2b... 1 tmp:name ice
> ZINTERSTORE tmp:result tmp:price tmp:name
> ZCOUNT tmp:result -INF +INF
> ZRANGE tmp:result 0 100
I use the ZCOUNT call to know in advance how many result pages I'll have, doing count / 100.
As I said before, this works nicely with Redis 3.2. But when I tried to run the code at AWS, which only supports Redis up to 2.8, I couldn't make it work anymore. I'm not sure how to iterate with HSCAN cursor without using a script or without writing from the script. There is a way to make it work on Redis 2.8?
Some considerations:
I know I can do part of the processing outside Redis (like iterate the cursor or intersect the matches), but it'll affect the application overall performance.
I don't want to deploy a Redis instance by my own to use version 3.2.
The criteria above (price range and name) is just an example to keep things simple here. I have other fields and type of matches, not only those.
I'm not sure if the way I'm storing the data is the best way. I'm willing to listen suggestion about it.
The only problem I found here is storing the values inside a lua scirpt. So instead of storing them inside a lua, take that value outside lua (return that values of string[]). Store them in a set in a different call using sadd (key,members[]). Then proceed with intersection and returning results.
> ZUNIONSTORE tmp:price 1 product:price WEIGHTS 1
> ZREVRANGEBYSCORE tmp:price 0 40
> ZREVRANGEBYSCORE tmp:price 50 +INF
> nameSet[] = EVALSHA b81c2b... 1 ice
> SADD tmp:name nameSet
> ZINTERSTORE tmp:result tmp:price tmp:name
> ZCOUNT tmp:result -INF +INF
> ZRANGE tmp:result 0 100
IMO your design is the most optimal one. One advice would be to use pipeline wherever possible, as it would process everything at one go.
Hope this helps
UPDATE
There is no such thing like array ([ ]) in lua you have to use the lua table to achieve it. In your script you are returning ids right, that itself is an array you can use it as a separate call to achieve the sadd.
String [] nameSet = (String[]) evalsha b81c2b... 1 ice -> This is in java
SADD tmp:name nameSet
And the corresponding lua script is the same as that of your 1st one.
local cursor = 0
local fields = {}
local ids = {}
local key = 'product:name'
local value = '*' .. ARGV[1] .. '*'
repeat
local result = redis.call('HSCAN', key, cursor, 'MATCH', value)
cursor = tonumber(result[1])
fields = result[2]
for i, id in ipairs(fields) do
if i % 2 == 0 then
ids[#ids + 1] = id
end
end
until cursor == 0
return ids
The problem isn't that you're writing to the database, it's that you're doing a write after a HSCAN, which is a non-deterministic command.
In my opinion there's rarely a good reason to use a SCAN command in a Lua script. The main purpose of the command is to allow you to do things in small batches so you don't lock up the server processing a huge key space (or hash key space). Since scripts are atomic, though, using HSCAN doesn't help—you're still locking up the server until the whole thing's done.
Here are the options I can see:
If you can't risk locking up the server with a lengthy command:
Use HSCAN on the client. This is the safest option, but also the slowest.
If you're want to do as much processing in a single atomic Lua command as possible:
Use Redis 3.2 and script effects replication.
Do the scanning in the script, but return the values to the client and initiate the write from there. (That is, Karthikeyan Gopall's answer.)
Instead of HSCAN, do an HKEYS in the script and filter the results using Lua's pattern matching. Since HKEYS is deterministic you won't have a problem with the subsequent write. The downside, of course, is that you have to read in all of the keys first, regardless of whether they match your pattern. (Though HSCAN is also O(N) in the size of the hash.)
I'm evaluating the use of lua scrips in redis, and they seem to be a bit slow. I a benchmark as follows:
For a non-lua version, I did a simple SET key_i val_i 1M times
For a lua version, I did the same thing, but in a script: EVAL "SET KEYS[1] ARGV[1]" 1 key_i val_i
Testing on my laptop, the lua version is about 3x slower than the non-lua version. I understand that lua is a scripting language, not compiled, etc. etc. but this seems like a lot of performance overhead--is this normal?
Assuming this is indeed normal, are there any workaround? Is there a way to implement a script in a faster language, such as C (which redis is written in) to achieve better performance?
Edit: I am testing this using the go code located here: https://gist.github.com/ortutay/6c4a02dee0325a608941
The problem is not with Lua or Redis; it's with your expectations. You are compiling a script 1 million times. There is no reason to expect this to be fast.
The purpose of EVAL within Redis is not to execute a single command; you could do that yourself. The purpose is to do complex logic within Redis itself, on the server rather than on your local client. That is, instead of doing one set operation per-EVAL, you actually perform the entire series of 1 million sets within a single EVAL script, which will be executed by the Redis server itself.
I don't know much about Go, so I can't write the syntax for calling it. But I know what the Lua script would look like:
for i = 1, ARGV[1] do
local key = "key:" .. tostring(i)
redis.call('SET', key, i)
end
Put that in a Go string, then pass that to the appropriate call, with no key arguments and a single non-key argument that is the number of times to loop.
I stumbled on this thread and was also curious of the benchmark results. I wrote a quick Ruby script to compare them. The script does a simple "SET/GET" operation on the same key using different options.
require "redis"
def elapsed_time(name, &block)
start = Time.now
block.call
puts "#{name} - elapsed time: #{(Time.now-start).round(3)}s"
end
iterations = 100000
redis_key = "test"
redis = Redis.new
elapsed_time "Scenario 1: From client" do
iterations.times { |i|
redis.set(redis_key, i.to_s)
redis.get(redis_key)
}
end
eval_script1 = <<-LUA
redis.call("SET", "#{redis_key}", ARGV[1])
return redis.call("GET", "#{redis_key}")
LUA
elapsed_time "Scenario 2: Using EVAL" do
iterations.times { |i|
redis.eval(eval_script1, [redis_key], [i.to_s])
}
end
elapsed_time "Scenario 3: Using EVALSHA" do
sha1 = redis.script "LOAD", eval_script1
iterations.times { |i|
redis.evalsha(sha1, [redis_key], [i.to_s])
}
end
eval_script2 = <<-LUA
for i = 1,#{iterations} do
redis.call("SET", "#{redis_key}", tostring(i))
redis.call("GET", "#{redis_key}")
end
LUA
elapsed_time "Scenario 4: Inside EVALSHA" do
sha1 = redis.script "LOAD", eval_script2
redis.evalsha(sha1, [redis_key], [])
end
eval_script3 = <<-LUA
for i = 1,2*#{iterations} do
redis.call("SET", "#{redis_key}", tostring(i))
redis.call("GET", "#{redis_key}")
end
LUA
elapsed_time "Scenario 5: Inside EVALSHA with 2x the operations" do
sha1 = redis.script "LOAD", eval_script3
redis.evalsha(sha1, [redis_key], [])
en
I got the following results running on my Macbook pro
Scenario 1: From client - elapsed time: 11.498s
Scenario 2: Using EVAL - elapsed time: 6.616s
Scenario 3: Using EVALSHA - elapsed time: 6.518s
Scenario 4: Inside EVALSHA - elapsed time: 0.241s
Scenario 5: Inside EVALSHA with 2x the operations - elapsed time: 0.5s
In summary:
scenario 1 vs. scenario 2 show that the main contributor is the round trip time as scenario 1 makes 2 requests to Redis while scenario 2 only makes 1 and scenario 1 is ~2x the execution time
scenario 2 vs. scenario 3 shows that EVALSHA does provide some benefit and I am sure this benefit increases the more complex the script gets
scenario 4 vs scenario 5 shows the overhead of invoking the script is near minimal as we doubled the number of operations and saw a ~2x increase in execution time.
so there is now a workaround using a module created by John Sully. It works for Redis and KeyDB and allows you to use the V8 JIT engine which runs complex scripts much faster than Lua scripts. https://github.com/JohnSully/ModJS
I am a beginner to lua scripting. I tried executing the below command in redis client
eval "print('hello world')" 0
and it works fine returning me
hello
(nil)
Firstly i don't understand why does the nil is returned/gets printed always?? is it because of the lua-datatype to redis-datatype inter-conversion which returns nil??
Then i used if constructs and executed the below script
eval "local t = 0; if t == 0 then print('hello'); end" 0 0
the above command prints as expected
hello
(nil)
But when i tried assigning the arguments to the local variable using the below code
eval "local t = ARGV[1]; if t == 0 then print('hello'); end" 0 0
the output is only
(nil)
So can someone explain me how should the assignments of the external arguments to the local variable be carried out??
The redis eval command does nothing more than split your input into tokens, hence your argument 0 is a string, and not a number.
However, the comparison 0 == "0" is false in Lua, so what you need to do is change to either of those variants:
eval "local t = ARGV[1]; if t == '0' then print('hello'); end" 0 0
eval "local t = ARGV[1]; if tonumber(t) == 0 then print('hello'); end" 0 0
Note: Since is was kind of mentioned by Kamiccolo; note that in Lua's eval you never have access to local variables from outside of the eval because code is always evaluated in the global context!
According to Redis EVAL documentation, global variable access from chunk executed using eval() to global variables is disabled. It's not mentioned, but I suspect that accessing local ones is even bigger violation and possible leakage. So, You should follow official documentation and use Redis keys instead. Or, of course, re-using returned valies.
Relevant chunk from eval() docs:
Global variables protection
Redis scripts are not allowed to create global variables, in order to avoid leaking data into the Lua state. If a script needs to maintain state between calls (a pretty uncommon need) it should use Redis keys instead.
Just realised, that question's body does not match the question's title, so starting again...
Proper eval() usage (from Redis docs):
The first argument of EVAL is a Lua 5.1 script. The script does not need to define a Lua function (and should not). It is just a Lua program that will run in the context of the Redis server.
The second argument of EVAL is the number of arguments that follows the script (starting from the third argument) that represent Redis key names. This arguments can be accessed by Lua using the KEYS global variable in the form of a one-based array (so KEYS[1], KEYS[2], ...).
All the additional arguments should not represent key names and can be accessed by Lua using the ARGV global variable, very similarly to what happens with keys (so ARGV[1], ARGV[2], ...).
In Your example You're setting number of arguments to 0, so, nothing gets passed to Your Lua script. Quick fix:
eval "local t = ARGV[1]; if t == 0 then print('hello'); end" 1 0
Not sure about nil, but that might be just because Your Lua chunk provided does not return anyhing.