I want to run many Lua scripts one after another, without allowing any commands to run in between. I also need to pass the result of the first script to the second one, etc.
I've solved the problem temporarily by putting all my scripts in one file. However, the second script modifies a key returned by the first script. Because of this, putting everything in one file violates the EVAL command semantics as all the keys that the second script uses should be passed using the KEYS array.
Actually, it is possible. Redis has an undocumented feature that allows doing just that. The basic premise is that once you EVAL or SCRIPT LOAD a script, you can call that script from another one by invoking the function f_<sha1 hash> (where sha1 hash is the SHA1 hash of the first script).
Credit for this goes to Josiah Carlson (who, in turn, gives credit to Nathan Fritz). Dr. Josiah was kind enough to provide all the details in here (this file is a part of a Python package that helps managing Lua scripts that call other scripts).
You cannot do that. However violating EVAL semantics this way should not be a problem as long as you do not use Redis Cluster.
Related
My plan is to store some existing Redis keys in a hash, that would be later fetched from within a Redis Lua script and acted upon. I read that it is best practice to provide all keys that are used in a script when invoking EVAL.
My question is, is it safe to run a script that doesn't have any keys provided when running EVAL, but operates on some keys that are fetched from within the script? If not, what are the consequences and how can I avoid this shortcoming?
I mention that at the time of EVAL, there is no way of knowing what the keys in that specific hash are. I can get all the keys from the hash in a step prior to EVAL, then provide them to EVAL, but that sounds like overkill.
Because of the ambiguity in the documentation this question gets asked a lot (see this and this, for example), but I'll use this occasion to try and specify the actual rule governing the use of keys in Redis scripts.
The actual rule is this: Your script must only access keys that reside on a single sever, and Redis must know what that server is so that it can route the script there.
If you're not using Redis Cluster then these conditions will always be met, since there's only one server.
If you are using Redis Cluster, then you must specify at least one key, and all keys that the script uses must be located on the same server as the specified keys.
So to answer your question: your script will work fine if you're not using Cluster. The consequences are that it won't work if you later switch to Cluster, since Redis won't know where to send the script.
It's also possible that Redis could change to enforcing the documented rule, rendering invalid all scripts that dynamically generate keys. I think that is very unlikely to happen. But specifying the keys is useful as documentation, if nothing else, so I always do it if I happen to know the keys in advance.
Suppose I have a shell script stored in a GCS bucket. Is it possible to execute it using Apache Beam? If yes, then how?
I haven't currently tried anything for it yet as I couldn't find anything of this sort in the documentations for Apache Beam or Dataflow. So just wanted to know what approach I must take for it.
Thanks.
It's unusual, but not unheard of to want to execute a whole shell script from something like a DoFn. Is this what you want to do? Do you want to run it once for each element in a PCollection?
If so, you'd want to use the GCS API, or the FileSystems API to obtain the whole contents of the shell script into a String or byte array, and then pass it as a side input into your ParDo.
Then you can execute it using a tool like subprocess in Python, or ProcessBuilder in Java.
Let me know if you need something more specific, and we can iterate a solution.
I am writing a script (a Rails runner, if it matters) that will run periodically. It uses a gem to query a SQL database. Because that database does not update existing rows, but merely adds new ones to reflect changes to data, the script will run a query that only finds objects with an id greater than the id that was in the database the last time the script was run.
In what file should that id be stored? Is it bad practice to store it in the script and have the script write over itself, and if so, why?
Store the ID in a separate file. Not only would the script writing over itself be more difficult to do correctly, but that practice would also be likely to confuse users, and could result in a whole host of other problems, such as additional friction when trying to version control the script or update it to a new version.
Under most circumstances, data and code should be separate.
I'm interested in using a lua script to post process elements being pushed to Redis list.
So what I want to know is if there is a way to tell Redis:
"Always run lua script X after a value is pushed into list Y"
No, you can't do that. Unless you make your own command (in lua), which will call LPUSH and perform your post-processing. You have to also make all developers use that command instead of original LPUSH (when corresponding keys are concerned). Good luck on that :)
Very new to Ruby so please try to look past my ignorance. Cause I have no idea what I am talking about currently. However I know the ability to do what I want exists. Essentially I have some JAVA server side that can be used via a command line. I am trying to figure out where and how to begin with communicating in the same notion of me typing it out in the cli without actually typing it out to the cli. Basicly I want to pass the commands like as if I was using the CLI but Im not. Does that make sense?
Its for a CLI to UI conversion. I have seen the process done RoR to JAVA in such a fashion but where to begin I couldn't tell ya to save my life.
First of all, I would suggest at least looking into jRuby, which can interact with java classes as though they were ruby classes.
If you still want the cli integration, the naive approach is extremely simple, all you need to do is wrap your cli command in backticks (`) and it will execute the command as if you typed it into a shell, and return the results as a string.
If you need to do this very frequently, check out https://github.com/rtomayko/posix-spawn which is a much more efficient way of doing it then the backtick approach.
If the Java program has a command prompt of its own, look into popen. It allows you to open a subprocess as an I/O stream allowing you to send it input and read its output. If all you need is to start the process and get its output then use backticks as suggested by Matt Briggs:
output = `the-command-to-start-the-java-program`