I'm attempting to constantly read and parse a log file (Minecraft log file) by using io.popen in tandem with Ubuntu's tail command so that I can send some messages upon certain events.
Now, I have mostly everything working here, except one small issue. After a while of reading, the entire program just freezes.
Here is the relevant code:
-- Open the tail command, return a file handle for it.
local pop = io.popen(config.listen_command)
-- Simply read a single line, I've pulled this into its own
-- function so that if this ever needs changing I can do so
-- easily.
local function get_line()
logger:log(4, "READ LINE")
return pop:read("*l")
end
-- For each line in the log file, check if it matches any
-- of a list of patterns, return the matches and the
-- pattern information if so.
local function match_line()
local line = get_line()
logger:log(4, "Line: %s", line)
-- This all works, and I've tested that it's not freezing
-- here. I've just included it for completion of the call
-- -stack.
for event_type, data in pairs(config.message_patterns) do
for event_name, pattern in pairs(data) do
local matches = {line:match(pattern)}
if matches[1] then
return event_type, event_name, matches
end
end
end
end
-- The main loop, simply read a line and send a message
-- if there was a match.
logger:log(4, "Main loop begin.")
while true do
local event_type, event_name, matches = match_line()
-- ...
-- The rest of the code here is not relevant.
config.listen_command = "tail -F --lines=1 latest.log"
The issue is in the get_line function. After a while of reading the log file, it completely freezes on the pop:read("*l"). It prints the READ LINE message, but never prints the Line: <whatever data here> line.
This is a really strange issue that I've been getting really confused about. I've tried swapping to different distributions of Lua (Luvit, LuaJIT, Lua) and a very large amount of debugging, changing small things, rerunning, ... But I cannot think of anything that'd be causing this.
Perhaps there's something small I've missed.
So my question here is this: Why is pop:read("*l") freezing, even though more data is being outputted to the logfile? Is there a way to fix this? Perhaps to detect if the next read will freeze indefinitely, so I can try closing the popen'ed file and re-open it (or to preferably stop it happening altogether?)
Related
I am learning how work with files in Lua, and I came across a problem.
I know that:
io.output() allows me to specify the file that I will write output to.
The io.write() function simply gets a number of string arguments and writes them to the current output file.
io.close() then closes the file I am reading or writing on.
I first set io.output() to "myNewFile.txt" then I did io.write("Hello World!").
io.output("myNewFile.txt")
io.write("Hello World!")
This part worked well and myNewFile.txt got Hello World written to it.
Then I wrote io.close() to close the myNewFile.txt.
After that, I wrote io.write("Hello World!"), but I got an error:
C:\\Program Files\\lua\\lua54.exe: ...OneDrive\\Documents\\learning lua\\steve's teacher\\main.lua:9: default output file is closed stack traceback: \[C\]: in function 'io.write' ...OneDrive\\Documents\\learning lua\\steve's teacher\\main.lua:9: in main chunk \[C\]: in ?
I wanted io.write("Hello World!") to write Hello World in the terminal. I know can use print() but print() function adds extra characters like \n and things like that.
So my question is, how do I write Hello World in the terminal using io.write("Hello World!") in this situation?
I tried to search this error up on Google, but there weren't any results. I also tried joining many Discord servers, but I didn't get a proper response. I am a new developer to Lua so this all is really confusing to me.
After io.close() do a io.output(io.stdout) to set it back.
The clean way
local oldout = io.output() -- without argument returns actual output
io.output("file") -- set to a filename
-- do file operations here
-- use io.flush() where it is necessary
io.close() -- close file also do a flush()
io.output(oldout) -- set back to oldout ( normally io.stdout )
-- changing io.output() back should also close() and flush() the "file"
-- But better doublecheck this when using userdata without experience
I'm currently trying to make a level loading system for a game.
function love.filedropped(file)
ofile=io.open(file:getFilename(),"r")
io.input(ofile)
file:close
levelc=io.read()
for i=1,levelc do
levels[i]=io.read()
print levels[i]
end
levelc should be the first line of the file, and file:getFilename is the file to open (path included) the project gives an error message on startup, and i've used a similar structure before, but for an output. The error is at line 30, which is the levelc=io.read().
I've tried changing the name of the file pointer (it was "f" before, now "ofile") and i've tried using io.read("*l") instead of io.read() but same result.
EDITS:
-this is a love.filedropped(file)
-i need to open other files from a .txt later and i don't really understand how do do that
The parameter given by love.filedropped is a DroppedFile.
In your case helpful could be File:lines().
For example:
function love.filedropped(file)
-- Open for reading
file:open("r")
-- Iterate over the lines
local i = 0
for line in file:lines() do
i = i + 1
levels[i] = line
print(i, levels[i]) -- Notice the parentheses missing in your code
end
-- Close the file
file:close()
end
Notice that love2d usually only allows reading/writing files within the save or working directory. Dropped files are an exception.
Unrelated to this answer but things I noticed in your code:
Use locals, oFile should be local
file:close() required parentheses as its a function call
Same for the print
The filedropped callback has no end
You mentioned reading other files too, to do so, you can either:
Use love.filesystem.newFile and a similar approach as before
The recommended one-liner love.filesystem.lines
Although I am quite familiar with Tcl this is a beginner question. I would like to read and write from a pipe. I would like a solution in pure Tcl and not use a library like Expect. I copied an example from the tcl wiki but could not get it running.
My code is:
cd /tmp
catch {
console show
update
}
proc go {} {
puts "executing go"
set pipe [open "|cat" RDWR]
fconfigure $pipe -buffering line -blocking 0
fileevent $pipe readable [list piperead $pipe]
if {![eof $pipe]} {
puts $pipe "hello cat program!"
flush $pipe
set got [gets $pipe]
puts "result: $got"
}
}
go
The output is executing go\n result:, however I would expect that reading a value from the pipe would return the line that I have sent to the cat program.
What is my error?
--
EDIT:
I followed potrzebie's answer and got a small example working. That's enough to get me going. A quick workaround to test my setup was the following code (not a real solution but a quick fix for the moment).
cd /home/stephan/tmp
catch {
console show
update
}
puts "starting pipe"
set pipe [open "|cat" RDWR]
fconfigure $pipe -buffering line -blocking 0
after 10
puts $pipe "hello cat!"
flush $pipe
set got [gets $pipe]
puts "got from pipe: $got"
Writing to the pipe and flushing won't make the OS multitasking immediately leave your program and switch to the cat program. Try putting after 1000 between the puts and the gets command, and you'll see that you'll probably get the string back. cat has then been given some time slices and has had the chance to read it's input and write it's output.
You can't control when cat reads your input and writes it back, so you'll have to either use fileevent and enter the event loop to wait (or periodically call update), or periodically try reading from the stream. Or you can keep it in blocking mode, in which case gets will do the waiting for you. It will block until there's a line to read, but meanwhile no other events will be responded to. A GUI for example, will stop responding.
The example seem to be for Tk and meant to be run by wish, which enters the event loop automatically at the end of the script. Add the piperead procedure and either run the script with wish or add a vwait command to the end of the script and run it with tclsh.
PS: For line-buffered I/O to work for a pipe, both programs involved have to use it (or no buffering). Many programs (grep, sed, etc) use full buffering when they're not connected to a terminal. One way to prevent them to, is with the unbuffer program, which is part of Expect (you don't have to write an Expect script, it's a stand-alone program that just happens to be included with the Expect package).
set pipe [open "|[list unbuffer grep .]" {RDWR}]
I guess you're executing the code from http://wiki.tcl.tk/3846, the page entitled "Pipe vs Expect". You seem to have omitted the definition of the piperead proc, indeed, when I copy-and-pasted the code from your question, I got an error invalid command name "piperead". If you copy-and-paste the definition from the wiki, you should find that the code works. It certainly did for me.
I wrote a admin script that tails a heroku log and every n seconds, it summarizes averages and notifies me if i cross a certain threshold (yes I know and love new relic -- but I want to do custom stuff).
Here is the entire script.
I have never been a master of IO and threads, I wonder if I am making a silly mistake. I have a couple of daemon threads that have while(true){} which could be the culprit. For example:
# read new lines
f = File.open(file, "r")
f.seek(0, IO::SEEK_END)
while true do
select([f])
line = f.gets
parse_heroku_line(line)
end
I use one daemon to watch for new lines of a log, and the other to periodically summarize.
Does someone see a way to make it less processor-intensive?
This probably runs hot because you never really block while reading from the temporary file. IO::select is a thin layer over POSIX select(2). It looks like you're trying to block until the file is ready for reading, but select(2) considers EOF to be ready ("a file descriptor is also ready on end-of-file"), so you always return right away from select then call gets which returns nil at EOF.
You can get a truer EOF reading and nice blocking behavior by avoiding the thread which writes to the temp file and instead using IO::popen to fork the %x[heroku logs --ps router --tail --app pipewave-cedar] log tailer, connected to a ruby IO object on which you can loop over gets, exiting when gets returns nil (indicating the log tailer finished). gets on the pipe from the tailer will block when there's nothing to read and your script will only run as hot as it takes to do your line parsing and reporting.
EDIT: I'm not set up to actually try your code, but you should be able to replace the log tailer thread and your temp file read loop with this code to get the behavior described above:
IO.popen( %w{ heroku logs --ps router --tail --app my-heroku-app } ) do |logf|
while line = logf.gets
parse_heroku_line(line) if line =~ /^/
end
end
I also notice your reporting thread does not do anything to synchronize access to #total_lines, #total_errors, etc. So, you have some minor race conditions where you can get inconsistent values from the instance vars that parse_heroku_line method updates.
select is about whether a read would block. f is just a plain old file, so you when get to the end reads don't block, they just return nil instantly. As a result select returns instantly rather than waiting for something to be appending to the file as I assume you're expecting. Because of this you're sitting in a tight busy loop, so high cpu is to be expected.
If you are at eof (you could either check f.eof? or whether gets returns nil), then you could either start sleeping (perhaps with some sort of back off) or use something like listen to be notified of filesystem changes
I need to use Lua to run a binary program that may write something in its stdout and also returns a status code (also known as "exit status").
I searched the web and couldn't find something that does what I need. However I found out that in Lua:
os.execute() returns the status code
io.popen() returns a file handler that can be used to read process output
However I need both. Writing a wrapper function that runs both functions behind the scene is not an option because of process overhead and possibly changes in result on consecutive runs. I need to write a function like this:
function run(binpath)
...
return output,exitcode
end
Does anyone has an idea how this problem can be solved?
PS. the target system rung Linux.
With Lua 5.2 I can do the following and it works
-- This will open the file
local file = io.popen('dmesg')
-- This will read all of the output, as always
local output = file:read('*all')
-- This will get a table with some return stuff
-- rc[1] will be true, false or nil
-- rc[3] will be the signal
local rc = {file:close()}
I hope this helps!
I can't use Lua 5.2, I use this helper function.
function execute_command(command)
local tmpfile = '/tmp/lua_execute_tmp_file'
local exit = os.execute(command .. ' > ' .. tmpfile .. ' 2> ' .. tmpfile .. '.err')
local stdout_file = io.open(tmpfile)
local stdout = stdout_file:read("*all")
local stderr_file = io.open(tmpfile .. '.err')
local stderr = stderr_file:read("*all")
stdout_file:close()
stderr_file:close()
return exit, stdout, stderr
end
This is how I do it.
local process = io.popen('command; echo $?') -- echo return code of last run command
local lastline
for line in process:lines() do
lastline = line
end
print(lastline) -- the return code is the last line of output
If the last line has fixed length you can read it directly using file:seek("end", -offset), offset should be the length of the last line in bytes.
This functionality is provided in C by pclose.
Upon successful return, pclose() shall return the termination status
of the command language interpreter.
The interpreter returns the termination status of its child.
But Lua doesn't do this right (io.close always returns true). I haven't dug into these threads but some people are complaining about this brain damage.
http://lua-users.org/lists/lua-l/2004-05/msg00005.html
http://lua-users.org/lists/lua-l/2011-02/msg00387.html
If you're running this code on Win32 or in a POSIX environment, you could try this Lua extension: http://code.google.com/p/lua-ex-api/
Alternatively, you could write a small shell script (assuming bash or similar is available) that:
executes the correct executable, capturing the exit code into a shell variable,
prints a newline and terminal character/string onto standard out
prints the shell variables value (the exit code) onto standard out
Then, capture all the output of io.popen and parse backward.
Full disclosure: I'm not a Lua developer.
yes , your are right that os.execute() has returns and it's very simple if you understand how to run your command with and with out lua
you also may want to know how many variables it returns , and it might take a while , but i think you can try
local a, b, c, d, e=os.execute(-what ever your command is-)
for my example a is an first returned argument , b is the second returned argument , and etc.. i think i answered your question right, based off of what you are asking.