I'm developing a system in python, and one functionality I need is the ability to have console output go to both the console and a user-specified file. This is replicating the Diary function in MATLAB. I have the following that works perfectly well on both IDLE on windows and python cmdline in ubuntu (this all exists inside a module that gets loaded):
class diaryout(object):
def __init__(self):
self.terminal = sys.stdout
self.save = None
def __del__(self):
try:
self.save.flush()
self.save.close()
except:
# do nothing, just catch the error; maybe it self was instantiated, but never opened
1/1
self.save = None
def dclose(self):
self.__del__()
def write(self, message):
self.terminal.write(message)
self.save.write(message)
def dopen(self,outfile):
self.outfile = outfile
try:
self.save = open(self.outfile, "a")
except Exception, e:
# just pass out the error here so the Diary function can handle it
raise e
def Diary(outfile = None):# NEW TO TEST
global this_diary
if outfile == None:
# None passed, so close the diary file if one is open
if isinstance(this_diary, diaryout):
sys.stdout = this_diary.terminal # set the stdout back to stdout
this_diary.dclose() # flush and close the file
this_diary = None # "delete" it
else:
# file passed, so let's open it and set it for the output
this_diary = diaryout() # instantiate
try:
this_diary.dopen(outfile) # open & test that it opened
except IOError:
raise IOError("Can't open %s for append!"%outfile)
this_dairy=none # must uninstantiate it, since already did that
except TypeError:
raise TypeError("Invalid input detected - must be string filename or None: %s"%Diary.__doc__)
this_dairy=none # must uninbstantiate it, since already did that
sys.stdout = this_diary # set stdout to it
Far superior to both IDLE and the plain python cmline, I'm using ipython; herein my problem lies. I can turn on the "diary" perfectly fine with no error but the display on the console gets messed. The attached screenshot shows this . The output file also becomes similarly garbled. Everything goes back to normal when I undo the redirection with Diary(None). I have tried editing the code so that it never even writes to the file, with no effect. It seems almost like something is forcing an unsupported character set or something I don't understand.
Anyone have an idea about this?
Related
I'm using papermill in a web app, and running execute_notebook inside of celery tasks. I'm logging the output, and the entire notebook finishes, I get the export I'm waiting for in GCS, and it all seems perfect. but my execute_notebook statement never returns, so my celery task never finishes either.
Here's some pared down code:
def execute_nb(parameters: NotebookParameters, product_type: ProductType, notebook_name: str = None):
try:
nb_path = f"{settings.NOTEBOOK_DIR_PATH}/{product_type}.ipynb"
if notebook_name:
nb_path = f"{settings.NOTEBOOK_DIR_PATH}/{notebook_name}.ipynb"
nb_content = NOTEBOOK_REPO.get_contents(nb_path, ref=settings.NOTEBOOK_REF).decoded_content
except Exception as e:
print(e)
raise NotebookDoesNotExistException(path=nb_path)
# Make sure these local notebook folders exist locally
if not os.path.isdir(settings.NOTEBOOK_DIR_PATH):
os.makedirs(settings.NOTEBOOK_DIR_PATH)
if not os.path.isdir(f"{settings.NOTEBOOK_DIR_PATH}/outputs"):
os.makedirs(f"{settings.NOTEBOOK_DIR_PATH}/outputs")
# Writes the notebook from git to a local file (at the same relative path as git, from main.py)
open(nb_path, "wb").write(nb_content)
parameters_dict = json.loads(parameters)
pm.execute_notebook(
nb_path, f"{settings.NOTEBOOK_DIR_PATH}/outputs/{product_type}_output.ipynb", parameters=parameters_dict, log_output=True
)
print('done!')
return True
It never prints that done statement, so my celery task never finishes. The logs in my container show this:
data-layer-generation-service-worker-1 | [2022-11-16 01:48:43,496: WARNING/ForkPoolWorkerExecuting: 100%|##########| 7/7 [00:39<00:00, 6.36s/cell]
So it's reaching the end. Am I supposed to do something to trigger the end of execute_notebook?
This is my code (recompilation of How do I temporarily redirect stderr in Ruby? (which can't be used because of native extension writes):
def silence_stdout(log = '/dev/null')
orig = $stdout.dup
$stdout.reopen(File.new(log, 'w'))
begin
yield
ensure
$stdout = orig
end
end
silence_stdout('ttt.log') do
#do something
end
But I have a problem, the file is filled with the code only after the puma stops (Ctrl + C).
Probably should I close the file? But I do not understand how to do it. All my attempts to close the file end as "log writing failed. closed stream" or "no block given (yield)".
I ask for advice.
My question is divided in 2 sections
Firstly,
There are multiple ways to create a loggers (I using ActiveSupport::Logger) which inherit from Logger.
# Approach 1
logger = ActiveSupport::Logger.new('test.log')
logger.info "Test1" # => I see the auto flushing happens in this
# Approach 2
logger = ActiveSupport::Logger.new(File.new('test.log','a'))
logger.info "Test2" ## No auto flushing unless done logger.close
So, I don't see a point with approach 2 no auto flushing happen in them.
Why does the auto flushing does not happen in approach 2?
Secondly,
If I happen to delete my log file none of above approach(mention above) ever create a new log file again and also not log the log message.
What should I do in the situation like this? How does Rails does this? Any clue?
Autoflushing is a property of the IO object, not of the Logger. Logger (via LogDevice) sets f.sync = true when it opens the file, so if you want to autoflush to a custom file, you need to do that yourself:
> f = File.open('test.log', 'a')
> f.sync = true
> l = ActiveSupport::Logger.new(f)
When you delete the file, the Logger still has a reference to the old file descriptor, which no longer has a place on disk to write. If you try to lstat it, you'll see an error:
> f.lstat
Errno::ENOENT: No such file or directory # rb_file_lstat - test.log
If you notice this, you can reopen to get a new file on disk:
> f.reopen('test.log', 'a')
> f.lstat
#<File::Stat dev=0x1000002, ino=22095609, mode=0100644, nlink=1, uid=501, gid=20, rdev=0x0, size=0, blksize=4096, blocks=0, atime=2016-04-19 08:09:45 -0700, mtime=2016-04-19 08:09:44 -0700, ctime=2016-04-19 08:09:44 -0700, birthtime=2016-04-19 08:09:44 -0700>
I need to call a command(in a sinatra or rails app) like this:
`command sub`
Some log will be outputed when the command is executing.
I want to see the log displaying continuously in the process.
But I just can get the log string after it's done with:
result = `command sub`
So, is there a way to implement this?
On windows i have the best experience with IO.popen
Here is a sample
require 'logger'
$log = Logger.new( "#{__FILE__}.log", 'monthly' )
#here comes the full command line, here it is a java program
command = %Q{java -jar getscreen.jar #{$userid} #{$password}}
$log.debug command
STDOUT.sync = true
begin
# Note the somewhat strange 2> syntax. This denotes the file descriptor to pipe to a file. By convention, 0 is stdin, 1 is stdout, 2 is stderr.
IO.popen(command+" 2>&1") do |pipe|
pipe.sync = true
while str = pipe.gets #for every line the external program returns
#do somerthing with the capturted line
end
end
rescue => e
$log.error "#{__LINE__}:#{e}"
$log.error e.backtrace
end
There's six ways to do it, but the way you're using isn't the correct one because it waits for the process the return.
Pick one from here:
http://tech.natemurray.com/2007/03/ruby-shell-commands.html
I would use IO#popen3 if I was you.
In shell, I can do
$ cat name_of_file_with_a_lot_of_text | grep "What I am looking for"
Inside the Rails Console, can I achieve something similar, say when I run a command and the output is huge, especially say a DB query.
I am aware of outputting it as YAML but that Is not what I am looking for.
Thanks.
Yes, you can. The method is called gr... wait for it ...ep. Ruby's grep works on String, Array and many other built-in objects. For example to get all to_xxx methods of a number, just do:
1.methods.grep(/to_/)
I had the same question and wasn't very satisfied with the somewhat snarky response from ream88, so I decided to take a crack at it.
# Allows you to filter output to the console using grep
# Ex:
# def foo
# puts "Some debugging output here"
# puts "The value of x is y"
# puts "The value of foo is bar"
# end
#
# grep_stdout(/value/) { foo }
# # => The value of x is y
# # => The value of foo is bar
# # => nil
def grep_stdout(expression)
# First we need to create a ruby "pipe" which is two sets of IO subclasses
# the first is read only (which represents a fake $stdin) and the second is
# write only (which represents a fake $stdout).
placeholder_in, placeholder_out = IO.pipe
# This child process handles the grep'ing. Its done in a child process so that
# it can operate in parallel with the main process.
pid = fork {
# sync $stdout so we can report any matches asap
$stdout.sync
# replace $stdout with placeholder_out
$stdin.reopen(placeholder_in)
# we have to close both placeholder_out and placeholder_in because all instances
# of an IO stream must be closed in order for it to ever reach EOF. There's two
# in this method; one in the child process and one in the main process.
placeholder_in.close
placeholder_out.close
# loop continuously until we reach EOF (which happens when all
# instances of placeholder_out have closed)
read_buffer = ''
loop do
begin
read_buffer << $stdin.readpartial(4096)
if line_match = read_buffer.match(/(.*\n)(.*)/)
print line_match[1].grep(expression) # grep complete lines
read_buffer = line_match[2] # save remaining partial line for the next iteration
end
rescue EOFError
print read_buffer.grep(expression) # grep any remaining partial line at EOF
break
end
end
}
# Save the original stdout out to a variable so we can use it again after this
# method is done
original_stdout = $stdout
# Redirect stdout to our pipe
$stdout = placeholder_out
# sync $stdout so that we can start operating on it as soon as possible
$stdout.sync
# allow the block to execute and save its return value
return_value = yield
# Set stdout back to the original so output will flow again
$stdout = original_stdout
# close the main instances of placeholder_in and placeholder_out
placeholder_in.close
placeholder_out.close
# Wait for the child processes to finish
Process.wait pid
# Because the connection to the database has a tendency to go away when calling this, reconnect here
# if we're using ActiveRecord
if defined?(ActiveRecord)
suppress_stdout { ActiveRecord::Base.verify_active_connections! }
end
# return the value of the block
return_value
end
The obvious drawback of my solution is that the output is lost. I'm not sure how to get around that without calling yield twice.
EDIT I've changed my answer to only call fork once, which allows me to keep the output of the block and return it at the end. Win.
EDIT 2 You can get all of this functionality (and more!) in this gem now https://github.com/FutureAdvisor/console_util