I am using subprocess32 3.2.6 with Python 2.6.6 on RHEL 6.5. A sequence like:
command = "sleep 20"
proc = subprocess.Popen(command, shell=True, bufsize=-1, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
std_out, std_err = proc.communicate(None, timeout=1)
works as expected, ie the one second timeout works. However, if
command = "sleep 20; echo Hello World"
The subprocess seems to run for the entire 20 seconds. I can work around this but it would be nice to either understand what I am doing wrong or why it works the way it does. BTW, this is in a very controlled, trusted environment so the "shell=True" is not risky.
I tried every thing, Just use this:
def reader(f,buffer):
while True:
line=f.readline()
if line:
buffer.append(line)
else:
break
command = subprocess.Popen (cmd, shell=True, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, stdin=subprocess.PIPE)
command.stdin.write (bytearray(testin, 'utf8'))
command.stdin.close ()
print ('Writing i/p')
linebuffer = []
t = Thread (target=reader, args=(command.stdout, linebuffer))
t.daemon = True
t.start ()
start = time.time ()
while start + timeout > time.time ():
if linebuffer:
ans += linebuffer.pop ()
Related
I am trying to solve the problem statement Jigsaw-multilingual-toxic-comment-classification but when I tried to preprocess the entire file ,I got the [Errorno 30].
I also tried to solve it by:
!cp -r /kaggle/input/jigsaw-multilingual-toxic-comment-classification /kaggle/working
Even then I am not able to solve the error.
Can someone please help me out solve this error?
The first code block defines the global variables.
!cp -r /kaggle/input/jigsaw-multilingual-toxic-comment-classification /kaggle/working
SEQUENCE_LENGTH = 128
DATA_PATH = "/kaggle/working/jigsaw-multilingual-toxic-comment-classification"
BERT_PATH = "../input/bert-multi"
BERT_PATH_SAVEDMODEL = os.path.join(BERT_PATH, "/kaggle/input/bert-multi/bert_multi_from_tfhub")
OUTPUT_PATH = "/kaggle/working"
The below code block is the block afer which I am getting the error
def preprocess_and_save_dataset(unprocessed_filename, text_label='comment_text',
seq_length=SEQUENCE_LENGTH, verbose=True):
"""Preprocess a CSV to the expected TF Dataset form for multilingual BERT,
and save the result."""
dataframe = pd.read_csv(os.path.join(DATA_PATH, unprocessed_filename),
index_col='id')
processed_filename = (unprocessed_filename.rstrip('.csv') +
"-processed-seqlen{}.csv".format(SEQUENCE_LENGTH))
pos = 0
start = time.time()
while pos < len(dataframe):
processed_df = dataframe[pos:pos + 10000].copy()
processed_df['input_word_ids'], processed_df['input_mask'], processed_df['all_segment_id'] = (
zip(*processed_df[text_label].apply(preprocess_sentence)))
if pos == 0:
processed_df.to_csv(processed_filename, index_label='id', mode='w')
else:
processed_df.to_csv(processed_filename, index_label='id', mode='a',
header=False)
if verbose:
print('Processed {} examples in {}'.format(
pos + 10000, time.time() - start))
pos += 10000
return
# Process the training dataset.
preprocess_and_save_dataset(wiki_toxic_comment_data)
I tried to move the directory to the Kaggle working folder.
I use google colab for execution.
In the following code, as all the arguments are passed by default. I can pass the args as an empty list, which can be seen from the last line:
DEFAULT_ENV_NAME = "PongNoFrameskip-v4"
MEAN_REWARD_BOUND = 19.5
parser.add_argument("--cuda", default=True, action="store_true", help="Enable cuda")
parser.add_argument("--env", default=DEFAULT_ENV_NAME,
help="Name of the environment, default=" + DEFAULT_ENV_NAME)
parser.add_argument("--reward", type=float, default=MEAN_REWARD_BOUND,
help="Mean reward boundary for stop of training, default=%.2f" % MEAN_REWARD_BOUND)
args = parser.parse_args(args = [])
when I use print(args) I got:
Namespace(cuda=True, env='PongNoFrameskip-v4', reward=19.5)
Also when I execute the following code:
DEFAULT_ENV_NAME = "PongNoFrameskip-v4"
parser.add_argument("-m", "--model", required=True, help="Model file to load")
parser.add_argument("-e", "--env", default=DEFAULT_ENV_NAME,
help="Environment name to use, default=" + DEFAULT_ENV_NAME)
parser.add_argument("-r", "--record", help="Directory to store video recording")
parser.add_argument("--no-visualize", default=True, action='store_false', dest='visualize',
help="Disable visualization of the game play")
args = parser.parse_args(args=['dqn_model'])
I am getting errors in syntax, which is because of the last line. I need to send required values for model argument (say dqn_model).
What is the correct syntax for passing the arguments?
I tried the following which gave me errors:
args = parser.parse_args(args=['dqn_model'])
args = parser.parse_args(args=[model='dqn_model'])
args = parser.parse_args(args=[m='dqn_model'])
this worked out for me
use :
args = argparse.Namespace(cuda="Argument=required",env="PongNoFrameskip-v4",reward=19.5)
make sure you have added all the argument keys with their respective value.
Hello people of stackoverflow, I've made a (very) simple programming language, it kind of looks like minecraft commands. Here is the code
function wait(cmdSleepGetNum)-- you can name this function
--what ever you want, it doesnt matter
local start = os.time()
repeat until os.time() > start + cmdSleepGetNum
end
local input = io.read()
if input == "/help" then
print"you'll find it out"
else if input == "/say" then
print"what do you want to say?"
local cmdSay = io.read()
print(cmdSay)
else if input == "/stop" then
os.exit()
else if input == "/sleep" then
print"how long?"
local cmdSleepGetNum = io.read()
wait(cmdSleepGetNum)
else if input == "/rand" then
local randNum = math.random()
print(randNum)
end
end
end
end
end
Now I know what you are thinking "what is the problem here?" the problem is that I can only execute one command and after that command is finished by the Lua interpreter, I cannot execute any other commands.
for example:
/rand
0.84018771715471 (the /rand command is executed and prints out a random number)
/rand
stdin:1: unexpected symbol near '/' (this happens when i try to execute another command)
If you replace else if with elseif, you won't need so many end's,
You can get rid of if's altogether, is you use a table,
As #Egor Skriptunoff said, you need to create a main loop to run many commands,
I suggest that you add an optional argument to /sleep and /say.
local function wait (cmdSleepGetNum) -- you can name this function
-- whatever you want, it doesn't matter.
local start = os.time ()
repeat until os.time () > start + cmdSleepGetNum
end
local commands = {
help = function ()
print "you'll find it out"
end,
say = function (arg)
if arg == '' then
print 'what do you want to say?'
arg = io.read ()
end
print (arg)
end,
stop = function ()
os.exit ()
end,
sleep = function (arg)
if arg == '' then
print 'how long?'
arg = tonumber (io.read ())
end
wait (tonumber (arg))
end,
rand = function ()
local randNum = math.random ()
print (randNum)
end,
[false] = function () -- fallback.
print 'Unknown command'
end
}
-- Main loop:
while true do
io.write '> '
local key, _, arg = io.read ():match '^%s*/(%S+)(%s*(.*))$' -- you can type /sleep 1, etc. in one line.
local command = key and key ~= '' and commands [key] or commands [false]
command (arg)
end
I have the following code to capture and process the Run command output.
How do I modify it such that the Run command window displays output and at the same time the output gets logged? Replacing #SW_HIDE with #SW_SHOW (or the equivalent) just shows a blank command window.
Something similar to the linux tee command which logs to file while it prints STDOUT.
$CurrentPID = Run(#ComSpec & ' /c ' & $CurrentLogCmd, "", #SW_HIDE, $STDERR_CHILD + $STDOUT_CHILD)
If Not ProcessWaitClose($CurrentPID,60) Then
WriteLog("[Warning] Timed-out.Finding date in current hour raw log -" & $CurrentLogFileName)
$F_LogWarningExist = 1
Return $C_SUCCESS ; Take chances and proceed with parsing raw logs
EndIf
$CurrentOutput = StdoutRead($CurrentPID)
ConsoleWrite(_getDOSOutput('ipconfig /all') & #CRLF)
Func _getDOSOutput($command)
Local $text = '', $Pid = Run('"' & #ComSpec & '" /c ' & $command, '', #SW_HIDE, 2 + 4)
While 1
$text &= StdoutRead($Pid, False, False)
If #error Then ExitLoop
Sleep(10)
WEnd
Return StringStripWS($text, 7)
EndFunc ;==>_getDOSOutput
Maybe this helps you out.
Something similar to the linux tee command which logs to file while it prints STDOUT.
Redirecting STDOUT makes receiving party (the script) responsible for display and logging. As per documentation:
StdoutRead() does not block, it will return immediately. In order to get all data, it must be called in a loop.
Example:
#AutoIt3Wrapper_Change2CUI=Y
#include <Constants.au3>
#include <MsgBoxConstants.au3>
Global Enum $EXIT_OK, _
$EXIT_NOCOMPILE, _
$EXIT_ABORT
Global Const $g_sPromptError = 'Compile this script and run resulting executable instead.', _
$g_sPromptInput = 'Enter a command:', _
$g_sInputDefault = 'ping localhost -n 10'
Global $g_sCMD = '', _
$g_sSTD = ''
Main()
Func Main()
If Not #Compiled Then
MsgBox($MB_OK + $MB_ICONERROR, #ScriptName, $g_sPromptError)
Exit $EXIT_NOCOMPILE
EndIf
$g_sCMD = InputBox(#ScriptName, $g_sPromptInput, $g_sInputDefault)
If #error Then Exit $EXIT_ABORT
$g_sSTD = _getCmdStd($g_sCMD)
MsgBox($MB_OK + $MB_ICONINFORMATION, $g_sCMD, $g_sSTD)
Exit $EXIT_OK
EndFunc
Func _getCmdStd(Const $sCMD, Const $sDir = '', Const $iType = $STDERR_MERGED, Const $bShow = False, Const $iDelay = 100)
Local $sTMP = ''
Local $sSTD = ''
Local $sCOM = #ComSpec & ' /c ' & $sCMD
Local Const $iWin = $bShow ? #SW_SHOW : #SW_HIDE
Local Const $iPID = Run($sCOM, $sDir, $iWin, $iType)
While True
$sTMP = StdoutRead($iPID, False, False)
If #error Then
ExitLoop 1
ElseIf $sTMP Then
$sTMP = StringReplace($sTMP, #CR & #CR, '')
$sSTD &= $sTMP
ConsoleWrite($sTMP)
EndIf
Sleep($iDelay)
WEnd
Return SetError(#error, #extended, $sSTD)
EndFunc
Returns STDOUT (and STDERR) after execution completes, while writing to console during execution. Replace MsgBox() as required (logging function).
As part of debugging a problem that might be related to my UIVIews, I want to write a python script to run from LLDB. I had thought to extract all settings for a view in a breakpoint and all view children, to allow me to compare states. I checked out the WWDC video on the topic and then spent time reading things at lldb.llvm.org/scripting.html, and didn't find them very helpful. A web search for examples led to nothing substantially different from those.
My problem is that I'm trying to figure out how to access iOS variables at my breakpoint. The examples I've seen do things like convert numbers and mimic shell commands. Interesting stuff but not useful for my purposes. I've been reading my way through the help info with "script help(lldb.SBValue)" and the like, but it is slow going as the results are huge and it is not clear what the use patterns are. I feel like one decent example of how to traverse a few iOS objects would help me understand the system. Does anyone know of one or can share a snippet of code?
UPDATE:
I wrote this to help me track down a bug in my UIView use. I want to do a bit more work to refine this to see if I could show the whole view tree, but this was sufficient to solve my problem, so I'll put it here to save others some time.
import lldb
max_depth = 6
filters = {'_view':'UIView *', '_layer':'CALayer *', '_viewFlags':'struct'}
def print_value(var, depth, prefix):
""" print values and recurse """
global max_depth
local_depth = max_depth - depth
pad = ' ' * local_depth
name = var.GetName()
typ = str(var.GetType()).split('\n')[0].split('{')[0].split(':')[0].strip()
found = name in filters.keys() # only visit filter items children
if found:
found = (filters.get(name) == typ)
value = var.GetValue()
if value is None or str(value) == '0x00000000':
value = ''
else:
value = ' Val: %s' % value
if var.GetNumChildren() == 0 and var.IsInScope():
path = lldb.SBStream()
var.GetExpressionPath(path)
path = ' pathData: %s' % path.GetData()
else:
path = ''
print '^' * local_depth, prefix, ' Adr:', var.GetAddress(), ' Name:', name, ' Type:', typ, value, path
if var.GetNumChildren() > 0:
if local_depth < 2 or found:
print pad, var.GetNumChildren(), 'children, to depth', local_depth + 1
counter = 0
for subvar in var:
subprefix = '%d/%d' % (counter, var.GetNumChildren())
print_value(subvar, depth - 1, subprefix)
counter += 1
def printvh (debugger, command_line, result, dict):
""" print view hierarchy """
global max_depth
args = command_line.split()
if len(args) > 0:
var = lldb.frame.FindVariable(args[0])
depth = max_depth
if len(args) > 1:
depth = int(args[1])
max_depth = depth
print_value(var, depth, 'ROOT')
else:
print 'pass a variable name and optional depth'
And I added the following to my .lldbinit :
script import os, sys
# So that files in my dir takes precedence.
script sys.path[:0] = [os.path.expanduser("~/lldbpy")]
script import views
command script add -f views.printvh printvh
so that I can just type "printvh self 3" at the LLDB prompt.
Maybe this will help. Here's an example of how to dump simple local variables when a breakpoint is hit. I'm not displaying char* arrays correctly, I'm not sure how I should get the data for these to display it like "frame variable" would display it but I'll figure that out later when I have a free minute.
struct datastore {
int val1;
int val2;
struct {
int val3;
} subdata;
char *name;
};
int main (int argc, char **argv)
{
struct datastore data = {1, 5, {3}, "a string"};
return data.val2;
}
Current executable set to 'a.out' (x86_64).
(lldb) br se -l 13
Breakpoint created: 1: file ='a.c', line = 13, locations = 1
(lldb) br comm add -s python
Enter your Python command(s). Type 'DONE' to end.
> def printvar_or_children(var):
> if var.GetNumChildren() == 0 and var.IsInScope():
> path = lldb.SBStream()
> var.GetExpressionPath(path)
> print '%s: %s' % (path.GetData(), var.GetValue())
> else:
> for subvar in var:
> printvar_or_children(subvar)
>
> print 'variables visible at breakpoint %s' % bp_loc
> for var in frame.arguments:
> printvar_or_children(var)
> for var in frame.locals:
> printvar_or_children(var)
>
> DONE
(lldb) r
variables visible at breakpoint 1.1: where = a.out`main + 51 at a.c:13, address = 0x0000000100000f33, resolved, hit count = 1
argc: 1
*(*(argv)): '/'
data.val1: 1
data.val2: 5
data.subdata.val3: 3
*(data.name): 'a'
Process 84865 stopped
* thread #1: tid = 0x1f03, 0x0000000100000f33 a.out`main + 51 at a.c:13, stop reason = breakpoint 1.1
frame #0: 0x0000000100000f33 a.out`main + 51 at a.c:13
10 int main (int argc, char **argv)
11 {
12 struct datastore data = {1, 5, {3}, "a string"};
-> 13 return data.val2;
(lldb)
Tip - for sanity's sake I worked on the python over in a side text editor and pasted it into lldb as I experimented.
If you use the frame variable command in lldb to explore your variables at a given stop location, that's the same basic way that you can access them via the SBFrame that is provided to your breakpoint python command in the 'frame' object.
Hope that helps to get you started.
Did you try looking at the python LLDB formatting templates stored in:
XCode.app/Contents/SharedFrameworks/LLDB.framework/Resources/Python/lldb/formatters/objc