Python 3 Popen.stdin.write encoding issues - character-encoding

I am attempting to use Popen to automate a simple telnet session. In python 2.6.5 the following code works:
openCmd = subprocess.Popen("telnet 192.168.1.1", shell=True, stdout=PIPE, stdin=PIPE)
time.sleep(1)
openCmd.stdin.write("username\r")
time.sleep(1)
openCmd.stdin.write("password\r")
time.sleep(1)
openCmd.stdin.write("some command\r")
openCmd.terminate()
In python 3 it complained of a type error, so I figured I just had to add .encode() to the end of each str object (as shown below). Adding the .encode() did fix the type error, and I don't get any exceptions, but the command I am trying to run on the remote machine doesn't get run.
openCmd = subprocess.Popen("telnet 192.168.1.1", shell=True, stdout=PIPE, stdin=PIPE)
time.sleep(1)
openCmd.stdin.write("username\r".encode())
time.sleep(1)
openCmd.stdin.write("password\r".encode())
time.sleep(1)
openCmd.stdin.write("some command\r".encode())
openCmd.terminate()
I also tried .encode("ascii") and .encode("UTF-8"). What am I doing incorrectly? I figure the issue is with the encoding, but I do not know for sure... I am running this program on a machine running Ubuntu 10.04.

Apparently all I needed to do was sleep on this one. It turns out that in Python 2.6.5 every
Popen.stdin.write()
Had a shorter delay before the buffer was flushed than Python 3 did! Here is the final working program:
def writeImmeadiatelyToPopen(openCmd, textToWrite):
openCmd.write(textToWrite.encode())
openCmd.flush()
openCmd = Popen("telnet 192.168.1.1", shell=True, stdout=PIPE, stdin=PIPE)
time.sleep(1)
writeImmeadiatelyToPopen(openCmd, "username\n")
time.sleep(1)
writeImmeadiatelyToPopen(openCmd, "password\n")
time.sleep(1)
writeImmeadiatelyToPopen(openCmd, "some command\n")
openCmd.terminate()
In case anyone wondered, I figured out how it worked by running Popen against 'cat' and carefully watching the output in Python 2.6.5 and Python 3 ^_^ .

Use python.pexpect to automate telnet session on Linux.

Related

Python Getting a SyntaxError: parsing command to Hython on windows

I'm trying to parse a few commands to Houdini's Python module called Hython.
And I'm doing this through os on windows.
My command looks like this:
command = '''"c:\\Program Files (x86)\\Steam\\steamapps\\common\\Houdini Indie\\bin\\hython.exe" -c \
"import sys; sys.path.append('d:\\Cloud\\OneDrive\\Dokumenter\\GitHub\\tutorialTools\\utils'); \
import houUtils; houUtils.runReduction(\'%s\',\'%s\')"''' % (assetDir, amount)
os.popen(command)
I've tried all the tricks on quotations I could think of, but I'm getting a :
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes
in position 49-50: truncated \uXXXX escape
Any kind of help would be appreciated
First I tried adding to the command by separate lines:
"command +=" but this stopped me quickly.
I've tried doing the initial path with r'c:\...'
The entire problem seems to be connected with writing paths inside a quotation.

xonsh "which" equivalent - how to test if a (subprocess mode) command is available?

I would like to test (from xonsh) if a command is available or not. If I try this from the xonsh command prompt:
which bash
Then it works:
user#server ~ $ which bash
/usr/bin/bash
But it does not work from xonsh script:
#!/usr/bin/env xonsh
$RAISE_SUBPROC_ERROR = True
try:
which bash
print("bash is available")
except:
print("bash is not available")
Because it results in this error:
NameError: name 'which' is not defined
I understand that which is a shell builtin. E.g. it is not an executable file. But it is available at the xnosh command prompt. Then why it is not available inside an xonsh script? The ultimate question is this: how can I test (from an xonsh script) if a (subprocess mode) command is available or not?
import shutil
print(shutil.which('bash'))
While nagylzs' answer led me to the right solution, I found it inadequate.
shutil.which defaults to os.environ['PATH']. On my machine, the default os.environ['PATH'] doesn't contain the active PATH recognized by xonsh.
~ $ os.environ['PATH']
'/usr/bin:/bin:/usr/sbin:/sbin'
I found I needed to pass $PATH to reliably resolve 'which' in the xonsh environment.
~ $ $PATH[:2]
['/opt/google-cloud-sdk/bin', '/Users/jaraco/.local/bin']
~ $ import shutil
~ $ shutil.which('brew', path=os.pathsep.join($PATH))
'/opt/homebrew/bin/brew'
The latest version of xonsh includes a built-in which command. Unfortunately, the version included will emit an error on stdout if the target isn't found, a behavior that is not great for non-interactive use.
As mentioned in another answer, which exists in the current version of xonsh (0.13.4 as of 15/12/2022) so your script would work. However, it outputs its own error message so it's necessary to redirect stderr to get rid of it.
Also, unless you redirect its stdout as well (using all>), it migh be a good idea to capture its output so the final version would look like this:
#!/usr/bin/env xonsh
$RAISE_SUBPROC_ERROR = True
try:
bash = $(which bash err> /dev/null)
print(f"bash is available: {bash}")
except:
print("bash is not available")

Why is my containerized Selenium application failing only in AWS Lambda?

I'm trying to get a function to run in AWS Lambda that uses Selenium and Firefox/geckodriver in order to run. I've decided to go the route of creating a container image, and then uploading and running that instead of using a pre-configured runtime. I was able to create a Dockerfile that correctly installs Firefox and Python, downloads geckodriver, and installs my test code:
FROM alpine:latest
RUN apk add firefox python3 py3-pip
RUN pip install requests selenium
RUN mkdir /app
WORKDIR /app
RUN wget -qO gecko.tar.gz https://github.com/mozilla/geckodriver/releases/download/v0.28.0/geckodriver-v0.28.0-linux64.tar.gz
RUN tar xf gecko.tar.gz
RUN mv geckodriver /usr/bin
COPY *.py ./
ENTRYPOINT ["/usr/bin/python3","/app/lambda_function.py"]
The Selenium test code:
#!/usr/bin/env python3
import util
import os
import sys
import requests
def lambda_wrapper():
api_base = f'http://{os.environ["AWS_LAMBDA_RUNTIME_API"]}/2018-06-01'
response = requests.get(api_base + '/runtime/invocation/next')
request_id = response.headers['Lambda-Runtime-Aws-Request-Id']
try:
result = selenium_test()
# Send result back
requests.post(api_base + f'/runtime/invocation/{request_id}/response', json={'url': result})
except Exception as e:
# Error reporting
import traceback
requests.post(api_base + f'/runtime/invocation/{request_id}/error', json={'errorMessage': str(e), 'traceback': traceback.format_exc(), 'logs': open('/tmp/gecko.log', 'r').read()})
raise
def selenium_test():
from selenium.webdriver import Firefox
from selenium.webdriver.firefox.options import Options
options = Options()
options.add_argument('-headless')
options.add_argument('--window-size 1920,1080')
ffx = Firefox(options=options, log_path='/tmp/gecko.log')
ffx.get("https://google.com")
url = ffx.current_url
ffx.close()
print(url)
return url
def main():
# For testing purposes, currently not using the Lambda API even in AWS so that
# the same container can run on my local machine.
# Call lambda_wrapper() instead to get geckodriver logs as well (not informative).
selenium_test()
if __name__ == '__main__':
main()
I'm able to successfully build this container on my local machine with docker build -t lambda-test . and then run it with docker run -m 512M lambda-test.
However, the exact same container crashes with an error when I try and upload it to Lambda to run. I set the memory limit to 1024M and the timeout to 30 seconds. The traceback says that Firefox was unexpectedly killed by a signal:
START RequestId: 52adeab9-8ee7-4a10-a728-82087ec9de30 Version: $LATEST
/app/lambda_function.py:29: DeprecationWarning: use service_log_path instead of log_path
ffx = Firefox(options=options, log_path='/tmp/gecko.log')
Traceback (most recent call last):
File "/app/lambda_function.py", line 45, in <module>
main()
File "/app/lambda_function.py", line 41, in main
lambda_wrapper()
File "/app/lambda_function.py", line 12, in lambda_wrapper
result = selenium_test()
File "/app/lambda_function.py", line 29, in selenium_test
ffx = Firefox(options=options, log_path='/tmp/gecko.log')
File "/usr/lib/python3.8/site-packages/selenium/webdriver/firefox/webdriver.py", line 170, in __init__
RemoteWebDriver.__init__(
File "/usr/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
self.start_session(capabilities, browser_profile)
File "/usr/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/usr/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/usr/lib/python3.8/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: Process unexpectedly closed with status signal
END RequestId: 52adeab9-8ee7-4a10-a728-82087ec9de30
REPORT RequestId: 52adeab9-8ee7-4a10-a728-82087ec9de30 Duration: 20507.74 ms Billed Duration: 21350 ms Memory Size: 1024 MB Max Memory Used: 131 MB Init Duration: 842.11 ms
Unknown application error occurred
I had it upload the geckodriver logs as well, but there wasn't much useful information in there:
1608506540595 geckodriver INFO Listening on 127.0.0.1:41597
1608506541569 mozrunner::runner INFO Running command: "/usr/bin/firefox" "--marionette" "-headless" "--window-size 1920,1080" "-foreground" "-no-remote" "-profile" "/tmp/rust_mozprofileQCapHy"
*** You are running in headless mode.
How can I even begin to debug this? The fact that the exact same container behaves differently depending upon where it's run seems fishy to me, but I'm not knowledgeable enough about Selenium, Docker, or Lambda to pinpoint exactly where the problem is.
Is my docker run command not accurately recreating the environment in Lambda? If so, then what command would I run to better simulate the Lambda environment? I'm not really sure where else to go from here, seeing as I can't actually reproduce the error locally to test with.
If anyone wants to take a look at the full code and try building it themselves, the repository is here - the lambda code is in lambda_function.py.
As for prior research, this question a) is about ChromeDriver and b) has no answers from over a year ago. The link from that one only has information about how to run a container in Lambda, which I'm already doing. This answer is almost my problem, but I know that there's not a version mismatch because the container works on my laptop just fine.
I have exactly the same problem and a possible explanation.
I think what you want is not possible for the time being.
According to AWS DevOps Blog Firefox relies on fallocate system call and /dev/shm.
However AWS Lambda does not mount /dev/shm so Firefox will crash when trying to allocate memory. Unfortunately, this handling cannot be disabled for Firefox.
However if you can live with Chromium, there is an option for chromedriver --disable-dev-shm-usage that disables the usage of /dev/shm and instead writes shared memory files to /tmp.
chromedriver works fine for me on AWS Lambda, if that is an option for you.
According to AWS DevOps Blog you can also use AWS Fargate to run Firefox/geckodriver.
There is an entry in the AWS forum from 2015 that requests mounting /dev/shm in Lambdas, but nothing happened since then.

Lua: forward output from Minicom

I have a Lua script and in there I open a minicom session which executes a script (with the -S" parameter).
local myFile = assert(io.popen('minicom -S myScript.sh ' myDevice ' -C myLogFile.log'))
local myFileOutput = myFile:read('*all')
myFile:close()
This works really fine.
But I would like to get the same output as if I execute the minicom command itself:
minicom -S myScript.sh ' myDevice ' -C myLogFile.log
Right now I don't get any output at all (I know that that's somehow obvious).
I would that the output should also occur at (at least nearly) the same time as with the minicom command itself. Not one big chuck of data at the end.
Does anyone know how to achieve that?
If I understand you correctly, you need something like
local myFile = assert(io.popen('minicom ...'))
for line in myFile:lines('l') do
print(line)
end
myFile:close()

Windows commands in ruby

How do I run a Windows command in an Ruby app?
I am trying to run something like:
output = `cd #{RAILS_ROOT}/lib && java HelloWorld #{param1} #{param2}`
I print the result of the line above and paste it to a command prompt in Windows and it works just fine. However, when i run app and hit this code, output is blank rather than have a string I get back from HellowWorld. In HelloWorld I do a System.out.print("helloworld")
The following:
output = `cmd.exe /C dir`
puts "OUTPUT #{output}"
Returns:
OUTPUT
Issue in JRuby 1.5.3 fixed in JRuby 1.5.5:
http://www.jruby.org/2010/11/10/jruby-1-5-5.html
Try to use File#join here. It will generate crossplatform path for you
http://apidock.com/ruby/File/join/class
my_path = File.join(RAILS_ROOT, "lib")
output = `cd #{my_path} && java HelloWorld #{param1} #{param2}`
Also you can execute your system commands this way:
`cd #{my_path} && java HelloWorld #{param1} #{param2}`
system("cd #{my_path} && java HelloWorld #{param1} #{param2}")
%x[cd #{my_path} && java HelloWorld #{param1} #{param2}]
Related topic: System call from Ruby
Backticks work fine for me. Try:
output = `dir`
to prove to yourself that it's working. At that point, your question is how to run a Java app from the command line, or why your particular app isn't work. Note that you can temporarily change the working directory like this:
Dir.chdir(File.join(RAILS_ROOT,'lib')) do
output = `...`
end

Resources