Unable to perform Bokeh export_png from docker image - docker

When attempting to run a Jupyter notebook that runs export_png via nbconvert in a Docker container, I get the following error.
Neither firefox and geckodriver nor a variant of chromium browser and chromedriver are available on system PATH. You can install the former with 'conda install -c conda-forge firefox geckodriver'.
I have a Linux vm configured with similar dependencies and it all works fine. I see there is a bug around finding firefox if the path points to a script, but mine does not, as shown in the last lines of output below. I'm starting to think the runtime error produced is a red herring and it's actually throwing a different error, but hardcoded to that message.
Are there any know issues of running bokeh export_png within a docker image? Any necessary config? Anyone doing that successfully with versions of firefox, geckodriver, selenium, and bokeh you could share? Installing chromium instead is not easy, as this is all behind a company conda mirror without those packages.
/opt/conda/lib/python3.7/site-packages/bokeh/io/webdriver.py in get(self)
116 if not self.reuse or self.current is None:
117 self.reset()
--> 118 self.current = self.create()
119 return self.current
120
/opt/conda/lib/python3.7/site-packages/bokeh/io/webdriver.py in create(self, kind)
120
121 def create(self, kind: Optional[DriverKind] = None) -> WebDriver:
--> 122 driver = self._create(kind)
123 self._drivers.add(driver)
124 return driver
/opt/conda/lib/python3.7/site-packages/bokeh/io/webdriver.py in _create(self, kind)
138 return driver
139
--> 140 raise RuntimeError("Neither firefox and geckodriver nor a variant of chromium browser and " \
141 "chromedriver are available on system PATH. You can install the former " \
142 "with 'conda install -c conda-forge firefox geckodriver'.")
RuntimeError: Neither firefox and geckodriver nor a variant of chromium browser and chromedriver are available on system PATH. You can install the former with 'conda install -c conda-forge firefox geckodriver'.
RuntimeError: Neither firefox and geckodriver nor a variant of chromium browser and chromedriver are available on system PATH. You can install the former with 'conda install -c conda-forge firefox geckodriver'.
[g_de_auto#0ff0d47b4a0f data_science_foundation]$ which firefox
/opt/conda/bin/FirefoxApp/firefox
[g_de_auto#0ff0d47b4a0f data_science_foundation]$ which geckodriver
/opt/conda/bin/geckodriver
[g_de_auto#0ff0d47b4a0f data_science_foundation]$ cat /opt/conda/bin/FirefoxApp/firefox
ELF>##1#8
######h###### # #-=#=#-=#=#0##DDPtd` ` #` #<<QtdRtd-=#=#PP/lib64/ld-linux-x86-64.so.2GNU GNUB
m5Jzq
ms- libpthread.so.0__errno_locationlibdl.so.2__gmon_start__libstdc++.so.6libm.so.6libgcc_s.so.1libc.so.6execv__stack_chk_fail_]`##D]fD`##UH`##HHHH?HHtHt ##(##0##8#####H#H/Ht5/%/#%/h%/h%/h%/h%/h%/h%/h%R/f1I^HHPTI#Hp#H#fDU`##H=`##HtHt
17dH%(H;tf.DAWAAVIAUIATL%(+UH-(+SL)1H]HtLLDAHH9u[]A\A]A^A_ff./proc/self/exeCouldn't find the application directory.
-binExec failed with error: %s
Update
The underlying error I get out of the _try_create_firefox_webdriver function is SessionNotCreatedException('Unable to find a matching set of capabilities')

Chose to use Xvfb. Looked into Kaleido, but I did not see an easy integration with Bokeh. Steps were:
Install the following dependencies.
yum install gtk3-devel xorg-x11-server-Xvfb.x86_64 selenium geckodriver=0.26.0 firefox=74.0
Telling firefox what display to use (the virtual one).
export DISPLAY=:0
Launch Xvfb along with my command.
Xvfb :0 -screen 0 1920x1080x24 & script_that_turns_charts_to_images

Related

installation of OpenCV(cv2) in anaconda3 environment on Ubuntu 20.04 python 3.9.13.final.0

I've been trying to install cv2 on my machine (linux-Ubuntu20.04) with environment manager conda
my python and conda information is pasted below
Conda info:
active environment : base
active env location : /home/anonymous/anaconda3
shell level : 1
user config file : /home/anonymous/.condarc
populated config files : /home/anonymous/.condarc
conda version : 22.9.0
conda-build version : 3.22.0
python version : 3.9.13.final.0
virtual packages : __linux=5.15.0=0
__glibc=2.31=0
__unix=0=0
__archspec=1=x86_64
base environment : /home/anonymous/anaconda3 (writable)
conda av data dir : /home/anonymous/anaconda3/etc/conda
conda av metadata url : None
channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
package cache : /home/anonymous/anaconda3/pkgs
/home/anonymous/.conda/pkgs
envs directories : /home/anonymous/anaconda3/envs
/home/anonymous/.conda/envs
platform : linux-64
user-agent : conda/22.9.0 requests/2.28.1 CPython/3.9.13 Linux/5.15.0-52-generic ubuntu/20.04.5 glibc/2.31
UID:GID : 1000:1000
netrc file : None
offline mode : False
# conda environments:
#
base * /home/anonymous/anaconda3
sys.version: 3.9.13 (main, Oct 13 2022, 21:15:33)
[G...
sys.prefix: /home/anonymous/anaconda3
sys.executable: /home/anonymous/anaconda3/bin/python
conda location: /home/anonymous/anaconda3/lib/python3.9/site-packages/conda
conda-build: /home/anonymous/anaconda3/bin/conda-build
conda-content-trust: /home/anonymous/anaconda3/bin/conda-content-trust
conda-convert: /home/anonymous/anaconda3/bin/conda-convert
conda-debug: /home/anonymous/anaconda3/bin/conda-debug
conda-develop: /home/anonymous/anaconda3/bin/conda-develop
conda-env: /home/anonymous/anaconda3/bin/conda-env
conda-index: /home/anonymous/anaconda3/bin/conda-index
conda-inspect: /home/anonymous/anaconda3/bin/conda-inspect
conda-metapackage: /home/anonymous/anaconda3/bin/conda-metapackage
conda-pack: /home/anonymous/anaconda3/bin/conda-pack
conda-render: /home/anonymous/anaconda3/bin/conda-render
conda-repo: /home/anonymous/anaconda3/bin/conda-repo
conda-server: /home/anonymous/anaconda3/bin/conda-server
conda-skeleton: /home/anonymous/anaconda3/bin/conda-skeleton
conda-token: /home/anonymous/anaconda3/bin/conda-token
conda-verify: /home/anonymous/anaconda3/bin/conda-verify
user site dirs:
CIO_TEST: <not set>
CONDA_DEFAULT_ENV: base
CONDA_EXE: /home/anonymous/anaconda3/bin/conda
CONDA_PREFIX: /home/anonymous/anaconda3
CONDA_PROMPT_MODIFIER: (base)
CONDA_PYTHON_EXE: /home/anonymous/anaconda3/bin/python
CONDA_ROOT: /home/anonymous/anaconda3
I've been trying several days installing opencv(cv2) on conda base environment with this line of code:
conda install -c conda-forge opencv
conda install -c "conda-forge/label/broken" opencv
conda install -c "conda-forge/label/cf201901" opencv
conda install -c "conda-forge/label/cf202003" opencv
conda install -c "conda-forge/label/gcc7" opencv
i had tried
conda install -c menpo opencv
to but it didn't work for me.
for last try i run this code:
conda install --channel https://conda.anaconda.org/menpo opencv3
and i received this massage:
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: |
Found conflicts! Looking for incompatible packages.
This can take several minutes. Press CTRL-C to abort.
failed
UnsatisfiableError: The following specifications were found
to be incompatible with the existing python installation in your environment:
Specifications:
- opencv3 -> python[version='2.7.*|3.4.*|3.5.*|3.6.*']
Your python: python=3.9
If python is on the left-most side of the chain, that's the version you've asked for.
When python appears to the right, that indicates that the thing on the left is somehow
not available for the python version you are constrained to. Note that conda will not
change your python version to a different minor version unless you explicitly specify
that.
The following specifications were found to be incompatible with your system:
- feature:/linux-64::__glibc==2.31=0
- feature:|#/linux-64::__glibc==2.31=0
Your installed version is: 2.31
Any help is appreciated.

RTL8188FU driver couldn't be installed because of a missing kernel tree module?

I am relatively new to Arch Linux.
So i have been dealing with this issue for quite a long time now. I got this new RTL8188FTV wireless USB network adapter and i have tried installing the drivers for it with Paru AUR helper. The installation seems to fail, as it throws this error at the end :
==> ERROR: Missing 5.15.70-1-lts kernel modules tree for module 8188fu/r112.89d7288.
I can confirm the wireless adapter is functional, since it works on Windows
Package Name: 8188fu-kelebek333-dkms-git
Is there a way to install this "kernel modules tree" so i could finish the installation?
Thanks!
Edit: back with more errors
Building module:
Cleaning build area...
'make' all KVER=6.0.5-arch1-1.............(bad exit status: 2)
Error! Bad return status for module build on kernel: 6.0.5-arch1-1 (x86_64)
Consult /var/lib/dkms/8188fu/r113.f5df53d/build/make.log for more information.
error: command failed to execute correctly
:: Running post-transaction hooks...
(1/3) Arming ConditionNeedsUpdate...
(2/3) Install DKMS modules
==> dkms install --no-depmod 8188fu/r113.f5df53d -k 6.0.5-arch1-1
Error! Bad return status for module build on kernel: 6.0.5-arch1-1 (x86_64)
Consult /var/lib/dkms/8188fu/r113.f5df53d/build/make.log for more information.
==> WARNING: `dkms install --no-depmod 8188fu/r113.f5df53d -k 6.0.5-arch1-1' exited 10
==> ERROR: Missing 5.15.75-1-lts kernel modules tree for module 8188fu/r113.f5df53d.
(3/3) Performing snapper post snapshots for the following configurations...
The command
echo 'alias usb:v0BDApF179d*dc*dsc*dp*icFFiscFFipFFin* rtl8188fu' | sudo tee /etc/modprobe.d/r8188eu-blacklist.conf
doesn't seem to do much.
Installation still fails.
Your on kernel 5.15 so you have to backlist the rl8188eu driver. You can do that with this command
echo 'alias usb:v0BDApF179d*dc*dsc*dp*icFFiscFFipFFin* rtl8188fu' | sudo tee /etc/modprobe.d/r8188eu-blacklist.conf
The code was taken from the same repo the aur package was made from
https://github.com/kelebek333/rtl8188fu

Why is my containerized Selenium application failing only in AWS Lambda?

I'm trying to get a function to run in AWS Lambda that uses Selenium and Firefox/geckodriver in order to run. I've decided to go the route of creating a container image, and then uploading and running that instead of using a pre-configured runtime. I was able to create a Dockerfile that correctly installs Firefox and Python, downloads geckodriver, and installs my test code:
FROM alpine:latest
RUN apk add firefox python3 py3-pip
RUN pip install requests selenium
RUN mkdir /app
WORKDIR /app
RUN wget -qO gecko.tar.gz https://github.com/mozilla/geckodriver/releases/download/v0.28.0/geckodriver-v0.28.0-linux64.tar.gz
RUN tar xf gecko.tar.gz
RUN mv geckodriver /usr/bin
COPY *.py ./
ENTRYPOINT ["/usr/bin/python3","/app/lambda_function.py"]
The Selenium test code:
#!/usr/bin/env python3
import util
import os
import sys
import requests
def lambda_wrapper():
api_base = f'http://{os.environ["AWS_LAMBDA_RUNTIME_API"]}/2018-06-01'
response = requests.get(api_base + '/runtime/invocation/next')
request_id = response.headers['Lambda-Runtime-Aws-Request-Id']
try:
result = selenium_test()
# Send result back
requests.post(api_base + f'/runtime/invocation/{request_id}/response', json={'url': result})
except Exception as e:
# Error reporting
import traceback
requests.post(api_base + f'/runtime/invocation/{request_id}/error', json={'errorMessage': str(e), 'traceback': traceback.format_exc(), 'logs': open('/tmp/gecko.log', 'r').read()})
raise
def selenium_test():
from selenium.webdriver import Firefox
from selenium.webdriver.firefox.options import Options
options = Options()
options.add_argument('-headless')
options.add_argument('--window-size 1920,1080')
ffx = Firefox(options=options, log_path='/tmp/gecko.log')
ffx.get("https://google.com")
url = ffx.current_url
ffx.close()
print(url)
return url
def main():
# For testing purposes, currently not using the Lambda API even in AWS so that
# the same container can run on my local machine.
# Call lambda_wrapper() instead to get geckodriver logs as well (not informative).
selenium_test()
if __name__ == '__main__':
main()
I'm able to successfully build this container on my local machine with docker build -t lambda-test . and then run it with docker run -m 512M lambda-test.
However, the exact same container crashes with an error when I try and upload it to Lambda to run. I set the memory limit to 1024M and the timeout to 30 seconds. The traceback says that Firefox was unexpectedly killed by a signal:
START RequestId: 52adeab9-8ee7-4a10-a728-82087ec9de30 Version: $LATEST
/app/lambda_function.py:29: DeprecationWarning: use service_log_path instead of log_path
ffx = Firefox(options=options, log_path='/tmp/gecko.log')
Traceback (most recent call last):
File "/app/lambda_function.py", line 45, in <module>
main()
File "/app/lambda_function.py", line 41, in main
lambda_wrapper()
File "/app/lambda_function.py", line 12, in lambda_wrapper
result = selenium_test()
File "/app/lambda_function.py", line 29, in selenium_test
ffx = Firefox(options=options, log_path='/tmp/gecko.log')
File "/usr/lib/python3.8/site-packages/selenium/webdriver/firefox/webdriver.py", line 170, in __init__
RemoteWebDriver.__init__(
File "/usr/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
self.start_session(capabilities, browser_profile)
File "/usr/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/usr/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/usr/lib/python3.8/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: Process unexpectedly closed with status signal
END RequestId: 52adeab9-8ee7-4a10-a728-82087ec9de30
REPORT RequestId: 52adeab9-8ee7-4a10-a728-82087ec9de30 Duration: 20507.74 ms Billed Duration: 21350 ms Memory Size: 1024 MB Max Memory Used: 131 MB Init Duration: 842.11 ms
Unknown application error occurred
I had it upload the geckodriver logs as well, but there wasn't much useful information in there:
1608506540595 geckodriver INFO Listening on 127.0.0.1:41597
1608506541569 mozrunner::runner INFO Running command: "/usr/bin/firefox" "--marionette" "-headless" "--window-size 1920,1080" "-foreground" "-no-remote" "-profile" "/tmp/rust_mozprofileQCapHy"
*** You are running in headless mode.
How can I even begin to debug this? The fact that the exact same container behaves differently depending upon where it's run seems fishy to me, but I'm not knowledgeable enough about Selenium, Docker, or Lambda to pinpoint exactly where the problem is.
Is my docker run command not accurately recreating the environment in Lambda? If so, then what command would I run to better simulate the Lambda environment? I'm not really sure where else to go from here, seeing as I can't actually reproduce the error locally to test with.
If anyone wants to take a look at the full code and try building it themselves, the repository is here - the lambda code is in lambda_function.py.
As for prior research, this question a) is about ChromeDriver and b) has no answers from over a year ago. The link from that one only has information about how to run a container in Lambda, which I'm already doing. This answer is almost my problem, but I know that there's not a version mismatch because the container works on my laptop just fine.
I have exactly the same problem and a possible explanation.
I think what you want is not possible for the time being.
According to AWS DevOps Blog Firefox relies on fallocate system call and /dev/shm.
However AWS Lambda does not mount /dev/shm so Firefox will crash when trying to allocate memory. Unfortunately, this handling cannot be disabled for Firefox.
However if you can live with Chromium, there is an option for chromedriver --disable-dev-shm-usage that disables the usage of /dev/shm and instead writes shared memory files to /tmp.
chromedriver works fine for me on AWS Lambda, if that is an option for you.
According to AWS DevOps Blog you can also use AWS Fargate to run Firefox/geckodriver.
There is an entry in the AWS forum from 2015 that requests mounting /dev/shm in Lambdas, but nothing happened since then.

Can't install image acquisition package on octave windows7

there I am using octave on windows7 machine and unable to install image-acquisition package using command :
>> pkg install -forge image-acquisition
and it shows error
system: unable to start subprocess for '"gzip -d -c "D:\Users\hello\image-acquisition-0.2.2\image-acquisition-0.2.2.tar.gz" | tar xf - 2>&1"'
error: called from
unpack at line 248 column 20
untar at line 47 column 5
install at line 81 column 9
pkg at line 437 column 9
version
>> version
4.4.0
The reason why your installation fails is, that the image-acquisition package for octave is not implemented for windows systems.

image::magick not installing with cpan in perl v5.14.2 from dwimperl

I am attempting to rebuild my development/test environment on a new laptop running windows7 32 bit. imageMagick is one of the modules I had installed in an (ancient) version on my old laptop. I have downloaded and installed perl from dwimperl, which is v5.14.2 and had a couple modules install, and several did not.
cpanm Image::Magick - failed
cpanm DB_File - failed
cpanm Time::HiRes - failed
I searched around and found a discussion on magick failing to install on v5.12, but couldn't tell if that was supposed to have been fixed, or if I need to attempt to recreate the fix discussed for 5.12
should I try installing perl from Strawberry? looks like it is a little newer v5.18 ?
is there something I can tweak and re-run cpan installs?
I installed the binaries from imagemagick, from Link first, then cpan again, and cpan still fails....
This is the top part of the build log down to where it really goes south and starts kicking out errors:
cpanm (App::cpanminus) 1.6941 on perl 5.014002 built for MSWin32-x86-multi-thread
Work directory is C:\Users\dtbaker/.cpanm/work/1377281741.8420
You have make C:\Dwimperl\c\bin\dmake.exe
You have LWP 6.03
Falling back to Archive::Tar 1.80
Searching Image::Magick on cpanmetadb ...
--> Working on Image::Magick
Fetching http://www.cpan.org/authors/id/J/JC/JCRISTY/PerlMagick-6.86.tar.gz
-> OK
Unpacking PerlMagick-6.86.tar.gz
Entering PerlMagick-6.86
META.yml/json not found. Creating skelton for it.
Configuring PerlMagick-6.86
Running Makefile.PL
################################### WARNING! ###################################
# It seems that you are trying to install Perl::Magick on a MS Windows box with
# perl + gcc compiler (e.g. strawberry perl), however we cannot find ImageMagick
# binaries installed on your system.
#
# Please check the following prerequisites:
#
# 1) You need to have installed ImageMagick Windows binaries from
# http://www.imagemagick.org/script/binary-releases.php#windows
#
# 2) We only support dynamic (DLL) ImageMagick binaries
# note: it is not possible to mix 32/64-bit binaries of perl and ImageMagick
#
# 3) During installation select that you want to install ImageMagick's
# development files (libraries+headers)
#
# 4) You also need to have ImageMagick's directory in your PATH
# note: we are checking the presence of convert.exe and/or identify.exe tools
#
# 5) You might need Visual C++ Redistributable Package installed on your system
# see instructions on ImageMagick's Binary Release webpage
#
# We are gonna continue, but chances for successful build are very low!
################################################################################
Checking if your kit is complete...
Looks good
Note (probably harmless): No library found for -lMagickCore-6.Q16
Writing Makefile for Image::Magick
Writing MYMETA.yml and MYMETA.json
-> OK
Checking dependencies from MYMETA.json ...
Checking if you have ExtUtils::MakeMaker 0 ... Yes (6.62)
Building and testing Image-Magick-6.86
cp Magick.pm blib\lib\Image\Magick.pm
AutoSplitting blib\lib\Image\Magick.pm (blib\lib\auto\Image\Magick)
C:\Dwimperl\perl\bin\perl.exe C:\Dwimperl\perl\lib\ExtUtils\xsubpp -typemap C:\Dwimperl\perl\lib\ExtUtils\typemap -typemap typemap Magick.xs > Magick.xsc && C:\Dwimperl\perl\bin\perl.exe -MExtUtils::Command -e mv -- Magick.xsc Magick.c
gcc -c -s -O2 -DWIN32 -DPERL_TEXTMODE_SCRIPTS -DUSE_SITECUSTOMIZE -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -fno-strict-aliasing -mms-bitfields -s -O2 -DVERSION=\"6.86\" -DXS_VERSION=\"6.86\" "-IC:\Dwimperl\perl\lib\CORE" -D_LARGE_FILES=1 -DHAVE_CONFIG_H Magick.c
Magick.xs:60:31: error: magick/MagickCore.h: No such file or directory
Magick.xs:167: error: expected specifier-qualifier-list before 'MagickRealType'
Magick.xs:188: error: expected specifier-qualifier-list before 'ImageInfo'
Magick.xs:210: error: 'MagickNoiseOptions' undeclared here (not in a function)
however we cannot find ImageMagick binaries installed on your system.
Is ImageMagick is in your path, as recommanded in the 4) point? Open cmd.exe and type convert -v or convert.exe -v. If you don't see informations about Image Magick (Windows have a built-in convert command), it is very likely you have to add it to your path.
You also have to check points 1), 3), and 5). After that try again to run installation process through cpanm.
Trying to install for Citrus Perl, I discovered on debugging through the Perl part of the install that the mingw64 install had not included 'pexports.exe'. Downloading that from https://sourceforge.net/projects/mingw/files/MinGW/Extension/pexports/
and placing it in the mingw64 directory was necessary to solve the problem of a long list of library export symbols not found.
Prior to that I had also set the environment variables CPATH, C_INCLUDE_PATH, and CPLUS_INCLUDE_PATH to point to the "include" directory of the ImageMagick install include directory in C:\Program Files (x86). (When you install ImageMagick you should check the box to install also for Strawberry Perl.)

Resources