I have a perl program taking two arguments. For example :
perl prog.pl -x A -y a
I want to use GNU parallel to combine two lists of arguments. For example :
perl prog.pl -x A -y a
perl prog.pl -x B -y b
perl prog.pl -x A -y b
perl prog.pl -x B -y a
etc...
I tried many syntaxes like :
parallel perl prog.pl -x ::: ABC -y ::: abc
parallel perl prog.pl -x -y ::: ABC ::: abc
I did not succeed to combine 2 or more arguments.
Thank you for your help !
I think you want this:
parallel --dry-run perlthing -x {1} -y {2} ::: A B C ::: a b c
perlthing -x A -y c
perlthing -x B -y a
perlthing -x B -y b
perlthing -x A -y b
perlthing -x A -y a
perlthing -x B -y c
perlthing -x C -y a
perlthing -x C -y b
perlthing -x C -y c
The --dry-run just shows you what GNU Parallel would do if you ran it.
Related
I am having an issue with installing dependencies with a Dockerfile. The image builds successfully, but when I attempt to test some of the dependencies installed in the Dockerfile below, they do not seem to be found. I am new to creating Dockerfiles so I am not sure if I am missing something. This may having something to do with environment variables? For example, I install sl as a test to confirm that something is not right. When a access a shell of my running image, running 'sl' returns "Command not found". Please let me know what I am doing wrong.
Here is my Dockerfile that I am using to build an image:
#FROM specifies the parent image from which we will construct our own custom image. This tomcat image contains a limited dist of Ubuntu, Tomcat9, and JRE11 ("temurin" is just a limited version of Java)
FROM tomcat:9-jre11-temurin
#add a maintainer label to the image
LABEL maintainer="Kerrick Cavanaugh - kerrickcavanaugh#ufl.edu"
#SciPy, Pandas, Numpy, other deps
RUN apt-get update && \
yes | apt-get -qq -y install \
git \
build-essential \
software-properties-common \
# python-dev \
python3 \
python3-dev \
python-dev \
libssl-dev \
# libffi-dev \
libxml2-dev \
libxslt1-dev \
apt-utils \
zlib1g-dev \
pip \
python3-pip \
sl \
gfortran \
libopenblas-dev \
liblapack-dev \
wget
# add /root/.local/bin to python path (causing warnings)
RUN python3 -c "import sys; sys.path.append('/root/.local/bin');"
RUN pip3 install --upgrade pip && pip3 install -vvv \
wheel \
matplotlib \
ipython \
jupyter \
sympy \
cython \
et-xmlfile==1.0.1 \
imbalanced-learn==0.5.0 \
imblearn==0.0 \
jdcal==1.4.1 \
joblib==0.14.0 \
numpy==1.16.4 \
scipy==1.3.1 \
openpyxl==2.6.4 \
pandas==0.24.2 \
python-dateutil==2.8.0 \
pytest==4.6.3 \
pytest-cov==2.7.1 \
pytz==2019.3 \
scikit-learn==0.21.3 \
six==1.12.0 \
xlrd==1.2.0
#R
RUN apt update && echo 'Y' | apt upgrade && \
#removed sudo from next 4
echo 'Y' | apt install dirmngr gnupg apt-transport-https ca-certificates software-properties-common && \
echo 'Y' | apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E298A3A825C0D65DFD57CBB651716619E084DAB9 && \
echo 'Y' | add-apt-repository 'deb https://cloud.r-project.org/bin/linux/ubuntu focal-cran40/' && \
echo 'Y' | apt install r-base && \
R --version
#MATLAB???
#FSL
COPY ./aidp_docker/fslinstaller.py /usr/local/tomcat
#printf '36' | <-- for keyboard
#removed sudo
RUN apt-get update && apt-get -qq install python2
RUN { echo; echo '36'; } | python2 /usr/local/tomcat/fslinstaller.py -E -v -q > /dev/null
#AFNI
RUN cd && \
curl -O https://raw.githubusercontent.com/afni/afni/master/src/other_builds/OS_notes.linux_ubuntu_20_64_a_admin.txt && \
curl -O https://raw.githubusercontent.com/afni/afni/master/src/other_builds/OS_notes.linux_ubuntu_20_64_b_user.tcsh && \
curl -O https://raw.githubusercontent.com/afni/afni/master/src/other_builds/OS_notes.linux_ubuntu_20_64_c_nice.tcsh && \
#removed sudo
bash OS_notes.linux_ubuntu_20_64_a_admin.txt 2>&1 | tee o.ubuntu_20_a.txt && \
tcsh OS_notes.linux_ubuntu_20_64_b_user.tcsh 2>&1 | tee o.ubuntu_20_b.txt
#dcm2niix
RUN curl -fLO https://github.com/rordenlab/dcm2niix/releases/latest/download/dcm2niix_lnx.zip
#ANTsR, ANTsRCore, ITKR
RUN git clone https://github.com/stnava/ITKR.git && \
git clone https://github.com/ANTsX/ANTsRCore.git && \
git clone https://github.com/ANTsX/ANTsR.git && \
R -e 'install.packages(c("Rcpp", "RcppEigen", "magrittr"))' && \
R CMD INSTALL ITKR && \
R CMD INSTALL ANTsRCore && \
R CMD INSTALL ANTsR
#COPY the specified folder structure and associated scripts to /usr/local/tomcat
COPY ./aidp_docker/ /usr/local/tomcat
#COPY wAIDP.war to the image
COPY ./wAIDP.war /usr/local/tomcat/webapps
#build complete
RUN echo 'Build complete!'
#runs the Docker image
CMD ["catalina.sh", "run"]
I'm trying to implentment openpilot on Jetson Xavier nx. So I'm following https://github.com/eFiniLan/xnxpilot instruction to install dependence.
But when I'm installing opencv4, I get the following errors in ".../opencv/build/CMakeFiles/CMakeError.log"
CMakeFiles/cmTC_ee78d.dir/CheckIncludeFile.c.o -c /home/tshu/opencv/build/CMakeFiles/CMakeTmp/CheckIncludeFile.c
/home/tshu/opencv/build/CMakeFiles/CMakeTmp/CheckIncludeFile.c:1:10: fatal error: sys/videoio.h: No such file or directory
#include <sys/videoio.h>
^~~~~~~~~~~~~~~
compilation terminated.
CMakeFiles/cmTC_ee78d.dir/build.make:65: recipe for target ‘CMakeFiles/cmTC_ee78d.dir/CheckIncludeFile.c.o’ failed
make[1]: * [CMakeFiles/cmTC_ee78d.dir/CheckIncludeFile.c.o] Error 1
make[1]: Leaving directory ‘/home/tshu/opencv/build/CMakeFiles/CMakeTmp’
Makefile:126: recipe for target ‘cmTC_ee78d/fast’ failed
make: * [cmTC_ee78d/fast] Error 2
The build command I used is
cmake -D CMAKE_BUILD_TYPE=RELEASE \
-D WITH_CUDA=ON \
-D CUDA_ARCH_PTX="" \
-D CUDA_ARCH_BIN="7.2" \
-D WITH_CUDNN=ON \
-D CUDNN_VERSION="8.0" \
-D BUILD_opencv_python3=ON \
-D BUILD_opencv_python2=OFF \
-D BUILD_opencv_java=OFF \
-D WITH_GSTREAMER=ON \
-D WITH_GTK=OFF \
-D BUILD_TESTS=OFF \
-D BUILD_PERF_TESTS=OFF \
-D BUILD_EXAMPLES=OFF \
-D BUILD_FFMPEG=ON \
-D OPENCV_DNN_CUDA=ON \
-D ENABLE_FAST_MATH=ON \
-D CUDA_FAST_MATH=ON \
-D WITH_QT=ON \
-D ENABLE_NEON=ON \
-D ENABLE_VFPV3=ON \
-D BUILD_TESTS=OFF \
-D INSTALL_PYTHON_EXAMPLES=OFF \
-D INSTALL_C_EXAMPLES=OFF \
-D OPENCV_ENABLE_NONFREE=ON \
-D OPENCV_GENERATE_PKGCONFIG=ON \
-D PYTHON_EXECUTABLE=/home/`whoami`/.pyenv/versions/3.8.5/bin/python \
-D PYTHON_DEFAULT_EXECUTABLE=/home/`whoami`/.pyenv/versions/3.8.5/bin/python \
-D PYTHON_PACKAGES_PATH=/home/`whoami`/.pyenv/versions/3.8.5/lib/python3.8/site-packages/ \
-D OPENCV_EXTRA_MODULES_PATH=/home/`whoami`/opencv_contrib/modules ..
The version of opencv I tried to install is opencv-4.5.2
Can someone give me some advices? Thank you.
Jetpack comes with opencv preinstalled. JetPack 4.4 includes OpenCV 4.1.1. JetPack 4.6 includes OpenCV 4.1.1.
Let me look at the link you sent and I get back to you. You may need to install and compile OpenCV 4.5.2 from source. I wrote some instructions a while a go.
What Jetpack version are you using? OPENCV 4.4, is CUDA GPU accelerated. Using version 4.4 or higher to fully use the Super Resolution function provided by OpenCV. If you are using Jetpack 4.4, you will need to delete the OpenCV 4.1.1 version of JetPack 4.4 and install 4.4 newly.
Try this script file:
#!/bin/bash
#
# Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved.
#
# NVIDIA Corporation and its licensors retain all intellectual property
# and proprietary rights in and to this software, related documentation
# and any modifications thereto. Any use, reproduction, disclosure or
# distribution of this software and related documentation without an express
# license agreement from NVIDIA Corporation is strictly prohibited.
#
if [ "$#" -ne 1 ]; then
echo "Usage: $0 <Install Folder>"
exit
fi
folder="$1"
user="nvidia"
passwd="nvidia"
echo "** Remove OpenCV4.1 first"
sudo apt-get purge *libopencv*
echo "** Install requirement"
sudo apt-get update
sudo apt-get install -y build-essential cmake git libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev
sudo apt-get install -y libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev
sudo apt-get install -y python2.7-dev python3.6-dev python-dev python-numpy python3-numpy
sudo apt-get install -y libtbb2 libtbb-dev libjpeg-dev libpng-dev libtiff-dev libjasper-dev libdc1394-22-dev
sudo apt-get install -y libv4l-dev v4l-utils qv4l2 v4l2ucp
sudo apt-get install -y curl
sudo apt-get update
echo "** Download opencv-4.5.1"
cd $folder
curl -L https://github.com/opencv/opencv/archive/4.5.1.zip -o opencv-4.5.1.zip
curl -L https://github.com/opencv/opencv_contrib/archive/4.5.1.zip -o opencv_contrib-4.5.1.zip
unzip opencv-4.5.1.zip
unzip opencv_contrib-4.5.1.zip
cd opencv-4.5.1/
echo "** Building..."
mkdir release
cd release/
cmake -D WITH_CUDA=ON -D ENABLE_PRECOMPILED_HEADERS=OFF -D CUDA_ARCH_BIN="7.2" -D CUDA_ARCH_PTX="" -D OPENCV_EXTRA_MODULES_PATH=../../opencv_contrib-4.4.0/modules -D WITH_GSTREAMER=ON -D WITH_LIBV4L=ON -D BUILD_opencv_python2=ON -D BUILD_opencv_python3=ON -D BUILD_TESTS=OFF -D BUILD_PERF_TESTS=OFF -D BUILD_EXAMPLES=OFF -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local ..
make -j6
sudo make install
echo "** Install opencv-4.5.1 successfully"
echo "** Bye :)"
If you are using NX -D CUDA_ARCH_BIN="7.2"
Run following script with on path:
$./opencv4.5_xavier_nx.sh /home/TH-Dev/src/
In order to build up a headless simulation cluster, we're working on containerization of our existing tools. Right now, the accessible server does not have any NVIDIA GPUs.
One problem, that we encounter is, that a specific application uses OpenGL for rendering. With an physical GPU, the simulation tool is running without any problem. To ship around the GPU dependencies, we're using Mesa 3D OpenGL Software Rendering (Gallium), LLVMpipe, and OpenSWR Drivers. For reference, we had a look at https://github.com/jamesbrink/docker-opengl.
The current Dockerfile, which builds mesa 19.0.2 (using gcc-8) from source, looks like this:
# OPENGL SUPPORT ------------------------------------------------------------------------------
# start with plain ubuntu as base image for testing
FROM ubuntu AS builder
# install some needed packages and set gcc-8 as default compiler
RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y \
llvm-7 \
llvm-dev \
autoconf \
automake \
bison \
flex \
gettext \
libtool \
python-dev\
git \
pkgconf \
python-mako \
zlib1g-dev \
x11proto-gl-dev \
libxext-dev \
xcb \
libx11-xcb-dev \
libxcb-dri2-0-dev \
libxcb-xfixes0-dev \
libdrm-dev \
g++ \
make \
xvfb \
x11vnc \
g++-8 && \
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-8 800 --slave /usr/bin/g++ g++ /usr/bin/g++-8
# get mesa (using 19.0.2 as later versions dont use the configure script)
WORKDIR /mesa
RUN git clone https://gitlab.freedesktop.org/mesa/mesa.git
WORKDIR /mesa/mesa
RUN git checkout mesa-19.0.2
#RUN git checkout mesa-18.2.2
# build and install mesa
RUN libtoolize && \
autoreconf --install && \
./configure \
--enable-glx=gallium-xlib \
--with-gallium-drivers=swrast,swr \
--disable-dri \
--disable-gbm \
--disable-egl \
--enable-gallium-osmesa \
--enable-autotools \
--enable-llvm \
--with-llvm-prefix=/usr/lib/llvm-7/ \
--prefix=/usr/local && \
make -j 4 && \
make install && \
rm -rf /mesa
# SIM -----------------------------------------------------------------------------------------
FROM ubuntu
COPY --from=builder /usr/local /usr/local
# copy all simulation binaries to the image
COPY .....
# update ubuntu and install all sim dependencies
RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y \
xterm \
freeglut3 \
openssh-server \
synaptic \
nfs-common \
mesa-utils \
xfonts-75dpi \
libusb-0.1-4 \
python \
libglu1-mesa \
libqtgui4 \
gedit \
xvfb \
x11vnc \
llvm-7-dev \
expat \
nano && \
dpkg -i /vtdDeb/libpng12-0_1.2.54-1ubuntu1.1_amd64.deb
# set the environment variables (display -> 99 and LIBGL_ALWAYS_SOFTWARE)
ENV DISPLAY=":99" \
GALLIUM_DRIVER="llvmpipe" \
LIBGL_ALWAYS_SOFTWARE="1" \
LP_DEBUG="" \
LP_NO_RAST="false" \
LP_NUM_THREADS="" \
LP_PERF="" \
MESA_VERSION="19.0.2" \
XVFB_WHD="1920x1080x24"
If we now start the container and initialize the xvfb session, all glx examples like glxgears are working. Also the output of glxinfo | grep '^direct rendering:' is yes, so OpenGL is working.
However, if we start our simulation binary (which is provided from some company and cannot be changed now), following error messages are provided:
uniform block ub_lights has no binding.
uniform block ub_lights has no binding.
FRAGMENT glCompileShader "../data/Shaders/roadRendererFrag.glsl" FAILED
FRAGMENT Shader "../data/Shaders/roadRendererFrag.glsl" infolog:
0:277(48): error: unsized array index must be constant
0:344(48): error: unsized array index must be constant
glLinkProgram "RoadRenderingBase_Program" FAILED
Program "RoadRenderingBase_Program" infolog:
error: linking with uncompiled/unspecialized shader
Any idea how to fix that? For us, the error message is kind of vacuous.
Did someone encountered a similar problem?
My pipeline fails at the sh """ element of the Jenkinsfile. Any idea where things go wrong?
stage('Install dependencies') {
when { expression { return params.dependencies } } }
steps {
sh """
apt-get update
apt-get install -y openssh-server net-tools inetutils-ping python-pip rubygems
apt-get install -y \
apt-transport-https \
ca-certificates \
curl \
gnupg2 \
software-properties-common
curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -
add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/debian \
$(lsb_release -cs) \
stable"
apt-get update
apt-get install -y docker-ce docker-ce-cli containerd.io
curl -L "https://github.com/docker/compose/releases/download/1.23.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
gem install serverspec pygmy
"""
}
}
The error message is:
WorkflowScript: 35: illegal string body character after dollar sign;
solution: either escape a literal dollar sign "\$5" or bracket the value expression "${5}" # line 35, column 17.
Replace double quotes """ with single quotes '''.
sh '''
apt-get update
//...
'''
Whenever Groovy sees $ inside the double quotes, it treats this string as GString and does string interpolation. However, in your case, a character $ is not used in the context of interpolation and it fails. Alternatively, you could escape \$ but it makes more sense to switch to single quoted string.
If you want to use groovy's string interpolation, you can leave the double-quotes, but you have to escape the dollar sign in your subshell expression because groovy doesn't know what to do with a round brace following a dollar sign:
change:
$(lsb_release -cs)
to:
\$(lsb_release -cs)
I have noticed that the error message indicates the wrong line. In my case:
sh """
echo "message: ${env.MESSAGE}" # <-- error message points here
pwd
echo "ls: $(ls)" <-- this line has the problem
"""
Look for the first dollar sign after the line indicated by the error message.
I have a following scenario. I want to use tensorflow for ML and OpenCV for some image processing. I recently learned about dockers and found out, that both TF and OCV are dockerized. I can easily pull the image and run eg. tensorflow script. Is there a way to somehow merge what both dockers offer? Or run on top of it. I want to write a piece of code that uses both OpenCV and Tensorflow. Is there a way to achieve this?
Or in more generic sense: Docker A image has preinstalled python package AA. Docker B has python package BB. How can I write script that uses functions from both AA and BB?
Really simple. Build your own docker image with both TF and OpenCV. Example Dockerfile (Based on janza/docker-python3-opencv):
FROM python:3.7
LABEL maintainet="John Doe"
RUN apt-get update && \
apt-get install -y \
build-essential \
cmake \
git \
wget \
unzip \
yasm \
pkg-config \
libswscale-dev \
libtbb2 \
libtbb-dev \
libjpeg-dev \
libpng-dev \
libtiff-dev \
libavformat-dev \
libpq-dev && \
pip install numpy && \
pip install tensorflow
WORKDIR /
ENV OPENCV_VERSION="3.4.2"
RUN wget https://github.com/opencv/opencv/archive/${OPENCV_VERSION}.zip \
&& unzip ${OPENCV_VERSION}.zip \
&& mkdir /opencv-${OPENCV_VERSION}/cmake_binary \
&& cd /opencv-${OPENCV_VERSION}/cmake_binary \
&& cmake -DBUILD_TIFF=ON \
-DBUILD_opencv_java=OFF \
-DWITH_CUDA=OFF \
-DWITH_OPENGL=ON \
-DWITH_OPENCL=ON \
-DWITH_IPP=ON \
-DWITH_TBB=ON \
-DWITH_EIGEN=ON \
-DWITH_V4L=ON \
-DBUILD_TESTS=OFF \
-DBUILD_PERF_TESTS=OFF \
-DCMAKE_BUILD_TYPE=RELEASE \
-DCMAKE_INSTALL_PREFIX=$(python3.7 -c "import sys; print(sys.prefix)") \
-DPYTHON_EXECUTABLE=$(which python3.7) \
-DPYTHON_INCLUDE_DIR=$(python3.7 -c "from distutils.sysconfig import get_python_inc; print(get_python_inc())") \
-DPYTHON_PACKAGES_PATH=$(python3.7 -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())") \
.. \
&& make install \
&& rm /${OPENCV_VERSION}.zip \
&& rm -r /opencv-${OPENCV_VERSION}
Of course, I don't know your exact requirements regarding this project and there is some probability that this Dockerfile won't work for you. Just adjust it to you needs. But I recommend creating from ground zero (just basing on some already existing image of some Linux distribution). Then you have full control what have you installed in which versions without redundant stuff that is often found in 3rd party images (I'm not saying they are bad, but often for people use cases most parts are redundant.)
There is also already combined docker image in official hub:
https://hub.docker.com/r/fbcotter/docker-tensorflow-opencv/
If you reaaaaly want to have it separate I guess you could link running containers of those images. Containers for the linked service are reachable at a hostname identical to the alias, or the service name if no alias was specified. But you would have to implement some kind of logic to use another package from another container (probably possible but difficult and complex).
Docker Networking