I've cross-compiled opencv3.4, and it is running well on board. The project is managed through Yocto. so I wrote this opencv-gl.bb file to copy prebuilt opencv files to the target filesystem. But after I burned the mirror image to the developed board, I got nothing. It seems the copy command has neveber been executed. Where am I wrong?
SUMMARY = "Install opencv 3.4.14 libraries"
LICENSE = "CLOSED"
LIC_FILES_CHKSUM = ""
SRC_URI = "\
file://etc \
file://usr \
"
S = "${WORKDIR}"
## prebuilt library don't need following steps
do_configure[noexec] = "1"
do_compile[noexec] = "1"
do_package_qa[noexec] = "1"
do_install[nostamp] += "1"
do_install() {
install -d ${D}/usr/local/bin
cp -rf ${S}/usr/bin/* ${D}/usr/local/bin/
install -d ${D}/usr/local/lib
cp -rf ${S}/usr/lib/* ${D}/usr/local/lib/
install -d ${D}/usr/local/include
cp -rf ${S}/usr/include/* ${D}/usr/local/include/
install -d ${D}/usr/local/share
cp -rf ${S}/usr/share/* ${D}/usr/local/share/
}
# let the build system extends the FILESPATH file search path
FILESEXTRAPATHS_prepend := "${THISDIR}/prebuilts:"
FILES_${PN} += " \
/usr/local/bin/* \
/usr/local/lib/* \
/usr/local/include/* \
/usr/local/share/* \
"
# INSANE_SKIP_${PN} += "installed-vs-shipped"
the file structure is as follows:
wb#ubuntu:~/Yocto/meta-semidrive/recipes-test/opencv-gl$ tree -L 3
.
├── opencv-gl.bb
└── prebuilts
├── etc
│ └── ld.so.conf
├── LICENSE
└── usr
├── bin
├── include
├── lib
└── share
7 directories, 3 files
Related
I am trying to build a docker image that contains all of the necessary plugins/providers that several source repos need, so that when an automated terraform validate runs, it doesn't have to download gigs of redundant data.
However, I recognize that this provides for a maintenance problem in that someone may update a plugin version, and that would needed to be downloaded, since the docker image would not contain it.
The question
How can I pre-download all providers and plugins
Tell the CLI use those predownloaded plugins AND
also tell it that, if it doesn't find what it needs locally, then it can go to the network
Below are the relevant file:
.terraformrc
plugin_cache_dir = "$HOME/.terraform.d/plugin-cache"
disable_checkpoint = true
provider_installation {
filesystem_mirror {
path = "$HOME/.terraform/providers"
}
direct {
}
}
tflint (not relevant to this question, but it shows up in the below Dockerfile)
plugin "aws" {
enabled = true
version = "0.21.1"
source = "github.com/terraform-linters/tflint-ruleset-aws"
}
plugin "azurerm" {
enabled = true
version = "0.20.0"
source = "github.com/terraform-linters/tflint-ruleset-azurerm"
}
Dockerfile
FROM ghcr.io/terraform-linters/tflint-bundle AS base
LABEL name=tflint
RUN adduser -h /home/jenkins -s /bin/sh -u 1000 -D jenkins
RUN apk fix && apk --no-cache --update add git terraform openssh
ADD .terraformrc /home/jenkins/.terraformrc
RUN mkdir -p /home/jenkins/.terraform.d/plugin-cache/registry.terraform.io
ADD .tflint.hcl /home/jenkins/.tflint.hcl
WORKDIR /home/jenkins
RUN tflint --init
FROM base AS build
ARG SSH_PRIVATE_KEY
RUN mkdir /root/.ssh && \
echo "${SSH_PRIVATE_KEY}" > /root/.ssh/id_ed25519 && \
chmod 400 /root/.ssh/id_ed25519 && \
touch /root/.ssh/known_hosts && \
ssh-keyscan mygitrepo >> /root/.ssh/known_hosts
RUN git clone git#mygitrepo:wrai/tools/g.git
RUN git clone git#mygitrepo:myproject/a.git && \
git clone git#mygitrepo:myproject/b.git && \
git clone git#mygitrepo:myproject/c.git && \
git clone git#mygitrepo:myproject/d.git && \
git clone git#mygitrepo:myproject/e.git && \
git clone git#mygitrepo:myproject/f.git
RUN ls -1d */ | xargs -I {} find {} -name '*.tf' | xargs -n 1 dirname | sort -u | \
xargs -I {} -n 1 -P 20 terraform -chdir={} providers mirror /home/jenkins/.terraform.d
RUN chown -R jenkins:jenkins /home/jenkins
USER jenkins
FROM base AS a
COPY --from=build /home/jenkins/a/ /home/jenkins/a
RUN cd /home/jenkins/a && terraform init
FROM base AS b
COPY --from=build /home/jenkins/b/ /home/jenkins/b
RUN cd /home/jenkins/b && terraform init
FROM base AS c
COPY --from=build /home/jenkins/c/ /home/jenkins/c
RUN cd /home/jenkins/c && terraform init
FROM base AS azure_infrastructure
COPY --from=build /home/jenkins/d/ /home/jenkins/d
RUN cd /home/jenkins/d && terraform init
FROM base AS aws_infrastructure
COPY --from=build /home/jenkins/e/ /home/jenkins/e
RUN cd /home/jenkins/e && terraform init
Staging plugins:
This is most easily accomplished with the plugin cache dir setting in the CLI. This supersedes the old usage with the -plugin-dir=PATH argument for the init command. You could also set a filesystem mirror in each terraform block within the root module config, but this would be cumbersome for your use case. In your situation, you are already configuring this in your .terraformrc, but the filesystem_mirror path conflicts with the plugin_cache_dir. You would want to resolve that conflict, or perhaps remove the mirror block entirely.
Use staged plugins:
Since the setting is captured in the CLI configuration file within the Dockerfile, this would be automatically used in future commands.
Download additional plugins if necessary:
This is default behavior of the init command, and therefore requires no further actions on your part.
Side note:
The jenkins user typically is /sbin/nologin for shell and /var/lib/jenkins for home directory. If the purpose of this Docker image is for a Jenkins build agent, then you may want the jenkins user configuration to be more aligned with the standard.
TL;DR:
Configure the terraform plugin cache directory
Create directory with a single TF file containing required_providers block
Run terraform init from there
...
I've stumbled over this question as I tried to figure out the same thing.
I first tried leveraging an implied filesystem_mirror by running terraform providers mirror /usr/local/share/terraform/plugins in a directory containing only one terraform file containing the required_providers block. This works fine as long as you only use the versions of the providers you mirrored.
However, it's not possible to use a different version of a provider than the one you mirrored, because:
Terraform will scan all of the filesystem mirror directories to see which providers are placed there and automatically exclude all of those providers from the implied direct block.
I've found it to be a better solution to use a plugin cache directory instead. EDIT: You can prefetch the plugins by setting TF_PLUGIN_CACHE_DIR to some directory and then running terraform init in a directory that only declares the required_providers.
Previously overengineered stuff below:
The only hurdle left was that terraform providers mirror downloads the providers in the packed layout:
Packed layout: HOSTNAME/NAMESPACE/TYPE/terraform-provider-TYPE_VERSION_TARGET.zip is the distribution zip file obtained from the provider's origin registry.
while Terraform expects the plugin cache directory to use the unpacked layout:
Unpacked layout: HOSTNAME/NAMESPACE/TYPE/VERSION/TARGET is a directory containing the result of extracting the provider's distribution zip file.
So I converted the packed layout to the unpacked layout with the help of find and parallel:
find path/to/plugin-dir -name index.json -exec rm {} +`
find path/to/plugin-dir -name '*.json' | parallel --will-cite 'mkdir -p {//}/{/.}/linux_amd64; unzip {//}/*.zip -d {//}/{/.}/linux_amd64; rm {}; rm {//}/*.zip'
Here is result tree on server after my script:
> pwd
/opt/atlassian/pipelines/agent/build
> tree -d
.
├── android-sdk-linux
│ ├── build-tools
│ │ └── 28.0.3
...
├── app
│ ├── build
...
└── readme
8005 directories
Here is my script from https://opatry.net/2017/11/06/bitbucket-pipelines-for-android/:
ci_install.sh
#!/usr/bin/env bash
set -eu
cur_dir=$(cd "$(dirname "$0")"; pwd)
origin_dir=$(cd "${cur_dir}/.."; pwd)
app_dir="${origin_dir}/android"
output_dir="${origin_dir}/artifacts"
default_android_sdk_zip_version="3859397"
android_sdk_zip_version=${1:-${default_android_sdk_zip_version}}
case $(uname -s) in
Linux)
os="linux"
;;
Darwin)
os="darwin"
;;
CYGWIN*|MINGW*)
os="windows"
;;
*)
echo "!! Unsupported OS $(uname -s)"
exit 1
;;
esac
export ANDROID_HOME="${origin_dir}/android-sdk-${os}"
if [ ! -f "${ANDROID_HOME}/tools/bin/sdkmanager" ]; then
# Download and unzip Android sdk
echo "Downloading Android SDK '${android_sdk_zip_version}' for '${os}'"
wget "https://dl.google.com/android/repository/sdk-tools-${os}-${android_sdk_zip_version}.zip"
unzip "sdk-tools-${os}-${android_sdk_zip_version}.zip" -d "${ANDROID_HOME}"
rm "sdk-tools-${os}-${android_sdk_zip_version}.zip"
fi
# Add Android binaries to PATH
export PATH="${ANDROID_HOME}/tools:${ANDROID_HOME}/tools/bin:${ANDROID_HOME}/platform-tools:${PATH}"
# Accept all licenses (source: http://stackoverflow.com/questions/38096225/automatically-accept-all-sdk-licences)
echo "Auto Accepting licenses"
mkdir -p "$ANDROID_HOME/licenses"
echo -e "\n8933bad161af4178b1185d1a37fbf41ea5269c55" > "${ANDROID_HOME}/licenses/android-sdk-license"
echo -e "\n84831b9409646a918e30573bab4c9c91346d8abd" > "${ANDROID_HOME}/licenses/android-sdk-preview-license"
# Update android sdk
echo "Downloading packages described by ${cur_dir}/package_file.txt"
cat "${cur_dir}/package_file.txt"
( sleep 5 && while [ 1 ]; do sleep 1; echo y; done ) | sdkmanager --package_file="${cur_dir}/package_file.txt"
package_file.txt
platform-tools
build-tools;26.0.2
platforms;android-26
bitbucket-pipelines.yml:
image: java:8
pipelines:
branches:
master:
- step:
caches:
- gradle
- android-sdk
script:
- bash ./build/ci_install.sh
- ANDROID_HOME=$PWD/android-sdk-linux bash ./build/android.sh
definitions:
caches:
android-sdk: android-sdk-linux
gradle: gradle
In result:
Build teardown
You already have a 'gradle' cache so we won't create it again
Assembling contents of new cache 'android-sdk'
But in Pipelines -> Caches->Dependency caches cache for android-sdk is not displayed:
And at next run:
Cache "android-sdk": Downloading
Cache "android-sdk": Not found
All works fine. My ci_install.sh file was in /MyProject/utils/pipelines/ci_install.sh so android-sdk-linux was created in /opt/atlassian/pipelines/agent/build/utils/android-sdk-linux folder
So I moved file ci_install.sh to /MyProject/pipelines/ci_install.sh and now android-sdk-linux is created in /opt/atlassian/pipelines/agent/build/android-sdk-linux folder
Removed folder utils from my project
I'm writing a Jenkins shared library.
I'm not a coder myself and because of that I bump into many errors, which usually I don't know how to solve.
My shared library structure looks like so:
itai#Itais-MBP ~/src/company/pipeline_utils - (master) $ tree -f
.
├── ./1
├── ./README.md
├── ./functions.groovy
├── ./src
│ └── ./src/com
│ ├── ./src/com/company
│ │ ├── ./src/com/company/pipelines
│ │ │ └── ./src/com/company/pipelines/standardPipeline.groovy
│ │ └── ./src/com/company/utils
│ │ ├── ./src/com/company/utils/Git.groovy
│ │ ├── ./src/com/company/utils/SlackPostBuild.groovy
│ │ ├── ./src/com/company/utils/dockerBuild.groovy
│ │ ├── ./src/com/company/utils/dockerTest.groovy
│ │ ├── ./src/com/company/utils/notifyEmail.groovy
│ │ ├── ./src/com/company/utils/notifySlack.groovy
│ │ ├── ./src/com/company/utils/pipeline.groovy
│ │ └── ./src/com/company/utils/pipelineFunctions.groovy
│ └── ./src/com/company-in-idea
├── ./test_utils.groovy
├── ./utils.groovy
└── ./vars
├── ./vars/standardPipeline.groovy
└── ./vars/utils.groovy
The pipeline file looks like so:
itai#Itais-MBP ~/src/company/pipeline_utils - (master) $ cat ./vars/standardPipeline.groovy
import com.company.utils.Git;
def call(body) {
def config = [:]
body.resolveStrategy = Closure.DELEGATE_FIRST
body.delegate = config
body()
node {
// Clean workspace before doing anything
deleteDir()
try {
stage ('Clone') {
checkout scm
def committer = getCommitter()
}
stage ('Build') {
sh "echo 'building ${config.projectName} ...'"
}
stage ('Tests') {
parallel 'static': {
sh "echo 'shell scripts to run static tests...'"
},
'unit': {
sh "echo 'shell scripts to run unit tests...'"
},
'integration': {
sh "echo 'shell scripts to run integration tests...'"
}
}
stage ('Deploy') {
sh "echo 'deploying to server ${config.serverDomain}...'"
sh "echo Itai ganot"
sh "echo Itai"
}
} catch (err) {
currentBuild.result = 'FAILED'
throw err
}
}
}
You can see in the pipeline file that I import "com.company.utils.Git", the git function file looks like so:
itai#Itais-MBP ~/src/company/pipeline_utils - (master) $ cat ./src/com/company/utils/Git.groovy
#!/usr/bin/env groovy
package com.company.utils;
def sh_out(command) {
sh(returnStdout: true, script: command).trim()
}
def getCommitter(){
node {
committer = this.sh_out('git show -s --format=\'%ce\' | tr -d "\'" | cut -d# -f1')
return committer
}
}
def getRepo(){
node {
reponame = this.sh_out('basename $(git remote show -n origin | grep Push | cut -d: -f2- | rev | cut -c5- | rev)')
return reponame
}
}
void gitClean(){
node {
this.sh_out('''
sudo chown -R ubuntu:ubuntu .
if [ -d ".git" ]; then
sudo git reset --hard &>/dev/null
sudo git clean -fxd &>/dev/null
sudo git tag -d $(git tag) &>/dev/null
fi
|| true ''')
}
}
return this
The Jenkinsfile looks like so:
#Library("company") _
standardPipeline {
projectName = "Project1"
serverDomain = "Project1 Server Domain"
}
When I run the job, it fails with the following error:
java.lang.NoSuchMethodError: No such DSL method 'getCommitter' found
among steps [AddInteractivePromotion, ArtifactoryGradleBuild,
ArtifactoryMavenBuild, ConanAddRemote, ConanAddUser, InitConanClient,
MavenDescriptorStep, RunConanCommand, ansiColor, ansiblePlaybook,
archive...
As far as I understand, I've imported the git package into the pipeline so I don't understand why this function is not recognized.
Another problem I have is that the pipeline only "looks" at the standardPipeline.groovy file at projectDir/vars and not under src/com/company/pipelines/standardPipeline.groovy ... I even tried removing the vars dir but the pipeline keeps looking there... why is that?
Any idea what I'm doing wrong?
It looks like your line def committer = getCommitter() is what is calling the error because it's looking for a step named getCommitter(), instead of calling the Git class's method.
Referencing the documentation here, you should do something like this in the pipeline file:
def gitUtil = new Git()
def committer = gitUtil.getCommitter()
I am working in the following Dockerfile:
RUN apt-get update && \
DEBIAN_FRONTEND=noninteractive \
apt-get install -y \
curl \
apache2 \
php5 \
php5-cli \
libapache2-mod-php5 \
php5-gd \
php5-json \
php5-mcrypt \
php5-mysql \
php5-curl \
php5-memcached \
php5-mongo \
zend-framework
# Install Composer
RUN curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer && \
chown www-data /usr/local/bin/composer && composer --version
# Install usefull PHP tools
RUN composer global require sebastian/phpcpd && \
composer global require phpmd/phpmd && \
composer global require squizlabs/php_codesniffer
# Install xdebug after we install composer since it cause issues
# see https://getcomposer.org/doc/articles/troubleshooting.md#xdebug-impact-on-composer
RUN apt-get install -y php5-xdebug
As you may notice this install PHP 5.5.x and it comes with the default configuration which I would like to override with my own values.
I have the following directory structure:
docker-php55/
├── container-files
│ ├── config
│ │ └── init
│ │ └── vhost_default
│ └── etc
│ └── php.d
│ ├── zz-php-directories.ini
│ └── zz-php.ini
├── Dockerfile
├── LICENSE
├── README.md
└── run
The files zz-php-directories.ini and zz-php.ini are my configurations that I should write to /etc/php5/apache2/php.ini upon image creation. The content of the files is the following:
zz-php.ini
; Basic configuration override
expose_php = Off
memory_limit = 512M
post_max_size = 128M
upload_max_filesize = 128M
date.timezone = UTC
max_execution_time = 120
; Error reporting
display_errors = stderr
display_startup_errors = Off
error_reporting = E_ALL
; A bit of performance tuning
realpath_cache_size = 128k
; OpCache tuning
opcache.max_accelerated_files = 32000
; Temporarily disable using HUGE PAGES by OpCache.
; This should improve performance, but requires appropriate OS configuration
; and for now it often results with some weird PHP warning:
; PHP Warning: Zend OPcache huge_code_pages: madvise(HUGEPAGE) failed: Invalid argument (22) in Unknown on line 0
opcache.huge_code_pages=0
; Xdebug
[Xdebug]
xdebug.remote_enable = true
xdebug.remote_host = "192.168.3.1" // this IP should be the host IP
xdebug.remote_port = "9001"
xdebug.idekey = "XDEBUG_PHPSTORM"
zz-php-directories.ini
; Configure temp path locations
sys_temp_dir = /data/tmp/php
upload_tmp_dir = /data/tmp/php/uploads
session.save_path = /data/tmp/php/sessions
uploadprogress.file.contents_template = "/data/tmp/php/upload_contents_%s"
uploadprogress.file.filename_template = "/data/tmp/php/upt_%s.txt"
How do I override the default php.ini parameters on the image with the ones on those files upon image creation?
EDIT: Improve the question
To leave an example, zz-php.ini is a local file placed in my laptop|PC. As soon as I install PHP in the image it comes with a default configuration file, this mean I should have a file under /etc/php5/apache2/php.ini.
This default configuration file already has default values like for example: expose_php = On (again this is the default, others comes as ;realpath_cache_size =) so what I want to do is to change the value for the default file with the value from my file, in other words:
default (as in /etc/php5/apache2/php.ini) expose_php = On
override (as in zz-php.ini) expose_php = Off
At the end I should have the values from zz-php.ini overwrited in /etc/php5/apache2/php.ini
As for the host IP address I think I could use a ENV var and pass to the build as an argument, I am right? If no, then how would you get the host IP address needed for that setup?
That's two questions.
1) Just use the COPY instruction to copy your local php.ini into the image location. Eg:
COPY php.ini /etc/php5/apache2/php.ini
2) You don't want to hardcode any ip into your image. That needs to be done when the container is started. The standard way of doing this with docker is to specify an environment variable like HOST_IP and you use a shell script to make the modifications on the container at start time. For instance:
Your inject.sh script:
#!/usr/bin/bash
sed -i -E "s/xdebug.remote_host.*/xdebug.remote_host=$HOST_IP/" /etc/php5/apache2/php.ini
You need to add the inject.sh file to your image when you build it.
COPY inject.sh /usr/local/bin/
Then you can initialize and start your container as follow:
docker run -e HOST_IP=53.62.10.12 mycontainer bash -c "inject.sh && exec myphpapp"
The exec is needed to make sure the myphpapp becomes the main process of the container (ie: it has PID 1) otherwise it won't receive kill commands (like Ctrl-C).
In big list of recursive directory i am searching for directory name that does not contain a file with pattern ending with pattern "mt"
how can i search for directory name in this case .
I search net and found command to find directory name containing file with f :
find . -type f -name '*f*' | sed -r 's|/[^/]+$||' | sort | uniq
But how can i search for directory name w/o mt
find command but didn't found right argument
i am able to search for file name in recursive directory but here i am interested in searching for directory name in recursive pattern that does not contain a file containing a pattern with mt
You can use:
find . -type f -name '*f*' | grep -v 'mt$'
grep -v filters out every match to the pattern that follows.
directory tree
∮ tree -F
.
├── a/
│ └── x.mt
├── b/
│ ├── x/
│ ├── y/
│ │ └── x.mt
│ └── z/
├── blacklist
└── c/
create blacklist
∮ find . -name '*mt' | sed 's#/[^/]*mt$##; s#^#^#' > blacklist
∮ cat blacklist
^./b/y
^./a
find directories
∮ find . -type d | grep -v -f blacklist
.
./b
./b/x
./b/z
./c
This solution ignores all directories which directly contain *mt file.
It's easy to write a script (using fgrep) to ignore those (., ./b) which indirectly contain *mt file.
Something like this should work
find -type d -exec bash -c \
'
shopt -s nullglob dotglob;
found=;
for f in "{}"/*mt; do
[[ -f $f ]] && found=true && break;
done;
[[ $found == true ]] || echo "{}";
' \;
Change || to && if you instead want directories that do contain *mt
Might be an easier way to do this, but you can't delay glob expansion in -exec without eval (using which would cause security issues), so couldn't think of a better way then putting it into bash -c.