We're using the BeagleBone Black running Angstrom Linux and the opkg package manager to power some of our systems. We need to ensure that we have consistent and reliable access to specific versions of opkg packages. I've set up an in-house opkg repository. Is there any way to sync packages between repositories ? e.g. I'd like to copy specific packages from public / not always accessible repositories to our internal repository, both for speed and reliable access.
After some fooling around with various packages, etc, I found a way of cloning (parts of) a repository using an Ubuntu system. Here's the steps I took:
# Install apache
sudo apt-get install apache2
# Install git
sudo apt-get install git
# Download the opkg-utils from the Yocto Project
git clone http://git.yoctoproject.org/git/opkg-utils
# Build the opkg-utils
cd opkg-utils && make; cd -
# Move them to a common directory
mv opkg-utils /usr/local/share\
# Add them to my path
echo "PATH=\"\$PATH:/usr/local/share/opkg-utils\"" >> /etc/environment
# Update my environment
source /etc/environment
# Create the structure of my repository
mkdir -p /var/www/repositories/opkg/beaglebone
# Create an index for the packages
opkg-make-index -l Packages.filelist -p Packages /var/www/repositories/opkg/beaglebone
cd /var/www/repositories/opkg/beaglebone
gzip -c Packages > Packages.gz
On my client BeagleBone Blacks, to setup access to this repository:
echo "src/gz reponame http://myserver/repositories/opkg/beaglebone" > /etc/opkg/rms-feed.conf
chmod 666 /etc/opkg/reponame-feed.conf
opkg update
On my developer machines, any time I need to backup a package:
#!/bin/bash
###############################################################################
#
# bbb_clone_package_to_internal_repo.sh
#
# Description:
# Clones an ipkg / opkg package to the internal repository server so that it can be deployed
# to BeagleBone Black clients on demand. This is so that we can have backups in
# the event that a public server becomes temporarily or permanently
# inaccessible.
#
# Pre-conditions:
# 1) The given package file must exist at the path specified.
#
# Post-conditions:
# 1) The given package file will be sent to the internal repository server.
# 2) The opkg repository indexes will all be updated
#
# Parameters:
# -p <file path.opk> : The package to be cloned
#
###############################################################################
PACKAGE_FILE_PATH=""
SERVER="myserver"
ERR_INVALID_PACKAGE_FILE_NAME=1
ERR_PACKAGE_FILE_NOT_ACCESSIBLE=2
ERR_FAILED_TO_COPY_PACKAGE_TO_SERVER=3
ERR_FAILED_TO_DEPLOY_PACKAGE_ON_SERVER=4
usage()
{
cat << EOF
usage: $0 [options]
This script copies a remote ipkg/opkg file to the $SERVER server for subsequent
deployment to BeagleBone Black boards.
OPTIONS:
-p <file path.[io]pk> The package file to be deployed
-h,? Show this message
EOF
}
while getopts “p:h?” OPTION
do
case $OPTION in
p)
PACKAGE_FILE_PATH="$OPTARG"
;;
h)
usage
exit
;;
?)
usage
exit
;;
esac
done
if [[ -z "$PACKAGE_FILE_PATH" || ! ( "$PACKAGE_FILE_PATH" =~ \.[io]pk$ ) ]]; then
echo "The package file must not be blank and must have an .ipk or .opk suffix"
exit $ERR_INVALID_PACKAGE_FILE_NAME
fi
# Retrieve the package
wget -q "$PACKAGE_FILE_PATH"
RESULT="$?"
if [[ $RESULT -ne 0 ]]; then
echo "Failed to retrieve file $PACKAGE_FILE_PATH with result $RESULT"
exit $ERR_PACKAGE_FILE_NOT_ACCESSIBLE
fi
# Deploy the package to myserver
PACKAGE_FILE_NAME="$(basename $PACKAGE_FILE_PATH)"
REPOSITORY_ROOT="/var/www/repositories/opkg/beaglebone"
scp "$PACKAGE_FILE_NAME" root#$SERVER:$REPOSITORY_ROOT
RESULT="$?"
if [[ $RESULT -ne 0 ]]; then
echo "Failed to copy file $PACKAGE_FILE_NAME to server with result $RESULT"
exit $ERR_FAILED_TO_COPY_PACKAGE_TO_SERVER
fi
ssh root#$SERVER "chmod 644 $REPOSITORY_ROOT/$PACKAGE_FILE_NAME; opkg-make-index -l $REPOSITORY_ROOT/Packages.filelist -p $REPOSITORY_ROOT/Packages -r $REPOSITORY_ROOT/Packages $REPOSITORY_ROOT && gzip -c $REPOSITORY_ROOT/Packages > $REPOSITORY_ROOT/Packages.gz"
RESULT="$?"
if [[ $RESULT -ne 0 ]]; then
echo "Failed to deploy file $PACKAGE_FILE_NAME in repository with result $RESULT"
exit $ERR_FAILED_TO_DEPLOY_PACKAGE_ON_SERVER
fi
exit 0
Related
I am trying to build a docker image that contains all of the necessary plugins/providers that several source repos need, so that when an automated terraform validate runs, it doesn't have to download gigs of redundant data.
However, I recognize that this provides for a maintenance problem in that someone may update a plugin version, and that would needed to be downloaded, since the docker image would not contain it.
The question
How can I pre-download all providers and plugins
Tell the CLI use those predownloaded plugins AND
also tell it that, if it doesn't find what it needs locally, then it can go to the network
Below are the relevant file:
.terraformrc
plugin_cache_dir = "$HOME/.terraform.d/plugin-cache"
disable_checkpoint = true
provider_installation {
filesystem_mirror {
path = "$HOME/.terraform/providers"
}
direct {
}
}
tflint (not relevant to this question, but it shows up in the below Dockerfile)
plugin "aws" {
enabled = true
version = "0.21.1"
source = "github.com/terraform-linters/tflint-ruleset-aws"
}
plugin "azurerm" {
enabled = true
version = "0.20.0"
source = "github.com/terraform-linters/tflint-ruleset-azurerm"
}
Dockerfile
FROM ghcr.io/terraform-linters/tflint-bundle AS base
LABEL name=tflint
RUN adduser -h /home/jenkins -s /bin/sh -u 1000 -D jenkins
RUN apk fix && apk --no-cache --update add git terraform openssh
ADD .terraformrc /home/jenkins/.terraformrc
RUN mkdir -p /home/jenkins/.terraform.d/plugin-cache/registry.terraform.io
ADD .tflint.hcl /home/jenkins/.tflint.hcl
WORKDIR /home/jenkins
RUN tflint --init
FROM base AS build
ARG SSH_PRIVATE_KEY
RUN mkdir /root/.ssh && \
echo "${SSH_PRIVATE_KEY}" > /root/.ssh/id_ed25519 && \
chmod 400 /root/.ssh/id_ed25519 && \
touch /root/.ssh/known_hosts && \
ssh-keyscan mygitrepo >> /root/.ssh/known_hosts
RUN git clone git#mygitrepo:wrai/tools/g.git
RUN git clone git#mygitrepo:myproject/a.git && \
git clone git#mygitrepo:myproject/b.git && \
git clone git#mygitrepo:myproject/c.git && \
git clone git#mygitrepo:myproject/d.git && \
git clone git#mygitrepo:myproject/e.git && \
git clone git#mygitrepo:myproject/f.git
RUN ls -1d */ | xargs -I {} find {} -name '*.tf' | xargs -n 1 dirname | sort -u | \
xargs -I {} -n 1 -P 20 terraform -chdir={} providers mirror /home/jenkins/.terraform.d
RUN chown -R jenkins:jenkins /home/jenkins
USER jenkins
FROM base AS a
COPY --from=build /home/jenkins/a/ /home/jenkins/a
RUN cd /home/jenkins/a && terraform init
FROM base AS b
COPY --from=build /home/jenkins/b/ /home/jenkins/b
RUN cd /home/jenkins/b && terraform init
FROM base AS c
COPY --from=build /home/jenkins/c/ /home/jenkins/c
RUN cd /home/jenkins/c && terraform init
FROM base AS azure_infrastructure
COPY --from=build /home/jenkins/d/ /home/jenkins/d
RUN cd /home/jenkins/d && terraform init
FROM base AS aws_infrastructure
COPY --from=build /home/jenkins/e/ /home/jenkins/e
RUN cd /home/jenkins/e && terraform init
Staging plugins:
This is most easily accomplished with the plugin cache dir setting in the CLI. This supersedes the old usage with the -plugin-dir=PATH argument for the init command. You could also set a filesystem mirror in each terraform block within the root module config, but this would be cumbersome for your use case. In your situation, you are already configuring this in your .terraformrc, but the filesystem_mirror path conflicts with the plugin_cache_dir. You would want to resolve that conflict, or perhaps remove the mirror block entirely.
Use staged plugins:
Since the setting is captured in the CLI configuration file within the Dockerfile, this would be automatically used in future commands.
Download additional plugins if necessary:
This is default behavior of the init command, and therefore requires no further actions on your part.
Side note:
The jenkins user typically is /sbin/nologin for shell and /var/lib/jenkins for home directory. If the purpose of this Docker image is for a Jenkins build agent, then you may want the jenkins user configuration to be more aligned with the standard.
TL;DR:
Configure the terraform plugin cache directory
Create directory with a single TF file containing required_providers block
Run terraform init from there
...
I've stumbled over this question as I tried to figure out the same thing.
I first tried leveraging an implied filesystem_mirror by running terraform providers mirror /usr/local/share/terraform/plugins in a directory containing only one terraform file containing the required_providers block. This works fine as long as you only use the versions of the providers you mirrored.
However, it's not possible to use a different version of a provider than the one you mirrored, because:
Terraform will scan all of the filesystem mirror directories to see which providers are placed there and automatically exclude all of those providers from the implied direct block.
I've found it to be a better solution to use a plugin cache directory instead. EDIT: You can prefetch the plugins by setting TF_PLUGIN_CACHE_DIR to some directory and then running terraform init in a directory that only declares the required_providers.
Previously overengineered stuff below:
The only hurdle left was that terraform providers mirror downloads the providers in the packed layout:
Packed layout: HOSTNAME/NAMESPACE/TYPE/terraform-provider-TYPE_VERSION_TARGET.zip is the distribution zip file obtained from the provider's origin registry.
while Terraform expects the plugin cache directory to use the unpacked layout:
Unpacked layout: HOSTNAME/NAMESPACE/TYPE/VERSION/TARGET is a directory containing the result of extracting the provider's distribution zip file.
So I converted the packed layout to the unpacked layout with the help of find and parallel:
find path/to/plugin-dir -name index.json -exec rm {} +`
find path/to/plugin-dir -name '*.json' | parallel --will-cite 'mkdir -p {//}/{/.}/linux_amd64; unzip {//}/*.zip -d {//}/{/.}/linux_amd64; rm {}; rm {//}/*.zip'
I'm using this container to set up X11 in GitPod.
ARG base
FROM ${base}
# Dazzle does not rebuild a layer until one of its lines are changed. Increase this counter to rebuild this layer.
ENV TRIGGER_REBUILD=1
# Install Xvfb, JavaFX-helpers and Openbox window manager
RUN sudo install-packages xvfb x11vnc xterm openjfx libopenjfx-java openbox
# Overwrite this env variable to use a different window manager
ENV WINDOW_MANAGER="openbox"
USER root
# Change the default number of virtual desktops from 4 to 1 (footgun)
RUN sed -ri "s/<number>4<\/number>/<number>1<\/number>/" /etc/xdg/openbox/rc.xml
# Install novnc
RUN git clone --depth 1 https://github.com/novnc/noVNC.git /opt/novnc \
&& git clone --depth 1 https://github.com/novnc/websockify /opt/novnc/utils/websockify
COPY novnc-index.html /opt/novnc/index.html
# Add VNC startup script
COPY start-vnc-session.sh /usr/bin/
RUN chmod +x /usr/bin/start-vnc-session.sh
USER gitpod
# This is a bit of a hack. At the moment we have no means of starting background
# tasks from a Dockerfile. This workaround checks, on each bashrc eval, if the X
# server is running on screen 0, and if not starts Xvfb, x11vnc and novnc.
RUN echo "export DISPLAY=:0" >> /home/gitpod/.bashrc.d/300-vnc
RUN echo "[ ! -e /tmp/.X0-lock ] && (/usr/bin/start-vnc-session.sh &> /tmp/display-\${DISPLAY}.log)" >> /home/gitpod/.bashrc.d/300-vnc
USER root
### checks ###
# no root-owned files in the home directory
RUN notOwnedFile=$(find . -not "(" -user gitpod -and -group gitpod ")" -print -quit) \
&& { [ -z "$notOwnedFile" ] \
|| { echo "Error: not all files/dirs in $HOME are owned by 'gitpod' user & group"; exit 1; } }
USER gitpod
This is where it gets sketchy :
# Install novnc
RUN git clone --depth 1 https://github.com/novnc/noVNC.git /opt/novnc \
&& git clone --depth 1 https://github.com/novnc/websockify /opt/novnc/utils/websockify
COPY novnc-index.html /opt/novnc/index.html
I get this output please help !
COPY failed: file not found in build context or excluded by .dockerignore: stat novnc-index.html: file does not exist
Knowing that my dockerfile is in /src and i'm building in /src . I tried to rebuild with the --no-cache flag and use export DOCKER_BUILDKIT=1 . But still I'm stuck with this problem .
I am trying to build an Docker base image using livemedia-creator on CentOS 7.5 with latest patches installed is failing. Below is the error I am getting.
# livemedia-creator --make-tar --no-virt --iso=CentOS-6.10-x86_64-netinstall.iso --ks=centos-6.ks --image-name=centos-root.tar.xz
Starting package installation process
The installation was stopped due to incomplete spokes detected while running in non-interactive cmdline mode. Since there cannot be any questions in cmdline mode, edit your kickstart file and retry installation.
The exact error message is:
CmdlineError: Missing package: firewalld.
The installer will now terminate.
The kickstart file which I am using is as below
url --url="http://mirrors.kernel.org/centos/6.9/os/x86_64/"
install
keyboard us
lang en_US.UTF-8
rootpw --lock --iscrypted locked
authconfig --enableshadow --passalgo=sha512
timezone --isUtc Etc/UTC
selinux --enforcing
#firewall --disabled
firewall --disable
network --bootproto=dhcp --device=eth0 --activate --onboot=on
reboot
bootloader --location=none
# Repositories to use
repo --name="CentOS" --baseurl=http://mirror.centos.org/centos/6.9/os/x86_64/ --cost=100
repo --name="Updates" --baseurl=http://mirror.centos.org/centos/6.9/updates/x86_64/ --cost=100
# Disk setup
zerombr
clearpart --all
part / --size 3000 --fstype ext4
%packages --excludedocs --nobase --nocore
vim-minimal
yum
bash
bind-utils
centos-release
shadow-utils
findutils
iputils
iproute
grub
-*-firmware
passwd
rootfiles
util-linux-ng
yum-plugin-ovl
%end
%post --log=/tmp/anaconda-post.log
# Post configure tasks for Docker
# remove stuff we don't need that anaconda insists on
# kernel needs to be removed by rpm, because of grubby
rpm -e kernel
yum -y remove dhclient dhcp-libs dracut grubby kmod grub2 centos-logos \
hwdata os-prober gettext* bind-license freetype kmod-libs dracut
yum -y remove dbus-glib dbus-python ebtables \
gobject-introspection libselinux-python pygobject3-base \
python-decorator python-slip python-slip-dbus kpartx kernel-firmware \
device-mapper* e2fsprogs-libs sysvinit-tools kbd-misc libss upstart
#clean up unused directories
rm -rf /boot
rm -rf /etc/firewalld
# Randomize root's password and lock
dd if=/dev/urandom count=50 | md5sum | passwd --stdin root
passwd -l root
#LANG="en_US"
#echo "%_install_lang $LANG" > /etc/rpm/macros.image-language-conf
awk '(NF==0&&!done){print "override_install_langs='$LANG'\ntsflags=nodocs";done=1}{print}' \
< /etc/yum.conf > /etc/yum.conf.new
mv /etc/yum.conf.new /etc/yum.conf
echo 'container' > /etc/yum/vars/infra
rm -f /usr/lib/locale/locale-archive
#Setup locale properly
localedef -v -c -i en_US -f UTF-8 en_US.UTF-8
#disable services
for serv in `/sbin/chkconfig|cut -f1`; do /sbin/chkconfig "$serv" off; done;
mv /etc/rc1.d/S26udev-post /etc/rc1.d/K26udev-post
rm -rf /var/cache/yum/*
rm -f /tmp/ks-script*
rm -rf /etc/sysconfig/network-scripts/ifcfg-*
#Generate installtime file record
/bin/date +%Y%m%d_%H%M > /etc/BUILDTIME
%end
I am not able to figure out from where firewalld is being picked. Any thought how to fix this issue.
I am new to Docker and working on developing the Docker image for our application in Ubuntu environment.
However, the below command is not working when executed from within the Dockerfile/ from within the docker-entrypoint file.
command: “jar xf ./abc.ear”
_/docker-entrypoint.sh: 69: /docker-entrypoint.sh: jar: not found**
I verified and ear file is present in the directory.
I tried passing the full path to ear or passed the full path to Jar command, however, no success.
Please help.
...............
#!/bin/sh
set -e
start=$(date +'%s')
# Setting Environment Variables
DEPLOY_DIR=/home/docker/xyz
SCRIPT_DIR=/usr/local/src
if [ "$(ls -A $DEPLOY_DIR/Install 2> /dev/null)" = "" ]; then
echo "The directory $DEPLOY_DIR/Install is empty."
# Fetch Installable from Artifactory
echo "[INFO] Extracting files from Artifactory"
mkdir -p $DEPLOY_DIR
cd $DEPLOY_DIR
wget -nv ArtifactoryPath
unzip "123.zip" -d $DEPLOY_DIR
# Install
cd $DEPLOY_DIR/
Install command
# Configure JAVA
echo "[INFO] Linking java folder"
ln -s /usr/lib/jvm/java-8-oracle $DEPLOY_DIR/Install/jdk
# Explode ear and war files
echo "[INFO] Explode ear and war files\n"
cd $DEPLOY_DIR/Install/jboss/deployments
ls -al
mv "$WFC_DEPLOY_DIR/Install/jboss/deployments/abc.ear" "$WFC_DEPLOY_DIR/Install/jboss/deployments/abc-old.ear"
mkdir -p abc.ear
cd $DEPLOY_DIR/Install/jboss/deployments/abc.ear
echo $PWD
mv "$DEPLOY_DIR/Install/jboss/deployments/abc-old.ear" ./
ls -al
jar xvf "$DEPLOY_DIR/Install/jboss/deployments/abc.ear/abc-old.ear"
rm -rf $DEPLOY_DIR/Install/jboss/deployments/abc.ear/abc-old.ear
else
echo "$DEPLOY_DIR/Install is not empty."
fi
Any idea why I'm getting this. I can successfully ping the server from my local machine. which sshpass is showing valid output and I'm using Cygwin64.
$ sshpass -p loriK0ba scp SSUA-HG.war SSUA-RCd.war loki12:/tmp/ARUNKS
lost connection
To install sshpass, I followed the following steps then, -V shows me valid output:
# IMPORTANT: You need to have sshpass utility installed on your local machine.
# ============================================================================
# HOW to get sshpass:
# STEPS: Run the following commands in Cygwin console window - This is a one time action.
# a. cd /cygdrive/c
# b. wget http://downloads.sourceforge.net/project/sshpass/sshpass/1.05/sshpass-1.05.tar.gz
# c. tar -xvzpf sshpass-1.05.tar.gz
# d. cd sshpass-1.05
# e. sh configure
# f. make install
# g. which sshpass
# ============================================================================
.
sh configure; sleep 5; make install; sleep 5; which sshpass
Version of sshpass is:
$ sshpass -V
sshpass 1.05 (C) 2006-2011 Lingnu Open Source Consulting Ltd.
This program is free software, and can be distributed under the terms of the GPL
See the COPYING file for more information.
PS: When I run sshpass with the following command, it doesnt' error out but also not creating giga.txt file on the target server (thus it's not reaching / doing anything). Same user (case sensitive) exist on both local box (where Cygwin is running) and on the target machine.
sshpass -p loriK0ba ssh -q loki12 'rm -rf /tmp/ARUNKS/* 2>/dev/null; mkdir /tmp/ARUNKS 2>/dev/null'