Duplicity backup script for S3 fails on CentOS 5 - duplicity

Hi I have not been able to setup backup to S3 using this guide:
http://www.cenolan.com/2008/12/how-to-incremental-daily-backups-amazon-s3-duplicity/
I get this error when I run the script:
Command line error: Expected 2 args, got 0
Enter 'duplicity --help' for help screen.
The duplicity log file:
2013-08-07_17:43:20: ... backing up filesystem
./duplicity-backup: line 60: --include=/home: No such file or directory
My script is like this:
!/bin/bash
# Set up some variables for logging
LOGFILE="/var/log/backup.log"
DAILYLOGFILE="/var/log/backup.daily.log"
HOST='hostname'
DATE=`date +%Y-%m-%d`
MAILADDR="user#domain.com"
# Clear the old daily log file
cat /dev/null > ${DAILYLOGFILE}
# Trace function for logging, don't change this
trace () {
stamp=`date +%Y-%m-%d_%H:%M:%S`
echo "$stamp: $*" >> ${DAILYLOGFILE}
}
# Export some ENV variables so you don't have to type anything
export AWS_ACCESS_KEY_ID="xxxxx"
export AWS_SECRET_ACCESS_KEY="xxx"
export PASSPHRASE="xx"
# Your GPG key
GPG_KEY=xxxx
# How long to keep backups for
OLDER_THAN="1M"
# The source of your backup
SOURCE=/
# The destination
# Note that the bucket need not exist
# but does need to be unique amongst all
# Amazon S3 users. So, choose wisely.
DEST="s3+http://xxxx"
FULL=
if [ $(date +%d) -eq 1 ]; then
FULL=full
fi;
trace "Backup for local filesystem started"
trace "... removing old backups"
duplicity remove-older-than ${OLDER_THAN} ${DEST} >> ${DAILYLOGFILE} 2>&1
trace "... backing up filesystem"
duplicity \
${FULL} \
--encrypt-key=${GPG_KEY} \
--sign-key=${GPG_KEY} \
--volsize=250 \
# --include=/vhosts \
# --include=/etc \
--include=/home \
--include=/root \
--exclude=/** \
${SOURCE} ${DEST} >> ${DAILYLOGFILE} 2>&1
trace "Backup for local filesystem complete"
trace "------------------------------------"
# Send the daily log file by email
cat "$DAILYLOGFILE" | mail -s "Duplicity Backup Log for $HOST - $DATE" $MAILADDR
# Append the daily log file to the main log file
cat "$DAILYLOGFILE" >> $LOGFILE
# Reset the ENV variables. Don't need them sitting around
export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
export PASSPHRASE=
Does anyone see what is wrong?

You should remove commented lines from script execution. From:
duplicity \
${FULL} \
--encrypt-key=${GPG_KEY} \
--sign-key=${GPG_KEY} \
--volsize=250 \
# --include=/vhosts \
# --include=/etc \
--include=/home \
--include=/root \
--exclude=/** \
${SOURCE} ${DEST} >> ${DAILYLOGFILE} 2>&1
to
duplicity \
${FULL} \
--encrypt-key=${GPG_KEY} \
--sign-key=${GPG_KEY} \
--volsize=250 \
--include=/home \
--include=/root \
--exclude=/** \
${SOURCE} ${DEST} >> ${DAILYLOGFILE} 2>&1

Related

Use .env file variables during Docker build

I am trying to use the sed command to replace variables during docker build. The variable I am attempting to do (to start) is $DATABASE_HOST. The value for that is coming from my .env file. I am reading online that environment variables are only available during run time if they come from the .env file. Due to this, my sed command is not registering.
Dockerfile:
# Dockerfile for Sphinx SE
# https://hub.docker.com/_/alpine/
FROM alpine:3.12
# https://sphinxsearch.com/blog/
ENV SPHINX_VERSION 3.4.1-efbcc65
# Install dependencies
RUN apk add --no-cache mariadb-connector-c-dev \
postgresql-dev \
wget \
sed
# set up and expose directories
RUN mkdir -pv /opt/sphinx/log /opt/sphinx/index
VOLUME /opt/sphinx/index
# http://sphinxsearch.com/downloads/sphinx-3.3.1-b72d67b-linux-amd64-musl.tar.gz
RUN wget http://sphinxsearch.com/files/sphinx-${SPHINX_VERSION}-linux-amd64-musl.tar.gz -O /tmp/sphinxsearch.tar.gz \
&& cd /opt/sphinx && tar -xf /tmp/sphinxsearch.tar.gz \
&& rm /tmp/sphinxsearch.tar.gz
# point to sphinx binaries
ENV PATH "${PATH}:/opt/sphinx/sphinx-3.4.1/bin"
RUN indexer -v
# redirect logs to stdout
RUN ln -sv /dev/stdout /opt/sphinx/log/query.log \
&& ln -sv /dev/stdout /opt/sphinx/log/searchd.log
# expose TCP port
EXPOSE 36307
EXPOSE 9306
# Copy base sphinx.conf file to container
VOLUME /opt/sphinx/conf
COPY ./sphinx.conf /opt/sphinx/conf/sphinx.conf
# Copy all docker sphinx.conf files
COPY ./configs/web-finder/docker/ /opt/sphinx/conf/
# look for and replace
RUN sed -i "s+DATABASE_HOST+${DATABASE_HOST}+g" /opt/sphinx/conf/sphinx.conf
# Concat the sphinx.conf files for all apps
# RUN cat /tmp/myconfig.append >> /etc/portage/make.conf && rm -f /tmp/myconfig.append
CMD indexer --all --config /opt/sphinx/conf/sphinx.conf \
&& searchd --nodetach --config /opt/sphinx/conf/sphinx.conf
.env file:
DATABASE_HOST=someport
DATABASE_USERNAME=someusername
DATABASE_PASSWORD=somepassword
DATABASE_SCHEMA=someschema
DATABASE_PORT=3306
SPHINX_PORT=36307
sphinx.conf:
searchd
{
listen = 127.0.0.1:$SPHINX_PORT
log = /opt/sphinx/searchd.log
query_log = /opt/sphinx/query.log
read_timeout = 5
max_children = 30
pid_file = /opt/sphinx/searchd.pid
seamless_rotate = 1
preopen_indexes = 1
unlink_old = 1
binlog_path = /opt/sphinx/
}
With sphinx the 'sphinx.conf' file can be 'executable'. Ie it can actully be a 'shell script' (or PHP, perl etc!)
Assuming your .env file makes real (runtime!) environment variables within the container (not overly familiar with Docker), then your sphinx.conf file could be ...
#!/bin/sh
set -eu
cat <<EOF
searchd
{
listen = 127.0.0.1:$SPHINX_PORT
log = /opt/sphinx/searchd.log
query_log = /opt/sphinx/query.log
read_timeout = 5
max_children = 30
pid_file = /opt/sphinx/searchd.pid
seamless_rotate = 1
preopen_indexes = 1
unlink_old = 1
binlog_path = /opt/sphinx/
}
EOF
And because it a shell script, the variables will automatically be expanded :)
Need it executable too!
RUN chmod a+x /opt/sphinx/conf/sphinx.conf
Then dont need the sed command in Dockerfile at all!

Drop DynamoDB if it exists

I am trying to setup dynamodb in my local using docker. I wish to control is initialization by using makefile. Here is the makefile I am using
TABLE_NAME="users"
create_db:
#aws dynamodb --endpoint-url http://localhost:8042 create-table \
--table-name $(TABLE_NAME) \
--attribute-definitions \
AttributeName=userID,AttributeType=N \
--key-schema \
AttributeName=userID,KeyType=HASH \
--provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5 >> /dev/null;
drop_db: check_db
check_db; if [test $$? -eq 1] then \
#aws dynamodb --endpoint-url http://localhost:8042 delete-table --table-name $(TABLE_NAME); \
fi
check_db:
-#aws dynamodb --endpoint-url http://localhost:8042 describe-table --table-name $(TABLE_NAME);
AWS does not give DROP IF EXISTS functionality like MYSQL, so I am trying to use output of describe table to check the presence of the table. However getting the following error
check_db; if [test $? -eq 1] then \
#aws dynamodb --endpoint-url http://localhost:8042 delete-table --table-name "requests"; \
fi
/bin/sh: -c: line 0: syntax error near unexpected token `fi'
/bin/sh: -c: line 0: `check_db; if [test $? -eq 1] then #aws dynamodb --endpoint-url http://localhost:8042 delete-table --table-name "requests"; fi'
make: *** [drop_db] Error 2
I am new to makefile and cannot figure out how to solve the error. What is wrong in the above makefile? And is there any better way to check presence of dynamo table
It's not a Makefile issue, it's a syntax error in your shell script. Basically you need a semicolon before then.
$ false; if [test $? -eq 1] then echo foo; fi
bash: syntax error near unexpected token `fi'
You also need to decide to either use [ or test, as the current syntax is also incorrect.
$ false; if [test $? -eq 1]; then echo foo; fi
Command '[test' not found, did you mean:
...
Working version:
$ false; if [ $? -eq 1 ]; then echo foo; fi
foo
I just made this workaround to mimic DROP IF EXISTS functionality with dynamoDB
drop_db: check_db
#if grep -q -i "active" a.out ; then \
aws dynamodb --endpoint-url http://localhost:8042 delete-table --table-name $(TABLE_NAME) >> /dev/null; \
rm a.out; \
fi
check_db:
#aws dynamodb --endpoint-url http://localhost:8042 describe-table --table-name $(TABLE_NAME) --output text &> a.out;

Grep sorted files in S3 folder

I'd like to sort files in S3 folders and then check if files contain a certain string.
When I usually want to grep a file I do the following:
aws s3 cp s3://s3bucket/location/file.csv.gz - | zcat | grep 'string_to_find'
I see I can sort files like this:
aws s3api list-objects-v2 \
--bucket s3bucket \
--prefix location \
--query 'reverse(sort_by(Contents,&LastModified))'
Tried something like this so far but got broken pipe:
aws s3api list-objects-v2 \
--bucket s3bucket \
--prefix location \
--query 'reverse(sort_by(Contents,&LastModified))' | cp - | zcat | grep 'string_to_find'
You can specify which fields to output and force them into text-only:
aws s3api list-objects-v2 \
--bucket s3bucket \
--prefix location \
--query 'reverse(sort_by(Contents,&LastModified))[].[Key]' \
--output text
Basically, the sort_by and reverse output the Contents array, and this extracts the Key element. I put [Key] in square brackets to force each result onto its own line.

Updating Heroku Buildpack to use ffmpegthumbnailer 2.2.0

The app I am working on is currently using a buildpack that is using ffmpegthumbnailer 2.0.8 and I need a feature that was added in version 2.1.2. I have forked the repo of the buildpack we are currently using https://github.com/akomic/heroku-buildpack-ffmpegthumbnailer and updated the bin/compile file to point the download_url to "https://github.com/dirkvdb/ffmpegthumbnailer/archive/2.2.0.tar.gz" but when I add my forked repo to the app and run heroku run "ffmpegthumbnailer -version" to verify that it worked I get a bash: ffmpegthumbnailer: command not found error.
Here is the original bin/compile:
#!/bin/sh
indent() {
sed -u 's/^/ /'
}
echo "-----> Install ffmpegthumbnailer"
BUILD_DIR=$1
VENDOR_DIR="vendor"
DOWNLOAD_URL="http://www.aksiom.net/stuff/ffmpegthumbnailer_2.0.8-2.bin.tar.gz"
echo "DOWNLOAD_URL = " $DOWNLOAD_URL | indent
cd $BUILD_DIR
mkdir -p $VENDOR_DIR
cd $VENDOR_DIR
curl -L --silent $DOWNLOAD_URL | tar zx
echo "exporting PATH and LIBRARY_PATH" | indent
PROFILE_PATH="$BUILD_DIR/.profile.d/ffmpeg.sh"
mkdir -p $(dirname $PROFILE_PATH)
echo 'export PATH="$PATH:$HOME/vendor/ffmpegthumbnailer_2.0.8-2/bin"' >> $PROFILE_PATH
echo 'export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$HOME/vendor/ffmpegthumbnailer_2.0.8-2/lib"' >> $PROFILE_PATH
Here is my updated bin/compile in the forked repo:
#!/bin/sh
indent() {
sed -u 's/^/ /'
}
echo "-----> Install ffmpegthumbnailer"
BUILD_DIR=$1
VENDOR_DIR="vendor"
DOWNLOAD_URL="https://github.com/dirkvdb/ffmpegthumbnailer/archive/2.2.0.tar.gz"
echo "DOWNLOAD_URL = " $DOWNLOAD_URL | indent
cd $BUILD_DIR
mkdir -p $VENDOR_DIR
cd $VENDOR_DIR
curl -L --silent $DOWNLOAD_URL | tar zx
echo "exporting PATH and LIBRARY_PATH" | indent
PROFILE_PATH="$BUILD_DIR/.profile.d/ffmpeg.sh"
mkdir -p $(dirname $PROFILE_PATH)
echo 'export PATH="$PATH:$HOME/vendor/ffmpegthumbnailer_2.0.8-2/bin"' >> $PROFILE_PATH
echo 'export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$HOME/vendor/ffmpegthumbnailer_2.0.8-2/lib"' >> $PROFILE_PATH
Other info: I add the buildpack using heroku buildpacks:add <link_to_forked_repo>
Thank you for any help!
This is because the original akomic tarball specified is not the source code, but rather a custom zip of compiled binaries and libraries. Notice that the compile script has no make step, for instance. There is no documentation for how this custom tarball created, but presumably if you go through the same process with a newer version of the source code, and then upload it to a place this script can access it, it will work again. I'm trying to figure this out now.

Minix 3.2.1 Makefile issue

I'm trying to run make with the below Makefile and I get the following error Makefile:2: *** missing separator. Stop.. This Makefile is the original file from Minix 3.2.1. I haven't done any changes. I check from the tabs problem. Can someone help me?
Thanks.
# Makefile for the kernel image.
.include <bsd.own.mk>
.include "nbsd.config"
.include <bsd.own.mk>
.include <bsd.sys.mk>
u=/usr
MDEC= /usr/mdec
GEN_FILES= *.bak image kernel *.iso *.iso.gz cdfdimage rootimage src
# LSC detect where were built the objects files
PROGROOT:= ..
.if "${MAKEOBJDIR:S,${.CURDIR},,}" != ""
PROGROOT:= ${MAKEOBJDIR:S,releasetools,,}
.endif
# Specify the programs that are part of the system image.
KERNEL= ${PROGROOT}/kernel/kernel
# PROGRAMS are in the order they should be loaded by boot
PROGRAMS+= ${PROGROOT}/servers/ds/ds
PROGRAMS+= ${PROGROOT}/servers/rs/rs
PROGRAMS+= ${PROGROOT}/servers/pm/pm
PROGRAMS+= ${PROGROOT}/servers/sched/sched
PROGRAMS+= ${PROGROOT}/servers/vfs/vfs
PROGRAMS+= ${PROGROOT}/drivers/memory/memory
.if ${MACHINE_ARCH} == "i386"
PROGRAMS+= ${PROGROOT}/drivers/log/log
.endif
PROGRAMS+= ${PROGROOT}/drivers/tty/tty
PROGRAMS+= ${PROGROOT}/servers/mfs/mfs
PROGRAMS+= ${PROGROOT}/servers/vm/vm
PROGRAMS+= ${PROGROOT}/servers/pfs/pfs
PROGRAMS+= ${PROGROOT}/servers/init/init
usage:
#echo " " >&2
#echo "Master Makefile to create new MINIX configuration." >& 2
#echo "Root privileges are required." >&2
#echo " " >&2
#echo "Usage:" >&2
#echo " make includes # Install include files" >&2
#echo " make depend # Generate dependency files" >&2
#echo " make services # Compile and install all services" >&2
#echo " make install # Make image, and install to hard disk" >&2
#echo " make hdboot # Make image, and install to hard disk" >&2
#echo " make bootable # Make hard disk bootable" >&2
#echo " make nbsd_fetch # Download current NetBSD reference sources" >&2
#echo " make nbsd_diff # Update minix-port.patch in NetBSD sources" >&2
#echo " make clean # Remove all compiler results, except libs" >&2
#echo " " >&2
#echo "To create a fresh MINIX configuration, try:" >&2
#echo " make clean install # new boot image" >&2
#echo " make fresh install # new everything" >&2
#echo " " >&2
all: services
# rebuild the program or system libraries
includes:
$(MAKE) -C ../ includes
depend: includes .gitignore
$(MAKE) -C ../ depend
.gitignore: Makefile
echo $(GEN_FILES) | tr ' ' '\n' >.gitignore
services: includes kernel servers .WAIT drivers
kernel: includes
$(MAKE) -C ../kernel
servers: includes
$(MAKE) -C ../servers all install
drivers: includes servers
$(MAKE) -C ../drivers all install
# make bootable and place system images
bootable:
exec su root mkboot bootable ${DESTDIR}
hdboot: services .WAIT do-hdboot
do-hdboot:
#rm -rf ${DESTDIR}/boot/minix/.temp/
${INSTALL_DIR} ${DESTDIR}/boot/minix/.temp
# mod_0 is used to make alphabetical order equal to the boot order
#n=0; \
for i in ${PROGRAMS}; \
do \
n=`expr $$n + 1`; \
[ "$$n" -ge 10 ] && prefix="mod" || prefix="mod0"; \
newname="${DESTDIR}/boot/minix/.temp/$${prefix}$${n}_`basename $$i`"; \
${INSTALL} $$i $$newname; \
done
#cp ${PROGROOT}/kernel/kernel ${DESTDIR}/boot/minix/.temp/
#if [ "${MKINSTALLBOOT:Uno}" != "no" ] ; then \
${STRIP} -s ${DESTDIR}/boot/minix/.temp/* ; \
gzip ${DESTDIR}/boot/minix/.temp/mod* ; \
${HOST_SH} mkboot hdboot ${DESTDIR}; \
${HOST_SH} ../commands/update_bootcfg/update_bootcfg.sh;\
else \
${INSTALL_DIR} ${DESTDIR}/multiboot; \
${INSTALL} ${DESTDIR}/boot/minix/.temp/* ${DESTDIR}/multiboot; \
fi
install:
${MAKE} includes services hdboot
# download and update NetBSD reference sources.
nbsd_fetch:
export CVS_RSH=ssh; \
export OLDPWD=`pwd`; \
echo "retrieving hierarchies from ${NBSD_CVSROOT}"; \
IFS=,; \
cd ..; \
cat releasetools/nbsd_ports | grep -v '^#' | while read port ; \
do set $$port; \
date=$$1; minixpath=$$2; origpath=$$3; \
if [ $$# -lt 3 ]; then origpath=$$2; fi; \
echo "retrieving $$origpath .."; \
cvs -q -d ${NBSD_CVSROOT} co -N -D "$$date UTC" -d nbsdsrc "src/$$origpath" ; \
done; \
cd $${OLDPWD};
nbsd_diff:
find .. -name minix-port.patch | xargs rm
cat nbsd_ports | grep -v '^#' | \
( cd .. && awk -F, '{ minixpath=$$2; origpath=$$3; if(NF < 3) { origpath=$$2; } system("sh releasetools/nbsd_diff.sh " \
"nbsdsrc/src/"origpath" "minixpath" "minixpath"/minix-port.patch");}' )
find .. -name minix-port.patch | xargs wc -l | sort -n
# clean up compile results
clean:
$(MAKE) -C ../kernel $#
$(MAKE) -C ../servers $#
$(MAKE) -C ../drivers $#
rm -rf $(GEN_FILES)
cleandepend::
$(MAKE) -C ../kernel $#
$(MAKE) -C ../servers $#
$(MAKE) -C ../drivers $#
MINIX uses BSD make, which typically uses . to prefix its specific extensions. You installed GNU make, which has another set of extensions, not prefixed with .; furthermore you put GNU make ahead in the PATH order without renaming it. This is not going to take off.
What we usually do is to name GNU make under the name gmake or gnumake, and invoke it under that name when needed (for example, to compile packages from GNU ecosystem, or with a strong Linux bias.)
Another possibility you might consider is to install (if not already done) the bmake package, and uses that command when trying to compile programs which expect the BSD breed of make (this includes MINIX itself.)

Resources