Makefile for building an rpm works locally, but not in Jenkins - jenkins

I have a makefile for building debian and rpm packages. I have two Jenkins environments, one for Ubuntu and one for CentOS. The debian package works no problem, and the rpm make command works on my machine, but not on Jenkins. Jenkins returns the following error:
cp: cannot stat /root/rpmbuild/SOURCES/myfile.file': No such file or directory
error: Bad exit status from /var/tmp/rpm-tmp.mII8KL (%install)
I was getting similar errors when developing the package but eventually figured everything out, and all was good. I think the problem may lie with $RPM_BUILD_ROOT, %{buildroot}, or _topdir options. Nothing I have tried has led me anywhere however.
Here is my (modified) Makefile:
# a list of tools we depend on and must install if they're missing
DEBTOOLS=/usr/bin/debuild-pbuilder
RPMTOOLS=/usr/bin/rpmbuild
# convenience target for "make deb"
deb: my-package_1.0_all.deb
# convenience target for "make rpm".
rpm: my-package-1.0-Public.x86_64.rpm
# the target package (on Ubuntu at least)
my-package_1.0_all.deb: $(DEBTOOLS)
cd my-package; debuild-pbuilder -us -uc
my-package-1.0-Public.x86_64.rpm: $(RPMTOOLS)
cd rpmbuild; rpmbuild -bb SPECS/my-package.spec
/usr/bin/debuild-pbuilder:
apt-get -y install pbuilder
/usr/bin/rpmbuild:
yum -y install rpm-build
This is my spec file:
Summary: My Package
Name: my-package
Version: 1.0
Release: Public
Group: Applications/System
License: Public
Requires: external-package
Source1: myfile.file
%description
blah blah
%files
%config /etc/myfile.file
%install
mkdir -p $RPM_BUILD_ROOT/etc/
cp %{SOURCE1} %{buildroot}/etc/myfile.file
%post
ln -sf /etc/myfile.file /etc/external-package.conf

The problem was in fact that the file wasn't being found (obviously). For me this had a lot to do with the confusing nature of building rpm files. When the make command is executed, and the rpmbuild command is called, I needed to be able to specify the directory. When reading the documentation, it was stated you could use rpmbuild -D '_topdir .' -bb path/to/spec.spec to set the _topdir variable to the local directory you call from. This made sense as . represents this in linux.
However the actual call needs to be
rpmbuild -D "_topdir `pwd`" -bb path/to/spec.spec
This doesn't look all that different except it is crucial to use double-quotes. Using this command will run the build within the directory you call it from. After this rpmbuild will copy and handle the files for you as it should (which is confusing in itself).

Related

how to rebuild the source code in docker after building the image and contaner?

I need to do some tracing on DeathStarBench. It is a micro-services benchmark, that is deployed on docker.
For my tracing I use lttng. Also, I have made some changes to the DeathStarBench source code to be able to perform this tracing. DeathStarBench also uses cmake.
My problem here is that to be able to perform this tracing task I need to first install lttng tracer on the docker image and then rebuild the code so that the changes in the source code are applied.
I am new to docker and don't really understand the workflow, but after lots of searches, I understood that with "docker build -t " I will be able to rebuild the docker image and apply the code changes, however, I encounter error while doing this because my code uses lttng/tracepoint.h (checks if lttng-ust is installed on the system prior to this), and since I am rebuilding the docker image for sure it is not yet installed and I encounter this error.
Isn't there any way that enables me to first build the docker image and container and then rebuild the code and deploy it inside that container?
To explain in more detail:
I tried the sample cmake tracing project in lttng-ust github and was successful to do the same
this is the sample: https://github.com/lttng/lttng-ust/tree/master/doc/examples/cmake-multiple-shared-libraries
here it uses FindLttngUst.cmake
I did all the same steps for my project
just this method has a requirement that lttng should be previously installed in the intended machine for this to work
As you know my application is deployed on top of docker
I add the following to my dockerfile to install lttng when rebuilding docker images
RUN apt-get update
RUN apt-get install -y git libpopt-dev uuid-dev libxml2-dev automake autoconf libtool flex
bison make libc6 libc6-dev libglib2.0-0 libglib2.0-dev libpopt-dev python-pip python-dev
#Install lttng related
RUN git clone git://git.lttng.org/userspace-rcu.git
RUN (cd userspace-rcu; ./bootstrap ; ./configure ; make ; make install ; ldconfig ; cd ..)
RUN git clone git://git.lttng.org/lttng-ust.git
RUN (cd lttng-ust; ./bootstrap ; ./configure ; make ; make install ; ldconfig; cd ..)
RUN git clone git://git.lttng.org/lttng-tools.git
RUN (cd lttng-tools; ./bootstrap ; ./configure ; make ; make install ; ldconfig; cd ..)
#RUN git clone git:////git.lttng.org/lttng-modules.git
#RUN (cd lttng-modules; make ; make install ; ldconfig; cd ..)
However when I try to rebuild the specific service image I encounter an error, which is not related to my changes at all (related to another service), so I think this is because of deployment dependencies between images
Then I tried to overcome this by building all images with
docker-compose up --build
Again the rebuild process seems to start, but from the logs displayed I think this is not what I am looking for (because when I use "docker build -t " the log shows that build commands like make are in progress, but as I told you I face issue in this method), however when I use "docker-compose up --build" to rebuild everything in logs I see the kind of commands in this picture
Which looks like database transactions to me and takes very long to execute (I waited for 1 day and it did not finish!!!)
so here I need to know what I am doing wrong

Can't figure out how to use newer version of g++ in Centos 6 Docker image

I have a Docker image that is used for running tests in Jenkins and Bamboo. I need to upgrade the version of g++ used (to something with C++11 support).
I tried using a Dockerfile that looks roughly like the following one:
FROM docker.blahblahblah/centos/6.6:latest
RUN yum install -y git gcc-c++ imake centos-release-scl-rh devtoolset-7-toolchain
# I've tried putting this into /etc/bashrc, ~/.bashrc, ~/.bash_profile
RUN echo "source scl_source enable devtoolset-7" >> ~/.bashrc
My issue is that when g++ is used within the container, it uses the older one, instead of the newer one in devtoolset-7, even though the newer one should be sourced from the bashrc. (Maybe I'm misunderstanding how Docker will try to run everything.)
Could anyone point me in the right direction here?

envsubst command getting stuck in a container

I have a requirement that before an application runs, some part of it needs to read the environmental variable. For this I have the following docker file
FROM nodesource/jessie:0.12.7
# install gettext for envsubst
RUN apt-get update
RUN apt-get install -y gettext-base
# cache package.json and node_modules to speed up builds
ADD package.json package.json
RUN npm install
# Add source files
ADD src src
# Substiture value for backend endpoint env var
RUN envsubst < src/js/envapp.js > src/js/app.js
ADD node_modules node_modules
EXPOSE 8000
CMD ["npm","start"]
The above envsubst line reads (should read) an env variable $MYENV and substitutes it. But when I open the file app.js, its empty.
I checked if the environmental variable exists in the container and it does. Any reason its value is not read and substituted?
I also tried the same command in teh container and it works. It only does not work when I run the image
This is likely because $MYENV is not available for envsubst when you run the image.
Each RUN command runs on its own shell.
From the Docker documentations:
RUN (the command is run in a shell - /bin/sh -c - shell form)
You need to source your profile as well, for example if the $MYENV environment variable is available in the .bashrc file, you can modify your Dockerfile like this:
RUN source ~/.bashrc && envsubst < src/js/envapp.js > src/js/app.js
I encountered the same issues, and after much research and fishing through the internet. I managed to find a few work arounds to this issue. Below I'll list them and identifiable risks at the time of this "Answer post"
Solutions:
1.) apt-get install -y gettext its a standard GNU package language library, one of these libraries that it includes is envsubst` and I can confirm that it works for docker UBUNTU:latest and it should work for every flavored version.
2.) npm install envsub dependent on the "use case" - this approach would be better supported by node based projects.
3.) enstub cli project lib in my opinion it seems a bit overkill to downloading a custom cli from a random stranger but, it's also another option.
Risk:
apt-get install -y gettext:
1.) gettext - this approach would NOT be ideal for VM's as with any package library, it requires maintenance and updates as time passes. However, this isn't necessary for docker because once an a container is initialized and up and running we can create a bashscript to add the package, substitute env vars and then remove the package.
2.) It's a bad idea for VM's because it can be used to execute arbitrary code
npm install ensub
1.) envsub - updating packages and this approach wouldn't be ideal if your dealing with a different stack and not using nodejs.
NOTE:
There's also a PHP version for those developing a PHP application and it seems to work PHP's cli if you need a custom environment.
Resources:
GetText package library info: https://www.gnu.org/software/gettext/
GetText Risk - https://ubuntu.com/security/notices/USN-3815-2
PHP-GetText - apt-get install -y php-gettext
Custom ensubst cli: https://github.com/a8m/envsubst
I suggest that since your are using Node, you use the npm envsub module.
This module is well tested and is developed with docker in mind.
It avoids the need for relying on other dependencies when you already have the full Node arsenal at your fingertips.
envsub is described as
envsub is envsubst for NodeJS
NodeJS global CLI module providing file-level environment variable substitution via Handlebars
I am the author of the package. I think you will enjoy it.
I had some issues with envsubst in Docker.
For some reasons envsubst doesn't work when I try to copy the output in the same file. For example, this is not working:
RUN envsubst < file.conf > file.conf
But when I when I tried to use a temp file the issue disappeared:
RUN envsubst < file.conf > file.conf.temp && cp -f file.conf.temp file.conf

Make no rule found 'kernel-toolchain' . Stop

I am trying got port FreeBSD on the ARMv8 foundation model.
I am following the wiki from [1]. But, I am not able to get past the step of building the tool chain.
a) According to step one, I could download all the binutils and it is in my home directory.
b) Next it is asking to change PATH of root Makefile. So I changed it as
**export PATH= $PATH:/aarch64-freebsd-sandbox/toolchain/build/aarch64-none-freebsd10/bin/**
c) Next, the step is to make kernel toolchain. But when I type
**make kernel-toolchain TARGET=arm64**
It gives an error saying
**make: *** No rule to make target `kernel-toolchain'. Stop.**
I did echo $PATH and found that the path is added correctly.
What might be the problem?
[1] https://wiki.freebsd.org/arm64
Thank you!
To do this, you have to start with a working FreeBSD system. Cross compiling from Linux won't work. If you are on FreeBSD 10, you can use the included svnlite, if you are on an earlier version, you need to install the /usr/ports/devel/subversion port.
First you need to build the binutils as described on the wiki.
Then you should download the branch that is mentioned on the wiki page. This branch should be installed at /usr/src (make a backup of the contents first in case you have to rebuil your current system!);
# mv /usr/src /usr/orig-src
# mkdir /usr/src
# svnlite co https://svn0.us-west.FreeBSD.org/base/projects/arm64 /usr/src
Then edit the Makefile in /usr/src to contain the path for the special binutils first. Otherwise the normal binaries for whatever architecture you're running will be found first, which will not work.
After then you can build the kernel toolchain;
# cd /usr/src
# make kernel-toolchain TARGET=arm64
# make _includes TARGET=arm64
Then you'll have to build the loader;
# make buildenv TARGET=arm64
This will open a new shell. From that shell you should run;
# make -C lib/libstand obj all
# make -C sys/boot -DWITHOUT_FORTH obj all
Don't exit that shell, because there is more. I assume that the kernel build procedure is more or less standard, it is not mentioned on the wiki;
# make buildkernel
This command needs to run in the shell that was opened by make buildenv.
Note: do not run make installkernel. That would presumably leave your x86 PC with an ARM kernel. :-)
The wiki doesn't mention building a userland, and it only shows the boot process, so I don't know if it even works.
You'll need a Linux box (or VM) to run the ARMv8 emulator. You will have to supply the kernel and boot loader that you built to this emulator, but I don't know how to do that. You definitly need to take that up on the freebsd-arm mailing list!

Portable erlang

Is there a recommended way to "bootstrap" an Erlang distribution? I would like to run erlang on the bunch of machines where I do not have root elevation nor development tool-set (no compilers etc ...) . My thinking was to pre-package (on the machine with the same architecture) as much as I can before. What are the minimal requirements for an usable Erlang environment?
You can use the different erlware tools. Using the latest Sinan, you can even create a standalone release with the erts bundled.
Ok, I should have read it before ... (from INSTALL.md)
* Install using the `release` target. Instead of doing `make install` you
can create the installation in whatever directory you like using the
`release` target and run the `Install` script yourself. `RELEASE_ROOT`
is used for specifying the directory where the installation should be
created. This is what by default ends up under `/usr/local/lib/erlang`
if you do the install using `make install`. All installation paths
provided in the `configure` phase are ignored, as well as `DESTDIR`,
and `INSTALL_PREFIX`. If you want links from a specific `bin` directory
to the installation you have to set those up yourself. An example where
Erlang/OTP should be located at `/home/me/OTP`:
$ ./configure
$ make
$ make RELEASE_ROOT=/home/me/OTP release
$ cd /home/me/OTP
$ ./Install -minimal /home/me/OTP
$ mkdir -p /home/me/bin
$ cd /home/me/bin
$ ln -s /home/me/OTP/bin/erl erl
$ ln -s /home/me/OTP/bin/erlc erlc
$ ln -s /home/me/OTP/bin/escript escript
You can look at Wing3D for example.
CouchDB is another example. (Credit to tbikeev.)

Resources