I need to do some tracing on DeathStarBench. It is a micro-services benchmark, that is deployed on docker.
For my tracing I use lttng. Also, I have made some changes to the DeathStarBench source code to be able to perform this tracing. DeathStarBench also uses cmake.
My problem here is that to be able to perform this tracing task I need to first install lttng tracer on the docker image and then rebuild the code so that the changes in the source code are applied.
I am new to docker and don't really understand the workflow, but after lots of searches, I understood that with "docker build -t " I will be able to rebuild the docker image and apply the code changes, however, I encounter error while doing this because my code uses lttng/tracepoint.h (checks if lttng-ust is installed on the system prior to this), and since I am rebuilding the docker image for sure it is not yet installed and I encounter this error.
Isn't there any way that enables me to first build the docker image and container and then rebuild the code and deploy it inside that container?
To explain in more detail:
I tried the sample cmake tracing project in lttng-ust github and was successful to do the same
this is the sample: https://github.com/lttng/lttng-ust/tree/master/doc/examples/cmake-multiple-shared-libraries
here it uses FindLttngUst.cmake
I did all the same steps for my project
just this method has a requirement that lttng should be previously installed in the intended machine for this to work
As you know my application is deployed on top of docker
I add the following to my dockerfile to install lttng when rebuilding docker images
RUN apt-get update
RUN apt-get install -y git libpopt-dev uuid-dev libxml2-dev automake autoconf libtool flex
bison make libc6 libc6-dev libglib2.0-0 libglib2.0-dev libpopt-dev python-pip python-dev
#Install lttng related
RUN git clone git://git.lttng.org/userspace-rcu.git
RUN (cd userspace-rcu; ./bootstrap ; ./configure ; make ; make install ; ldconfig ; cd ..)
RUN git clone git://git.lttng.org/lttng-ust.git
RUN (cd lttng-ust; ./bootstrap ; ./configure ; make ; make install ; ldconfig; cd ..)
RUN git clone git://git.lttng.org/lttng-tools.git
RUN (cd lttng-tools; ./bootstrap ; ./configure ; make ; make install ; ldconfig; cd ..)
#RUN git clone git:////git.lttng.org/lttng-modules.git
#RUN (cd lttng-modules; make ; make install ; ldconfig; cd ..)
However when I try to rebuild the specific service image I encounter an error, which is not related to my changes at all (related to another service), so I think this is because of deployment dependencies between images
Then I tried to overcome this by building all images with
docker-compose up --build
Again the rebuild process seems to start, but from the logs displayed I think this is not what I am looking for (because when I use "docker build -t " the log shows that build commands like make are in progress, but as I told you I face issue in this method), however when I use "docker-compose up --build" to rebuild everything in logs I see the kind of commands in this picture
Which looks like database transactions to me and takes very long to execute (I waited for 1 day and it did not finish!!!)
so here I need to know what I am doing wrong
Related
I'm currently working out a Dockerfile. So I am trying to build out a Centos 7.6 base image and I get a failure when I try to use any yum packages. I'm not sure what the cause of this is.
I've already attempted to make the user root to see if that makes a difference but it doesn't help the situation. I've also done a docker pull centos to recieve the latest version of centos.
I simplified the code and still the same error.
FROM centos
ARG MONGO-RAILS-VERSION="0.0"
RUN yum install vim
# curl -L get.rvm.io | bash -s stable \
# rvm install 2.3.1 \
# rvm use 2.3.1 --default \
# gem install rails
I get an error that looks something like this
One of the configured repositories failed (Unknown),
and yum doesn't have enough cached data to continue. At this point the only
safe thing yum can do is fail. There are a few ways to work "fix" this:
1. Contact the upstream for the repository and get them to fix the problem.
2. Reconfigure the baseurl/etc. for the repository, to point to a working
upstream. This is most often useful if you are using a newer
distribution release than is supported by the repository (and the
packages for the previous distribution release still work).
3. Run the command with the repository temporarily disabled
yum --disablerepo=<repoid> ...
4. Disable the repository permanently, so yum won't use it by default. Yum
will then just ignore the repository until you permanently enable it
again or use --enablerepo for temporary usage:
yum-config-manager --disable <repoid>
or
subscription-manager repos --disable=<repoid>
5. Configure the failing repository to be skipped, if it is unavailable.
Note that yum will try to contact the repo. when it runs most commands,
so will have to try and fail each time (and thus. yum will be be much
slower). If it is a very temporary problem though, this is often a nice
compromise:
yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true
Cannot find a valid baseurl for repo: base/7/x86_64
Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=container error was
14: curl#7 - "Failed to connect to 2001:1b48:203::4:10: Network is unreachable"
The command '/bin/sh -c yum install vim' returned a non-zero code: 1
You may want to have a look for Set build-time variables (--build-arg):
$ docker build --build-arg HTTP_PROXY=http://10.20.30.2:1234 --build-arg FTP_PROXY=http://40.50.60.5:4567 .
I have a Docker image that is used for running tests in Jenkins and Bamboo. I need to upgrade the version of g++ used (to something with C++11 support).
I tried using a Dockerfile that looks roughly like the following one:
FROM docker.blahblahblah/centos/6.6:latest
RUN yum install -y git gcc-c++ imake centos-release-scl-rh devtoolset-7-toolchain
# I've tried putting this into /etc/bashrc, ~/.bashrc, ~/.bash_profile
RUN echo "source scl_source enable devtoolset-7" >> ~/.bashrc
My issue is that when g++ is used within the container, it uses the older one, instead of the newer one in devtoolset-7, even though the newer one should be sourced from the bashrc. (Maybe I'm misunderstanding how Docker will try to run everything.)
Could anyone point me in the right direction here?
I have my ASP.NET Core app running beautifully (more or less) on microsoft/aspnetcore:2.0-jessie. Now I want to try to get it to deploy to amazonlinux:2.
So far, the biggest hurdle has been libicu. I tried setting Globalization to Invariant, but this caused weird failures in, e.g., mySQL database calls.
Here's the relevant step from my Dockerfile:
RUN curl -L --http1.1 http://download.icu-project.org/files/icu4c/57.1/icu4c-57_1-RHEL6-x64.tgz --output icu.tgz \
&& tar -xf icu.tgz -C / \
&& export LD_LIBRARY_PATH=/usr/local/lib \
&& rm icu.tgz
(SourceForge was down while I was trying to work on this yesterday, which didn't improve matters.)
In any case, I still get the message of doom from .NET Core:
FailFast: Couldn't find a valid ICU package installed on the system. Set the configuration flag System.Globalization.Invariant to true if you want to run with no globalization support.
Any suggestions how to proceed?
Well, I revisited this yesterday. I don't know if it's because the base .tar of the Amazon Linux image has been updated, or because I was doing something wrong last time, but I installed the following packages using yum and all was well:
libunwind
libicu
dotnet-hosting-2.0.5
Note that for the dotnet package I needed first to set up Microsoft's package repository for yum, i.e.
rpm --import https://packages.microsoft.com/keys/microsoft.asc
and copying the following file to /etc/yum.repos.d/dotnetdev.repo :
[packages-microsoft-com-prod]
name=packages-microsoft-com-prod
baseurl=https://packages.microsoft.com/yumrepos/microsoft-rhel7.3-prod
enabled=1
gpgcheck=1
gpgkey=https://packages.microsoft.com/keys/microsoft.asc
(see Microsoft's instructions for CentOS and other Linux distros)
I have a requirement that before an application runs, some part of it needs to read the environmental variable. For this I have the following docker file
FROM nodesource/jessie:0.12.7
# install gettext for envsubst
RUN apt-get update
RUN apt-get install -y gettext-base
# cache package.json and node_modules to speed up builds
ADD package.json package.json
RUN npm install
# Add source files
ADD src src
# Substiture value for backend endpoint env var
RUN envsubst < src/js/envapp.js > src/js/app.js
ADD node_modules node_modules
EXPOSE 8000
CMD ["npm","start"]
The above envsubst line reads (should read) an env variable $MYENV and substitutes it. But when I open the file app.js, its empty.
I checked if the environmental variable exists in the container and it does. Any reason its value is not read and substituted?
I also tried the same command in teh container and it works. It only does not work when I run the image
This is likely because $MYENV is not available for envsubst when you run the image.
Each RUN command runs on its own shell.
From the Docker documentations:
RUN (the command is run in a shell - /bin/sh -c - shell form)
You need to source your profile as well, for example if the $MYENV environment variable is available in the .bashrc file, you can modify your Dockerfile like this:
RUN source ~/.bashrc && envsubst < src/js/envapp.js > src/js/app.js
I encountered the same issues, and after much research and fishing through the internet. I managed to find a few work arounds to this issue. Below I'll list them and identifiable risks at the time of this "Answer post"
Solutions:
1.) apt-get install -y gettext its a standard GNU package language library, one of these libraries that it includes is envsubst` and I can confirm that it works for docker UBUNTU:latest and it should work for every flavored version.
2.) npm install envsub dependent on the "use case" - this approach would be better supported by node based projects.
3.) enstub cli project lib in my opinion it seems a bit overkill to downloading a custom cli from a random stranger but, it's also another option.
Risk:
apt-get install -y gettext:
1.) gettext - this approach would NOT be ideal for VM's as with any package library, it requires maintenance and updates as time passes. However, this isn't necessary for docker because once an a container is initialized and up and running we can create a bashscript to add the package, substitute env vars and then remove the package.
2.) It's a bad idea for VM's because it can be used to execute arbitrary code
npm install ensub
1.) envsub - updating packages and this approach wouldn't be ideal if your dealing with a different stack and not using nodejs.
NOTE:
There's also a PHP version for those developing a PHP application and it seems to work PHP's cli if you need a custom environment.
Resources:
GetText package library info: https://www.gnu.org/software/gettext/
GetText Risk - https://ubuntu.com/security/notices/USN-3815-2
PHP-GetText - apt-get install -y php-gettext
Custom ensubst cli: https://github.com/a8m/envsubst
I suggest that since your are using Node, you use the npm envsub module.
This module is well tested and is developed with docker in mind.
It avoids the need for relying on other dependencies when you already have the full Node arsenal at your fingertips.
envsub is described as
envsub is envsubst for NodeJS
NodeJS global CLI module providing file-level environment variable substitution via Handlebars
I am the author of the package. I think you will enjoy it.
I had some issues with envsubst in Docker.
For some reasons envsubst doesn't work when I try to copy the output in the same file. For example, this is not working:
RUN envsubst < file.conf > file.conf
But when I when I tried to use a temp file the issue disappeared:
RUN envsubst < file.conf > file.conf.temp && cp -f file.conf.temp file.conf
I have a makefile for building debian and rpm packages. I have two Jenkins environments, one for Ubuntu and one for CentOS. The debian package works no problem, and the rpm make command works on my machine, but not on Jenkins. Jenkins returns the following error:
cp: cannot stat /root/rpmbuild/SOURCES/myfile.file': No such file or directory
error: Bad exit status from /var/tmp/rpm-tmp.mII8KL (%install)
I was getting similar errors when developing the package but eventually figured everything out, and all was good. I think the problem may lie with $RPM_BUILD_ROOT, %{buildroot}, or _topdir options. Nothing I have tried has led me anywhere however.
Here is my (modified) Makefile:
# a list of tools we depend on and must install if they're missing
DEBTOOLS=/usr/bin/debuild-pbuilder
RPMTOOLS=/usr/bin/rpmbuild
# convenience target for "make deb"
deb: my-package_1.0_all.deb
# convenience target for "make rpm".
rpm: my-package-1.0-Public.x86_64.rpm
# the target package (on Ubuntu at least)
my-package_1.0_all.deb: $(DEBTOOLS)
cd my-package; debuild-pbuilder -us -uc
my-package-1.0-Public.x86_64.rpm: $(RPMTOOLS)
cd rpmbuild; rpmbuild -bb SPECS/my-package.spec
/usr/bin/debuild-pbuilder:
apt-get -y install pbuilder
/usr/bin/rpmbuild:
yum -y install rpm-build
This is my spec file:
Summary: My Package
Name: my-package
Version: 1.0
Release: Public
Group: Applications/System
License: Public
Requires: external-package
Source1: myfile.file
%description
blah blah
%files
%config /etc/myfile.file
%install
mkdir -p $RPM_BUILD_ROOT/etc/
cp %{SOURCE1} %{buildroot}/etc/myfile.file
%post
ln -sf /etc/myfile.file /etc/external-package.conf
The problem was in fact that the file wasn't being found (obviously). For me this had a lot to do with the confusing nature of building rpm files. When the make command is executed, and the rpmbuild command is called, I needed to be able to specify the directory. When reading the documentation, it was stated you could use rpmbuild -D '_topdir .' -bb path/to/spec.spec to set the _topdir variable to the local directory you call from. This made sense as . represents this in linux.
However the actual call needs to be
rpmbuild -D "_topdir `pwd`" -bb path/to/spec.spec
This doesn't look all that different except it is crucial to use double-quotes. Using this command will run the build within the directory you call it from. After this rpmbuild will copy and handle the files for you as it should (which is confusing in itself).