We use travis-ci for continuous integration. I'm troubled by the fact that our build process takes too long (~30 minutes). We depend on several Ubuntu packages which we fetch using apt-get, among others python-pandas.
We also have some of our own debs which we fetch over HTTPS and dpkg install. Finally, we have several pip/pypi requirements, such as Django, Flask, Werkzeug, numpy, pycrypto, selenium.
It would be nice to be able to at least pre-package some of these requirements. Does travis support something like this? How can I prepackage some of these requirements? Is it possible to build a custom travis base VM and start the build from there (perhaps using docker)? Especially the apt-get requirements from the default Ubuntu precise repository as well as the pip requirements should be easy to include.
So while this question is already answered, it's doesn't actually provide a solution path. You can use cache directives in travis to cache your built packages for future travis runs.
cache:
directories:
- $HOME/.pip-cache/
- $HOME/virtualenv/python2.7
install:
- pip install -r requirements.txt --download-cache "$HOME/.pip-cache"
Now your package content is saved for your next travis build. You can similarly store slow-to-retrieve resources in other directories and cache them.
Currently Travis-CI doesn't support such a feature. There are related issues currently open though such as custom VMs, running Docker in an OpenVz container - (Spotify seems to have a somewhat working example links in this issue), using Linux Containers (LXC), using KVM.
Some of those have workarounds mentioned in the issues, I'd give those a try until something more substantial is supported by Travis-CI. I'd also suggest reaching out to Travis-CI support and see if they have any suggestions (maybe there's something coming out soon that could help).
Related
Let's say I have old, unmaintained application that lives on a VPS (i.e. Symfony 3 PHP app that relies on PHP 5).
If some changes are needed I have to clone this app to my desktop, build it, change and re-deploy. As time goes, recreating desktop dev environment gets harder - in this example I can't simply build the app as I use PHP7 in my CLI that breaks building process.
I tried to dockerize the app, so I added Ubuntu 18 to my docker-compose file... and it doesn't work as latest Ubuntu that has PHP5 support is 14.04. 14.04 is also the oldest (official) version available on DockerHub. But will it be still available in 3 years? If not, Docker won't build a container.
So, my question is: is Docker a right tool to solve described problem at all?
If so, should I backup docker images described that my build relies on?
If not, beside proper maintenance, what tool is better?
You can install PHP5 in newer ubuntu versions, but it means adding an external repository.
You could also create your own docker image, containing only the libraries you want. If so, I'd advise to try and use alpine as a base image. There is a bit of a learning curve to adapt, but once you do it you'll have a small image tailored to your needs.
Given that containers allow you to isolate processus and conf with minimal footprint compared to a VM, I think it is the best option. Tailoring and maintaining your own image is not that expensive in terms of maintenance if you document it correctly, and it will allow you to always have a system 'maintaining' all your precise requirements.
I have to install a few dependencies on my docker container, I want to use python:3.6-alpine version to have it as light as possible, but apk package manager which comes with alpine is giving me trouble so I would like to get the apt-get package manager. I tried:
apk add apt-get
and it didnt work.
how can I get it on the container?
Using multiple package systems is usually a very bad idea, for many reasons. Packages are likely to collide and break and you'll end up with much greater mess than you've started with.
See this excellent answer for more detail: Is there a pitfall of using multiple package managers?
A more feasible approach would be troubleshooting and resolving the issues you are having with apk. apk is designed for simplicity and speed, and should take very little getting used to. It is really an excellent package manager, IMO.
For a good tutorial, I warmly recommend the apk introduction page at the Alpine Wiki site:
https://wiki.alpinelinux.org/wiki/Alpine_Linux_package_management
If you're determined not to use apk, and for the sake of experiment want try bringing up apt instead, as a first step, you'll have first to build apt from source: https://github.com/Debian/apt. Then, if it is produces a functional build (not likely since it's probably not compatible with musl libc), you'll have to wire it to some repositories, but Alpine repositories are only fit for apk, not apt. As you can see, this is not really feasible, and not the route you want to go to.
I am looking for a way to set up or modify an existing Docker image for installing tensorflow that will install it such that the SSE4, AVX, AVX2, and FMA instructions can be utilized for CPU speed up. So far I have found how to install from source using bazel How to Compile Tensorflow... and CPU instructions not compiled.... Neither of these explain how to do this within Docker. So I think what I am looking for is what you need to add to an existing docker image that installs without these options so that you can get a compile version of tensorflow with the CPU options enabled. The existing docker images do not do this because they want the image to run on as many machines as possible. I am using Ubuntu 14.04 on linux PC. I am new to docker but have installed tensorflow and have it working without getting the CPU warnings I get when I use the docker images. I may not need this for speed, but I have seen posts that claim the speed up can be significant. I searched for existing docker images that do this and could not find anything. I need this to work with gpu so needs to be compatible with nvidia-docker.
I just found this docker support for bazel and it might provide an answer, however I do not understand it well enough to know for sure. I believe this is saying that you can not build tensorflow with bazel inside a Dockerfile. You have to build a Dockerfile using bazel. Is my understanding correct and is this the only way to get a docker image with tensorflow compiled from source? If so, I could still use help in how to do it and still get the other dependencies that I would get if using an existing docker image for tensorflow.
Dockerfiles that build with CPU support can be found here.
Hope that helps! Spent many a late night here on Stack Overflow and Github Issues and stuff. Now it's my turn to give back! :)
The GPU stuff in particular is really hairy - especially when enabling the XLA/JIT/AOT stuff as well as the Graph Transform Tools.
Lots of hacks embedded in my Dockerfiles. Feel free to review and ask me questions!
The contributing guidelines mention building TensorFlow from source with Docker to run the unit tests:
Refer to the
CPU-only developer Dockerfile and
GPU developer Dockerfile
for the required packages. Alternatively, use the said
Docker images, e.g.,
tensorflow/tensorflow:nightly-devel and tensorflow/tensorflow:nightly-devel-gpu
for development to avoid installing the packages directly on your system.
In order to ease the installation process of RabbitMQ, I want to use a simple, zero dependencies installer for erlang.
There is a great project on Github from the guys at RabbitMQ - erlang-rpm, but this is RPM only.
I'm aware of erlang-solutions, but it not a single, no dependencies installer.
EDIT: Adding some context here. I need this to be able to prepare a simple, offline install to my app, which needs RabbitMQ. Offline means NO access to internet for downloading dependencies.
Any ideas?
There is no Zero-dependency Erlang Ubuntu/Debian for RabbitMQ.
But the Erlang Solution package can be installed easily.
We built the Centos package because the dependencies are hard to install on Centos (if you don't want to use the epel)
If you want to create an offline setup, you can do it like for other packages (https://askubuntu.com/questions/974/how-can-i-install-software-or-packages-without-internet-offline). Or you can create your own package by building Erlang from source code.
Note also RabbitMQ has dependencies as socat
I am working on a Rails project and the Architect has asked me to investigate bundling CouchDB into to application so that it can be deployed by Capistrano across multiple platforms and managed by Rake.
My expectation was that I could set up the Erlang VM on the various environments and then distribute the CouchDB application with Capistrano. However I can't find any option to download CouchDB without the Erlang runtime. I can, however see an option to build CouchDB from source which I assume is platform dependent.
I am new to Erlang and CouchBD, am I missing something? Is there a way to bundle CouchDB into a Rails app and distribute it across multiple platforms?
Have a look at some of the tools for provisioning Rails services (such as passenger_stack). Passenger Stack will download, make and install the ancillary services for your Rails app ... might be something you can adapt or use as a base to install Erlang and CouchDB.
There are a bunch of alternatives to this as well. Deprec contains recipes for provisioning with Capistrano. Essential idea is the same though.
I think you will not find a silver bullet. Distributing Erlang is similar to distributing Ruby; however Ruby has the advantage of being included in many default OS installs.
I know ejabberd has pre-built binaries for many distros. You might investigate how they do it.
The correct solution probably depends on how many "multiple platforms" you are targeting. If it's "Ubuntu 8.04 plus Ubuntu 10.04" that is different from several Linux distros, plus OSX, plus FreeBSD. Typically only open source projects support those many platforms and ideally you can get patches from the community. For internal projects, I have seen teams standardize on a Linux build and use virtualization on Mac/Windows.
But back to your question:
Building from source is a reasonable option. You could build when you deploy, or pre-build for all platforms and then deploy the binaries. Both Erlang and CouchDB use Autoconf which means you can --prefix them to a dedicated location (more-or-less standalone apps). It will take some trial and error but your build script can
Platform-specific dependency setup: gcc, make, autoconf, everything you need. apt-get on Ubuntu, yum on RHEL, Macports, whatever you need to get a common platform on your development and deployment system
Compile and install the rest using the tools from step 1. Use configure --prefix=/opt/my_software to keep it all in one place. (You can totally uninstall with rm -rf.)
This is an medium-level challenge--mostly trial and error. If possible, work within a build framework such as Rake or Toby's suggestion passenger_stack. Good luck!