I installed the nix package manager on my machine (macOS 10.12.6 Sierra) several weeks ago.
I want to update mylocal nixpkgs collection to bring it in sync with any upstream updates in the channel. My understanding is that this can be achieved by running: nix-channel --update. However when I run this I get the following output:
unpacking channels...
created 0 symlinks in user environment
Suggesting no expressions were updated in the channels my system is subscribed to. If I run nix-channel --list I don't see any channels listed. What channel is my system subscribed to by default? and should I expect it to be listed?
Is it the case that generally nix-channel --update will only produce local changes if I have modified the channels I'm subscribed to or if I'm subscribed to the unstable channel?
The following might be specific to OSX:
Nix channels are managed on a per-user basis (source). nix-channel --list's output is empty because by default you are not subscribed to any channels - only root is subscribed to nixpkgs-unstable (source).
You can run nix-channel --list as root to see his subscriptions. But
do not run it through sudo (see: https://github.com/NixOS/nix/issues/1548) and
do not use a simple sudo su because it doesn't load /etc/profile (source) and thus will not have the nix env variables set up (variables-setup).
Working example:
user$ nix-channel --list
user$ sudo su -
root# nix-channel --list
nixpkgs https://nixos.org/channels/nixpkgs-unstable
Your nix-channel --update suggests that you have zero channels in your channel list, not zero packages. You can see your channel configuration with nix-channel --list. You probably need to configure a channel.
The Nix install script currently configures a single channel with the name nixpkgs:
"$nix/bin/nix-channel" --add https://nixos.org/channels/nixpkgs-unstable
So that's the default channel and it should be listed in nix-channel --list after installation, until you nix-channel --remove nixpkgs or rm ~/.nix-channels.
The nix-channel command only updates your Nix expressions. It will not update any package installations. It will only affect future invocations of nix-build, nix-env and everything else that uses $NIX_PATH. (It resembles apt-get update in this respect, or brew update, except nix-channel will not update the installed version of Nix.)
How do I set the default channel in NixOS's /etc/configuration.nix?
There is a command to set it and rebuild with
sudo nix-channel --add https://nixos.org/channels/nixpkgs-unstable
sudo nixos-rebuild switch -I nixpkgs=/nix/var/nix/profiles/per-user/root/channels/nixpkgs
but I'd like to have it setup in configuration.nix so I don't have to remember how to do this everytime. Thanks!
system.autoUpgrade.channel is what you might be looking for
set it to any channel e.g.
system.autoUpgrade.channel = "https://nixos.org/channels/nixos-16.03-small/";
the documentation says:
by default, this is the channel set using nix-channel (run
nix-channel --list to see the current value)
an up to date list of channels can be found on https://nixos.org/channels/
ref: https://nixos.org/nixos/manual/options.html#opt-system.autoUpgrade.channel
https://nixos.org/nixos/manual/index.html#idm140737317454064
Set nixPath = [ "nixpkgs=http://nixos.org/channels/nixos-unstable/nixexprs.tar.xz" ];, see https://github.com/snabblab/snabblab-nixos/blob/d8b9761b107293891b19021f2f0f77a0e3ba3746/modules/common.nix#L39
The nix.nixPath (ref) option looks like it will do what you're after.
Also the nixos-unstable channel might be more appropriate for you, rather than nixpkgs-unstable. I believe the pkgs in the nixpkgs channel are tested and built for non-nixOS systems, though I can't remember a reference for that at the moment.
nix-channel --add https://nixos.org/channels/nixos-unstable/
nix-channel --update nixos-unstable
# /etc/nixos/configuration.nix
# Put nixos-unstable at the front of nixPath
{ lib, ... }:
{
nix.nixPath = lib.mkDefault (lib.mkBefore [ "nixpkgs=/nix/var/nix/profiles/per-user/root/channels/nixos-unstable" ]);
}
If you want the imperative nix-channel commands in your configuration.nix as well you could write a small systemd service to do so, as shown here.
Edit: To ensure configuration.nix is built from the latest unstable channel just set the value of nixpkgs as in the answer from #iElectric and Nix will use the expressions contained at that URL whenever it evaluates configuration.nix.
PS I realised you could also just point the nixos path to the nixos-unstable channel by doing nix-channel --add https://nixos.org/channels/nixos-unstable/ nixos but I think the first solution is clearer.
I'm on the unstable nix channel:
# nix-channel --list
nixos https://nixos.org/channels/nixos-unstable
# nix-channel --update
downloading Nix expressions from ‘https://nixos.org/releases/nixos/unstable/nixos-16.09pre78848.9549257//nixexprs.tar.xz’...
downloading ‘https://nixos.org/releases/nixos/unstable/nixos-16.09pre78848.9549257//nixexprs.tar.xz’... [ [6192/6921 KiB, 6143.1 KiB/s]
unpacking channels...
For some reason, command-not-found doesn't work:
$ command-not-found
/home/fabian command-not-found
DBI connect('dbname=/nix/var/nix/profiles/per-user/root/channels/nixos/programs.sqlite','',...) failed: unable to open database file at /run/current-system/sw/bin/command-not-found line 13.
cannot open database `/nix/var/nix/profiles/per-user/root/channels/nixos/programs.sqlite' at /run/current-system/sw/bin/command-not-found line 13.
Indeed, /nix/var/nix/profiles/per-user/root/channels/nixos/programs.sqlite doesn't exist. What could be the reason for this?
FWIW, it seems nixos-install doesn't properly generate programs.sqlite.
I was seeing the same error on a fresh 17.03 install, even though it's working on other systems I have that are also on 17.03.
Manually running nix-channel --update was all that was needed to fix it for me. (I didn't run nixos-rebuild or anything that would have changed the versions of command-not-found.)
Apparently it's removed in the most recent channels/nixos builds and awaiting a replacement - refs (1) (2)
I want to use apt to install some packages for the test, however, it fails due to that the sudo is disabled. I found the following in the test output:
Sudo, the FireFox addon, setuid and setgid have been disabled.
It seems that the output comes from this line in travic-ci, but setting paranoid_mode to false in .travis.yml does not work.
How to enable sudo access?
PS: I am using private repo.
EDIT: The following .travis.yml will fail due to sudo: must be setuid root when running sudo apt-get update -qq
language: python
python:
- "3.4"
before_install:
- sudo apt-get update -qq
script:
- nosetests
Setting sudo: true and/or paranoid_mode: false does not work.
Sudo access is turned off on our Docker based architecture, which will be used in two contexts:
repositories opt in using sudo: false in their .travis.yml file (it additionally needs to be turned on on our side)
on our educational program (see http://education.travis-ci.com)
Builds running on our Docker based architecture currently cannot be allowed sudo access due to certain security concerns in the LXC/Docker layer. We hope this will be fixed in the near future, but unfortunately the issue is out of our own hands.
We are also working on improving the Firefox addon, which currently uses sudo itself, but shouldn't. We'll post on our blog once this has happened.
To extend the existing answer, if you put in .travis.yml:
sudo: required
Travis should switch your build to use their "standard infrastructure" (rather than their "container based infrastructure") and then you can use sudo.
References:
https://docs.travis-ci.com/user/workers/standard-infrastructure
https://docs.travis-ci.com/user/workers/container-based-infrastructure/
Nov 2018 Update
It seems that container-based infrastructure is getting deprecated. From the docs:
Container-based infrastructure is currently being deprecated. Please
use the fully-virtualized infrastrstructure via sudo: required
instead.
As explained in "Combining The Linux Infrastructures"
Going forward, we will slowly transition the container-based environment out, in favor of a build environment that is entirely virtual machine-based.
Folks using container-based infrastructures will be the only ones affected, and this transition will roll out slowly, depending on whether you specify sudo: false in your .travis.yml.
This is illustrated in Git 2.20 (Q4 2018), with:
Travis CI will soon deprecate the container-based infrastructure enabled by sudo: false in ce59dff (Git 2.8.0, Jan. 2016).
See commit 0f0c511 (01 Nov 2018) by SZEDER Gábor (szeder).
(Merged by Junio C Hamano -- gitster -- in commit 57f06d5, 13 Nov 2018)
travis-ci: install packages in 'ci/install-dependencies.sh'
Ever since we started using Travis CI, we specified the list of packages to install in '.travis.yml' via the APT addon.
While running our builds on Travis CI's container-based infrastructure we didn't have another choice, because that environment didn't support 'sudo', and thus we didn't have permission to install packages ourselves.
With the switch to the VM-based infrastructure in the previous patch we do get a working 'sudo', so we can install packages by running 'sudo apt-get -y install ...' as well.
Let's make use of this and install necessary packages in 'ci/install-dependencies.sh', so all the dependencies (i.e. both packages and "non-packages" (P4 and Git-LFS)) are handled in the same file.
Install gcc-8 only in the 'linux-gcc' build job; so far it has been unnecessarily installed in the 'linux-clang' build job as well.
Print the versions of P4 and Git-LFS conditionally, i.e. only when they have been installed; with this change even the static analysis and documentation build jobs start using 'ci/install-dependencies.sh'
to install packages, and neither of these two build jobs depend on and
thus install those.
This change will presumably be beneficial for the upcoming Azure Pipelines integration preliminary versions of that patch series run a couple of 'apt-get' commands to install the necessary packages before running 'ci/install-dependencies.sh', but with this patch it will be sufficient to run only 'ci/install-dependencies.sh'.
I'd rather not have to push every little change to .travis.yml and every little change I make to the source in order to run the build. With jenkins you can download jenkins and run locally. Does travis offer something like this?
Note: I've seen the travis-ci cli and downloaded it, but all it seems
to do is call their API, which then connects to my GitHub repo, so if
I don't push, it won't matter that I restart the last build.
This process allows you to completely reproduce any Travis build job on your computer. Also, you can interrupt the process at any time and debug. Below is an example where I perfectly reproduce the results of job #191.1 on php-school/cli-menu
.
Prerequisites
You have public repo on GitHub
You ran at least one build on Travis
You have Docker set up on your computer
Set up the build environment
Reference: https://docs.travis-ci.com/user/common-build-problems/
Make up your own temporary build ID
BUILDID="build-$RANDOM"
View the build log, open the show more button for WORKER INFORMATION and find the INSTANCE line, paste it in here and run (replace the tag after the colon with the newest available one):
INSTANCE="travisci/ci-garnet:packer-1512502276-986baf0"
Run the headless server
docker run --name $BUILDID -dit $INSTANCE /sbin/init
Run the attached client
docker exec -it $BUILDID bash -l
Run the job
Now you are now inside your Travis environment. Run su - travis to begin.
This step is well defined but it is more tedious and manual. You will find every command that Travis runs in the environment. To do this, look for for everything in the right column which has a tag like 0.03s.
On the left side you will see the actual commands. Run those commands, in order.
Result
Now is a good time to run the history command. You can restart the process and replay those commands to run the same test against an updated code base.
If your repo is private: ssh-keygen -t rsa -b 4096 -C "YOUR EMAIL REGISTERED IN GITHUB" then cat ~/.ssh/id_rsa.pub and click here to add a key
FYI: you can git pull from inside docker to load commits from your dev box before you push them to GitHub
If you want to change the commands Travis runs then it is YOUR responsibility to figure out how that translates back into a working .travis.yml.
I don't know how to clean up the Docker environment, it looks complicated, maybe this leaks memory
Travis-ci offers a new container-based infrastructure that uses docker. This can be very useful if you're trying to troubleshoot a travis-ci build by reproducing it locally. This is taken from Travis CI's documentation.
Troubleshooting Locally in a Docker Image
If you're having trouble tracking down the exact problem in a build it often helps to run the build locally. To do this you need to be using our container based infrastructure (ie, have sudo: false in your .travis.yml), and to know which Docker image you are using on Travis CI.
Running a Container Based Docker Image Locally
Download and install the Docker Engine.
Select an image from Docker Hub. If you're not using a language-specific image pick ci-ruby. Open a terminal and start an interactive Docker session using the image URL:
docker run -it travisci/ubuntu-ruby:18.04 /bin/bash
Switch to the travis user:
su - travis
Clone your git repository into the / folder of the image.
Manually install any dependencies.
Manually run your Travis CI build command.
UPDATE: I now have a complete turnkey, all-in-one answer, see https://stackoverflow.com/a/49019950/300224. Only took 3 years to figure out!
According to the Travis documentation: https://github.com/travis-ci/travis-ci there is a concoction of projects that collude to deliver the Travis CI web service we know and love. The following subset of projects appears to allow local make test functionality using the .travis.yml in your project:
travis-build
travis-build creates the build
script for each job. It takes the configuration from the .travis.yml file and
creates a bash script that is then run in the build environment by
travis-worker.
travis-cookbooks
travis-cookbooks holds the
Chef cookbooks that are used to provision the build environments.
travis-worker
travis-worker is responsible for
running the build scripts in a clean environment. It streams the log output to
travis-logs and pushes state updates (build starting/finishing)
to travis-hub.
(The other subprojects are responsible for communicating with GitHub, their web interface, email, and their API.)
Similar to Scott McLeod's but this also generates a bash script to run the steps from the .travis.yml.
Troubleshooting Locally in Docker with a generated Bash script
# choose the image according to the language chosen in .travis.yml
$ docker run -it -u travis quay.io/travisci/travis-jvm /bin/bash
# now that you are in the docker image, switch to the travis user
sudo - travis
# Install a recent ruby (default is 1.9.3)
rvm install 2.3.0
rvm use 2.3.0
# Install travis-build to generate a .sh out of .travis.yml
cd builds
git clone https://github.com/travis-ci/travis-build.git
cd travis-build
gem install travis
# to create ~/.travis
travis version
ln -s `pwd` ~/.travis/travis-build
bundle install
# Create project dir, assuming your project is `AUTHOR/PROJECT` on GitHub
cd ~/builds
mkdir AUTHOR
cd AUTHOR
git clone https://github.com/AUTHOR/PROJECT.git
cd PROJECT
# change to the branch or commit you want to investigate
travis compile > ci.sh
# You most likely will need to edit ci.sh as it ignores matrix and env
bash ci.sh
Use wwtd (what would travis do) ruby gem to run tests on your local machine roughly as they would run on travis.
It will recreate the build matrix and run each configuration, great to sanity check setup before pushing.
gem i wwtd
wwtd
tl;dr Use image specified at https://docs.travis-ci.com/user/common-build-problems/#troubleshooting-locally-in-a-docker-image in combination with https://github.com/travis-ci/travis-build#use-as-addon-for-travis-cli.
EDIT 2019-12-06
#troubleshooting-locally-in-a-docker-image section was replaced by #running-builds-in-debug-mode which also describes how to SSH to the job running in the debug mode.
EDIT 2019-07-26
#troubleshooting-locally-in-a-docker-image section is no longer part of the docs; here's why
https://github.com/travis-ci/docs-travis-ci-com/issues/2342
https://blog.travis-ci.com/2018-10-04-combining-linux-infrastructures
https://blog.travis-ci.com/2018-11-30-announcing-xenial-build-environment-for-enterprise
Though, it's still in git history: https://github.com/travis-ci/docs-travis-ci-com/pull/2193.
Look for (quite old, couldn't find newer) image versions at: https://travis-ci.org/travis-ci/docs-travis-ci-com/builds/230889063#L661.
I wanted to inspect why one of the tests in my build failed with an error I din't get locally.
Worked.
What actually worked was using the image specified at Troubleshooting Locally in a Docker Image docs page. In my case it was travisci/ci-garnet:packer-1512502276-986baf0.
I was able to add travise compile following steps described at https://github.com/travis-ci/travis-build#use-as-addon-for-travis-cli.
dm#z580:~$ docker run --name travis-debug -dit travisci/ci-garnet:packer-1512502276-986baf0 /sbin/init
dm#z580:~$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
travisci/ci-garnet packer-1512502276-986baf0 6cbda6a950d3 11 months ago 10.2GB
dm#z580:~$ docker exec -it travis-debug bash -l
root#912e43dbfea4:/# su - travis
travis#912e43dbfea4:~$ cd builds/
travis#912e43dbfea4:~/builds$ git clone https://github.com/travis-ci/travis-build
travis#912e43dbfea4:~/builds$ cd travis-build
travis#912e43dbfea4:~/builds/travis-build$ mkdir -p ~/.travis
travis#912e43dbfea4:~/builds/travis-build$ ln -s $PWD ~/.travis/travis-build
travis#912e43dbfea4:~/builds/travis-build$ gem install bundler
travis#912e43dbfea4:~/builds/travis-build$ bundle install --gemfile ~/.travis/travis-build/Gemfile
travis#912e43dbfea4:~/builds/travis-build$ bundler binstubs travis
travis#912e43dbfea4:~/builds/travis-build$ cd ..
travis#912e43dbfea4:~/builds$ git clone --depth=50 --branch=master https://github.com/DusanMadar/PySyncDroid.git DusanMadar/PySyncDroid
travis#912e43dbfea4:~/builds$ cd DusanMadar/PySyncDroid/
travis#912e43dbfea4:~/builds/DusanMadar/PySyncDroid$ ~/.travis/travis-build/bin/travis compile > ci.sh
travis#912e43dbfea4:~/builds/DusanMadar/PySyncDroid$ sed -i 's,--branch\\=\\\x27\\\x27,--branch\\=master,g' ci.sh
travis#912e43dbfea4:~/builds/DusanMadar/PySyncDroid$ bash ci.sh
Everything from .travis.yml was executed as expected (dependencies installed, tests ran, ...).
Note that before running bash ci.sh I had to change --branch\=\'\'\ to --branch\=master\ (see the second to last sed -i ... command) in ci.sh.
If that doesn't work the command bellow will help to identify the target line number and you can edit the line manually.
travis#912e43dbfea4:~/builds/DusanMadar/PySyncDroid$ cat ci.sh | grep -in branch
840: travis_cmd git\ clone\ --depth\=50\ --branch\=\'\'\ https://github.com/DusanMadar/PySyncDroid.git\ DusanMadar/PySyncDroid --echo --retry --timing
889:export TRAVIS_BRANCH=''
899:export TRAVIS_PULL_REQUEST_BRANCH=''
travis#912e43dbfea4:~/builds/DusanMadar/PySyncDroid$
Didn't work.
Followed the accepted answer for this question but didn't
find the image (travis-ci-garnet-trusty-1512502259-986baf0) mentioned by instance at https://hub.docker.com/u/travisci/.
Build worker version points to travis-ci/worker commit and its travis-worker-install references quay.io/travisci/ as image registry. So I tried it.
dm#z580:~$ docker run -it -u travis quay.io/travisci/travis-python /bin/bash
travis#370c23a773c9:/$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 12.04.5 LTS
Release: 12.04
Codename: precise
travis#370c23a773c9:/$
dm#z580:~$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
quay.io/travisci/travis-python latest 753a216d776c 3 years ago 5.36GB
Definitely not Trusty (Ubuntu 14.04) and not small either.
You could try Trevor, which uses Docker to run your Travis build.
From its description:
I often need to run tests for multiple versions of Node.js. But I don't want to switch versions manually using n/nvm or push the code to Travis CI just to run the tests.
That's why I created Trevor. It reads .travis.yml and runs tests in all versions you requested, just like Travis CI. Now, you can test before push and keep your git history clean.
I'm not sure what was your original reason for running Travis locally, if you just wanted to play with it, then stop reading here as it's irrelevant for you.
If you already have experience with hosted Travis and you want to get the same experience in your own datacenter, read on.
Since Dec 2014 Travis CI offers an Enterprise on-premises version.
http://blog.travis-ci.com/2014-12-19-introducing-travis-ci-enterprise/
The pricing is part of the article as well:
The licensing is done per seats, where every license includes 20 users. Pricing starts at $6,000 per license, which includes 20 users and 5 concurrent builds. There's a premium option with unlimited builds for $8,500.
I wasn't able to use the answers here as-is. For starters, as noted, the Travis help document on running jobs locally has been taken down. All of the blog entries and articles I found are based on that. The new "debug" mode doesn't appeal to me because I want to avoid the queue times and the Travis infrastructure until I've got some confidence I have gotten somewhere with my changes.
In my case I'm updating a Puppet module and I'm not an expert in Puppet, nor particularly experienced in Ruby, Travis, or their ecosystems. But I managed to build a workable test image out of tips and ideas in this article and elsewhere, and by examining the Travis CI build logs pretty closely.
I was unable to find recent images matching the names in the CI logs (for example, I could find travisci/ci-sardonyx, but could not find anything with "xenial" or with the same build name). From the logs it appears images are now transferred via AMQP instead of a mechanism more familiar to me.
I was able to find an image travsci/ubuntu-ruby:16.04 which matches the OS I'm targeting for my particular case. It does not have all the components used in the Travis CI, so I built a new one based on this, with some components added to the image and others added in the container at runtime depending on the need.
So I can't offer a clear procedure, sorry. But what I did, essentially boiled down:
Find a recent Travis CI image in Docker Hub matching your target OS as closely as possible.
Clone the repository to a build directory, and launch the container with the build directory mounted as a volume, with the working directory set to the target volume
Now the hard work: go through the Travis build log and set up the environment. In my case, this meant setting up RVM, and then using bundle to install the project's dependencies. RVM appeared to be already present in the Travis environment but I had to install it; everything else came from reproducing the commands in the build log.
Run the tests.
If the results don't match what you saw in the Travis CI logs, go back to (3) and see where to go.
Optionally, create a reusable image.
Dev and test locally and then push and hopefully your Travis results will be as expected.
I know this is not concrete and may be obvious, and your mileage will definitely vary, but hopefully this is of some use to somebody. The Dockerfile and a README for my image are on GitHub for reference.
It is possible to SSH to Travis CI environment via a bounce host. The feature isn't built in Travis CI, but it can be achieved by the following steps.
On the bounce host, create travis user and ensure that you can SSH to it.
Put these lines in the script: section of your .travis.yml (e.g. at the end).
- echo travis:$sshpassword | sudo chpasswd
- sudo sed -i 's/ChallengeResponseAuthentication no/ChallengeResponseAuthentication yes/' /etc/ssh/sshd_config
- sudo service ssh restart
- sudo apt-get install sshpass
- sshpass -p $sshpassword ssh -R 9999:localhost:22 -o StrictHostKeyChecking=no travis#$bouncehostip
Where $bouncehostip is the IP/host of your bounce host, and $sshpassword is your defined SSH password. These variables can be added as encrypted variables.
Push the changes. You should be able to make an SSH connection to your bounce host.
Source: Shell into Travis CI Build Environment.
Here is the full example:
# use the new container infrastructure
sudo: required
dist: trusty
language: python
python: "2.7"
script:
- echo travis:$sshpassword | sudo chpasswd
- sudo sed -i 's/ChallengeResponseAuthentication no/ChallengeResponseAuthentication yes/' /etc/ssh/sshd_config
- sudo service ssh restart
- sudo apt-get install sshpass
- sshpass -p $sshpassword ssh -R 9999:localhost:22 -o StrictHostKeyChecking=no travisci#$bouncehostip
See: c-mart/travis-shell at GitHub.
See also: How to reproduce a travis-ci build environment for debugging