Travis CI : How to allow failures with a customized environment variable? - travis-ci

Question

Following this answer, I wrote this Travis configuration file :
language: php
php:
- 5.3
- 5.4
- 5.5
- 5.6
- 7
- hhvm
- nightly
branches:
only:
- master
- /^\d+\.\d+\.\d+$/
matrix:
fast_finish: true
include:
- php: 5.3
env: deps="low"
- php: 5.5
env: SYMFONY_VERSION=2.3.*
- php: 5.5
env: SYMFONY_VERSION=2.4.*
- php: 5.5
env: SYMFONY_VERSION=2.5.*
- php: 5.5
env: SYMFONY_VERSION=2.6.*
- php: 5.5
env: SYMFONY_VERSION=2.7.*
- php: 5.5
env: SYMFONY_VERSION=2.8.*#dev TEST_GROUP=canFail
allow_failures:
- php: nightly
- env: TEST_GROUP=canFail
before_script:
- composer self-update
- if [ "$SYMFONY_VERSION" != "" ]; then composer require --dev --no-update symfony/symfony=$SYMFONY_VERSION; fi
- if [ "$deps" = "low" ]; then composer update --prefer-lowest; fi
- if [ "$deps" != "low" ]; then composer update --prefer-source; fi
script: phpunit
But Travis CI counts only the php nightly version as an "allowed to fail" version. Am I using the environment variables in a wrong way ?
UPDATE
Just a precision, I know that I can directly write the environment like that:
matrix:
include:
- php: 5.5
env: SYMFONY_VERSION=2.8.*#dev
allow_failures:
- env: SYMFONY_VERSION=2.8.*#dev
but still I don't get why the other way doesn't work.

Answer 1

What you specify in allow_failures: is your allowed failures
"You can define rows that are allowed to fail in the build matrix. Allowed failures are items in your build matrix that are allowed to fail without causing the entire build to fail. This lets you add in experimental and preparatory builds to test against versions or configurations that you are not ready to officially support."
Unfortunately, I believe the way the matrix reads your first set of code is as php nightly version as the "allowed to fail" version with the environment as part of nightly.
Because of how Travis allows failures it must be an exact match, you can not just specify env: as an allowed failure you have to specify for each php version with the env: you want to allow as a failure for example
allow_failures:
- php: 5.3
env: SYMFONY_VERSION=2.8.*#dev TEST_GROUP=canFail
- php: 5.4
env: SYMFONY_VERSION=2.8.*#dev TEST_GROUP=canFail
- php: 5.5
env: SYMFONY_VERSION=2.8.*#dev TEST_GROUP=canFail
- php: 5.6
env: SYMFONY_VERSION=2.8.*#dev TEST_GROUP=canFail
- php: 7.0
env: SYMFONY_VERSION=2.8.*#dev TEST_GROUP=canFail
- php: hhvm
env: SYMFONY_VERSION=2.8.*#dev TEST_GROUP=canFail
- php: nightly # Allow all tests to fail for nightly

Answer 2

According to this issue, the php and env keys must match perfectly. env can be either a single value or an array, but in both case it must be a perfect match. So if you want your build:
- php: 5.5
env: SYMFONY_VERSION=2.8.*#dev TEST_GROUP=canFail
to be allowed to fail, you have to either give the whole env key SYMFONY_VERSION=2.8.*#dev TEST_GROUP=canFail and the whole env key and the PHP version (if you had the same env key for different PHP versions).