GitHub Actions: homebrew slow - homebrew

I have never used macOS in my life so I'm not sure if this is a problem with the image used by GitHub actions, homebrew itself or how I'm using it:
I'm trying to use GitHub actions to build and test a C++ project on Ubuntu and macOS. Building the code using CMake only takes around one minute in both cases, however, installing the dependencies (Boost and Lua) takes more than three minutes on macOS as opposed to 20 seconds on Ubuntu. My workflow.yml looks as follows:
jobs:
build-and-test:
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-latest, macos-latest]
include:
- os: ubuntu-latest
INSTALL_DEPS: sudo apt-get install -yq libboost-dev libboost-graph-dev lua5.3 liblua5.3-dev
CC: gcc
CXX: g++
- os: macos-latest
INSTALL_DEPS: brew install boost lua
CC: clang
CXX: clang++
steps:
- uses: actions/checkout#v2
- name: install-dependencies
run: ${{ matrix.INSTALL_DEPS }}
# ...
It seems as though boost is the only package I can install using homebrew, as opposed to the more fine-grained libboost-dev + libboost-graph-dev.
Is there anything I can do here if I don't want to build Boost from source on macOS? I know that it's possible to cache dependencies using GitHub actions but that seems overly complicated for my use-case and I don't actually know how to use that with homebrew in the first case.

I made small test to check how long it does take to use brew on Ubuntu
jobs:
build-and-test:
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-latest, macos-latest]
include:
- os: ubuntu-latest
INSTALL_DEPS: /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)" && date && brew install boost lua && date
#INSTALL_DEPS: sudo apt-get install -yq libboost-dev libboost-graph-dev lua5.3 liblua5.3-dev
CC: gcc
CXX: g++
- os: macos-latest
INSTALL_DEPS: brew install boost lua
CC: clang
CXX: clang++
steps:
- uses: actions/checkout#v2
- name: install-dependencies
run: ${{ matrix.INSTALL_DEPS }}
And to run brew install boost lua it took about 26 seconds
Fri Dec 18 11:16:02 UTC 2020
Fri Dec 18 11:16:28 UTC 2020
Where on MacOS I got 2:33, 46s, 45s. So comparing to highest value this is a big difference. But comparing 45s to 26s is not (still almost 2 times, but it doesn't sound such bad). I think that there is no point actually to optimizing that. There can be plenty of aspects which may impact and even this message may also play role on that
If you are on good way to finish your pipeline focus on that, as you may loose plenty of time fighting for best performance without a luck as some aspect could be out of your control.

Related

Can I install arm64 libraries on x86_64 with homebrew?

I am cross-compiling on x86_64 MacOS 11 for arm64 architecture.
clang/XCode support it but I face an issue when external library is required.
Let it be boost, for example. I know that the bottle for arm64 is available but it looks like there is no way to select it for installation.
arch -arm64 brew ... says that this architecture is unknown which sounds fair.
So the question is wheather there is an option to force brew install bottles for 'foreign' architecture ?
Thanks to Homebrew team
https://github.com/Homebrew/discussions/discussions/2843
I made it work with the code like this:
setopt sh_word_split
mkdir arm-homebrew && curl -L https://github.com/Homebrew/brew/tarball/master | tar xz --strip 1 -C arm-homebrew
alias arm-brew='$(pwd)/arm-homebrew/bin/brew'
response=$(arm-brew fetch --force --bottle-tag=arm64_big_sur boost | grep "Downloaded to")
parsed=($response)
arm-brew install $parsed[3]
Native system brew can be asked to download any desired bottle version with two switches:
--force ignores the compatibility check
--bottle-tag=arm64_big_sur downloads the ARM Big Sur build. This assumes the formula has this build, check the formula .rb for available tags. Big Sur is the earliest macOS with ARM support.
Then use brew --cache to get the filename of the downloaded bottle and pass to brew install:
brew fetch --force --bottle-tag=arm64_big_sur boost
brew install $(brew --cache --bottle-tag=arm64_big_sur boost)
You can verify that it worked with cd $(brew --prefix boost) then use file on any .dylib files to see what architecture they're built for.
If the formula has any dependencies then they will still be installed normally (for the native architecture), so you may have to follow the same process for them.

Cannot run genhtml in dart project

I'm a newbie, I wanna learn Dart language.So basically I wanna display my coverage report. Follow in this https://pub.dev/packages/test_coverage. When I using genhtml command it not working
'genhtml' is not recognized as an internal or external command,
operable program or batch file.
And in my pubspec.yaml
name: fordart
description: A sample command-line application.
# version: 1.0.0
# homepage: https://www.example.com
environment:
sdk: '>=2.7.0 <3.0.0'
#dependencies:
# path: ^1.6.0
dev_dependencies:
pedantic: ^1.8.0
test: ^1.6.0
test_coverage: ^0.4.1
dependencies:
lcov: ^*
What I missing and can all you guys advice more coverage tools supported Dart language.I searched in Google there are many coverage tools like CodeCOV, Coverall, Codacy,... but it's so hard to set up in Dart projects beacause Dart language is not a popular language and the coverage tools maybe not supported for it .If you guys know how to set up please contact me.Thanks
Right now doesn't seem like repositories with only Dart work with Codacy
The lcov package puts down the genhtml command.
If you're on OSX:
❯ brew install lcov
Warning: lcov 1.15 is already installed and up-to-date.
To reinstall 1.15, run:
brew reinstall lcov
❯ command -v lcov
/opt/homebrew/bin/lcov
❯ command -v genhtml
/opt/homebrew/bin/genhtml
If you're on other Linux, including Amazon Linux 2:
❯ yum install lcov
❯ command -v genhtml
/usr/bin/genhtml
❯ command -v lcov
/usr/bin/lcov
If you get a message the package cannot be found, you might need to enable the repo, especially true on Amazon Linux 2:
❯ yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
See: Add repositories on an Amazon Linux instance

Travis CI not using master branch cache

I'm using Carthage as my project package manager and Travis CI.
The problem that I'm facing is that yesterday, Travis did create a cache from master branch correctly, as you can see in the picture below, and everything was working fine.
But today, when I did switch to another branch and pushed changes to it, Travis is not fetching the cache that it has made from master branch. Instead, it's creating a new cache for this branch that I've created. I want it to use master cache always, and don't try to create a cache for each branch that I create from master.
How should I proceed to achieve that?
Source Files:
.travis.yml
#Xcode 9.4 image default contents: https://blog.travis-ci.com/2018-07-19-xcode9-4-default-announce
#Need to create: "if: branch = master" and "if: branch = development"
language: swift
os: osx
osx_image: xcode10
sudo: required
cache:
directories:
- Carthage
before_install:
# Update Homebrew
- brew update
# Clean install Carthage
- brew uninstall --force carthage
- brew install carthage
- brew cask install fastlane
# Provide travis machines with github access
- echo -e "machine github.com\n login $CI_USER_TOKEN" >> ~/.netrc
# Print script to avoid travis_wait from stopping build process
- while sleep 480m; do echo "=====[ $SECONDS seconds, Carthage still building... ]====="; done &
# Carthage Bootstrap
- carthage bootstrap --cache-builds --platform ios,watchos --no-use-binaries
# Stop Print script
- kill %1
script:
- fastlane beta

docker version number 18.09

See link on explanation of release naming
https://docs.docker.com/install/
My understanding is its using YY.mm. e.g. 18.09.0
So I go to install for ubuntu following steps here
https://docs.docker.com/install/linux/docker-ce/ubuntu/#install-docker-ce-1
where in 3b it gives an example of installing a specific version:
docker-ce=18.03.0~ce-0~ubuntu
sudo apt-get install docker-ce=<VERSION>`
So I go ahead and do this:
sudo apt-get install docker-ce=18.09.0~ce-0~ubuntu
and get
Version '18.09.0~ce-0~ubuntu' for 'docker-ce' was not found
So checking cache I have to install
5:18.09.0~3-0~ubuntu-bionic
What is the 5 about?
It's the epoch. Think of it as a versioning for the version number e.g. if the package decides to change versioning from date-based to semver style, they can begin prefixing version numbers with 1:1.0.1.
Here's the official policy manual, some examples are given here.
Have to specify the 5.
Running the following works
sudo apt-get install docker-ce=5:18.09.0~3-0~ubuntu-bionic
Docker version then shows up as 18.09

Travis CI: PHP 7.2 don't support Argon2i

I've an error in my project with Travis CI:
Argon2i algorithm is not supported. Please install the libsodium extension
or upgrade to PHP 7.2+.
But, Argon2i is present in the PHP 7.2 version and Travis CI install the PHP 7.2 version:
$ phpenv global 7.2 2>/dev/null
7.2 is not pre-installed; installing
Downloading archive: https://s3.amazonaws.com/travis-php-archives/binaries/ubuntu/14.04/x86_64/php-7.2.tar.bz2
$ curl -s -o archive.tar.bz2 $archive_url && tar xjf archive.tar.bz2 --directory /
$ phpenv global 7.2
$ php --version
PHP 7.2.0 (cli) (built: Dec 2 2017 17:12:55) ( ZTS )
Copyright (c) 1997-2017 The PHP Group
Zend Engine v3.2.0, Copyright (c) 1998-2017 Zend Technologies
with Zend OPcache v7.2.0, Copyright (c) 1999-2017, by Zend Technologies
with Xdebug v2.6.0-dev, Copyright (c) 2002-2017, by Derick Rethans
Someone have an idea ?
I faced the same problem recently in a Symfony 4 project and posted an issue on Travis' Github.
However, the problem does not seem to come from Travis but from PHP 7.2 default build itself.
Quoting myself:
I was locally using a pre-configured PHP, so to be sure I just compiled PHP 7.2 from sources.
$ ./php -v
PHP 7.2.0 (cli) (built: Dec 6 2017 15:26:29) ( NTS )
Copyright (c) 1997-2017 The PHP Group
Zend Engine v3.2.0, Copyright (c) 1998-2017 Zend Technologies
Then I tryed to use the ARGON2I algorithm such as described in official docs:
$ ./php -r 'echo password_hash("test", PASSWORD_ARGON2I) . "\n";'
Warning: Use of undefined constant PASSWORD_ARGON2I - assumed
'PASSWORD_ARGON2I' (this will throw an Error in a future version of PHP) in Command line code on line 1
Warning: password_hash() expects parameter 2 to be integer, string given in Command line code on line 1
While not having any problem with BCRYPT:
$ ./php -r 'echo password_hash("test", PASSWORD_BCRYPT) . "\n";'
$2y$10$wsWe3BhyzenVqDs6JV/fPOB0XKh0oTuGdrgLp61MnUPzOUdw4jZey
This is strange. I would have expected this algorithm to be part of the default PHP 7.2 build, just as other hash algorithms. And nothing seems to indicate the opposite in the docs. I'll investigate. Maybe I understood something wrong... but this looks like a bug to me, since they say here that PASSWORD_ARGON2I is part of PHP core.
Edit:
In the light of Sheppard's comment, it appears indeed that 7.2 does not implement the PASSWORD_AGRON2Ialgorithm in its default build. PHP has to be compiled with option -with-password-argon2, such as described in https://wiki.php.net/rfc/argon2_password_hash.
It appear PHP 7.2 haven't argon2i password hasher by default, it's a compile option, but TravisCi don't use it in their PHP images, then we can avoid this error by adding a library (initially used in PHP < 7.2 versions) as mensionned in https://symfony.com/blog/new-in-symfony-3-4-argon2i-password-hasher, but not in our composer.json directly, call it in .travis.yml file:
before_install:
# Fix Argon2i password hasher in TravisCi PHP version
- composer require paragonie/sodium_compat
EDIT:
Because Symfony has been updated, the previous solution no longer works, then I choose to directly add the libsodium extension with PECL. This method is better I think, because we install and enable the PHP extension.
We must download sources of the libsodium library because ubuntu 14.04 haven't the library, then compile it, and compile the PHP extension with pecl, and enable it.
It works well, but it take more time than the previous solution.
before_install:
# Manually compile the libsodium library
- sudo apt-get update -qq
- sudo apt-get install build-essential git -y
- git clone -b stable https://github.com/jedisct1/libsodium.git
- cd libsodium && sudo ./configure && sudo make check && sudo make install && cd ..
- '[[ "$TRAVIS_PHP_VERSION" == "nightly" ]] || phpenv config-rm xdebug.ini'
- composer self-update
install:
# Manually install libsodium, because the TravicCi image doesn't provide PHP7.2 with libsodium
- pecl install libsodium
- echo "extension=sodium.so" >> ~/.phpenv/versions/$(phpenv version-name)/etc/php.ini
This is indeed a silly dropped-ball by PHP. Fortunately there is a reasonable workaround - while libargon2 is not compiled in by default (because it's not available on all the platforms that PHP is), libsodium is, and though libsodium doesn't provide the PASSWORD_ARGON2I constant, it has the sodium_crypto_pwhash function, which uses the Argon2id variant of the Argon2 password hashing algorithm, which is stronger than the stock PHP 7.2 Argon2i, but is not compatible with it. You use it like this:
$hash = sodium_crypto_pwhash_str(
$password,
SODIUM_CRYPTO_PWHASH_OPSLIMIT_INTERACTIVE,
SODIUM_CRYPTO_PWHASH_MEMLIMIT_INTERACTIVE
);
You can get this function in PHP versions prior to PHP 7.2 using the paragonie/sodium_compat library via composer.
I think transparent fallback through this was a missed opportunity in Symfony - it ends up failing to provide an implementation in exactly the same way that PHP itself does, which is no help to anyone.

Resources