Travis.ci config and execution of npm scripts - travis-ci

I'm trying to use Travis for an open source build of a PR. The configuration is quite simple and the logs seem to show that the appropriate modules are installed upon running yarn install and I am installing the same version of yarn as that which is used locally. The issue is that when I try to execute the scripts defined in the package.json scripts object the modules are not found when running in Travis. Below is the config and the errors I receive at build time.
before_install:
- curl -o- -L https://yarnpkg.com/install.sh | bash -s -- --version 1.7.0
- export PATH="$HOME/.yarn/bin:$PATH"
cache:
yarn: true
directories:
- "node_modules"
env:
- NODE_ENV=production
language: node_js
node_js:
- 8
- 9
- "stable"
install:
- yarn install
script:
- yarn run lint
- yarn test
The above producing the following output in the build logs:
yarn run v1.7.0
$ ./node_modules/.bin/eslint src/**
/bin/sh: 1: ./node_modules/.bin/eslint: not found
error Command failed with exit code 127.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
The command "yarn run lint" exited with 1.
0.54s$ yarn run test
yarn run v1.7.0
$ ./node_modules/.bin/jest
/bin/sh: 1: ./node_modules/.bin/jest: not found
error Command failed with exit code 127.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
The command "yarn run test" exited with 1.
The package.json for this project is as follows:
"scripts": {
"test": "./node_modules/.bin/jest",
"lint": "./node_modules/.bin/eslint src/**",
"precommit": "lint-staged",
"format": "prettier --trailing-comma es5 --single-quote --write 'src/*/*.js' '!(node_modules)/**/*.js'"
},

Was having the same issue and was able to solve it by doing the following in package.json file
"scripts": {
"test": "test --passWithNoTests",
}

Related

Mocha --watch with Docker

I'm trying to use --watch on mocha but when I save code source or test source, it doesn't re run tests. I have an enviroment using docker-compose with node:16-slim image and my tests run inside it. This same config works with bare metal enviroment.
The dev docker image run the app with:
USER node
CMD ["npm", "run", "dev"]
And this npm script is:
"dev": "npx nodemon --inspect=0.0.0.0:1080 src/index.js",
test npm script:
"test:tdd": "cross-env NODE_ENV=test mocha --config .mocharc.tdd.js",
.mocharc.tdd.js:
module.exports = {
"reporter": "dot",
"watch": true,
"watch-ignore": [],
"file": 'test/common.js',
"recursive": true
};
output:
> test-app#1.0.0 test:tdd
> cross-env NODE_ENV=test mocha --config .mocharc.tdd.js
!
0 passing (6ms)
1 failing
1) Events
abc:
MissingParamError: Missing param: Data
at updated (src/app/events.js:8:22)
at Context.<anonymous> (test/app/events.test.js:13:28)
ℹ [mocha] waiting for changes...
Versions:
➜ test-app git:(master) ✗ npx mocha --version
10.0.0
➜ test-app git:(master) ✗ node --version
v16.15.0
➜ test-app git:(master) ✗ npx nodemon --version
2.0.15
What can I do to fix this? Thanks in advance =)
I solved it. I added watch-files attr to config file.
mocharc.tdd.js:
module.exports = {
"reporter": "dot",
"watch": true,
"watch-files": ['test/**/*.js', 'src/**/*.js'],
"watch-ignore": ['node_modules'],
"file": 'test/common.js',
"recursive": true
};

How to properly run Docker containers and YARN server with WSL 2 in Windows 10?

The question may be poorly formed, but let me explain.
Initial setup
I had my web-dev Laravel project in C:\dev\gitlab.our-company.com\laravel-backend with a Vue app in .\public\vue-frontend.
I use Docker (using WSL 2 based engine) with the following commands:
// First time
export AUTH_TOKEN='glpat-XXX'
export COMPOSER_AUTH='{"http-basic":{"gitlab.our-company.com": {"username": "oauth2", "password": "${AUTH_TOKEN}"}}}'
docker build -f deployment/app.Dockerfile --build-arg COMPOSER_AUTH -t laravel_backend_app .
docker build -f deployment/web.Dockerfile --build-arg AUTH_TOKEN -t laravel_backend_web .
// Always
docker-compose -f deployment/docker-compose.yml -f deployment/docker-compose.override.yml -p laravel_backend up
Then I open container's bash with
docker-compose -f deployment/docker-compose.yml -f deployment/docker-compose.override.yml -p laravel_backend exec app bash
which in turn allows me to run php artisan test.
I can also run
yarn --cwd ./public/vue-frontend/ install && yarn --cwd ./public/vue-frontend/ serve
to serve the frontend.
Why I made changes
Running php artisan test or vendor/bin/grumphp run in container's bash was ridiculously slow (x10).
Current setup
$ wsl docker --version
Docker version 20.10.12, build e91ed57
Using Ubuntu for Windows with explorer.exe . I copied entire C:\dev to (Ubuntu)~/dev
I rebuilt Docker containers and up-ed them, executed app bash and tried php artisan test. It is now lightning fast.
Question 1
I can do the procedure in the last paragraph in 2 ways.
I can open VS Code integrated terminal in //wsl$/Ubuntu/home/{USER}/dev/gitlab.our-company.com/laravel-backend and do the above procedure directly there, or
as an extra step, when in terminal, I can execute wsl command and my location becomes ~/dev/gitlab.our-company.com/laravel-backend before continuing with said procedure.
In both cases, I get my php artisan test improvement speed.
Which is correct, or for some reason better?
Problem
I had issues with yarn, so in Ubuntu for Windows I installed nvm and then with it nodejs, npm, yarn.
Yarn is installed in both cases, thought a tad different version.
//wsl$/Ubuntu/home/{USER}/dev/gitlab.our-company.com/laravel-backend (205-issue-title)
$ yarn --version
1.22.17
$ wsl
{USER}#DESKTOP-NAME:~/dev/gitlab.our-company.com/laravel-backend$ yarn --version
1.22.15
Now the problem is serving the app.
//wsl$/Ubuntu/home/{USER}/dev/gitlab.our-company.com/laravel-backend (205-issue-title)
$ yarn --cwd ./public/vue-frontend/ install && yarn --cwd ./public/vue-frontend/ serve
yarn install v1.22.17
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
[-/3] ⡀ waiting...
[2/3] ⡀ ejs
error \\wsl$\Ubuntu\home\{USER}\dev\gitlab.our-company.com/laravel-backend\public\vue-frontend\node_modules\yorkie: Command failed.
Exit code: 1
Command: node bin/install.js
Arguments:
Directory: \\wsl$\Ubuntu\home\{USER}\dev\gitlab.our-company.com/laravel-backend\public\vue-frontend\node_modules\yorkie
Output:
'\\wsl$\Ubuntu\home\{USER}\dev\gitlab.our-company.com/laravel-backend\public\vue-frontend\node_modules\yorkie'
CMD.EXE was started with the above path as the current directory.
UNC paths are not supported. Defaulting to Windows directory.
node:internal/modules/cjs/loader:936
throw err;
^
Error: Cannot find module 'C:\Windows\bin\install.js'
at Function.Module._resolveFilename (node:internal/modules/cjs/loader:933:15)
at Function.Module._load (node:internal/modules/cjs/loader:778:27)
at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12)
at node:internal/main/run_main_module:17:47 {
code: 'MODULE_NOT_FOUND',
requireStack: []
or with wsl:
{USER}#DESKTOP-3OMP0G1:~/dev/gitlab.our-company.com/laravel-backend$ yarn --cwd ./public/vue-frontend/ install && yarn --cwd ./public/vue-frontend/ serve
yarn install v1.22.15
[1/4] Resolving packages...
[2/4] Fetching packages...
error An unexpected error occurred: "https://gitlab.our-company.com/api/v4/projects/{PROJECT_ID}/packages/npm/#our-company/case-messaging/-/#our-company/case-messaging-1.0.3.tgz: Request failed \"404 Not Found\"".
info If you think this is a bug, please open a bug report with the information provided in "/home/{USER}/dev/gitlab.our-company.com/laravel-backend/public/vue-frontend/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
I now believe the second way ie. in wsl is the right way.
If anyone has a deeper, better or different explanation I will review it, and likely mark it as the accepted answer.
The problem I had with running yarn install in wsl is somewhat unrelated to the question and was caused by an NPM package that resides in Our Company's self hosted GitLab (as opposed to NPM public registry) and npm had no access to download the case-messaging package.
Solved it with
npm config set #our-company:registry https://gitlab.our-company.com/api/v4/projects/{PROJECT_ID}/packages/npm/
npm config set -- '//gitlab.our-company.com/api/v4/projects/{PROJECT_ID}/packages/npm/:_authToken' "${AUTH_TOKEN}"

Installing private npm package works on local laptop but not in gitlab pipeline

Why is this build stage in my gitlab pipeline failing with
npm Err! 401: Unable to authenticate, need: Basic realm="Artifactory Realm"
When I run the command $ npm-cli-login -u $USERNAME -p $API_KEY -e $EMAIL -r $REPOSITORY it seems like I get correcly logged in. My correct username gets displayed and the global .npmrc file gets created in my home directory. But when I run npm install or npm i --registry=https://<my_private_repo>.jfrog.io/<my_private_repo> it fails with the 401.
Following output am I seeing in the logs of my failed pipeline stage:
Pulling docker image node:14.15.4 ...
Using docker image sha256:924763541c... for node:14.15.4 with digest node#sha256:cb01e9d98a...
Preparing environment 00:01
Running on runner-ffeacb89-project-...-concurrent-0 via ....r.gitlab.host...
Getting source from Git repository 00:01
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/gravity/my_test_project/.git/
Checking out 5f7... as dev...
Removing .gradle/
Skipping Git submodules setup
Restoring cache 00:06
Checking cache for my_test_project...
cache.zip is up to date
Successfully extracted cache
Executing "step_script" stage of the job script 01:07
Using docker image sha256:924763541c0c8b3839132... for node:14.15.4 with digest node#sha256:cb01e9d9... ...
$ export GRADLE_USER_HOME=`pwd`/.gradle
$ pwd
/builds/my_test_project
$ cd ~
$ npm install -g npm-cli-login
npm WARN deprecated request#2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated har-validator#5.1.5: this library is no longer supported
/usr/local/bin/npm-cli-login -> /usr/local/lib/node_modules/npm-cli-login/bin/npm-cli-login.js
> core-js#3.10.1 postinstall /usr/local/lib/node_modules/npm-cli-login/node_modules/core-js
> node -e "try{require('./postinstall')}catch(e){}"
+ npm-cli-login#0.1.1
added 634 packages from 451 contributors in 34.086s
$ npm-cli-login -u $USERNAME -p $API_KEY -e $EMAIL -r $REPOSITORY
info attempt registry request try #1 at 2:32:54 PM
http request PUT https://<my_private_repo>.jfrog.io/<my_private_repo>/api/npm/npm/-/user/org.couchdb.user:<my_correct_username>
http 201 https://<my_private_repo>.jfrog.io/<my_private_repo>/api/npm/npm/-/user/org.couchdb.user:<my_correct_username>
$ pwd
/root
$ cat .npmrc
//<my_private_repo>.jfrog.io/<my_private_repo>/api/npm/npm//:_authToken=eyJ2...(very long token, looks correct)
$ echo 'always-auth = true' >> .npmrc
$ cat .npmrc
//<my_private_repo>.jfrog.io/<my_private_repo>/api/npm/npm//:_authToken=eyJ2...(very long token, looks correct)
always-auth = true
$ npm i -g #angular/cli
npm WARN deprecated request#2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated har-validator#5.1.5: this library is no longer supported
/usr/local/bin/ng -> /usr/local/lib/node_modules/#angular/cli/bin/ng
> #angular/cli#11.2.9 postinstall /usr/local/lib/node_modules/#angular/cli
> node ./bin/postinstall/script.js
+ #angular/cli#11.2.9
added 242 packages from 181 contributors in 12.287s
$ cd /builds/my_test_project/ui
$ printenv NPM_TOKEN
eyJ2Z...(my long token)
$ npm config set //registry.npmjs.org/:_authToken ${NPM_TOKEN}
$ npm install
npm WARN deprecated debug#4.1.1: Debug versions >=3.2.0 <3.2.7 || >=4 <4.3.1 have a low-severity ReDos regression when used in a Node.js environment. It is recommended you upgrade to 3.2.7 or 4.3.1. (https://github.com/visionmedia/debug/issues/797)
npm WARN deprecated axios#0.20.0: Critical security vulnerability fixed in v0.21.1. For more information, see https://github.com/axios/axios/pull/3410
npm WARN deprecated request#2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated fsevents#2.1.3: "Please update to latest v2.3 or v2.2"
npm WARN deprecated chokidar#2.1.8: Chokidar 2 will break on node v14+. Upgrade to chokidar 3 with 15x less dependencies.
npm WARN deprecated har-validator#5.1.5: this library is no longer supported
npm WARN deprecated fsevents#1.2.13: fsevents 1 will break on node v14+ and could be using insecure binaries. Upgrade to fsevents 2.
npm WARN deprecated urix#0.1.0: Please see https://github.com/lydell/urix#deprecated
npm WARN deprecated resolve-url#0.2.1: https://github.com/lydell/resolve-url#deprecated
npm ERR! code E401
npm ERR! Unable to authenticate, need: Basic realm="Artifactory Realm"
npm ERR! A complete log of this run can be found in:
npm ERR! /root/.npm/_logs/2021-04-16T14_33_25_786Z-debug.log
Cleaning up file based variables 00:00
ERROR: Job failed: exit code 1
This is my build stage in the .gitlab-ci.yml file (Spring Boot Project)
build:
stage: build
image: node:14.15.4
script:
- pwd
- cd ~
- npm install -g npm-cli-login
- npm-cli-login -u $USERNAME -p $API_KEY -e $EMAIL -r $REPOSITORY
- cat .npmrc
- echo 'always-auth = true' >> .npmrc
- cat .npmrc
- npm i -g #angular/cli
- cd /builds/myProjectFolder
- printenv NPM_TOKEN
- npm config set //myrepo.jfrog.io/myrepo/api/npm/npm//:_authToken ${NPM_TOKEN}
- npm install --registry=https://myrepo.jfrog.io/<my_private_repo>/api/npm/npm/
artifacts:
paths:
- app/src/ui/dist/dm-consent-page
expire_in: 5 days
I solved it by using this method: https://gruchalski.com/posts/2020-09-09-authenticate-to-private-jfrog-npm-registry/
After running this curl command I received everything that I needed to put into my global .npmrc file:
curl -u ${JFROG_USER}:${JFROG_ENCRYPTED_PASSWORD} https://${JFROG_ORG}.jfrog.io/${JFROG_ORG}/api/npm/auth
For anyone who's interested, the full script in my gitlab ci pipeline stage now looks like this:
script:
- npm -v
6.14.10
- node -v
v14.15.4
- cd ~
- pwd
/root
# install angular globally
- npm i -g #angular/cli
# create the config file '.npmrc' for authenticating at jFrog when running 'npm install'.
- cat > .npmrc
- echo _auth = ${NPM_AUTH_TOKEN} >> .npmrc <- This is the token that I received after running the curl command from the tutorial / link above
- echo always-auth = true >> .npmrc
- echo email = ${EMAIL} >> .npmrc
# the next line makes npm look for the packages that are annotated with #<my-private-repo> at the JFrog Repo.
- echo #<my-private-repo>:registry=${UI_JFROG_REGESTRY} >> .npmrc
# change back to the project folder.
- cd /builds/<my-project-folder>/ui
# install all packages + the <my-private-repo> package from JFrog.
- npm install

Configure .yml to execute multiple commands

Hello i have simple configuration in my project:
version: 2
jobs:
build:
docker:
- image: circleci/node:7
steps:
- checkout
- run:
name: install-dependencies
command: npm install
- run:
name: tests
command: npm test
- deploy:
name: digital-ocean
command: ssh -o "StrictHostKeyChecking no" user#hostname "cd ~/profile-store; git pull; npm install; forever start app.js"
The problem is it need multiply command:
cd client
npm start
cd ..
(in second iteration should install packages from server and in the next run unit tests in client)
I tried these syntax:
command: ["cd client", "npm install", "cd .."]
But getting an error. The question is :
How can i write to execute 3 commands in one command instruction?
command: cd client && npm install && cd ..
For enhanced readability, you can use a folded block scalar (folds linebreaks into spaces):
command: >-
cd client &&
npm install &&
cd ..
Note that you do not really need the final cd .. since the shell instance executing the command is not re-used.

"./gradlew: Permission denied" when deploying a jhipster 5.1.0 project on Gitlab-CI

I am using jhipster 5.1.0, I used "jhipster ci-cd" in order to generate the .gitlab-ci.yml file.
I am running Gitlab and Gitlab-CI on a private Ubuntu 18.04LTS server in my company. I configured the Gitlab Runner to execute the builds with docker.
My .gitlab-ci.yml file is as follows (I did not modify it much):
image: jhipster/jhipster:v5.1.0
cache:
key: "$CI_COMMIT_REF_NAME"
paths:
- .gradle/wrapper
- .gradle/caches
stages:
- build
before_script:
- export GRADLE_USER_HOME=`pwd`/.gradle
- ./gradlew yarn_install -PnodeInstall --no-daemon
gradle-build:
stage: build
script:
- ./gradlew compileJava -x check -PnodeInstall --no-daemon
- ./gradlew test -PnodeInstall --no-daemon
- ./gradlew yarn_test -PnodeInstall --no-daemon
- ./gradlew bootJar -Pprod -x check -PnodeInstall --no-daemon
artifacts:
paths:
- build/libs/*.jar
# Uncomment following to expire the artifacts after defined period, https://docs.gitlab.com/ee/ci/yaml/README.html#artifacts-expire_in
# expire_in: 90 day
Here is the output of the gitlab-ci runner:
...
Successfully extracted cache
$ export GRADLE_USER_HOME=`pwd`/.gradle
$ ./gradlew compileJava -x check -PnodeInstall --no-daemon
/bin/bash: line 60: ./gradlew: Permission denied
ERROR: Job failed: exit code 1
As the problem seems obvious, I tried to add " - chmod +x gradlew", before the ".gradlew" call in the "before_script" section. I thought it would be a good idea, because it was generated by the "jhipster ci-cd" command before 5.1.0, but not anymore. No success: Gitlab-CI output became as follows:
...
Successfully extracted cache
$ export GRADLE_USER_HOME=`pwd`/.gradle
$ chmod +x gradlew
chmod: changing permissions of 'gradlew': Operation not permitted
ERROR: Job failed: exit code 1
So I tried to switch to the docker image "openjdk:8" instead of "jhipster/jhipster:v5.1.0", in the .gitlab-ci.yml file. Much better, gradle runs the "yarn install" command, but it stops at some point, because that container does not contain "libpng-dev" (which was added recently into the jhipster container, no luck !):
...
[5/5] Building fresh packages...
error An unexpected error occurred:
"/builds/epigone/exportCCN/node_modules/pngquant-bin: Command failed.
Exit code: 1
Command: sh
Arguments: -c node lib/install.js
Directory: /builds/epigone/exportCCN/node_modules/pngquant-bin
Output:
⚠ The `/builds/epigone/exportCCN/node_modules/pngquant-bin/vendor/pngquant`
binary doesn't seem to work correctly
⚠ pngquant pre-build test failed
ℹ compiling from source
✔ pngquant pre-build test passed successfully
✖ Error: pngquant failed to build, make sure that libpng-dev is installed
at Promise.all.then.arr (/builds/epigone/exportCCN/node_modules/pngquant-bin/node_modules/bin-build/node_modules/execa/index.js:231:11)
at <anonymous>
at process._tickCallback (internal/process/next_tick.js:188:7)".
info If you think this is a bug, please open a bug report with the information provided in "/builds/epigone/exportCCN/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
:yarn_install FAILED
You need to modify the permissions on your git repo.
Run:
git update-index --chmod=+x gradlew
then commit and push.

Resources