BitBucket pipeline is not using cache for npm install - bitbucket

I have a single BitBucket repository containing the code for an Angular app in a folder called ui and a Node API in a folder called api.
My BitBucket pipeline runs ng test for the Angular app, but the node_modules folder isn't being cached correctly.
This is my BitBucket Pipeline yml file:
image: trion/ng-cli-karma
pipelines:
default:
- step:
caches:
- angular-node
script:
- cd ui
- npm install
- ng test --watch=false
definitions:
caches:
angular-node: /ui/node_modules
When the builds runs it shows:
Cache "angular-node": Downloading
Cache "angular-node": Extracting
Cache "angular-node": Extracted
But when it performs the npm install step it says:
added 1623 packages in 41.944s
I am trying to speed the build up and I can't work out why npm needs to install the dependencies assuming they are already contained in the cache which has been restored.

my guess is, your cache position is not correct. there is a pre-configured node cache (named "node") that can just be activated. no need to do a custom cache for that. (the default cache fails, because your node build is in a sub folder of the clone directory, so you need a custom cache)
cache positons are relative to the clone directory. bitbucket clones into /opt/atlassian/pipelines/agent/build thats probably why your absolute cache-path did not work.
simply making the cache reference relative should do the trick
pipelines:
default:
- step:
caches:
- angular-node
script:
- cd ui
- npm install
- ng test --watch=false
definitions:
caches:
angular-node: ui/node_modules
that may fix your issue

Related

GitLab CI : Runner Docker and cache Maven repository does not working

I'm trying to setup a GitLab CI pipeline on a Docker runner.
I have a Docker runner with a Maven image (maven:3.8.6-openjdk-11) that I'm trying to use for my pipeline that compiles a Maven project.
I have set maven repository cache but I feel like this cache is not working properly. Every time my pipeline runs, it downloads the dependencies that are useful for compiling my project, onto my Nexus artifactory... I expected it to download only the "new" dependencies, and for the others, they use the cache without downloading them.
Below is the content of the .gitlab-ci.yml file:
variables:
MAVEN_OPTS: >-
-Dhttps.protocols=TLSv1.2
-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository
-Dorg.slf4j.simpleLogger.showDateTime=true
-Djava.awt.headless=true
cache:
paths:
- .m2/repository/
build:
tags:
- tdev
script:
- mvn --settings $MAVEN_SETTINGS clean compile
On each run it download the dependencies:
[...]
Downloading from infogreffe: https://xxxx/nexus/repository/infogreffe/org/wildfly/core/wildfly-core-security/19.0.0.Final/wildfly-core-security-19.0.0.Final.jar
Downloaded from infogreffe: https://xxxx/nexus/repository/infogreffe/org/wildfly/core/wildfly-controller-client/19.0.0.Final/wildfly-controller-client-19.0.0.Final.jar (214 kB at 70 kB/s)
Downloading from infogreffe: https://xxxx/nexus/repository/infogreffe/org/wildfly/security/wildfly-elytron-auth/2.0.0.Final/wildfly-elytron-auth-2.0.0.Final.jar
Downloaded from infogreffe: https://xxxx/nexus/repository/infogreffe/com/oracle/ojdbc5/11.2.0.2.0/ojdbc5-11.2.0.2.0.jar (2.0 MB at 666 kB/s)
[...]
Note that I use a local cache.
Thanks.
I expect that with using a cache for my maven repository, it won't try to download dependencies on every pipeline run unless there are new dependencies. But maybe I misunderstood how caching works...

Gitlab CI /CD: Cannot override private project npm package registry URL for dependencies

I was wondering if someone could help me with my CI/CD configuration for a multi-JS project setup.
Project js-core has some core JS libaries, and project vue-core has some reusable vue components. They are both published as npm packages to the "js-core" project's repository (so that we can use a deploy token with npm publish, as opposed to using a group deploy token, where you have to script the package creation and push directly to the API).
vue-core has a dependency of js-core, so it needs to access the npm package registry of the js-core project to download it inside of the Docker CI/CD instance.
However, gitlab does not allow me to override the package registry's URL. According to some google research, setting the values with yarn config set #myorg:registry <gitlab-url> / yarn config set //gitlab.com/api/v4/... <deploy token> or npm config set #myorg:registry <gitlab-url> / npm config set //gitlab.com/api/v4/... <deploy token> should work. However I can see that it is still trying to download the packages from the vue-core package registry, even though it is disabled as a feature there.
This is the part of my .gitlab-ci.yml that runs before it fails:
image: node:17-alpine
stages:
- build
- test
before_script:
- yarn config set #myorg:registry https://gitlab.com/api/v4/projects/${NPM_PACKAGES_PROJECT_ID}/packages/npm/
- yarn config set //gitlab.com/api/v4/projects/${NPM_PACKAGES_PROJECT_ID}/packages/npm/:_authToken ${NPM_PACKAGES_PROJECT_TOKEN}
- yarn install
It fails at yarn install with:
[2/4] Fetching packages...
error An unexpected error occurred: "https://gitlab.com/api/v4/projects/<vue-core_project_id>/packages/npm/#myorg/js-core/-/#myorg/js-core-1.0.0.tgz: Request failed \"404 Not Found\"".
Where the project ID should be the value of <js-core_project_id>.
I have tried writing to all possible .npmrc file paths (~/.npmrc, ./.npmrc, ${CI_BUILD_DIR}/.npmrc, setting the values with npm config set and yarn config set, deactivating the packages feature in the gitlab project itself. I have also not found any predefined environment variables that would override my configs (see https://docs.gitlab.com/ee/ci/variables/predefined_variables.html). #myorg is correct and matches the gitlab url, as the actual value for "myorg" is a single word...
I am pretty much out of ideas at this point, any help is appreciated!
Update 1: I don't think it is a npm / yarn issue, maybe a gitlab caching problem? If I execute npm config ls -l and yarn config list the correct URLs are output in a format that works locally. I will attempt to clear the yarn cache (globally and locally) and pray that that works.

Bitbucket pipeline custom cache definition not working

I have a project with two sub folders. Here's the structure:
src
|- api
|- ui
|-node_modules
bitbucket-pipelines.yml
Since my node modules folder is not in the same directory as the pipeline file, I tried creating a definition and included the path. Here's where I'm trying to cache it in my pipeline:
pipelines:
default:
- step:
name: "Frontend"
image: node:14.17.1
caches:
- frontendnode
script:
- cd src/ui
- npm install
- CI=false npm run build
definitions:
caches:
frontendnode: src/ui/node_modules
My guess is that my definition is wrong but I have tried multiple things and I'm getting the same error:
Assembling contents of new cache 'frontendnode'
Cache "frontendnode": Skipping upload for empty cache
Thanks!
Try to print all files of machine, call:
apt-get install -y tree
tree
Also cache will fill after success complete of pipeline
Or your src/ui/node_modules folder is empty

Why doesn't gradlew build use local cache in docker (bitbucket) pipeline?

I'm using a bitbucket docker pipeline to validate my builds for an android app on push. One of my dependancies is a private package which I am hosting on another bitbucket repository. For builds on the user machine, I use Gradle's private maven repository plugin which can resolve my dependency with encrypted username and password.
This works well for developer machines, but I want to avoid hard coding usernames and passwords in the pipeline. Instead, since bitbucket support sshkeys across repositories for authentication, I have built in my pipeline script to clone the repository with the my private packages, and copy them over to the gradle cache. I have tried both:
/home/gradle/.gradle/caches/modules-2/files-2.1/com.mycompany
~/.gradle/caches/modules-2/files-2.1/com.mycompany
as caches. The clone and copy work just fine as I can see the files in their respective directories in the cache with by adding an ls in the pipeline, but gradlew still tries to go to the internet (the other bitbucket repository) to resolve the dependancies, as if there is no cache. Further, i'm using gradle docker image gradle:3.4.1 (the version of gradle in my project level build.gradle file), but gradle build fails with a google() is not a function.
Gradlew build fails trying to resolve my package.pom file, with a lack of a username (because there is no gradle.properties in the pipeline). But why doesn't it use the cache, instead of trying to go to the repository?
I have tried the standard java:8 docker image, the gradle docker image up to 5.1.1, and i have tried copying the package files into various gradle caches in the docker image. I have also tried altering permissions with chmod 775 to no avail. I have also tried gradlew assembleDebug with the same results as gradlew build. I'm a bit new to gradle, docker and bitbucket so i'm not sure what is causing the issue.
image: gradle:3.4.1
pipelines:
default:
- step:
caches:
- gradle
- android-sdk
script:
# Download and unzip android sdk
- wget --quiet --output-document=android-sdk.zip https://dl.google.com/android/repository/sdk-tools-linux-3859397.zip
- unzip -o -qq android-sdk.zip -d android-sdk
# Define Android Home and add PATHs
- export ANDROID_HOME="/opt/atlassian/pipelines/agent/build/android-sdk"
- export PATH="$ANDROID_HOME/tools:$ANDROID_HOME/tools/bin:$ANDROID_HOME/platform-tools:$PATH"
# Download packages.
- yes | sdkmanager "platform-tools"
- yes | sdkmanager "platforms;android-27"
- yes | sdkmanager "build-tools;27.0.3"
- yes | sdkmanager "extras;android;m2repository"
- yes | sdkmanager "extras;google;m2repository"
- yes | sdkmanager "extras;google;instantapps"
- yes | sdkmanager --licenses
# Build apk
- git clone git#bitbucket.org:myorg/myrepo.git
- scp -r myrepo/com/mycomp/* /home/gradle/.gradle/caches/modules-2/files-2.1/com.mycomp
#- gradle build
- ./gradlew build
#- ./gradlew assembleDebug
definitions:
caches:
android-sdk: android-sdk
Gradlew build error:
> Could not resolve all files for configuration ':app:debugCompileClasspath'.
> Could not resolve com.mycomp:mypackage:1.0.1.
Required by:
project :app
> Could not resolve com.mycomp:mypackage:1.0.1.
> Could not get resource 'https://bitbucket.org/myorg/myrepo/raw/releases/com/mycomp/mypackage/1.0.11/package-1.0.1.pom'.
> Username may not be null

How to cache the entire directory (platformio dependencies) in bitbucket pipelines?

I am running a CI pipeline to build firmware for ESP8266 using plaitformio and bitbucket pipelines, my code builds successfully and now I want to cache the directory that contains the platformio libraries (.piolibdeps). Here are the contains of my platform.ini file.
[env:nodemcuv2]
platform = espressif8266
board = nodemcuv2
framework = arduino
upload_port = 192.168.1.108
lib_deps =
ESPAsyncTCP#1.1.0
OneWire
Time
FauxmoESP
Blynk
DallasTemperature
ArduinoJson
Adafruit NeoPixel
How to cache this directory in BitBucket pipelines? Please see below the contents of bitbucket-pipelines.yml file, with this it is not caching the defined directory, what's wrong here?
image: eclipse/platformio
pipelines:
branches:
develop:
- step:
name: Build Project
caches: # caches the depende
- directories
script: # Modify the commands below to build your repository.
- pio ci --project-conf=./Code/UrbanAquarium.Firmware/platformio.ini ./Code/UrbanAquarium.Firmware/src
- pwd
definitions:
caches:
directories: ./Code/UrbanAquarium.Firmware/.piolibdeps
And here my folder structure.
in case you're still looking for an answer - I think you got it almost right, but probably need to specify a custom --build-dir (so that you can specify the same path for your cache) as well as --keep-build-dir (see https://docs.platformio.org/en/latest/userguide/cmd_ci.html). Also, I'm not sure why you specified a ./Code/UrbanAquarium.Firmware/ prefix.
That said, I've tried the above and it became quickly ugly - for now I'll only cache ~/.platformio, as well as the default pip cache:
image: python:2.7.16
pipelines:
default:
- step:
caches:
- pip
- pio
script:
- pip install -U platformio
- platformio update
- platformio ci src/ --project-conf=platformio.ini
definitions:
caches:
pio: ~/.platformio

Resources